Replacing OpenSSH/SFTP with SFTPGo
Python/Nim/Go coder, Linux admin, hacker who is always learning new things, studies Operating systems specifically Unix derivatives
I run a pair of GNU/Linux VPS's, one running Plex as a client and the other as a storage server for media. They both run on "the cloud" hosted by different providers. I've been noticing lately that transfer rates between them over SSHFS have been lagging, causing buffering and drop issues for some of my media streams. I tested directly between the hosts using Iperf3 and while not spectacular between two large hosting providers, it's fine for streaming SDTV content.
me@plex:~$ iperf3 -c 10.10.10.100 -p 5201 --bytes 1000000000
Connecting to host 10.10.10.100 , port 5201
[ 4] local 10.10.10.99 port 56634 connected to 10.10.10.100 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 57.0 MBytes 478 Mbits/sec 165 3.94 MBytes
[ 4] 1.00-2.00 sec 45.0 MBytes 378 Mbits/sec 17 1.99 MBytes
[ 4] 2.00-3.00 sec 43.8 MBytes 367 Mbits/sec 0 2.02 MBytes
[ 4] 3.00-4.00 sec 42.5 MBytes 356 Mbits/sec 0 2.15 MBytes
[ 4] 4.00-5.00 sec 48.8 MBytes 409 Mbits/sec 0 2.41 MBytes
[ 4] 5.00-6.00 sec 52.5 MBytes 440 Mbits/sec 0 2.76 MBytes
[ 4] 6.00-7.00 sec 60.0 MBytes 503 Mbits/sec 0 3.02 MByte
I then moved on and tested the mount point. I mount the media storage filesystem from plex client to storage server using SSHFS. So I performed a crude check on the SFTP mount point using cat and a standard utility called pipe viewer (pv).
me@plex:/mnt/media/tv/show/S01$
cat show.mkv | pv -rb --progress >/dev/null
4.88MiB [ 300KiB/s]
indeed, file read hovers around ~300KB/s (with multiple stalls) were just bad. Now there are a slew of tweaks that can be done, via the ssh protocol lighter MACs, Ciphers, via SSHFS buffer cache, VFS tweaks etc.. and the network, buffer, window sizes etc.. I decided to sidestep all the tweaks and try a different SFTP implementation entirely. I remembered a hacker news post not too long ago and decided download a binary release of SFTPGo.
SFTPGo is a standalone SFTP server written in Go, that does not rely on the system authentication mechanisms and can utilize multiple SQL authentication back-ends such as SQLite, Mysql etc.. but I wanted a quick way to use public/private key authentication and good thing it supports this via its "portable" option.
First we'll create an ssh public/private key pair on the Plex client VPS
me@plex:~ ssh-keygen -t ecdsa -f ~/.ssh/id_ecdsa_sftpgo
We then add a section in ~/.ssh/config on the client that describes the storage server
Host storage_server
Hostname 10.10.10.100
Port 4444
User sftpdude
IdentityFile ~/.ssh/sftpgo_ecdsa
Ciphers aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com
MACs umac-128-etm@openssh.com,umac-128@openssh.com,hmac-sha1-etm@openssh.com,hmac-sha1,hmac-sha2-512-etm@openssh.com,hmac-sha2-512
Now we'll move to the storage server and create a dedicated directory and download the latest release of SFTPGo and install it manually
me@storage:/ sudo mkdir /opt/sftpgo
me@storage:/ sudo chown me:me !$
me@storage:/ cd !$
me@storage:/opt/sftpgo wget -O - https://github.com/drakkan/sftpgo/releases/download/0.9.6/sftpgo_0.9.6_linux_x86_64.tar.xz | tar Jxf -
Next create a wrapper script on the storage server and copy in the client's public key that was created earlier in ~/.ssh/id_ecdsa_sftpgo.pub
me@storage:/opt/sftpgo/ cat sftpgo_start.sh
#!/bin/sh
sftpgo=/opt/sftpgo/sftpgo
port=4444
pubkey='ecdsa-sha2-nistp256 AAAAE2V....3YPCHWceWD2QcQFG='
dir=/mnt/media
$sftpgo portable --username sftpdude --public-key "$pubkey" --sftpd-port $port --directory $dir --permissions '*'
We run the newly created script on the storage server
me@storage:/opt/sftpgo/ /opt/sftpgo/ssftpgo_start.sh
We connect to the storage_server via sshfs
me@plex:~ /usr/bin/sshfs -f storage_server:/tv /mnt/media/tv2 -o ServerAliveInterval=15 -o rw,reconnect,idmap=user
and retry via the crude cat | pv
me@plex:/mnt/media/tv2/show/S01$
cat show.mkv | pv -rb --progress >/dev/null
23.8MiB [2.84MiB/s]
and lo and behold .. much faster transfer rates without tweaking and just switching implementations. Of course it's far slower than a raw byte transfers (iperf3) but we are utilizing FUSE which in itself is a bottleneck traversing user-space/kernel boundaries, but it's good enough for this use case.
Securing things
OpenSSH has decades of security under its belt and comes from the iron clad OpenBSD folks and we want to protect the system as much as possible from any weird side effects, bugs, and/or exploits typical of long running programs. While it is written in a garbage collected language (Go) which eliminates many classes of memory induced bugs, it is better to be safe than sorry. So if you're running a popular Linux distro today you most likely have systemd installed by default. Systemd is an initd/framework for starting/managing the system and it's services. It is what we'll use to to monitor, manage and secure the SFTPGo service. First thing we do is think about that what the service needs to do to perform its job; it starts up, creates a pub/priv key file if /opt/sftpgo/id_ecdsa doesn't exist, listens on a specified port, and reads/writes to /mnt/media/. So let's create a systemd service file that incorporates this behavior.
So here's the systemd service file
me@storage:/etc/systemd/system cat sftpgo@me.service
[Unit]
Description=SFTP Go
AssertDirectoryNotEmpty=/mnt/media/
[Service]
# Username is extracted after @ symbol of service filename
User=%i
ExecStart=/opt/sftpgo/sftpgo_start.sh
ExecReload=/bin/kill -s HUP $MAINPID
WorkingDirectory=/opt/sftpgo
Type=simple
Restart=on-failure
RestartSec=10
KillMode=mixed
StartLimitInterval=20s
StartLimitBurst=3
NoNewPrivileges=yes
PrivateTmp=yes
PrivateDevices=yes
DevicePolicy=closed
ProtectSystem=strict
ProtectHome=yes
ReadWritePaths=/opt/sftpgo /mnt/media/tv
RestrictAddressFamilies=AF_INET
#requires systemd 235+ and kernel 4.11+
IPAccounting=yes
IPAddressDeny=any
IPAddressAllow=10.10.10.0/24
[Install]
WantedBy=multi-user.target
Alias=sg.service
Basically we prevented read/write access anywhere except /opt/sftpgo and /mnt/media/tv. We restricted access to /home or any other filesystems via the ProtectHome, ProtectSystem stanzas. We also restricted socket creation to the AF_INET protocols while only accepting connections from Plex client ip address range. Systemd provides a host of facilities one can use to secure their services without that much effort in tandem with the kernel 4.x+ namespacing, eBPF, cgroup controls. Most of systemd's lock down parameters are explained here, there's even more we can do such as system call, namespace filtering, but we'll need to profile the binary some more before implementing that.
Now we enable and start our new service
me@storage:~ sudo systemctl enable sftpgo@me
me@storage:~ sudo systemctl start sftpgo@me
me@storage:~ sudo systemctl status sftpgo@me
And check some transfer stats after transferring some files (since we enabled IPAccounting=yes)
me@storage:~ sudo systemctl show sftpgo@me -p IPIngressBytes -p IPEgressBytes
IPIngressBytes=20382896
IPEgressBytes=677305852
99% of the time I have no need to replace the standard SFTP server implementation on my Linux/*BSD hosts, but it was worth it this time for some flexibility and the bump in speed. Though the OpenSSH SFTP implementation is actually faster than SFTPGo, so clearly something is wrong between these hosts that I need to troubleshoot down the road. But, just the fact that I can decouple the standard GNU/Linux user authentication mechanism from SFTP and name the user whatever I want as long as the script has the correct read/write permissions is a plus for me.