I’ve just finished installing four 4 TB hard drives into my NAS, and configured them as a zpool of mirrored vdevs (conceptually RAID10).
Sharing the pool via SMB, a simple file copy test revealed the transfer speeds to be unusually poor - only reaching ~7 - 10 MB/s. Extremely slow; and suspicous.
I reviewed the ZFS configuration for hours - double- and triple-checking configurations, deleting and recreating the pool, reading articles/blogs/forums on ZFS performance optimizations and tuning.
By random chance, I noticed that my phone and laptop both lost internet connectivity through their wireless connections. The access point was still powered on and working, and the SSID was still being broadcast fine, but no internet connectivity through it. I logged into my router’s web UI (an ERLite-3), and discovered the CPU usage was pinned at 100%. The router was barely able to keep its web UI responsive.
It turns out that every time I transfer a file to the network share, the CPU of the router is maxed out for the duration of the transfer. The router was somehow so busy with the transfer that it couldn’t keep up with directing traffic for the wireless clients.
After some further searching of the internet, I discovered the culprit: I make use of VLANs in my network, and the ERLite-3 router does not have Hardware Offloading enabled by default for VLAN traffic. I needed to use the router’s CLI to enable hardware offloading for the VLAN packets.
So after hours of troubleshooting and testing, scouring the internet for unrelated advice, a few CLI commands on my router enabled network transfer speeds to skyrocket from 7 - 10 MB/s to 95 - 105 MB/s, while the router’s CPU now hovered at a modest 45% - 60% during the transfer.
The ports among my networking hardware are 1 GbE, so the zpools seem to be achieving close to the theoretical line speed. FTP transfers (instead of SMB) have consistently higher speeds - 105 to 110 MB/s - so there’s some loss due to overhead in the network/protocols/etc.
Some simple file copy/transfer tests using the NAS’s OS directly - so networking is not a factor - had the zpool reaching ~250 MB/s on writes. I should have tested the zpool locally from the OS of the NAS much sooner; it would have made the network a much more obvious culprit. I’m still very new to ZFS, so I had assumed I made a mistake with configuring the vdevs/pools/datasets.
Although now I’ve got a new “problem” - with the array able to support such fast speeds, I’ve got an itch to upgrade my network to 10 GbE.
10 GbE is becoming (or even has already become) affordable to the home user/enthusiast, so someday I’ll upgrade my network when I crave more speed (or legitimately need it, e.g. bigger backups)