by I recently “upgraded” my home file server to OpenIndiana 151a. It was more of a sideways migration, really, since it was previously on Solaris 10 u8.
Anyway, I noticed a big performance hit on NFS after switching to OpenIndiana. Initially I thought it was an issue with some slower disks I installed, as I also added two 3TB 5400 rpm Hitachi disks in a new ZFS mirror immediately after the migration. Further testing showed the existing 3 disk raidz pools also exhibiting the poor NFS write speed, though, so I rejected that theory.
Since I’d also upgraded VirtualBox at the same time (what was that I heard? too many changes at once?), I thought maybe a regression had been introduced there. On Solaris 10 I been using an old release of VirtualBox because of a number of regressions I’d stumbled across in newer versions, one of which was bad enough to lock up the whole system. Since I was poking around the VirtualBox site to see what versions would run on OpenIndiana, I took a look in the forum, and one of the first posts addressed my exact problem.
Turns out there’s some sort of kernel contention between NFS running on the host and a VM running in bridged mode. One resolution is to bridge the VM to a second NIC. Another is to bridge it to a VNIC. I chose to do the latter. The solution I’ve summarized below was posted up by “carbs” in the VirtualBox forum here: https://forums.virtualbox.org/viewtopic.php?f=11&t=45416#p204773
You have to use one VNIC per VM, and the MAC of the VNIC must be used in the VM’s network configuration.
For example:
dladm create-vnic -l e1000g0 vnic1
dladm show-vnic
The second command should show output like the following (note that the MAC has no zero padding):
LINK OVER SPEED MACADDRESS MACADDRTYPE VID
vnic1 e1000g0 1000 2:8:20:7a:f7:42 random 0
Next, apply the change to the VM “myserver” (must be shut down):
VBoxManage modifyvm myserver --nic1 bridged --bridgeadapter1 vnic1 --macaddress1 0208207af742
The changes using dladm and VBoxManage are both written out automatically to configs so they will survive a reboot. This two minute fix has thus far resolved the issue, and my NFS write speeds are back to normal.
Update,11/6: Seems this is just a delaying tactic. After about a week I noticed the same slow down. Rebooting the VM brought the speed back.
Update,12/24 I forgot to update with my final fix. I noticed the slowdowns again after only about 2 days for the VM running. I ended up using a second physical interface on the system, since I had another NIC on the motherboard. I’ve not seen any further issues since switching the configuration.