Terrible Virtualbox disk performance

Monday, March 14th, 2016

For a while I’ve noticed that Virtualbox has terrible performance when install Debian / Ubuntu packages. A little top, iotop and atop research later, and it turns out the culprit is the disk I/O, which is just ludicrously slow. The cause is the fact that Virtualbox doesn’t have “Use host IO cache” turned on by default for SATA controllers.Turning that option on gives a massive improvement to speed.

To turn “Host IO  cache” on:

• Open the Virtual Machine’s Settings dialog
• Go to “Storage”
• Click on the “Controller: SATA” controller
• Enable the “Host I/O Cache” setting.
• You may also want to check any other controllers and / or disks to see if this option is on there.
• Save the settings and start the VM and you’ll see a big improvement.

 

 

Update:

Craig Gill was kind enough to mail me with the reasoning behind why the “Host I/O cache” setting is off by default:

Hello, thanks for the blog post, I just wanted to comment on it below:

In your blog post “Terrible Virtualbox disk performance”, you mention
that the ‘use host i/o cache’ is off by default.
In the VirtualBox help it explains why it is off by default, and it
basically boils down to safety over performance.

Here are the disadvantages of having that setting on (which these
points may not apply to you):

1. Delayed writing through the host OS cache is less secure. When the
guest OS writes data, it considers the data written even though it has
not yet arrived on a physical disk. If for some reason the write does
not happen (power failure, host crash), the likelihood of data loss
increases.

2. Disk image files tend to be very large. Caching them can therefore
quickly use up the entire host OS cache. Depending on the efficiency
of the host OS caching, this may slow down the host immensely,
especially if several VMs run at the same time. For example, on Linux
hosts, host caching may result in Linux delaying all writes until the
host cache is nearly full and then writing out all these changes at
once, possibly stalling VM execution for minutes. This can result in
I/O errors in the guest as I/O requests time out there.

3. Physical memory is often wasted as guest operating systems
typically have their own I/O caches, which may result in the data
being cached twice (in both the guest and the host caches) for little
effect.

“If you decide to disable host I/O caching for the above reasons,
VirtualBox uses its own small cache to buffer writes, but no read
caching since this is typically already performed by the guest OS. In
addition, VirtualBox fully supports asynchronous I/O for its virtual
SATA, SCSI and SAS controllers through multiple I/O threads.”

Thanks,
Craig

That’s important information to keep in mind. Thanks Craig!

My reply:

Thanks for the heads up!

For what it’s worth, I already wanted to write the article a few weeks ago, and have been running with all my VirtualBox hosts with Host I/O cache set to “on”, and I haven’t noticed any problems. I haven’t lost any data in databases running on Virtualbox (even though they’ve crashed a few times; or actually I killed them with ctrl-alt-backspace), the host is actually more responsive if VirtualBoxes are under heavy disk load. I do have 16 Gb of memory, which may make the difference. 

The speedups in the guests is more than worth any other drawbacks though. The gitlab omnibus package now installs in 4 minutes rather than 5+ hours (yes, hours).

I’ll keep this blog post updates in case I run into any weird problems.

Update: I’ve been running with Host I/O caching on for almost 6 months now. I’ve noticed no performance degradations on the host or other Virtual Machines. I regularly (about 3 times a week) have to hard power-cycle my laptop due to a kernel bug for my Wifi driver. I’ve noticed no data loss in my VMs. Then again, I don’t use them for mission critical applications, nor do they do a lot of write disk I/O. The performance increase on disk I/O in the VM remains extremely high.