Sunday, January 11, 2009

zfs sparse provisioning with compression - results

I thought I'd quickly share an update with the results after a few days of using this solution.

My ESX servers are hosting 5 VMs, with 30GB,15GB,15GB,10GB,10GB disks respectively for a total of 90GB.
ESX is showing the iscsi volume as 93GB/500GB used. I have a couple of GB on the volume for the ESXi userworld swap , which explains the descrepancy.

ZFS on the other hand is showing 35.0GB/220GB used on the zvol.
What's more, it's also reporting 1.28x compression!

This is a great space saving, so what are we seeing here?

On the ESX/VMFS side, the vmdks are actually being sparsely provisioned onto the storage from what I can tell. VMFS makes sure that it accounts for the full space of each vmdk in it's totals so that it doesn't overcommit itself, but it isn't actually zeroing/preallocating the contents of each vmdk as it makes them, despite what you may think when you hear the term "thick-provisioning".

I actually confirmed this by testing with another zvol with compression off to rule out zfs compression reclaiming any "000000....0" blocks that VMFS may have been writing. Same result as my production compression=on iscsi volume.

On the Solaris side, ZFS is COWing only the blocks that actually get touched. As you would expect most of my VMs are not using the full capacity of their file systems yet, which explains the big difference between ~90GB and the 35GB that ZFS is actually using.

It's also interesting to see that I'm getting 1.28x compression too. It was higher earlier on but I've been doing quite a bit of work on a Solaris vm which is using ZFS-root, with compression=on in the guest, so we can't compress any blocks for that VM a second time.
This means that this compression figure is for 4 windows VMs of the 5, which is worth having in my book!

Scaling this out: 35GB-for-90GB is a space saving of bit over 2.5x.
I thinly provisioned a 500GB volume with around 250GB of actually storage available with was only a 2x overcommit. Maybe I should have aimed higher!

Snapshots

Just a quick note on ESX snapshots. They will burn through your space savings as they are COWing in a totally different area on the disk, raising the "high water mark" that I referenced in my earlier post.
That's not to say that you shouldn't use them, but just be aware that they really are quite costly both in terms of performance to the VM as we all know, but also to your thin provisioning savings :)

No comments: