It’s been said many times and many ways that storage is an essential element in virtualized environments and there may be no better way to make this point than to use real world examples.
At one VMWare customer, the storage and virtualization engineers were different teams and communication between these two groups was perhaps not as good as it could have been.
After relocating many VM’s to a different datacenter (with a different SAN) the customer began to experience problems on the BES (BlackBerry Enterprise Server) virtual machines running under ESX 3.5. When the backups would run it would take hours for the snapshot to be closed and this would cause an interruption in Blackberry services.
The new environment featured IBM storage front-ended by an IBM SVC with terabytes of cache (the IBM SVC is a storage virtualization solution somewhat similar to the EMC VPLEX but yet quite a bit different). This SAN supports many mission critical environments, so the quick knee-jerk reaction was to blame VMWare and to suggest that VMWare couldn’t handle such critical workloads, despite that these same VMs ran just fine on the original SAN.
What’s going on here? First let’s consider the BES VM’s. These VM had several large drives 500GB and more which contained log files, which of course are highly transactional. So you have a snapshot being held open for a long period of time (for the backups to run) and then a high volume of transactions that needed to be merged when the snapshot was closed.
Now let’s take a closer look at the SAN. The customer learned that the storage pool was being comprised of SATA disk rather than FC disk. This may have been done because the storage team may have surmised that SATA would be “fast enough” since it had massive amounts of cache to drive better speed and indeed disk I/O performance was very good – but not for all data patterns…
Caching is very effective for random I/O workloads, but not for sequential workloads. DBA’s are very familiar with this concept which is discussed here on the MySQL Performance Blog.
To prove that there was a performance penalty for sequential I/O patterns, I ran a series of tests using SQLIO on the SAN. There were many interesting results from the tests but perhaps the most telling result was that throughput (MB/sec) dropped to just 25% of normal speeds when using a large (100GB) sequential file (and to 50% of the original SAN). The difference here is simply that caching cannot effectively accelerate large sequential disk operations, which exposed the performance limitations of the SATA drives. And closing a snapshot is a sequential disk operation. The delta log file is merged with the VMDK to create a new VMDK, and the larger the disk and more transaction the data (more delta blocks) the longer that sequential close operation is going to take. A cloning of a VM would be another example of a sequential I/O operation.
In this case the customer had several remediation options including:
- Don’t use snapshots for backups and use a traditional backup agent inside the VM
- Don’t backup the log disks and mark them as independent (no redo/delta log)
- Move the VMDK’s to higher performing disks (i.e. 15K FC).
It is in situations just like this where we commonly see people say “virtualization is the problem”, but more often than not that’s really not the case. In this case the problem was that the storage system did not effectively meet the demands for the types of activities that were being performed.
There are many cases like this one where virtualization and/or the virtualization vendor was blamed, when the real problem was in the storage subsystem. Especially with vSphere 4 and current CPU capabilities there are very few x86 servers that cannot be effectively virtualized (more on that in future posts). Make sure you understand how virtualization uses storage and also have a full understanding of your own storage infrastructure.