Why Microsoft?

This is a question that can be explored from many different angles, but I’d like to focus on it from not JUST a virtualization perspective, and not JUST a cloud perspective, and not JUST from my own perspective as a vExpert joining Microsoft, but a more holistic perspective which considers all of this, as well

Top 6 Features of vSphere 6

This changes things. It sounds cliché to say “this is our best release ever” because in a sense the newest release is usually the most evolved.  However as a four year VMware vExpert I do think that there is something special about this one.  This is a much more significant jump than going from 4.x

vSphere 6.0 Public Beta — Sign Up to Learn What’s New

Yesterday, VMware announced the public availability of vSphere 6.0 Beta 2.  I can’t tell you what’s all in it due to the NDA, but you can still register for the beta yourself, read about what’s new and download the code for your home lab. There’s some pretty exciting stuff being added to vSphere 6.0 in

Will VMware Start Selling Hardware? Meet MARVIN

The Register is running a story that VMware is preparing to launch a line of hardware servers.

VMware Pursues SDN With Upcoming NSX Offering

Earlier this week VMware announced VMware NSX – an upcoming offering that takes network virtualization to new levels. NSX appears to be somewhat of a fusion between Nicria’s SDN technology (acquired last year by VMware) and vCloud Network and Security (vCNS – formerly known as vShield App and Edge). Since I already had intentions to

What Really Is Cloud Computing? (Triple-A Cloud)

What is cloud computing?  Ask a consumer, CIO, and salesman and you’ll likely get widely varying responses. The consumer will typically think of the cloud as a hosted service, such as Apple’s iCloud, or uploading pictures to Photobucket, and scores more of like services (just keep in mind that several such services existed before it

Agility Part 2 — The Evolution of Value in the Private Cloud

When an IT project is commissioned it can be backed by a number of different statements such as: “It will reduce our TCO” “This is a strategic initiative” “The ROI is compelling” “There’s funds in the budget” “Our competitors are doing it” Some of these are better reasons than others, but here’s a question.  Imagine a

Stacks, the Vblock and Value — A Chat with EMC’s Chad Sakac

…I reached out to EMC’s Chad Sakac to gain more insights from his perspective on how the various stacks…well…stacked up….

Tips for Linux Performance in Hyper-V

[Disclosure:  Microsoft employee]

Every now and then we are asked how well Linux performs on Hyper-V.  We’ve done extensive testing and most of the time it runs quite well, but we’ve found a few cases, where some tuning can make all the difference.

I was recently involved with a scenario where a Linux application on Hyper-V (Windows Server 2016) was showing benchmarks of between 30-35% slower than other hypervisors. We instinctively felt something was wrong, so we rolled up our sleeves and got to work.

After some effort from our outstanding product engineering team, we found that this gap was essentially eliminated by tuning some settings in the BIOS, tweaking some Linux kernel parameters as well as keeping the in-guest tools current. Here’s a quick summary of some considerations if you’re looking to maximize the performance of some of your Linux applications.


Many systems have power management profiles in the BIOS, and while many environments correctly have this set to “performance”, many systems will still have additional options for individual C-states.  If you want a primer on what a C-state is, there’s a good article here from Intel, but basically its a power management method. If you want the best possible CPU performance on your host servers, turn it all off!

In this case, we observed a 15% performance improvement after disabling all C-states in the host server’s BIOS.

Upgrade Linux Integration Services (LIS)

Linux Integration Tools consists of optimized synthetic device drivers and other “virtualization helpers” that go into the guest OS.  Windows servers have integration services built in and get can updates via Windows Update. Linux is similar in that the various distributions will build LIS into their images, and — depending on what version your are running — updates are distributed through their respective patch channels.

LIS 4.1.3 was released in December of 2016 (download here) and we found that simply updating to this version of LIS there was a 10% performance increase.

A few notes here — some Linux distributions will not support LIS unless it is released through their patch distribution. Before you manually install LIS, make sure you understand if there are any support implications from your Linux vendor.

On the topic of LIS one final important point to keep in mind is that it makes a big difference what version of Linux you are running, as not all features are supported on all versions of Linux. If you visit the Hyper-V Linux support page, you will see sections for RHEL/CentOS, Debian, Oracle, Ubuntu, SUSE and FreeBSD.  In each of these sections you can find detail on which Hyper-V features are supported in which Linux versions.

Bottom line again, is that in this case, simply upgrading LIS to the current version of 4.1.3 delivered a 10% performance improvement.

Set the Linux Clock Source

Let’s say you didn’t want to upgrade to the latest LIS.  You can still capture much of the performance improvement here simply by tuning the Linux clock.

Changing the Linux kernel’s clock source to “tsc” which stands for “Time Stamp Counter” resulted in a performance improvement of 6%.  This particular application had a lot of context switches and made heady use of clocks and timers, so your mileage may vary here.

A bit of information on this parameter is available here.

How Many CPUs?

Linux kernels have an optional parameter where you limit the number of possible CPUs by setting the “possible_cpus” kernel parameter.  For example, if you have 8 virtual CPUs in the guest, edit the kernel parameters to include “possible_cpus = 8”. By setting this parameter, additional overhead processing was removed from the Linux CPU scheduler.

In this application a performance improvement of 2% was observed after applying this setting.

The Bottom Line

IT is often complicated.  There’s so many distros, Linux versions and workload patterns generated by applications and other variables to consider — your mileage may vary.

If you are concerned that your Linux workloads may be running slower than they could be on Hyper-V check these three basic things:

  • Disable C-States in BIOS
  • Upgrade to current version of LIS (If not an option, change clock source to “tsc”)
  • Set “maximum_cpus” in the Linux kernel

For one application, these three steps resulted in a performance improvement of nearly 30% (with another 5 – 8% from correcting some misconfigurations).

So if you have any concerns about Linux performance in Hyper-V, just remember these tweaks. And of course that….


Understanding The Latest Cloud Announcements (AWS to Azure Mapping)

(Disclaimer – this blog post was written by a Microsoft employee)

It’s a real exciting time in the cloud computing industry with so many new solutions being offered at a dizzying pace.

Several new announcements were made last week and @stufox tried to provide some clarity by mapping some AWS products to Azure:

Building on this, I quickly made the following table which attempts to map several of these newer offerings.  Of course not all features here are going to have perfect parity across vendor implementations – as always perform your due diligence – but this should help to relate to these features across multiple clouds.

Offerings in preview or “coming soon” at the time of this post are noted below.

AWS Feature Azure Feature Notes
Step Functions Logic Apps Both enable scalable, visual workflows for distributed applications.
X-Ray (preview) Application Insights Rich application performance monitoring (APM) leveraging machine learning
AWS Glue (coming soon) Data Factory Data processing and transformation
AWS Batch (preview) Azure Batch Cloud scale batch processing
CodeBuild Visual Studio Team Services Developer Tools
Pinpoint Mobile Engagement Track mobile engagement
AWS Shield Built-in, also Azure Security Center Mitigate DDOS attacks
Alexa Skills Cortana Intelligence Suite Alexa Skills is focused more on leveraging voice capabilities, while Cortana is more focused on analytics.
Health Dashboard Azure Security Center and Azure Advisor (preview) Health, performance, and security status
Lambda Functions “Serverless” computing or “Functions as a Service”

Also, while some of the newer offerings are not yet listed, you can view a broader list of AWS to Azure service mapping here:



Record Breaking Hyper-Converged Performance with Windows Server 2016

(Disclaimer – this blog post was written by a Microsoft employee)

As Gartner has noted, hyper-converged SANs (often referred to as Software Defined Storage, SDS or HCI) have been taking off in popularity in the mid-market and even for side projects in larger environments as well. One big reason is cost – the cost to deploy a hyper-converged SAN is far less than purchasing and maintaining a traditional array.

OK, but how well do they perform?

Using Intel NVMe SSDs, Intel engineers were able to achieve 3 million IOPS using an all flash configuration on Windows Storage Spaces Direct, with Hyper-V VMs creating the workload on the same hardware.

We’ve seen fast hyper-converged SANs before with 32 and 64 node clusters, but have you seen 3 million IOPS with just 4 hosts?



Just 4 nodes with Windows Storage Spaces 2016 producing over 3 million IOPS

I know what you’re thinking – these massive IOPS tests are interesting for reads, but what about writes?  Intel also tested 144 SQL 2016 virtual machines deployed to a similar 4 node cluster, and achieved a rate of over 28,000 transactions per second – suitable for many OLTP scenarios.

So what made these record performance numbers possible?

Well first of all the Intel NMVe SSDs offer incredible performance, but we need a way to sustain this activity across multiple hosts.  40GB network interfaces can help but how can you further reduce latency and approach line speed?

Windows Storage Spaces Direct leverages SMB Direct (RDMA) which allows for direct memory access to the network cards, significantly reducing network latency.  And SMB Multichannel is built in – allowing all network adapters to dynamically load balance. Of course there’s a lot more to Windows Storage Spaces Direct from erasure coding to caching tiers as noted in the video below:

And let’s not forget about ReFS 2.0 which has been designed as a file system for the cloud era. Cloning, snapshots, and provisioning can normally create a lot of write IOPS, but ReFS offloads this I/O by making these operations a simple metadata operation. A short 2 minute demo of this in action (no sound) is available here.

The bottom line is that Storage Spaces Direct (S2D), SMB Direct (RDMA) and ReFS 2.0 go a long way to enabling strong hyper converged performance – especially with state of the art NVMe solutions.

Now throw the power of Azure on top of all this, and you’ve got Azure Stack – everything you need to run a private cloud – that is consistent with the Azure public cloud – in just one mini rack.

Stay tuned for more Windows Server and Azure Stack news as MSIgnite unfolds this week.

Active Directory in the Cloud

Active Directory capabilities have been a key differentiatior for the Microsoft Cloud, which includes the capabilities of Azure.  Some may be inclined to think of Active Directory as a legacy on-premise technology that grinds against cloud principles, but when it comes to business execution the reality is quite different. There are some exciting new capabilities being announced, but first a quick overview.

Azure Active Directory is the foundation for a myriad of B2C and B2B scenarios, with one of the more exciting use cases being Real Madrid who uses Azure Active Directory B2C to handle identity management for their 450 million fans.  When combined with Azure IaaS/PaaS, Power BI and Dynamics CRM the capabilities are nothing short of amazing:

B2B and B2C are great use cases, but what about extending Active Directory to leverage cloud resources? Over 90% of the Fortune 1000 use Active Directory in an on-premise capacity. Azure Active Directory allows these organizations to quickly enable federated access to Office 365, Salesforce.com, Concur and many other SaaS applications — or even in-house cloud applications.  It takes just 4 clicks to link Azure Active Directory with your organization’s Active Directory, to provide a means to quickly provision access to these applications to your existing users.  On top of this, Azure Active Directory provides for multi-factor authentication, self-service password and group management, application alerting and monitoring, role based access control and more.


Click to expand


While Azure Active Directory is focused on extending identity management to the cloud, this never included core features of on-premise Active Directory such as Group 101415_1620_AzureADDoma1Policy, Kerberos, NTLM and so forth.  Many legacy applications in use today may not have support for cloud-friendly methods such as OAuth2.0 or SAML. Those that needed this classic functional level of Active Directory for their applications and databases would have to deploy a domain controller in Azure for this capability.

Now with Azure AD Domain Services, organizations no longer need to maintain domain controllers in the cloud for this functionality.  Think of it as the on-premise Active Directory you already know, but as-a-service in the cloud. In the example below, Litware deployed Azure AD Connect to provide classic AD functions like LDAP read & bind, NTLM, Kerberos, and Group Policy to their Azure IaaS workloads — no domain controllers required!



Click to expand

Azure AD Domain Services are currently available as public preview, meaning that there is no SLA guarantee in place, but you still have the opportunity to pilot and explore the capabilities and prepare for the GA release. Follow the link for more information on Azure AD Domain Services including how to get started.


Also just announced is that many preview features of Azure AD App Proxy such as custom domain names, conditional access policies, Intune NDES — are now in GA.  In addition the following new features in preview:

  1. Support for Remote Desktop
  2. Support for complex networks and data center topologies using connector grouping
  3. Support for non-Windows applications using Kerberos over SPNego

For more information on these new features, please see this post on Azure AD App Proxy.


The Active Directory capabilities in Azure are a key differentiator — allowing customers to extend their existing Active Directory to leverage cloud and SaaS applications, as well as new B2C and B2B capabilities.  Now with Azure AD Domain Services in public preview, organizations will soon no longer need to maintain IaaS domain controllers in Azure to support classic Active Directory functions.

New Azure Capabilities and AzureCon (Sept. 29)

downloadAzure has evolved and grown remarkably over the years,  now offering first class support for Linux VMs, containers, machine learning and more, making it a great choice to extend your on-premises environment –whether it be VMware, Windows Hyper-V, or something else.

In my previous post I noted how Microsoft was ranked by Gartner as one of two leaders in EACH of the following categories:

  • x86 Server Virtualization
  • Cloud Infrastructure as a Service
  • Application Platform as a Service
  • Public Cloud Storage Services

Azure recently introduced a few new standout capabilities which I wanted to briefly explore, starting with the new GS series of Azure virtual machines.

Azure GS-VM Series

32 Cores. 448GB RAM. 80,000 Peak IOPS. 2,000 MB/sec disk bandwidth.

That’s the largest GS series VM currently available and to put that into perspective that’s more than double the storage throughput offered by any other cloud provider.  Network throughput of more than 20 gbps is also more than double what is available from any other cloud provider.

That’s large enough for some fairly heavy databases and keep in mind Azure supports SQL 2014 AlwaysOn clusters.

For more details see Scott Guthrie’s blog post on the GS series as well as the table below.


Azure Backup for Azure VMs

What if you wanted to backup your Azure VMs?  Azure Backup has been around for some time for backing up on-premises workloads to Azure, but now these capabilities have been extended to Azure VMs running both Linux and Windows.  

Your Linux and Windows VMs will be backed up hot (online) and stored in the Azure vault with support for GFS retention schemes and more. Using this capability Azure VMs can be restored to a prior state with a single click.

For more details, see the Azure Backup announcement here.

Azure Active Directory extends to support B2B and B2C Scenarios

Real_Madrid_CF.svgAzure Active Directory is a great way to provide single-sign-on from your on-premises Active Directory to cloud hosted apps including Office 365. But what about a B2C scenario where a business wants to provide consumer identity services.  Real Madrid is leveraging Azure Active Directory B2C to allow millions of soccer fans (yes, I’m American) to register and create accounts. B2B capabilities are also being added which will allow companies to securely exchange data and collaborate.

Both Azure AD B2C and Azure AD B2B are now in public preview. For more information please see the Azure B2C/B2B announcement post.


More features and capabilities are constantly being developed for Azure, so you may want to “attend” AzureCon this coming Tuesday, September 29th for all the new announcements, customer stories and technical sessions.

AzureCon is a virtual event so all you have to do is sign up and join from wherever you are. For more info, click below.


Why Microsoft?

Why Microsoft?

This is a question that can be explored from many different angles, but I’d like to focus on it from not JUST a virtualization perspective, and not JUST a cloud perspective, and not JUST from my own perspective as a vExpert joining Microsoft, but a more holistic perspective which considers all of this, as well as the core questions of where value exists for the enterprise.

As someone who focused on VMware technologies for over 8 years, I get questions which are variations of the following:

“Has your opinion of VMware changed?”

“Why would you give up all your VMware knowledge and experience, to have to learn a new platform that has less market share today?”

There’s a couple different answers here. One of them is that my thoughts about VMware haven’t really changed.  I still remain a big fan of the solutions which have accomplished so much, and especially the community which I’ve enjoyed being a part of these last several years. I don’t view myself as leaving the community – at the end of the day we all share the same goals – but rather I’m viewing it from a different vantage point.

What I REALLY want to talk about here is what I think Microsoft is doing right and how they are incredibly well positioned to help enterprises unlock their potential.


The Gartner Magic Quadrant for Server Virtualization may not be the final word on all things, but I think few would dispute the positioning as VMware and Microsoft being the only two leaders:


I agree with VMware being the clear leader in this space, however this chart does not take into consideration next year’s release of Windows Server 2016 with MANY new capabilities ranging from Nano Server, and improved ReFS, Azure Stack and many improvements to both Hyper-V and System Center (more on these in upcoming posts).  In short, I expect this gap to tighten significantly in the near future.


On-premise virtualization and cloud solutions are important, but so are those which are hosted and with rich automation, allowing enterprises to quickly scale and use capacity on-demand – something that is much more challenging in an on-premises environment.

Once an IT Director was telling me about how a critical application was spread across 16 different “tiers” of development, test, QA and production such that it was impossible to get a maintenance window and advance changes into production.  Today in Azure, customers can position their dev/QA/production elements into logical slots and flip them into production with a simple click.

microsoft-azureThat’s just a simple example that doesn’t even scratch the amazing depth of automation and function that is available on Azure.  And let’s not forget Docker, containers, microservices and more.  At one point Azure had a reputation for being a “Windows-only platform”, but this is clearly no longer the case. Many Linux distros are Azure certified and you’ll find Ubuntu, CentOS, Oracle Linux and more, not to mention containerized versions of nginx, rabbitmq, MySQL.


If you haven’t looked at Azure lately you owe it to yourself to take a closer look.

Getting back to the big picture we notice that once again Microsoft is in the leader quadrant, this time behind Amazon Web Services.  Notice the pattern?

mq_iaasAnother market segment where Microsoft is identified as one of two leaders that have pulled away from the pack.


Let’s look at Gartner’s Magic Quadrant for Application Platform as a Service…


Again Microsoft is one of two leaders in the market.

And here’s the MQ for Public Cloud Storage Services…


Put another way, here are the leaders Gartner has identified in the 4 quadrants…


The only other company that appeared in a leadership quadrant more than once was Amazon. Only Microsoft can claim to be recognized as being in a leadership position in each of these 4 market segments. Being in a leadership position in EACH quadrant provides an amazing breadth, synergy and an ability to execute on many fronts.

Add to this the success and growth with Surface, Windows 10, XBOX and emerging technologies like Hololens and you have a pretty exciting company to be a part of with great solutions to offer to customers.

The New Microsoft

Just a few years ago some had written off Microsoft as dead or dying based on the proliferation of mobile devices and cloud, but Microsoft has transformed into a leader in the cloud space, which all those mobile applications need on the back end.  Microsoft has embraced development on platforms like Android and iOS while Microsoft embraces Linux in its own cloud.

Microsoft is now spending in excess of their R&D budget of $10 billion on building data centers.  Again that’s over $10 billion each year on datacenters at the current rate.

And what I’ve already witnessed within Microsoft’s culture is nothing short of impressive. It is a culture that encourages people to interact and build relationships beyond the silos and the org charts.  It is a culture that encourages collaboration, teamwork, responsibility and excellence while enabling “fast zebras” to successfully execute their mission. I may expand on this in a future post, but Microsoft’s leadership team set out to remake the culture a few years ago and those efforts have clearly begun to bear fruit.

Somewhat of a sidebar here but when I look at the tech market, there seems to be an unbalance of sorts in “the force”.

Consider this chart of market capitalization. Apple’s position in the mobile device market is well deserved, but now look at this chart a different way – both Amazon and Microsoft play in both the consumer and the enterprise markets, but Apple for the most part is focused on consumer devices.


Is this level of market capitalization for Apple justified when they do not have a presence in the enterprise cloud space – the very cloud space that powers the back end for so many mobile applications? Financials aside, it seems to me that a mobile device maker can’t ultimately sustain this valuation simply by maintaining a leading market share in mobile devices. And let’s not overlook that Microsoft is preparing to re-enter the mobile market with Windows 10 Mobile.


My focus at Microsoft will mostly revolve around Hyper-V and Azure and between the upcoming release of Windows Server 2016 and continuing Azure developments I couldn’t be more excited about what we will be able to do for customers building hybrid clouds as a leader in BOTH x86 virtualization and hosted IaaS (Azure).

On a personal level, I am taking some risk in stepping away from the familiarity of VMware to learn the nuances of a platform which is more unfamiliar to me, but I expect this risk to pay off in the long run.  While Microsoft may compete with VMware at some levels, both also co-exist at many more levels.  In my mind I’m not stepping away from the VMware community as much as I’m changing my vantage point. I have great respect for VMware and the community I’ve enjoyed being a part of over the years and none of that has changed. But my focus will be changing to Microsoft platforms for the reasons noted above and I couldn’t be more excited.

When CPU metrics with Hyperthreading, Monster VMs and VMware Make No Sense

CPU seems like such a simple thing, but in the age of virtualization, hyper-threading and vNUMA, it can get quite complicated. In fact looking at some metrics can get you to lose your mind until you realize what’s really going on. Let’s jump right in to the original problem.

I encountered a VM with 16 vCPUs on a server with 16 physical cores (2 x 8). The VM was frequently getting alarms in vROPS (vRealize Operations Manager) as at times it would be at 90% or more for sustained periods, but the host server would show just under 50% utilization. This is illustrated in the graph below.


Click Image to Expand

Why would a VM with 16 vCPU be at 80% when the 16 core host is only at 41%. What’s going on here?

The first things we will need to explore are hyper-threading, NUMA and how different CPU metrics in VMware are calculated.


Hyper-threading was intended to solve an issue with a waste of potential resources in the CPU. It does not change the physical resources (the CPU cores) but more resources can be potentially tapped into by allowing two threads to be processed by the same execution resource simultaneously — with each physical core being an execution resource.

When hyper-threading is enabled it doubles the number of logical processors presented by the BIOS. In this example, our 2 socket, 8 core system with 16 cores total now presents 32 cores to VMware. Chris Wahl has an excellent post on this topic which I strongly encourage you to read, but for now I’m just going to “borrow” one of his graphics.


Image taken from Chris Wahl’s post (link in article)

VMware’s CPU scheduler can make efficient use of hyper-threading and generally it should be enabled. The number of logical processors now doubles, providing a performance benefit in the range of 10-15% in most vSphere environments (depending on workload/applications, etc.).

But what about our scenario of a host which has only 1 “monster VM”?

Sizing a Monster VM with Hyper-Threading

The general rule here is that you should not provision more vCPUs than the number of PHYISCAL cores the server has. In our scenario there are 32 logical processors presented due to hyper-threading, but only 16 physical cores. If we provision more than 16 vCPUs to the VM it means that execution resources will now be shared for the VM. Now there are some exceptions here (test your workloads!), but is it generally recommended not to exceed the number of physical cores for this reason.

VMware has a blog post on this topic.  What is their guidance?

VMware’s conservative guidance about over-committing your pCPU:vCPU ratio for Monster virtual machines is simple – don’t do it.

For a deeper dive on the issues here please see Chris Wahl’s post (again) or this post on the VMware blogs about Monster VMs.


In the interest of time I’m going to go too deep here, but let’s just say NUMA is a technology designed to assign affinity between CPUs and memory banks in order to optimize memory access times.

vNUMA was introduced with vSphere 5.0 which allows this technology to be extended down to guest virtual machines.

The bottom line here is that the mix of virtual sockets and virtual cores assigned matters. As this article shows, processing latency can be increased if these settings are not optimal.

First you’ll want to make sure that hot-CPU add is disabled as this disables vNUMA in any virtual machine and then you’ll want to make sure that your allocation of virtual sockets and virtual cores matches the underlying physical architecture or you could be adding some processing latency to your VM as noted in this VMware blog post.

One more point here. There’s a setting in VMWare called PerferHT. You can read about it here, but it basically changes the preferences in vNUMA. There’s no universal answer here as it will vary from application to application, but this setting is a trade off between additional compute cycles and more efficient access to processor cache and memory via vNUMA. If your application needs faster memory access more than it needs compute cycles, you may want to experiment with this setting.


As it turned out all of our settings here were optimal. We had one vCPU socket with 16 cores – matching the 16 physical cores on the server – and vNUMA enabled. If you are using a Windows guest you can download Coreinfo.exe from Sysinternals and get more detail on how vNUMA is configured within your VM.

But we still don’t have an answer to our question – why is VM CPU at 80% when the host is at 41% given 16 physical cores (host) and 16 virtual cores (VM)?

Is it possible that not all the cores are being used?  Let’s check — here is a graph from vCenter showing that all 32 logical cores are being used (and tracking within 10% of each other) but average CPU is 19% peaking at 27% over the past hour:


But then we look at the VM for the same time period and we see the same pattern except that CPU peaks at over 90% and averages 45%:




How can this be? The VM is triggering high utilization alarms when the host is at less than 50% utilization.

Let’s go to ESXTOP to get some additional metrics, but first we need to understand the difference between PCPU and “CORE” in ESXTOP:


Click to Expand

So PCPU refers to all 32 logical processors while “CORE” refers to only the 16 physical cores.

Now lets look at ESXTOP for this host.


Click to Expand

Notice how CORE UTIL % is reported at 78% while PCPU util is only 40%. That’s a big difference! Which one is right?

If we look at the Windows OS, we see that CPU at that same instant was aligned with the CORE UTIL% metric:


It seems that there’s a couple things going on here. First, the CORE UTIL metric more accurately reflects utilization for THIS Monster VM scenario as it averages across 16 physical cores and not 32 logical cores. Second it seems that the CPU utilization metrics which we tend to rely on in vCenter and other tools tend to follow the PCPU (hyper-threaded) statistics and not the “core” utilization.

A few graphs to quickly illustrate this. First once more here’s CPU for both the host and the 1 VM that is on it as reported by vROPS 6:


Same pattern in both but the host is averaged across 32 logical processors while the VM is averaged across 16 vCPUs, which results in the numbers being almost double for the VM.

We can also see this by looking at MHz rather than percent utilization:


Without breaking down the math, the number of Mhz consumed by the VM divided by the capacity of the host, does align with the CORE UTIL% metric.

One thing I could not figure out about this chart is why the host shows LESS Mhz utilized. There should be no averaging here – just raw Mhz consumed – so it’s escaping me why the host would show less consumed than the VM (not possible in raw Mhz). If anyone has an answer for this I’ll gladly update this post with attribution.

So if I’m using vROPS 6, what metric do I use to see actual core utilization without factoring for hyper threading? The documentation I must confess lost me a bit. Allegedly this metric exists but I couldn’t find it anywhere:


After some trial and error I did find a CPU Workload % metric which does appear to focus on the cores (no hyper-threading):


Again the pattern is identical except “Usage” (top) is averaged over 32 cores – not accurate for our scenario – and “Workload” (bottom) is averaged over the 16 physical cores.  Here the Workload metric (bottom) gives a far more accurate picture which aligns with the VM level metrics.  If we look at just the default Usage % metric we are left with the impression that the host has far more resources to give and that our vCPU allocation (or something else) may not be efficient, but that does not seem to be the case here.

So what would these metrics look like on a host with many workloads and no Monster VMs (more common)?


Different scales here which makes the bottom chart appear more volatile, but the gap between the two is not a doubling like we saw before. The numbers are much closer.

Now here’s a question that troubles me. The default CPU metrics in vSphere count all the logical cores but look at the peak above. If I looked at the default CPU graph, I’d think I was at 74% when the physical cores were actually at 88%. I can see how averaging across all logical cores can provide a better view of utilization, but it seems to me that the Workload metric (physical cores only) provides a better watermark for detecting bottlenecks.


We’ve jumped about a bit here but if you’re still with me let’s try to nail down some conclusions from all this:

  • Hyper-threading does not increase execution resources, but it many cases it allows them to be used more efficiently depending on the workload (this benefit is often 10-15% in VMware environments).
  • The default CPU metrics in vSphere are averaged across all logical cores which includes those added by hyper-threading. This can result in confusing results when a single “Monster VM’ is running from a host
  • VMware’s guidance is to NOT exceed the number of physical cores on a host, when provisioning vCPUs to a Monster VM.
  • In vROPS 6 the Workload % metric appears to only look at physical cores and thus may be a better indicator for CPU bottlenecks in some cases.
  • vNUMA considerations including virtual to physical core allocation can impact performance.

As for our VM which was triggering CPU alarms, it appears that it is using an appropriate amount of resources on the host after all.  Now there’s a possibility that we could experiment with more cores and possibly get better results, but they key is we can throw out the 50% disparity between CPU utilization and VM utilization as bad data in this scenario.

And last but not least –

  • Measuring CPU utilization is not nearly as simple as we had thought.

One final note — this is my interpretation of what I am seeing. If anyone can offer better guidance (especially corrections) to anything I’ve posted here, please do so and I will be glad to update the post with attribution.

VM Snapshots — They can be a problem, but VVols in vSphere 6 can help

Snapshots in VMware have been an invaluable tool for years.  The ability to create an application consistent point-in time snapshot of a virtual machine has significant OPEX (or DevOps — take your pick) benefits. It can be used as an “undo” button for upgrades, it can facilitate clones and some replication solutions, but perhaps most commonly it is used to facilitate backups of virtual machines.

For me snapshots have always been a love-hate thing.  Wonderful feature but in some cases they’ve caused a lot of pain and disruption. In my vSphere 6 What’s New post I talk a bit about vVols — what VVols mean for snapshots just might be one of the least known and discussed features.

Here’s the issue.  A snap is created, the backup runs, and then the snap is closed.  It is this closing of the snap where the VM can become “stunned” for significant lengths of time.  I’ve seen this become an issue for highly transactional servers ranging from web servers, databases and email mail systems as well.

There’s even a VMware KB article that discusses this problem:


So what’s happening here? Think of this this way.

First the snap is opened.  From this point forward writes are not committed to the base virtual disk (VMDK) but a child VMDK.  The more writes that occur while the snap is open (often how long the backup takes), the greater the size of this child VMDK which will have to be consolidated.



Above you can see a VMDK with three snapshots open.  Writes go to the most recent snapshot, and the live state of the VMDK is actually a real-time calculation across this entire chain.  Once I discovered an Exchange server for which the backups were not properly configured — there was a chain of 58 snaps supporting a production Microsoft Exchange server! Yikes!

There’s actually an additional snapshot file that is created for application quiescing (Microsoft VSS) but there’s no need to go into that here.  Hopefully you already have an appreciation for how closing a snapshot can be problematic for transactional workloads.  These snapshots — and the child VMDKs created for them — need to be written back into the base VMDK.

For many VMs you might be able to backup and run snapshots just fine.  But in my experience, just an IIS server creating a steady output of IIS log files, can experience disruption during a snap close event — especially if you are doing a full backup. I’ve watched the snap close process freeze IIS servers to where web transactions are dropped and lost. And for large transactional databases you can just forget about it.

vSphere 6 and VVols

With the new VVol feature in vSphere 6 several things change.  First of all the base VMDK is ALWAYS the base VMDK.  It is always the write target.  The snapshots are now read only reference files that do NOT exist with a chain.  When the snap is closed, there’s nothing to ingest back into the base VMDK — it already has it!



This image is from Cormac Hogan’s post (link in article) which shows how snapshots are read-only references in vSphere 6 VVols.

This is a huge change from the previous method where writes went into the most recent snap in the chain and would have to be consolidated back into the base VMDK.  Now there’s nothing to consolidate when the snap is closed — the base VMDK is always the live state. VMware’s Cormac Hogan has an excellent post on this which goes into far greater detail on how this process works.

But that’s not all. VVols also enable the ability to offload snapshot functions to the array controller.  The implementation details may  vary among storage vendors, but the whole snapshot process can be offloaded to the storage array itself in some cases, providing instant and non-disruptive snapshots.

This is a huge change from vSphere 5 which should allow for backups and snap close operations on highly transactional servers where this might not have been possible in the past.  Impact free snapshot (and backuptennesse-vols-fans(2)) operations.

Now in full disclosure I’ve not had the opportunity to work with VVols in production yet, but perhaps you can see why I’m rather excited.  Non disruptive snaps and backups for ALL workloads would be a welcome feature.

Do you have any experience with snapshots with vSphere 6 and VVols? Post in the comments below.  Go VVols!

vSphere 6 (and VSAN 6) Now Available

The wait is over.  You can now download vSphere 6 and VSAN 6 from VMware.com.  I’ve written about the great features in vSphere 6 here and a lot of people are anxious to upgrade to start realizing the benefits of the new capabilities,  but hold on….before you do there’s some things you should probably review first.

First of all you’ll want to review this KB article which details compatibility considerations, upgrade considerations and much more.

If you’re using vCNS (vCloud Networking and Security) you will most definitely want to be reading this KB article before you start even making plans to upgrade.

You’ll also want to check out the vSphere HCL (Hardware Compatibility List) to see if things like your processors or your storage array will support vSphere 6.  The HCL even goes into specifics like VAAI primitives to show you which features are supported with which firmware releases on your storage array.  Also note that many storage arrays will support ESXi 6.0 with their current firmware release but support for vVols in some cases will not be available until a future firmware release.  As always check with your storage vendor.


Example of level of support detailed for a specific storage array varying with firmware version

You’ll also want to check with 3rd party vendors that integrate with vSphere.  This is basically any VIBs installed on host servers and anything that registers with vCenter Server, especially any plugins.  Backup/recovery and monitoring solutions are some of the most common here, but there’s also replication, storage acceleration and several others.  I’ve reached out to 5 vendors on the issue of vSphere 6 support it and none of them formally support it yet, but plan on delivering support in a matter of weeks or months depending on the vendor.  Whether you need to wait for formal support or want to be a pioneer depends on the use case and your tolerance for risk.  In some cases you may be able to “cheat” formal support while in other cases the risk may be just too great without formal support.

And last but not least read the releases notes!

Happy upgrading!

Top 6 Features of vSphere 6

Top 6 Features of vSphere 6

This changes things.

It sounds cliché to say “this is our best release ever” because in a sense the newest release is usually the most evolved.  However as a four year VMware vExpert I do think that there is something special about this one.  This is a much more significant jump than going from 4.x to 5.x for example.  It’s not just feature packed or increasing the maximums, although it does accomplish both of these.  vSphere 6 introduces a few new paradigms which have the potential to create a lot of value, efficiency, and also good old-fashioned performance.

In our clickbait, social media driven world, “listsicles” seem to be a favorite article style . When I began to look at vSphere 6 with all of the new features I thought to myself “where does one start”?  Perhaps just this once I’ll go full “buzzfeed” and list what I feel are the top 6 new features of vSphere 6. All without any diet tips, Tumblr feeds or embarrassing celebrity photos — I promise.

I think there’s some really game changing stuff here. Let’s dive in.

UPDATE (2/9/15):  This post has been updated to reflect details which have changed from the beta to GA. Changed information will be highlighted.

1) vVols (virtual volumes)

This is arguably the biggest new feature and has the potential to fundamentally transform how storage in approached in vSphere, so it demands that we spend a bit more time exploring this one.

VASA 1.0 ( vSphere Storage APIs for Storage Awareness) was introduced with vSphere 5 which enabled many features ranging from array integration, offloading of copying and zeroing operations, multipathing, and storage awareness, which gave vSphere insight into the relative performance of your storage tiers.

While these features were great, there were several limits, including that datastores could not offer granularity to individual virtual machines, but rather all virtual machines would inherit the capabilities of a datastore.  And while we could offload some functions to the array, snapshots were still based on delta files with copy-on-write mechanics.

While this is “OK”, what if every VM could have it’s own storage container and storage policy?

Today we spend a fair amount of time managing LUNs and Volumes in vSphere which in turn determine the storage characteristics.  My VM is on “SAN02-VOL03” but what does that mean to me as an application owner?

What if the storage array through APIs could become “aware” of vSphere elements?  What if each VM was it’s own container and vSphere administrators no longer had to deal with the management overhead and complexity of LUNs and file systems?  Just provision a server and choose “Gold”, “Silver” or “Bronze” storage — or have this predetermined by a policy.

This is what vVols along with VASA 2.0 aim to provide. Chuck Hollis has a great post going into more detail on this but for now I’m going to “borrow” one of the slides from his post to illustrate how this facilitates providing the right capabilities to the right consumers.


[Click to expand]

vVols and VASA 2.0 could be a blog post in and of itself, but to keep things simple let’s just focus on a few key characteristics of vVols:

  • VMDKs are native storage objects

That sounds good, but what does this mean exactly? Well it means that the storage array is “aware” of each VMDK and that the complexity of LUNS and mount points are no more. This layer of complexity is now removed from vSphere administration — going forward administrators only need to focus on VMs and storage policies.


Traditional Storage versus vVols (click to expand)


  • Virtual Volumes

Each virtual volume maps to a specific VMDK.  Because of this exclusivity, SCSI locking is no longer necessary.

  • Storage Containers

In vSphere 6 a new logical construct is a Storage Container which can contain multiple virtual volumes.  Storage containers are managed by the storage array and can be used to group together storage that will share common characteristics and/or a common storage policy.

  • Single Protocol Endpoint

All storage is unified behind a single logical construct for I/O—called a Protocol Endpoint. With all storage traffic passing through this logical element


vVol — Protocol Endpoint (Click to Expand)


  • Policy Based Management

Now we can have policies that we apply to VMs to govern capacity, performance and availability. Rather than managing this on the back end with LUNs and volumes we can now simply apply policies that provide the desired capacity/performance/availability configurations on a per-VM basis.  We used to do this with scripts (or CLI) against hosts for specific LUNs — now we can simply define a policy and assign it to storage objects as desired.

  • Storage Array Integration

Storage vendors can integrate with the VASA APIs to offload I/O functions (array acceleration) and granular capabilities. This existed for some functions with VASA 1.0, but now with VASA 2.0 the opportunities to unlock the full capabilities of the storage array are available throughout the vSphere ecosystem.

For just one example of this, think of the way snapshots work today – a separate file is created which is basically copy-on-write which must then be reconstituted back into the VMDKs when the snap is closed. Many of you are already familiar with the performance impacts of these operations – which are especially common with backup and replication operations. Imagine if all this could be offloaded to the storage array for fast and space efficient snapshots!

And let’s not stop there as these benefits can be extend to provisioning, replication, deduplication, caching and more.  In my opinion this is HUGE — you may be familiar with the benefits of space efficient snaps and clones, but these were always outside the domain of native vSphere snapshots.  Now all storage vendors have the ability to provide hardware accelerated snapshots (big impact on backups) as well as instantly deploy space efficient clones for test/dev and more.   There’s a lot of implications here for replication and disaster recovery as well. Pictured below are just some of the vendors that have made commitment to supporting vVols.

vendorsIn a nutshell, the complexity of LUNs and volumes is removed from the vSphere administrator, while enabling policy-based management and hardware acceleration from storage arrays for many common functions. Fast and space efficient snapshots. Space efficient instant clones for test/dev.

We’ve had storage APIs for a few releases now, but this level of integration between the storage array and the hypervisor is new.  In many ways it’s a game changer.

(I’ll update this post in the future with links to more detailed vVol articles as they become available).

2) Fault Tolerance

VMware fault tolerance was always a fantastic solution but it’s use was always limited due to the restriction to only a single vCPU, no snapshots and more. Now these restrictions are being removed, opening up new possibilities.

If you’re not familiar with Fault Tolerance, a second clone of a VM is maintained in CPU-lockstep such that either VM in the pair could become unavailable and a single CPU cycle would not be missed, nor would any TCP connection be dropped. This is critical for transactional applications, e-commerce, VoIP and many more mission critical applications.


Now with vSphere 6, Fault Tolerance is now available for VMs with up to four (4) vCPUs and 64GB of RAM, enabling it to be used for larger web servers, VOIP, databases and more.  You’ll want to test your applications first for latency and failover response, but this opens up fault tolerance to a whole new set of VMs that couldn’t leverage this previously.  Some may also want to consider this as an alternative to Microsoft Cluster Server (MSCS) in some scenarios.

vCenter Server is an obvious potential use case here, especially with the retirement of the vCenter Heartbeat product.  Some deployments of vCenter Server will be supported for use with FT based on the size and scale.  Exact details of the requirements for vCenter Server support are still being worked out. (Separate from FT, Microsoft Clustering for vCenter Server will also be supported).

vSphere 6 also adds support for VADP based snapshots (not user snapshots), enabling backups and replication. Also added are support for paravirtualized devices, and storage redundancy for Fault Tolerant VMs which is critical for many use cases.

3) vMotion Improvements

vMotion has always been an incredible feature in vSphere which helps to provide both flexibility and availability, but now several new features will allow its use to be significantly expanded:

  1. vMotion across virtual switches
  2. vMotion across vCenter Servers
  3. Long Distance vMotion

The last one refers to a dramatic increase in latency tolerance as noted in this tweet from this past VMworld:

Put those three together and you now have the ability to vMotion to different regions. I worked on a large datacenter migration project (petabytes) where we had to populate data mules and ship them to remote datacenters to “seed” the replication process. I can only imagine how much time and money could have been saved if this technology were available then.

Future enhancements will support for active-passive replication as well as vSphere Replication.

4) Policy Based Management

Update:  While Policy Based Management was featured in the beta, it seems that it has been witheld from the 6.0 release and will be introduced in a later update.  The Content Libraries feature mentioned below will still be in the initial 6.0 release as I understand it.

There’s a few components to this including a new Virtual Datacenter Object which is essentially a resource pool which can span multiple vSphere clusters and facilities the assignment of policies to VMs. For example you might want to create virtual datacenters for production and another for test/dev and have these span multiple sites (and clusters). In the initial release this will be limited to a single vCenter server, with plans to support multiple vCenter servers in a future release.

polixyAnother new logical construct is tags which can be applied to any VM. These tags can be used to automate the initial deployment of VMs and ensure that the proper policies are maintained throughout the VM’s lifecycle.

Also worth a mention here is the new Content Libraries feature. Very often in VMware environments administrators will carve out datastores and/or folders for VM templates, ISOs, vApps, scripts and more. Now you can have a full content library for your virtual datacenters that can even be published across them.

With the ability to aggregate this content into a library, which can be shared and published to multiple vCenter servers, content can be standardized and made more accessible. You might even want to have different content libraries for different teams, business units and/or applications.

5) Installation and Usability Improvements

This is sort of a collection of multiple features, but I’d like to briefly touch on each as they are significant:

a) vCenter Server Appliance with guided install from ISO image

In vSphere 6 the vCenter Server Appliance has made improvements with feature parity and is now provisioned using a guided process from a self-contained ISO. I went through the guided process and it is much more quickly deployed than in prior versions.

b) Infrastructure Controller

With vSphere 6 a new Infrastructure Controller (IC) service is introduced which provides the following functions:

  • Single Sign-On (SSO)
  • Licensing
  • Certificate Authority
  • Certificate Store
  • Service Registration

Depending on your topology and requirements the Infrastructure Controller can be deployed within a vCenter Server or as its own independent server. This not only facilitates scale and more complex topologies but it simplifies both deployment and management.

c) vSphere Web Client Improvements

vSphere 6 still ships with a traditional thick client (C++) but newer functionality specific to 5.5 and later will require the Web Client which has been substantially improved in this release. The login time has been reduced to about 3 seconds while other common functions within vSphere have been improved by several full seconds (such as from 4 seconds to 1 second for invoking the Data Center Action menu).


The task pane returns to the bottom in the improved vSphere Web Client (click to expand)


Not only is the Web Client significantly more responsive in this release but navigation has been significantly improved by providing more right-click menus and adding the tasks pane back to the bottom of the screen.

The combination of the performance and usability improvements makes it easier to be more productive in vSphere as well as making the experience more enjoyable.

6) vCloud Air Integration

Hybrid_CouldYou already have a vSphere infrastructure but what if you could quickly add for unplanned capacity using vCloud Air and make a hybrid cloud?

What if you could quickly set up full disaster recovery capability for your most important virtual machines using vCloud Air with 15 minute recovery points?

vSphere 6 has built in integration with the vCloud Air service allowing you to quickly tap into these hybrid cloud capabilities.  On the backup and DR front, vSphere 6 features RPOs as low as 15 minutes, allowing you to effectively use the vCloud Air service as a hot site for your production workloads, with support for both failover and failback operations.

Recently I wrote a review of VMware’s vCloud Air OnDemand service and I was honored that VMware had elected to share it. Rather that talk about it here, I’ll just link here to my post on vCloud Air for more information on that service.

Note:  an earlier version had stated that RPO’s would be 5 minutes.  This was based on information communicated during the beta.  In the initial (GA) release the RPO will be 15 minutes.

Honorable Mention

Of course there’s many more features than just these six, so here’s a few I want to just briefly mention:

  • Storage I/O granularity improved to per -VM basis (was per LUN).
  • Network I/O control allows bandwidth reservations for the VMs that need it.
  • 64 node clusters hosting up to 8,000 VMs
  • VMs up to 128 vCPUs and 4TB of RAM
  • Hosts with up to 12TB RAM, 64TB datastores and up to 1,000 VMs
  • NFS 4.1 client enables multipathing, improved security, improved locking and less overhead for NFS storage.
  • vCenter Server resiliency — vCenter Server will now attempt to “self-heal” at several different levels in order to improve availability.
  • vSphere Replication now supports RPOs of as little as 15 minutes.

There’s a lot here, and the combination of vVols with VM level policy management and tagging will be huge.  Performance benefits aside, administrators can now organize and combat the configuration drift of VM sprawl by designing policies that will automatically place VMs on the desired class of storage, with the appropriate performance and availability policies.

Many of these features are worthy of their own blog post, but I hope this quick list introduced some of the reasons why I think vSphere 6 is one of the more significant releases in VMware’s history.

Best of 2014

These lists typically annoy me but I was curious what some of my most popular content was and since I have it right here….

The most popular blog post by far was “Exploring VMware’s New OnDemand Private Cloud” which was shared over 650 times on social media.

Below are some of the most popular tweets of the year, several which came at the Vice President’s expense:

Exploring VMware’s New OnDemand Private Cloud (Part 1)

Screenshot_108UPDATE:  vCloud Air OnDemand is out of beta and has now entered an Early Access Program for which you can sign up here.

Recently I’ve had the opportunity to explore a beta of VMware’s upcoming cloud offering – vCloud Air OnDemand through their Ambassador program. I wanted to share my observations and experiences, but there’s so much to talk about, I found it better to start with an introductory post and drill deeper with a walk through some of the details in a future post.

The quick version is that vCloud Air’s Virtual Private Cloud OnDemand is pretty much what it sounds like. Hosted IaaS (Infrastructure as a Service) running on VMware, enhanced with SDN, with on-demand availability and pricing — meaning that you are billed only for what resources (CPU, memory, disk, etc.) are actually consumed. It’s like the electricity meter on our homes, but this is measuring the resource utilization of your virtual datacenter in the “cloud”.

Amazon (AWS), Azure and Google are on most everyone’s short list for IaaS service providers but there may be some good reasons to put VMware on your short list as well.

The vCloud Air service is compelling for several reasons. To start, it runs VMware vSphere which provides easy and familiar methods for integrating with existing on-prem infrastructure. Perhaps you have a new project but don’t have time to wait to add more hardware and capacity, but still need to maintain operational methods and security. For many use cases vCloud Air Private Cloud will be seen as compelling — especially where vSphere is already used. And with over 99% of the Fortune 1000 using VMware, that’s…well…most of us.

Before we explore Virtual Private Cloud OnDemand in more detail, I’d first like to step back and review different cloud types, use cases.

Private, Public, Hybrid

The original key distinctions between private and public cloud were mostly control and multi-tenancy. With a private cloud the hosted infrastructure was exclusively yours and therefore afforded more control, whereas in a public cloud your workloads might be shared with those of others on the same hardware (multi-tenancy) which could lead to the “noisy neighbor” problem.

Advances in hypervisors, I/O virtualization, SDN and orchestration have made this a bit less of distinction now days as more control is available to the consumer and the “noisy neighbor” is not the threat that it once was.Hybrid_Could

A Hybrid cloud then is essentially a combination of an “in-house” private cloud and infrastructure from an external service provider. A perfect example is a business that runs VMware vSphere internally in their datacenter. Let’s say a new project comes along, and rather than buy new infrastructure (and incur the associated delays) they could just logically extend and scale their existing vSphere infrastructure to a hosted offering, and be billed only for what is consumed.

Is vCloud Air Hybrid or Private?

In 2013, VMware launched the vCloud Hybrid Service (vCHS) which was positioned as the the hosted cloud infrastructure needed to evolve an on-premises environment into a hybrid cloud.  The vCloud Connector facilitated building a unified view of the hybrid cloud, allowing the ability to view, manage and migrate workloads from either the on-premises side or the hosted side.

Just this past September the service was re-branded as vCloud Air with the service offering now called Virtual Private Cloud (a dedicated option is available). What changed that it’s now called a private and not a hybrid cloud? Yes, there’s a bit of marketing here but also a pretty important point.  Private cloud is all about control.  Do you control the security, the operations, the processes?vchs-vca1

When you start with the vCloud Air service you create a virtual datacenter.  There is no external access until you establish firewall rules, public IPs, SNAT/DNAT rules, routing and more.  There’s also VPN and load balancing services built in.

If that sounds like a lot, it’s not and it’s quite straight forward as you’ll see in the next post, but the point is that you have such a strong level of control that really can be considered a private cloud.  It’s like the difference between ordering a sandwich someone else designed versus building your own.   As an engineer who has encountered the friction and delays that silos bring, I found it liberating to be able to quickly design the virtual datacenter — network, storage, compute — to my requirements.  And of course if  you integrate the Virtual Private Cloud with an on-premises environment, you still have a hybrid cloud spanning those two environments.

Introducing vCloud Air Virtual Private Cloud OnDemand

The “original” vCloud Air Service that went live last year is Virtual Private Cloud.  It is powered by vCloud Director, providing VMware users with a familiar construct and interface with their on-premises environment.  With this service, capacity is purchased in “blocks”.   For example a starting block might consist of 20GB of memory, 10Ghz of CPU and 2TB of storage (pricing as of November 16, 2014 shown below).Screenshot_111

The new OnDemand service has many similarities with the original service.  They both run vSphere and vCloud Director.  They both employ SDN using VMware’s own offerings. They both integrate into vCenter Server using the vCloud Air plug-in.  They both allow stretched Layer 2 and Layer 3 so that you can “bring your own IPs” and also feature Direct Connect options (private circuit).

My understanding is that the OnDemand service is a new “pod” within the vCloud Air service meaning that it is a new and separate rack design and configuration.  The new OnDemand service — as it’s name would suggest — uses an OnDemand pricing model.  Rather than purchasing “blocks” of capacity you will be billed for what you consume as you consume it.  I haven’t done much for the past 24 hours but below you can see a screen shot of my billing for that period, broken down by CPU, memory, storage (SSD and standard tiers), and public IPs.


Click to expand

Each account has a single billing point but as we’ll see in a future post, it is possible to create multiple virtual data centers (VDCs) within your account to both track internal costs and well as to control access.

Use Cases for Virtual Private Cloud OnDemand

There’s many different use cases that are a very good fit for the OnDemand service.  If you’re a new company without much capital you might want to just use the virtual datacenter as your primary datacenter.

If you’re a medium or large business with an established on-prem vSphere infrastructure, you might elect to keep your most critical applications and data on premises, but still leverage the OnDemand service for seasonal capacity, test/dev and new projects.

I was working at a Fortune 500 once when a new project came up which required a large amount of  web servers, databases and middleware.  How nice it would have been — and how much more quickly we would have been able to execute — if we could have simply defined our vApps in Virtual Private Cloud OnDemand and then clone and distribute them as needed in the vCloud Air service.  You might even choose to keep the databases on-premises but put the web tier out in the cloud.  You have the flexibility to align your workloads between on-premises and vCloud Air with whatever balance and topology works best for your organization, your security and operational requirements — you have the flexibility to allocate as you see fit.

Disaster Recovery

As you could imagine, disaster recovery for on-premises vSphere deployments is a very popular use case and quite straight forward to setup.  Today, the Disaster Recovery option is offered as a discounted tier on the original Virtual Private Cloud Service but it is my understanding that this will move to the OnDemand service in the future.  This would be a very effective pricing model as when using the capacity for hot-site replication, most of your resources in a passive state will be storage.  CPU and memory would be at relatively low levels until a fail-over occurred at which point they would increase with all the instances coming on-line.  OnDemand capacity when you need it.

Sign Up and Getting Started

I’ll go through a detailed walk through later, but the effort required to start creating VMs and consuming resources is relatively low.  I simply registered for the service, supplied a credit card, and once I was confirmed I was off creating my virtual data center and spinning up virtual machines and vApps.  This was my first time using vCloud Air but it was not my first time using VMware and as a result it didn’t take me much time to quickly find my way around and be productive within the vCloud Director interface within the vCloud Air service.   Within a few hours of signup, most should be able to define their networks and start provisioning VMs.

VMs, vApps and Catalogs

Within vCloud Air there is a public catalog from which you can instantly provision new VMs.  At this time, the public catalog includes multiple editions of CentOS, Ubuntu and Windows Server.  The Windows Server VMs will incur a licensing surcharge for their use which is prorated to an hourly rate.  In other words you are effectively renting the Windows Server license cost by the hour.

There’s two other important ways to populate your own private catalog within vCloud Air.  First you can import any OVA into your private catalog as either a URL link or a local file — which includes the over 1700 virtual appliances available on the VMware Marketplace.  The second way is to simply upload your own ISO to your catalog.  Just to prove a point that it could be easily done, I uploaded a Windows ISO to my private catalog in vCloud Air and I was able to build a VM from scratch right form ISO.  Also using the vCloud Connector you can even keep your catalogs in sync between our on-premises vSphere environment and vCloud Air.

vApps are a vCloud Director construct which solves several problems.  You can add multiple VMs and define rules for how they should work together.  A vApp can be an n-tier application or just a set of servers that need to be managed by a common team.  You can define leases on vApps as a cost control measure (i.e. power off after x hours, delete storage when off for x days) and even fencing, which ensures VM clones which exist in multiple vApps have unique MACs and IP addresses.  More on this later but there’s a lot of rich capability here for designing and managing your virtual datacenter.

Unified View

The vCloud Air plugin that is built into current versions of the vSphere client provides support for administering vCloud Air right from within the vSphere Web client.  The video below provides a walkthrough of the functionality available in the vCloud Air plugin.


Having run administered many vSphere environments I’ve been somewhat spoiled by the ability to quickly extract rich metrics on VMs and hosts using vCenter and even more with vCOPS.  In the vCloud Air environment you can see your CPU, memory and storage utilization for your virtual datacenters and vApps but that’s about it.  The hosts really don’t need to be in the picture (that’s sort of the point of a cloud service) but it would be nice to know some key VM metrics (what’s my storage latency or memory allocation over time?

Two things here.  One is that there’s nothing stopping you from using the monitoring solution of our choice. Want to use Microsoft System Center, CA UIM, Nagios, etc?  Use whatever processes you use today in house.  The second thing is that VMware has a robust monitoring solution in vCOPS.  I would not at all be surprised if VMware were to release a version of this that would work within vCloud Air in the future.

UPDATE:  The vCloud Air adapter for vCOPS was released in July, 2014.  Below are some screenshots of vCOPS monitoring vCloud Air with more at the link:


Click to expand



There’s much more here in terms of features and even connection options that I haven’t drilled into here and which I’ll try to explore in future posts. But just to back this up a bit, many IT consultancies have suggested that hybrid cloud is the new normal — the business having the ability to consume on-prem and hosted capacity as needs arise, with use-case flexibility and functional integration (i.e. the vCloud Air plug-in in vSphere). Some cloud providers will require you to make adjustments to operational procedures and security, but vCloud Air does a good job of making this feel seamless for VMware shops. Also keep in mind the appeal of multi-cloud (using more than one cloud service provider) which can be used to mitigate risk, provide flexibility and expand DR options.  And if you don’t already have a DR solution you may want to take a look at vCloud Air’s Disaster Recovery Service.

Most companies will want to explore options for both hybrid cloud and multi-cloud scenarios for many compelling reasons.  As a long time VMware vSphere engineer, I found the vCloud Air service very accessible and easy to quickly get started with.   If you have a significant VMware vSphere deployment in our organization or even if you are just starting out, you owe it to yourself to include vCloud Air in your short list of options.  With the new OnDemand service with its utility pricing model being prepared for launch and more datacenters being added globally, the vCloud Air solution is worth taking a close look.