Expensive Hypervisors - a bad idea even if you can afford them

There's a saying in politics People will fall more easily for a big lie than for a small one*. I think someone at VMware has picked this up because they're trying a really big lie.
VMWare doesn't need memory.

Or to quote the post more precisely: VMware Infrastructure's exclusive ability to overcommit memory gives it an advantage in cost per VM the others can't match.**

They were able to start 40 instances of Windows XP to achieve the 40 VMs, with 512MB of memory on a machine with only 4GB of RAM - a 5 times over commitment ratio.  Of course they didn't actually run anything in them, because if you and I fired up Outlook, and IE (with our own mail boxes and choice of pages)  you open word and I open PowerPoint very few memory pages will be sharable (I've got 47 pages open in IE right now, and it's using over 300MB of RAM, almost all for data). That means a lot of paging will have to happen in the virtualization stack. Brace yourself for really poor performance.

When I read they were using Windows 2000 in the server test I got suspicious, why use an OS which is out of support ? to get better results of course. (As I understand it they don't support Windows Server 2008 yet). To get as much page overlap as possible they ran identical OS, and application on all the VMs, and gave them only a little memory - 512MB for a server is pretty meager. That's important because it ensures memory is occupied by OS and programs  rather than data. Programs will have a lot of sharable pages of memory: data and caches won't.  Suppose they had done the same over commitment with 1.5GB machines and got 4  on their system to 2 on Microsoft Virtualization.  Spare memory will get used as a cache and this will have almost no sharable pages. Data is in from disk to fill memory which isn't there and has to be paged out in the Virtualization stack - that's two sets of disk I/O for nothing.

The writer then cooks the figures: a $6000 for a  2-way server with 4GB of RAM ? I went onto Dell's web site, and got a quote for 2xDual Core Processors, 16GB of RAM and 2x500GB drives in an SC1435 for under £2,000. Given that the UK prices for hardware are always higher than US prices I'm very dubious of that server price. But let's allow that. According to the VMware post their software costs an extra $5,750 (note that's without support). Again I'm working from the memory upgrade prices on Dells web site, an upgrade from 4 to  32 GB upgrade adds £2,720 to the price. At today's Exchange rate, and ignoring the cheaper prices in the US, that's $5,440. So according to VMware a $17,000 system based on their software would have 4GB of ram and run 14VMs. On 4GB the system would run 7 VMs on Microsoft software. But the same money using Microsoft Software would buy 32GB of RAM, the extra 28GB which allows you to run 56 more VMs. That's 63 VMs on $17,000 of system with Microsoft ~ that $270 per VM for Microsoft, against $1268 for per VM for VMware. And with physical RAM for every byte the VMs see, the Microsoft system will perform better.
(To save smarter readers from pointing it out, a $6,000 server might run out of steam between the 7 VMs initially supported, and 63 VMs. The VMware work assumed there would enough CPU, DISK and network resource and the system would be RAM limited. If 14 VMs is where CPU becomes a limitation then we'd be looking at a 4GB upgrade costing £260 /$500, so the Microsoft system would come out about $5,250 cheaper for the same VMs - that's about $375 per VM.)

That's the argument ? Over $5000 of VMWare software is a good investment because it saves $500 on RAM ?

Further Cooking the figures. If you have a Microsoft support contract Microsoft Virtualization Software is covered. One of my colleagues tells me that VM platinum support for 3 years, would add $3666 to the total cost. I haven't checked this numbers but divided over 14VMs that's $261 per VM for VMware support... yes you read the numbers correctly support for VMware costs nearly as much per VM as the hardware, software and for the Microsoft solution. That's before you allow for training, maintaining a separate management infrastructure for the virtual world, and so on.

If someone can show me a customer who is running, in production, a VMware VI3 Enterprise system with a 2:1 memory overcommit ratio on all the VMs, where spending the cost of VMware on RAM wouldn't remove the need to use overcommitment then I'll give... lets say $270 to their choice of charity. ***

Update. Someone asked me to remove a term which is OK in British usage, but tends to offend other English speakers.

Technorati Tags: Windows Server 2008,Windows Server,Virtulization,Hyper-v,VMware,Fud

* Foot note 1. Yes I know who wrote that, and I'm not going to suggest that VMware have been reading him. Like me, someone there might collect books of quotes and aphorisms.

**Footnote 2. As one of the commenters to the post points out VMware's published recommendation is  "Make sure the host has more memory than the total amount of memory that will be used by ESX plus the sum of the working
set sizes that will be used by all the virtual machines." I.e. Don't use this feature, if you want decent performance.

*** One offer only. Void where prohibited. Over 18s only. My cause drowsiness; if affected to not operate virtual machinery. etc.

Comments

• Anonymous
  January 01, 2003
  In the comments an earlier post , "PMunro" raised an interesting point, and I mentioned it

• Anonymous
  January 01, 2003
  The comment has been removed

• Anonymous
  January 01, 2003
  Today's post on Edge is an interview I did recently with Michael Niehaus and Jeremy Chapman regarding

• Anonymous
  January 01, 2003
  The comment has been removed

• Anonymous
  January 01, 2003
  @"ME" Not sure which table you're refering to. "You need half the number of servers to run a given number of VMs on ESX than on Hyper-V." No you don't. You need half the amount of RAM. You don't need half the amount of CPU, you don't need half the disk I/Os (in fact you need more disk I/Os) If you were going to double the memory on the Microsoft system by doubling the number of servers the servers could (a) Run Windows Server Enterprise instead of Data-center (b) Use one Processor instead of 2 (c) Use half the disk capacity, and slightly less than half the number of spindles (cheaper disks). but they would produce more than half the heat / power so the running cost for two severs would be more than one, but less than 2 consolidated servers. Once you to take that into account the rest of your numbers just unravel. @ Mike, I've replied on your blog. Assuming they're in production I'll pay and post something here.

• Anonymous
  January 01, 2003
  It's quite amusing to see the VMware blogger ( Citrix's blog names him as Eric Horschmann) has come back

• Anonymous
  January 01, 2003
  These past few days, we've seen some back and forth postings taking place on various corporate blogs around the topics of virtualization ROI and a unique feature found in VMware ESX Server called memory overcommitment. VMware's Eric Horschman posted an

• Anonymous
  January 01, 2003
  I'm only quoting the numbers that were on the blog post. But even if you use $1000 for the cost of VMware that's still an 8GB upgrade for the Microsoft system, and the numbers in the post still don't hold water. It's always dangerous to try to match two vendors products too closely, but with Hyper-V at $28 and having a better management story to offset the lack of a "v-motion" alike, the number of good reasons to buy VMware will shrink. They won't all vanish with the first version.

• Anonymous
  January 01, 2003
  It has come to my attention that some of our competitors are making various claims about SCVMM 2008 and

• Anonymous
  January 01, 2003
  The comment has been removed

• Anonymous
  January 01, 2003
  Your point would be what ? As far as I'm concerned when people develop any kind of intellectual property (from Music and Photos to software). They should at liberty to give it away, bundle it, or charge a kings ransom for it AS THEY SEE FIT. (Courts and Governments modify that in ways I don't agree with ....) If other people want that intellectual property they take it on the terms offered. We went into Release candidate for Hyper-V today. VMware already has some features which we won't have until (at least) the next version, and (rightly) can charge a premium for those features.  Customers decide if the features justify the premium - for example to one set of customers, Vmotion is of no value whatsoever and to another set its worth many times the cost of VMware. I'm not going to criticize anyone's reading of the demand curve - so Let them "make their buck, their way" What I do object to, and will do my best to shoot down is spurious arguments being advanced for the premium priced features.

• Anonymous
  January 01, 2003
  We're two dates into our roadshow and I've twice been asked to do a comparison of VMware and Microsoft

• Anonymous
  January 01, 2003
  @ "Me" Hyper-V handles running as many VMs as you've got memory (disk/CPU/Network for). That's why this test is completely bogus, artificially constrain one resource [memory] and show that a piece of software gets round that constraint. However the cost of removing the constraint via software is 10 times that of removing it by hardware. I think the product group are aiming for better numbers than virtual server and we've shown that in public with hundreds of VMs on it. @Pmunro. Well, VMware's own perf guides say it's a bad idea. Do you want to be in the support call with an app vendor who has taken steps to prevent critical bits of code being paged to disk by the OS only to have those self same bits paged out by the virtualization stack ? (actually that should be another post in its own right). Give some thought to what apps do when they see spare memory - they USE it. So you're reading stuff from disk to cache it, only to have the virtualization stack page it back out again. There are exceptions - for example they only way to get a training lab going is to over commit a little.   If BobMu says we have to have it in the product is that because it's expected or because it would be a good idea for customers to use it ...

• Anonymous
  March 13, 2008
  So I take it that Microsoft won't be adding this feature :)

• Anonymous
  March 13, 2008
  The comment has been removed

• Anonymous
  March 14, 2008
  The comment has been removed

• Anonymous
  March 14, 2008
  >Over $5000 of VMWare software is a good >investment because it saves $500  on RAM ? It actually is a good investment.

• Anonymous
  March 14, 2008
  The comment has been removed

• Anonymous
  March 14, 2008
  Is it true that Hyper-V only handles six running VMs?

• Anonymous
  March 14, 2008
  The comment has been removed

• Anonymous
  March 14, 2008
  I've got a wide variety of VM's running in ESX.  They differ by OS, patch levels, applications running.  I still see the benefit of memory overcommitment, despite these differences. Also, with regards to having to purchase the Enterprise version to do all of this, if you aren't using the advanced features (which Microsoft doesn't have yet), then the cost does drop by $2000 to $3750.  And now that the 8GB limit has been removed from the Foundation Edition (formerly Starter Edition), the price drops again to $1000.  That being said, the $$$ that you say VMware will cost is very misleading. I look forward to when Microsoft's offering is as capable.   That's when it will really get interesting.

• Anonymous
  March 17, 2008
  >Over $5000 of VMWare software is a good >investment because it saves $500  on RAM ? Your math is off.  If your virtual machines are bound by memory and not by CPU, than you are saving much more.  There is a good chance you can reduce the entire size of your farm based on memory over commitment.  Now you are saving the fully burdened cost of one or more servers which is significantly more than 5k.  Consider labor, real-estate, hardware, and hardware acquisitions costs.  Add in incremental costs such as power and administration.  

• Anonymous
  March 17, 2008
  jamesone, you're missing joes point. Yeah, the cost for an extra 4GB of memory is less than the cost of the hypervisor...in a single server. But over an entire farm, you're going to be able to reduce the number of servers required to do the job, along with the accompanying reduced costs in power, support, etc. Most virtualization projects I've been a part of have utilized servers with 16GB+, 32GB is seeming to be the usual...Now we're talking about much larger discrepancies in the number of VM's that can be hosted, as well as much larger cost savings in hardware and service/support costs.

• Anonymous
  March 17, 2008
  James: I cannot agree with VMware with a general over commitment ratio of 2, but the question here is not the 4GB RAM machine: take a server of whatever type and put in the maximum possible RAM (it is so cheap...). In that case you don't have the option to put in cheap RAM to run more VMs: you have to buy a new server. And a technical question: how much overhead does a 1GB, 1VCPU, 32bit VM has in Hyper-V? (in case of ESX it is 97,35MB) How much RAM does Windows Server 2008 need if used for virtualization? (ESX Console needs ~272MB)

• Anonymous
  March 17, 2008
  ... "it's a really, really bad idea to use it". James, tell us please, how many users of your software have you met in the past 6 months? Memory is the number 1 resource constraint in virtualised production environments! And when you virtualise desktops, memory overcommit is a godsend. We can put 15-20 desktops per core comfortably under ESX. What do your tests suggest you can do with Hyper-V?  

• Anonymous
  March 18, 2008
  Thanks for your reply. One more question: why have you omitted VI3 Foundation from the last comparison table...?  :-) Buy only 16GB RAM and VI3 Foundation for the same number of VMs as Hyper-V and you still have some free capacity in your server (e.g. if you need to run more VMs, you won't have to buy a new server, you only have to buy some cheap RAM). Now let's look at some more practical approaches/numbers. At first let's stick at VI3 Enterprise and the standard "VM blogger prices" ($5998 Windows cost/server, $6000/server) and 2:1 ratio. You need half the number of servers to run a given number of VMs on ESX than on Hyper-V. Let's say you need 2 ESX Servers, thus you need 4 Hyper-V servers: ESX cost: 2 *(5998 + 6000 + 5750) = $35,496 Hyper-V cost: 4 * (5998 + 6000) = $47,992 That is you save about $12,000 (25%) with ESX and you have:

• more functionality
• a proven technology Now you can say that 2:1 over commitment ratio is not real in general. Ok, I agree. But if that ratio is let's say 1.2:1 compared to Hyper-V and you buy five $10,000 price ESX server: VI3 Enterprise cost: 5 * (5998 + 10000 + 5750) = $108,740 VI3 Foundation cost: 5 * (5998 + 10000 + 995) = $84,965 Hyper-V cost: 6 * (5998 + 10000) = $95,988 Thus you pay
• 10% less for a proven technology or
• 10% more for more functionality and proven technology. You can choose. If you buy more expensive servers (or add related infrastructure costs such as UPS, networking, space, ...) the difference between VI3 Enterprise and Hyper-V will be even less.

• Anonymous
  March 18, 2008
  James, You asked for a customer that was using memory overcommit in the real world. We just posted such an example. Go here for the details: http://blogs.vmware.com/virtualreality/2008/03/memory-overcomm.html. And thank you for the kind donation to the less fortunate in our world.

• Anonymous
  March 19, 2008
  It would be great if Microsoft would adopt this "free" model to their CALs for all of their backoffice/server products.  Maybe if that happened they could have a valid platform upon which to stand and chastise VMware!  Until then, you (MS) are making your buck your way, and VMware is making their's their way.