First off, I would like to take a moment to thank Mr. Irish Spring for the level of access provided today! You are a saint among men! I really appreciate the transparency and the conversation about everything! This was an amazing opportunity for me and I appreciate it.
Second off, I have a very sweet spot in my heart for labs like this (see PDX VMUG Performance Lab – 2010 I organized).
So… I was sitting in a session when my phone “blew up” (aka – I saw a tweet that the lab had just hit 100,000 VMs deployed). That’s a huge deal, right!? So, I thought I would write up a couple (or 8 ) questions for me to interrupt some important person’s day and ask for a response. Fate, so it seems, was on my side. When I went to the lab environment, Irish Spring was ready and willing to answer my questions. However, not only that, he offered a tour of the area, answered every single question (and more), and let me participate in the celebration of both the person that completed the 30 labs first as well as the person completing the 11,000th lab.
Instead of just answering the questions, Irish and I had some very exciting conversations. The following are the questions I had asked. However, I would not call that the end of this posting by any means. These labs go much deeper… to levels that most people do not realize.
Question and Answer
Q1) Which lab was #100,000 VM deployed on? View 4.5
Q2) What did the person win? A Golden Ticket and a coffee mug
Q3) What were you and your team thinking when #100,000 hit? Completely amazed by the speed of accomplishing such a feat. We were nervous before the labs and completely caught off-guard.
Q4) Projected lab count for VMworld now that you have hit 100,000? Around 14,000 attendees and around 140,000 VMs deployed (at this rate).
Q5) How has the lab performed compared to your expectations? Surprisingly, the East Coast based datacenters have been performing better than the SFO based datacenter.
Q6) Any failures / failover events? 2 instances
- (VirtualBill – I can confirm this as I was in the labs when it happened) There was a network issue on the first day that caused some of the sessions to hang/stall. Paul Maritz showed up at the exact time. But, he was able to witness a failover to another site until the issue resolved itself.
- We lost a rack of compute nodes in the SFO datacenter. The conference power was not consistent enough and a rack went offline. However, HA kicked in and only 2 sessions were truly lost. All others hosted on the compute rack were offloaded elsewhere and the sessions resumed! Once the rack was brought online, all of the compute nodes were restored and HA moved the operations back to the rack.
Q7) Any changes for the VMworld Europe setup? The scale of the lab setup is much smaller. So, the sheer size is dropping. The entire lab setup for Copenhagen is going to be leaving from Moscone West by this coming Sunday for arrival (via sea) on site. Additionally, a direct circuit between VMworld Europe and Ashburn, VA is going to be setup… although, it will still be accessed via Cloud methodology… just fewer hops to ensure proper connectivity for lab operation.
Q8) Any ideas, yet, for the next VMworld setup? Good model to keep? (now, I just want to make this very clear that this is not any official direction from VMware, or anything like that. This is just a “pie in the sky” commentary. Under no circumstance should you consider this an option as to what may happen in VMworld 2011) Obviously, this is a great model to keep for next year. However, I think it would be neat to host 3rd party labs in a duplicated, but similar environment.
Alright… now that the Q&A is over with, I would like to take a couple moments of your time to try and enlighten you to some aspects of the Lab environment that you may not be aware of… but that everyone needs to know. This is a massive undertaking that was done to benefit ALL OF US!
Lab Setup and Operations
The team of workers that put on the Lab environment for VMworld are amazing. Simply amazing. This entire setup (from planning months ago to execution this week) was completed in their free time. Seriously, these guys (and gals) have their usual day jobs. They got together of their own free will and in their free time to get this put together for us. That is time they spent away from wives, husbands, children, family, friends, and other personal commitments. Au Gratis.
I completely understand the time commitment for putting together a lab environment. However, to the scale of the VMworld 2010 labs, is just mind boggling.
Once everything was ready logistically, the VMworld lab team got access 72 hours in advance of VMworld. 3 days to ensure 480 clients setup properly, connectivity, connections, and services. Amazing…
Many of us know, already, that the Labs are being provided via 3 cloud locations. Moscone West, Ashburn (Virginia), and Miami (Florida). However, what most people do not realize is that the Labs are operating in a true cloud configuration.
Obviously, the Ashburn and Miami locations are remote… but what about the Moscone West location? Believe it, or not, the local datacenter is being accessed as a WAN environment. No 10Gb connectivity from the lab environment. The connections are technically being accessed via a secure VPN tunnel to the datacenter itself. So, while the lab hardware is technically a couple hundred feet away, the connectivity to the infrastructure is identical to the East Coast locations. This is truly a cloud environment!
Look all around the lab… or picture yourself looking all around the lab. Got it?! Every computer interface you have seen (projections and terminals) as well as many you have not seen are completely custom. The lab environment is so complex and custom, many developers spent their free time developing the environment you use to interact with the labs as well as monitoring and administration services. This is thanks to Clair and Curtis. I was privileged enough to get a tour of the operations center and see some of the interfaces first-hand. The interfaces look production ready. Hard to believe that this is not a member of a money making product right now.
Custom Monitoring and Administration
Via the use of common APIs in the framework products used to provide the Lab environment, developers were able to create some amazingly simple and informative monitoring and administration utilities. The view into the Lab environment from the tools allows for a fairly minimal amount of admin staff.
The client stations are using the new View 4.5 product for presentation. To think about where we have come… in 2007, the labs were comprised of a handful of laptops. This was to address the primary lab needs… the desktop products. A mere 4 years later, we are using cloud infrastructure, thin clients, and 480 simultaneous clients.
So, for all you lab workers and organizers at VMworld 2010 labs, I would like to take a moment to thank you, on behalf of the VMworld attendees, for your personal sacrifice to make the lab environment work so well. The end result was amazing. You have had an impact on the careers of so many people. You should be very proud of what you have accomplished.