Unmount VMFS Datastore

With all the wicked-cool new functions in vSphere 5, one of the most understated but highly functional lies with the ability to unmount an iSCSI share. Seemingly a simple function, this has not been available in non-vSphere 5 hosts until now.

The problem I have faced in the past is that there is a need to remove iSCSI stores from an ESXi host. In those rare instances, I have needed to migrate some VMs off of a SAN while keeping other VMs on the same SAN (ex: moving a development SAN to another site). svMotion handles the hard work of moving the VMs to the new datastores (easy-peasy, right?). However, unlike an NFS share, a VMFS share could not be unmounted. I ran into 2 options to remove the share:

1) Right-click the datastore and select “Delete”!

Uh… the point of this is to not delete these VMs!

2) Remove the initiator IP address, remove access to the ESXi host initiators via the SAN interface, vMotion VMs to other hosts (if you’re lucky), and reboot the host.

- Host downtime, SAN maintenance (which, yes, I know initiators not being used should be cleaned up… but not as a requirement to save my VMs), host downtime, etc… I can add a datastore live, why not remove it live?!

 

To my surprise this morning, while removing some iSCSI stores after some over-the-weekend SAN migration, I was presented with a new option via vSphere 5!

Following this new function leads me to a functional check to ensure that the unmount requirements are green and good to go:

Now, the downside to this procedure is that in my environment, I have a couple non-DRS clustered hosts (thank you Oracle VMware licensing) that I am unable to take offline to upgrade to ESXi 5.0 right now. So, the same iSCSI volumes are available on both ESXi 4.1 and 5.0 hosts. Thus, the unmount process is only partially useful. Due to those darn ESXi 4.1 hosts, I still need to delete the datastore to get rid of the iSCSI volume!

Thanks Oracle Licensing!

Lucky for me, I do not have any VMs to save on the datastore!

This was a great way to start a Monday morning! I look forward to being able to unmount VMFS volumes as necessary… once everything is up to vSphere 5.0!

ESXi 5.0–1.5 Hour Boot Time During Upgrade

I have to say, I am quite shocked that I am on the tail end of waiting 1.5 hours for an ESXi 5.0 upgrade to complete booting. Seriously… 1.5 hours.

I have been waiting for some time to get some ESXi 5.0 awesomeness going on in my environment. vCenter has been sitting on v5 for some time and I have been deploying ESXi 5 in a couple stand-alone situations without any issues. So, now that I have more compute capacity in the data center, it is time to start rolling the remaining hosts to ESXi 5… or so I thought!

I downloaded ESXi 5.0.0 Kernel 469512 a while back and have been using that on my deployments. So far, so good… until today. Update Manager configured with a baseline –> Attach –> Scan –> Remediate –> back to business. Surely, Update Manager processes should take more time than the actual upgrade. About 30 minutes after starting the process, vCenter was showing that the remediation progress was a mere 22% complete and the host was unavailable. I used my RSA (IBM’s version of HP ILO or Dell DRAC) to connect to the console. Sure enough, it was stuck at loading some kernel modules. About 20 minutes later IT WAS STILL THERE!

Restarting the host did not resolve the issue. During the ESXi 5 load screen, pressing Alt + F12 loads the kernel messages. It turns out that iSCSI was having issues loading the datastores in an acceptable amount of time. I was seeing messages similar to:

A little research turned me onto the following knowledgebase article in VMware’s KB: ESXi 5.x boot delays when configured for Software iSCSI (KB2007108)

To quote:

This issue occurs because ESXi 5.0 attempts to connect to all configured or known targets from all configured software iSCSI portals. If a connection fails, ESXi 5.0 retries the connection 9 times. This can lead to a lengthy iSCSI discovery process, which increases the amount of time it takes to boot an ESXi 5.0 host.

So, I have 13 iSCSI stores on that specific host and multiple iSCSI VMkernel Ports (5). So, calling the iSCSI lengthy is quite the understatement.

The knowledgebase states that the resolution is applying ESXi 5.0 Express Patch 01. Fine. I can do that. And… there is a work around described in the article that states you can reduce the number of targets and network portals. I guess that is a workaround… after you have already dealt with the issue and the ridiculously long boot.

Finally, to help mitigate the issue going forward, VMware has released a new .ISO to download that includes the patch. However, this is currently available in parallel with the buggy .ISO ON THE SAME PAGE! Seriously. Get this… the only way to determine which one to download is:

As a virtualization admin, I know that I am using the Software iSCSI initiator in ESXi. But, why should that even matter at all?! There is a serious flaw in the boot process in version 469512  and that should be taken offline. Just because someone is not using Software iSCSI at the current time does not mean they are not going to in the future. So, if they download the faulty .ISO, they are hosed in the future. Sounds pretty crummy to me!

My Reaction

I am quite shocked that this made it out of the Q/A process at VMware in the first place. My environment is far from complex and I expect that my usage of the ESXi 5.0 hypervisor would be within any standard testing procedure. I try to keep my environment as vanilla as possible and as close to best practices as possible. 1.5 hours for a boot definitely should have been caught before release to the general public.

Additionally, providing the option to download the faulty ISO and the fixed ISO is a complete FAIL! As mentioned on the download page, this is a special circumstance due to the nature of the issue. I would expect that if this issue is as serious as the download page makes it out to be, the faulty ISO should no longer be available. There has to be a better way!

Conclusion

I have since patched the faulty ESXi 5.0 host to the latest/safest version, 504890, and boot times are back to acceptable. I will proceed with the remainder of the upgrades using the new .ISO and have deleted all references to the old version from my environment.

I have never run into an issue like this with a VMware product in my environment and I still have all the confidence in the world that VMware products are excellent. In the scheme of things, this is a bump in the road.

VMware vSphere 5–Using Image Builder For Custom Installation

Hard to believe the vSphere 5 release is coming down the pipe. In anticipation of the official availability of the bits for use and VMworld 2011, I thought it would be a great idea for everyone to get their house in order and prep for deploying vSphere 5.

One of the cool new features with the vSphere 5 release was the inclusion of new PowerCLI functions called Image Builder.

Image Builder allows VMware Admins to customize their installation media by adding and removing components. These components, called VMware Infrastructure Bundles (VIBs), comprise the base image, drivers, CIM providers, and other necessary components to make the vSphere go-‘round. Plus, 3rd party vendors can release VIBs in the future for new devices, providers, or whatever (can someone make a Minesweeper VIB?). This results in:

1. VMware not needing to keep updating just to add code for new devices.
2. VMware Admins no longer need to kludge through cramming the driver support for 3rd party products using Linux-based utilities and concepts (although, good job for knowing how to do it)
3. VMware Admins can create a single custom installation with the appropriate drivers without having to install ESXi on a host and immediately patch to add the components.

As mentioned above, Image Builder is included with the latest and greatest version of the PowerCLI utilities… well… the latest and greatest vSphere 5 PowerCLI utilities. So, don’t rush out and download right now.

Note: When installing the new PowerCLI for vSphere 5 over an existing PowerCLI installation, you may find that the Image Builder cmdlets do not appear to be available. If this is the case, be sure to uninstall ALL PowerCLI installations on your workstation prior to installing the new PowerCLI. I ran into this problem during the installation of the pre-release bits and it drove me crazy. Heck, why not just uninstall first to be on the safe side?!

Image Builder introduces two new terms to our VMware verbiage

1. VIB – (as mentioned above) bundles of files that can comprise any base image, driver, CIM provider, or another component. VIBs are certified by VMware and fit a very specific format.
2. Depot – A location where Image Builder can find installation components (aka – an offline bundle). An offline bundle is just a .zip file containing the installation files for a specific version of ESXi. These can be downloaded from the vSphere download page (typically, you are provided with the option of a .iso or .zip download of the media – the .zip is the offline bundle/depot). However, a depot can also be a URL to an offline bundle!!! During Image Builder sessions, multiple depots can be added to a session.
3. Profile – A profile is the entity that comprises the image you are working with. Offline Bundles contain multiple profiles that can be used as a basis to copy. The profile, essentially, tells Image Builder which components to pack into a custom installation.

Finally, Image Builder understands that creating a custom image does not just involve adding and removing VIBs. Rather, you also need some way to get the custom image out in a usable format. Image Builder allows for the export of the custom image to an offline bundle (.zip) or a usable CD/DVD image (.iso). The offline bundle

So, now that we know what Image Builder does and some new terminology, let’s get down and dirty with creating a new Image Builder custom installation!

Procedure

• Start up PowerCLI

• Connect to a depot
  • This example will use a locally saved .zip file.
  • Command: Add-EsxSoftwareDepot –DepotUrl C:\Downloads\VMware\Depot\vmware-ESXi-5.0.0-381646-depot.zip

• The offline bundle contains a number of profiles. These profiles are read-only and cannot be edited. However, that does not mean that it cannot be copied to a new profile and customize the copied profile!
• Get a list of the available depot profiles:
  • Command: Get-EsxImageProfile

  

  • As you can see, we have two profiles: ESXi5.0.0-381646-no-tools and standard.
• Create a copy of a profile
  • Command: New-EsxImageProfile –CloneProfile ESXi-5.0.0-381646-standard –Name “Custom_vSphere5_Installation”

• Now that the profile has been copied, it is time to wreckcreate a new custom installation. First, let’s check on which components are included in the Depot added earlier.
  • Command: Get-ESXSoftwarePackage

  

  • Note: This will load all software packages for all depots loaded in the session.
  • Each of the packages listed are VIBs! (NEAT!!!)
• At this point, the question becomes: What is it about the default installation that you do not like? Are you missing some drivers/VIBs? In most instances, you are going to be missing some VIBs. However, there may be a need to remove a VIB for some reason. In the next step, we will be removing a VIB from the custom profile.
  • Note: You are the master of your universe. This example only shows you how to do something. I do not suggest you removing the VIB from the custom profile unless you know you need to. If you remove the VIB and screw up your environment, you only have yourself to blame because you are the master (right?!).
  • Note:The availability of 3rd party VIBs prior to vSphere 5 release is provided by the 3rd parties themselves. I do not have a connection to a 3rd party that could provide a VIB (wamp wamp wamp). So, I will include the command to add one. Once a VIB is available to me, I will update the post.
  • Command (Add a VIB): Add-EsxSoftwarePackage –ImageProfile Custom_vSphere5_Installation
  • Command (Remove a VIB):  Remove-EsxSoftwarePackage -ImageProfile
    Custom_vSphere5_Installation -SoftwarePackage sata-sata-promise

• Alright, we have a depot, copied an existing ImageProfile, and messed with the clone so it looks like we want it to. Now, we need to get the profile in some form that we can do something with. How about exporting it?! Fantastic idea. Let’s do it!
• The customized profiles need to be exported in a format that can be used for installation. Otherwise, you just wasted precious time and bandwidth on something that just dead-ended. Recall that we have 2 options for exporting:
  • ISO – Traditional disk image. These are burned onto CD/DVD media and ESXi can be installed.
  • ZIP – These can be stored on network locations and used for PXE installations, VUM upgrades, and a basis for future Image Builder customizations.
• Exporting as a .ZIP or .ISO is as simple as changing a value and extension in the PowerCLI command:
  • Command (ISO): Export-EsxImageProfile –ImageProfile Custom_vSphere5_Installation –FilePath C:\downloads\vmware\depot\Custom_vSphere5_Installation.iso –ExportToIso
  • Command (ZIP): Export-EsxImageProfile –ImageProfile Custom_vSphere5_Installation –FilePath C:\downloads\vmware\depot\Custom_vSphere5_Installation.zip –ExportToBundle

• Recall that earlier, we wanted to remove the ‘sata-sata-promise’ VIB from our customized installation media? (I would suggest going back a little bit in the post to refresh your memory). This is a great time to make sure it was removed.
  • Browse to the .zip location in Windows Explorer and open the .zip file.
  • Browse to the ‘vib20’ directory.

• Look around for ‘sata-sata-promise’ VIB. Can you find it?

• Nope! It’s not there! Talk about customization!

At this point, you have viable installation media to streamline your installations and save you time and headaches.

Thanks VMware for the awesome utility. Happy customizations

VMware vSphere 5 Licensure–Take 2!

The release of the vSphere 5 products on July 12th was amazing. Some very cool new functions and updated existing functions! However, all of the new features were tossed by the wayside in favor of focusing on the changes in licensure.

While there is not necessarily a need to go into major detail… a refresher should suffice…

Previously, VMware had used a license that restricted servers to 6-cores per socket. With new processor technology coming out and a need to remove themselves from that equation, the decision was made to drop the core limit on processors and move to something else. What that something else should be was a topic of discussion internally. Mathematically, the next-best option for customers and VMware was the concept of vRAM pooling.

vRAM pooling takes how much virtual RAM is assigned to your virtual machines into consideration versus how much is physically installed. The vRAM pools were based on the editions of vSphere running in your environment and pooled together. If multiple sites had Enterprise licensure, all of the RAM was pooled together in an Enterprise vRAM pool. The licensure allotment was spread across all Enterprise hosts, regardless of site.

The issue was that each edition of vSphere was really lacking in how much vRAM was allowed. The previous thought of high server consolidation by using memory dense physical hosts was, now, being challenged. Licensure costs for immediate upgrades and future projects were being questioned.

VMware had done the math and it really only should affect, roughly, 4% of the customer-base. But, the future looking questions still remained… and the impact to VMware was looking bad.

Those of you who work closely with VMware in one fashion or another know that VMware actually cares about customer experience and empowering customers to adopt their products. While the math penciled out for 96% of the customers, the reaction was poor and VMware recognized a need to resolve the issue…

Which brings us to the announcement from today. VMware has adjusted the licensure scheme as a response to the feedback and needs of the customers. This is the low-down:

vRAM Entitlement Changes

As you can see, the Enterprise and Enterprise Plus licensure has been doubled while the Standard, Essentials, and Essentials Plus have been increased by 1/3. The Free Edition has been quadrupled from 8GB to 32GB.

I was very pleased to see the changes above. a 48GB vRAM entitlement for the Enterprise Plus license was a little tough to swallow. Servers are available to run a ridiculous amount of RAM. So, to think that the Enterprise Plus licensure was only 2x better than Standard was a little off-putting. However, by moving to a 96GB entitlement, those higher consolidation projects can still continue. Seeing as most Enterprise Plus users would utilize dual CPUs, allowing for 192GB of vRAM per host versus 96GB is a major increase in value.

Monster VM!

So, we all know that the vSphere 5 RAM max per VM is crazy. If, for some reason, you have the need to run a 1TB RAM allotted VM, you were going to be hit hard. You would need to buy a ton of Enterprise Plus licensure (or even more Enterprise / Standard edition) to meet the vRAM entitlement licensure.

The fact is that running a single virtual machine should not cost more than a single Enterprise Plus license. So, to help reduce the impact of the new licensure on the monster VMs running out there, VMware came up with a new scheme: The most vRAM entitlement a single VM can consume is 96GB. So, if you run that monster 1TB RAM VM, you do not need to license for 1TB entitlement. Rather, it will only hit the entitlement for 96GB.

Major cost savings for the small group (right now). Although, these license decisions are probably made to be resilient to the future. So, if you are in 2021 and reading this (welcome to the past), maybe every VM is 1TB in size.

The change in the 96GB entitlement cap per VM is not going to be included in the general release of the vSphere 5 product. An update to the product is going to be coming out to add this to the reporting capabilities of vCenter. Also, recall that the licensure change will not stop you from running your virtual machines. So, fire up the 1TB VM and reporting will catch up with you later.

Average Usage vs. High Water

The previous measurement of the entitlement focused on high watermarks of usage. This was thought to have caused too much of a pain point for the TEST/DEV environments and VDI environments.

I could not agree more. In those environments where developers can spin up virtual machines on demand, the number that can be running at a given time for a short amount of time can end up being cost prohibitive.

To combat this issue, VMware decided to calculate the average vRAM pool usage over a 12-month period. Infrequent spikes would be absorbed into the average. More frequent spikes would have a more mild impact on the bottom line.

The View on VDI vRAM Entitlement

Dealing with VDI is another beast altogether. Personally, I feel like running virtual workstations is no different than running standard servers… there is more potential for higher density per physical host, though. So, the entitlement is still valid.

However, VMware decided to take a different route and promote their product line for the vSphere Desktop edition… which is licensed per user, not per vRAM pool. So, $65 per desktop ends up being easier on the company wallet for VDI implementations.

Check out the following blog posts from Raj Mallempati (Director, Product Marketing, End-User Computing). He really dives deep into the pricing and comparison between vRAM entitlement and the per-user license models:

vSphere Desktop Licensing Overview

Desktop Virtualization with vSphere 5: Licensing Overview

 

Virtual Bill’s Thoughts On The Changes

This is a great move on VMware’s part. It just goes to show that VMware really listens to the customer base. They had numbers to show that it was not going to be that hard on customers but decided to adjust and allow for more flexibility.

Hindsight is 20/20 and we all think that this should have been accounted for prior to the original announcement of the licensure change. However, the adjustments made now, prior to the product being available, is the result of the user community feedback. The licensure is now more refined, flexible, and easier on all of us.

One neat side effect of the change in entitlement is that there is a nice upgrade path for users on versioning. For example, if a company with Standard licensure needs more vRAM made available to them, they end up getting more functionality by moving to an Enterprise license rather than buying a new Standard license. Same with Enterprise to Enterprise Plus. Previously, magic version upgrades happened when an edition was discontinued and users were “upgraded” OR when a new feature was needed. Now, just by doing the math and determining that the next edition up the ladder has the same vRAM entitlement as the 2 lower licenses, the users can get more functionality. Now, it may be easier to get to get VAAI in environments, for example.

Finally, I encourage you to look at Bob Plankers’ posts on the licensure change from the first go-around. While the posts are not necessarily going to reflect the newest licensure changes as described in this post, Bob has put a lot of thought into impact of the vRAM pooling in his environment.

Lone Sysadmin – The Five Stages Of VMware Licensing Greif

Lone Sysadmin – A Look At VMware Licensing Environment Growth

This is all going to work out in the end. The increase in vRAM entitlement per version is a great step and provides VMware an easy way to adapt with improvements in technology.

 

Footnote: The images in the post were taken from a VMware licensure presentation. The content and data from the images were obtained and created by VMware. I just used it here.

vSphere 5 Release–HA Improvements

High availability has become one of those functions that many companies take for granted. The ability for a mission critical virtual machine to re-spawn elsewhere in the event of a host failure is really useful. While there is some downtime associated with the virtual machine restarting and recovering itself, the reaction time is fantastic.

This functionality is accomplished by the host maintaining a “heartbeat” with other servers in the HA cluster. In the event that the other servers stop receiving the heartbeat signal, the cluster assumes that the server is down and reboots the virtual machines on the server onto other, available virtual machines.

Issues arise when a network is not designed properly or the server is somehow isolated from the other servers (perhaps a specific switch failure) or failure of the management network. Suddenly, there are major issues with multiple copies of the same virtual machine running. Not good at all. It takes just a second of thought to understand how complicated the repercussions of this situation is. However, never fear, VMware has heard your cries and incorporated another level of host detection in this round of vSphere versions.

Master / Slave Relationship

No longer are the days of primary and secondary nodes. Rather, all nodes in the HA cluster can participate automatically. The following are the criteria that are used to determine which host is going to be the master in the cluster:

• Which host has the access to the most datastores in the cluster.
• ESXi host with the highest MOID

Master elections occur when:

• HA functionality being enabled initially
• Master node fails
• Master node enters maintenance mode
• Management network partitioning
• If the management network is somehow split up (failed switch, for example), hosts that cannot see the original master will elect a new one and operate within the same HA environment.
• Upon resolution of management network partitioning, the multiple master nodes will consolidate into a single management node.

In the new master / slave relationship, the master node is responsible for monitoring the activities of the slave nodes via the heartbeat. Additionally, it will maintain a list of the VMs running on each ESXi host. The slave node, on the other hand, monitors the run state of the local VMs and monitors the health of the master node (see, even the master node needs a little love and attention sometime…. it’s hard work being the master).

Storage Heartbeats

All this talk about heartbeat monitoring is a great segue into a new type of heartbeat… the storage heartbeat.

Previously, heartbeat relied upon the IP network to pass the status information around to the other nodes. But, we all can come up with ways in which this can fail. If the virtualization environment utilizes fibre storage or is architected such that IP storage is on a separate physical network, it is possible for management to fail but the VMs continue to run uninterrupted. To the VMs, nothing has happened. They may see a drop in client connectivity, but they can access their server resources. The other hosts will freak out and start up multiple copies on other ESXi hosts. Not good.

VMware has introduces a new heartbeat type that is able to address this issue. Storage heartbeats utilizes the datastore level to maintain heartbeat information. So, in the event of a network failure, ESXi hosts will look to the datastores to determine if the ESXi host is still active. If so, the VMs remains in the same state. If the ESXi host is no longer actively using the datastores, the master node will start the VMs elsewhere. This is accomplished by storage heartbeats writing to specific locations on the VMFS datastores or to a specific file location on a NAS datastores.

The datastores are selected by random during initialization of the functionality. The datastores can be changed manually, but it is not suggested to alter the default behavior. When a new datastore is introduced into the environment or a change to the environment would allow for greater/less preference, vCenter will recalculate the proper datastore for the storage heartbeats. In manual mode, you would need to manually change it.

The storage heartbeat is meant to be a final catch-all function and ends up being a great diagnostic feature as well. It will drastically protect the integrity of your server environment and help protect the virtual machines from having multiple versions running at the same time.

This is one of those features that rely upon a properly designed network… especially if utilizing IP-based storage.

HA States

A new host property exists that will inform you of the ESXi host’s HA state in the HA cluster.

• NA (HA not configured)
• Election (Master election in process)
• Master (remember, in the event of a network partition, there can be more than one master)
• Connected (Connected to the master – aka “Slave”)
• Network partitioned
• Network isolated
• Dead
• Agent Unreachable
• Initialization Error
• Unconfig Error

These new properties can be useful in ascertaining the HA state of your virtualization infrastructure… especially useful if you are experiencing an HA failure at the moment.

Conclusion

The new HA operating states and functions in vSphere 5 provide for a more secure HA environment in your virtual datacenter. The new master/slave election process allows for resiliency during a management network partition. Those hosts that can see each other become a new HA sub-environment until the partitioning has been resolved. The storage heartbeat allows the protection of virtual machines in the event of a network partition or IP connectivity failure.

HA will continue to work with multiple versions of ESXi. However, the functions available are a limit of the version running. So, if HA is critical to you and you like what you see, you better start evaluating vSphere 5 at your earliest convenience and roll it out!

vSphere 5 Release–Initial Thoughts

Well… it may not be much of a surprise that the Raising The Bar event on July 12th was regarding the release of the latest version of the vSphere product line.

So, what is this release all about? Excellent question! I am glad you asked.

When I think about all of the new features and updated functions, I cannot help but characterize the release as focusing more on storage than any release before. While the software iSCSI stack was heavily improved with vSphere 4, we are seeing more and more storage-related functions in vSphere 5.

One of my sweet spots in virtualization is how virtualization products can really benefit the SMB environment. Previously, VMware has provided SKUs for the ESXi and vCenter components. However, this time around, there are very specific product decisions and options available for the SMB markets. This is absolutely fantastic and really helps cement the role of VMware in the SMB markets. Plus, in many instances, what is good for the SMB market is also good for the Enterprise Remote Office / Branch Office configurations.

Additionally, this release of vSphere really pushes the cloud-platform along very nicely. We are going to see some more flexibility in how we can implement and operate within our private clouds as well as integrate with public clouds.

Finally, the one thing that people hate to see… changing in licensure. I promise to cover this topic in more detail in a blog post soon. However, suffice it to say that with any licensure change, people are not going to be happy. One little goodie is the Advanced SKU… say good bye to Advanced and hello to Enterprise!

When it comes down to it, I really feel like vSphere 5 is a compelling product line for new customers and existing customers to adopt. The component product lines (SRM, vCloud Director, ESXi, vSphere, etc…) have great additions and changes that should increase the adoption rate across all customer markets. Larger customers or people with higher density server deployments are going to have to evaluate the new features and licensure requirements to determine proper deployment… but, that is to be expected.

I encourage you to reach out to the vExpert community, the VMware community, local VMUGs, and VMware Account Representatives to get more information on the new product line as the posts roll out over the next couple of weeks.

I cannot wait to get my grubby hands on the production release and get it rolled out.

Happy vSphere’ing!