2015 in review

Posted on Updated on


The WordPress.com stats helper monkeys prepared a 2015 annual report for this blog.

Here’s an excerpt:

A New York City subway train holds 1,200 people. This blog was viewed about 6,600 times in 2015. If it were a NYC subway train, it would take about 6 trips to carry that many people.

Click here to see the complete report.

How to fix corrupt image profile issues on an ESXi host?

Posted on Updated on


Recently I happen to deal with such an issue while installing the patches on an ESXi host using vSphere Update Manager. This issue is rare and will occur if you interrupt the patches remediation task abruptly, in my case thanks to one of my colleague.

I will mention the steps which helped me to recover the image profile for the ESXi host.

Issue: No image profile is found on the host or image profile is empty. In my case image profile shows Unknown – no profile defined

image profile issue

Resolution:

This issue has been seen on systems where the image database file, imgdb.tgz, is corrupt. An image profile is required to install or remove VIBs.

How to confirm if you are proceed with the mentioned solution:

To confirm that the imgdb.tgz file is corrupt:

Connect to the ESXi host via an SSH session.

Change directory to /vmfs/volumes by cd /vmfs/volumes

Search for the imgdb.tgz file: find * | grep imgdb.tgz

 

Note: This command normally results in two matches. For example:

0ca01e7f-cc1ea1af-bda0-1fe646c5ceea/imgdb.tgz

edbf587b-da2add08-3185-3113649d5262/imgdb.tgz

 

Run this command on each match:

ls -l match_result

For example:

ls -l 0ca01e7f-cc1ea1af-bda0-1fe646c5ceea/imgdb.tgz

-rwx——   1 root root  26393 Jul 20 19:28 0ca01e7f-cc1ea1af-bda0-1fe646c5ceea/imgdb.tgz

 

The default size for the imgdb.tgz file is approximately 26 KB. If one of the files is only a couple of bytes, it indicates that the file is corrupt.

 

There are 2 ways you can fix this issue. To work around this issue, perform one of these options:

1. Rebuild the ESXi host

OR

2. Copy an imgdb.tgz file from a known good ESXi host to the host having the issue

 

You may rebuilt if you want but that is not what you are looking for!

Now let us see how we can proceed with the second option:

 

To copy the imgdb.tgz file from a known good ESXi host perform the following:

On the working ESXi host, copy a good copy of imgdb.tgz by perform the following

cp /bootbank/imgdb.tgz /vmfs/volumes/<shared-LUN>

On the corrupt host, copy the good copy of the imgdb.tgz to /tmp:

cp /vmfs/volumes/<shared LUN>/imgdb.tgz /tmp

cd /tmp

tar -xzf imgdb.tgz

Copy the good profile files to the profile directory:

cp /tmp/var/db/esximg/profiles/* /var/db/esximg/profiles/

Copy the good VIBs to the VIB repository:

cp /tmp/var/db/esximg/vibs/* /var/db/esximg/vibs/

Remove the corrupt imgdb.tgz from the bootbank:

rm /bootbank/imgdb.tgz

Move the good copy of imgdb.tgz into the bootbank:

cp /tmp/imgdb.tgz /bootbank/

Backup configuration changes made:

/sbin/auto-backup.sh

Restart the ESXi host, Attempt to install or patch the host again using vSphere Update Manager.

How to fix Lost connectivity to the device backing the boot filesystem on an ESXi host?

Posted on


Error: Lost connectivity to the device naa.60xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx backing the boot filesystem /vmfs/devices/disks/naa.60xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx. As a result, host configuration changes will not be saved to persistent storage.

If we lose connectivity to the NIC that runs the boot LUN (switch reboot, cable disconnect, etc.), we will see the above error. This error is being displayed because connectivity is lost and the iSCSI boot does not support Multi pathing, which means that if connectivity is lost between the Storage Processor (SP) on the VNXe and the NIC on the host, the host can no longer access its boot lun and cannot write logs etc.

There is no impact as whole ESXi OS is loaded into memory so there is no outage for the VMs. Once the connectivity is restored the host can access the storage again. The error was for the fact that the error does not clear automatically.

The simplest solution is to put the host into maintenance mode, reboot it and the problem is solved Or restart the Management Agents on the ESXi host.

Check How to restart Management agents on ESXi host

How to rename the local administrator with Group Policy

Posted on


To improve security in your Active Directory domain, you should rename the administrator account because this lowers the risk of brute force attacks. Renaming the administrator account and resetting its password on all computers in your AD domain can be easily done via Group Policy.

Open the Active Directory Group Policy Management console, create a new GPO, and link it to your desired OU. Of course, you can also work with an existing GPO.

Linking a GPO to an OU

Right-click the new GPO or an existing GPO and select Edit. This will launch the Group Policy editor. Now, browse to the following Group Policy setting: Computer Configuration > Preferences > Control Panel Settings > Local Users and Groups.

Renaming the administrator account

As you can see in the screenshot above, right-click Local Users and Groups and then navigate toNew > Local User.

On the next screen, you select the user name you would like to use for the administrator account:

Selecting the user name

Select the following:

Action – Select Update.

User name – Select Administrator (built-in).

Rename to – Enter the new user name.

Full name – Enter your desired name.

Description – Add a description (optional).

Password – Set a new password (optional).

Check boxes – Verify that the check boxes comply with your company policies.

The GPO is now configured and can be deployed in your network. The refresh interval for computer settings is 90 minutes. If you want to apply the GPO immediately on a client computer, open a command prompt and type gpupdate /force at the command line.

Alternatively, you can reboot the computer. If you are finding that a computer isn’t applying the policy, simply run gpresult /r at a command line to see whether your new GPO is listed:

 Checking if the GPO has been applied

If it’s not listed or if you see a permission error message, go back to Active Directory Users and Computers and check the OU to which you have the policy applied. Also check whether the computer contains that OU. Perhaps the computer is in a different OU and therefore doesn’t pick up the policy.

Also check the GPO settings. In the Security Filtering section, ensure that the GPO is applied to Authenticated Users; in the Links section, verify that the correct OU is linked to the GPO :

GPO security filtering

If the policy is still not applied to some of your computers and you have checked all the above, then your domain controllers might not replicate the GPO properly.

 

How to reset ESXi 5.x root password using Host Profiles

Posted on Updated on


According to VMware, the only supported way to reset a lost password is to do a fresh install. However, there are ways around it if your host is already connected to vCenter.

If you do NOT know the host password but it’s currently connected to vCenter, you can use Host Profiles to reset the password. This is only possible because the vpxa user on each ESXi host, added when the ESXi host is connected to vCenter Server, has root privileges.

Host Profiles are a feature of Enterprise Plus licensing only.

The is a VMware KB which mentions root password recovery is this one and it clearly states that it’s not supported to reset passwords on ESXi 5.x and ESXi in general as there is no longer the Linux console where you would use the single-user mode for the job:

Reinstalling the ESXi host is the only supported way to reset a password on ESXi. Any other method may lead to a host failure or an unsupported configuration due to the complex nature of the ESXi architecture. ESXi does not have a service console and as such traditional Linux methods of resetting a password, such as single-user mode.

But using host profiles to change the root password on ESXi host is supported and if you got the appropriate licensing then you should be able to change the root password.

Steps:

1. Right click the host, choose All vCenter Actions, Host Profiles, and select Extract Host Profile. Run through the wizard to create the new profile.

2. At the top of the vSphere client, click Home and Host Profiles under the Management section. Right click the newly created profile and choose Edit.

3. Click Next to the Edit Host Profile step and expand Security and Services, then expand Security Settings. Click on Security Configuration. Modify the dropdown list and select the “Configure a fixed administrator password” option. Enter the new password.

4. Complete the wizard which will save all your changes.

5. Back in the Hosts and Clusters view, right click your host and go to All vCenter Actions, Host Profiles, and Attach Host Profile. Select your profile you created and customized and finish the wizard.

6. Put your host in maintenance mode.

7. Right click the host again, All vCenter Actions, Host Profiles, and Remediate. If your host is not in maintenance mode, you’ll get the message “Remediate operation is allowed only for hosts in maintenance mode”

8. Once the Host Profile is applied, the host will reboot and your password will now be updated.

How to Remove Storage Devices from ESXi Hosts

Posted on Updated on


Unmounting a LUN checklist

Before unmounting a LUN, ensure that:

  • If the LUN is being used as a VMFS datastore, all objects (such as virtual machines, snapshots, and templates) stored on the VMFS datastore are unregistered or moved to another datastore.Note: All CD/DVD images located on the VMFS datastore must also be unregistered from the virtual machines.
  • The datastore is not used for vSphere HA heartbeat.
  • The datastore is not part of a datastore cluster.
  • The datastore is not managed by Storage DRS.
  • The datastore is not configured as a diagnostic coredump partition.
  • Storage I/O Control is disabled for the datastore.
  • No third-party scripts or utilities running on the ESXi host can access the LUN that has issue. If the LUN is being used as a datastore, unregister all objects (such as virtual machines and templates) stored on the datastore.
  • If the LUN is being used as an RDM, remove the RDM from the virtual machine. Click Edit Settings, highlight the RDM hard disk, and click Remove. Select Delete from disk if it is not selected, and click OK.Note: This destroys the mapping file, but not the LUN content.
  • Check if the LUN/datastore is used as the persistent scratch location for the host.This PowerCLI script can be used to check the current scratch location:

$vcServer = “vCenter01”
$cluster = “CL01”
$esxCred = Get-Credential
Connect-VIServer $vcServer | Out-Null
#Connect to ESX hosts in cluster
foreach ($esx in Get-Cluster $cluster | Get-VMHost) {
Connect-VIServer $esx -Credential $esxCred | Out-Null
Get-VMHostAdvancedConfiguration -Name “ScratchConfig.ConfiguredScratchLocation”
}

Note: When using the vSphere Web Client with vSphere 5.1, 5.5 and 6.0, only these checks are performed during the datastore unmount operation:

  • Host should not have any virtual machines residing on this datastore
  • Host should not use the datastore for HA heartbeats

Obtaining the NAA ID of the LUN to be removed

From the vSphere Client, this information is visible in the Properties window of the datastore.

From the ESXi host, run this command:

# esxcli storage vmfs extent list

You see output similar to:

Volume Name VMFS UUID Extent Number Device Name Partition
———– ———————————– ————- ———————————— ———
datastore1 4de4cb24-4cff750f-85f5-0019b9f1ecf6 0 naa.6001c230d8abfe000ff76c198ddbc13e 3
Storage2 4c5fbff6-f4069088-af4f-0019b9f1ecf4 0 naa.6001c230d8abfe000ff76c2e7384fc9a 1
Storage4 4c5fc023-ea0d4203-8517-0019b9f1ecf4 0 naa.6001c230d8abfe000ff76c51486715db 1
LUN01 4e414917-a8d75514-6bae-0019b9f1ecf4 0 naa.60a98000572d54724a34655733506751 1

Make a note of the NAA ID of the datastore to use this information later in this procedure.

Note: Alternatively, you can run the esxcli storage filesystem list command, which lists all file systems recognized by the ESXi host.

Unmounting a LUN using the vSphere Client

To unmount a LUN from an ESXi 5.0 host using the vSphere Client:

  1. If the LUN is an RDM, skip to step 2. Otherwise, in the Configuration tab of the ESXi host, click Storage. Right-click the datastore being removed, and click Unmount.A Confirm Datastore Unmount window appears. When the prerequisite criteria have been passed, click OK.Note: To unmount a datastore from multiple hosts in the vSphere Client, click Hosts and Clusters > Datastores and Datastore Clusters view (Ctrl+Shift+D). Perform the unmount task and select the appropriate hosts that should no longer access the datastore to be unmounted.
  2. Click the Devices view (under Configuration > Storage):                          
  3. Right-click the NAA ID of the LUN (as noted above) and click Detach. A Confirm Device Unmount window is displayed. When the prerequisite criteria are passed, click OK. Under the Operational State of the Device, the LUN is listed as Unmounted.Note: The Detach function must be performed on a per-host basis and does not propagate to other hosts in vCenter Server. If a LUN is presented to an initiator group or storage group on the SAN, the Detach function must be performed on every host in that initiator group before unmapping the LUN from the group on the SAN. Failing to follow this step results in an all-paths-down (APD) state for those hosts in the storage group on which Detach was not performed for the LUN being unmapped.
  4. Confirm if the LUN is successfully detached. The LUN can then be safely unpresented from the SAN. For more information, contact your storage array vendor.
  5. Perform a rescan on all ESXi hosts which had visibility to the LUN. The device is automatically removed from the Storage Adapters.

When the device is detached, it stays in an unmounted state even if the device is re-presented (that is, the detached state is persistent). To bring the device back online, the device must be attached.

If you want the device to permanently decommission from an ESXi host, manually remove the NAA entries from the host configuration:

  1. To list the permanently detached devices, run this command:# esxcli storage core device detached listYou see output similar to:Device UID State
    ———————————— —–
    naa.50060160c46036df50060160c46036df off
    naa.6006016094602800c8e3e1c5d3c8e011 off
  2. To permanently remove the device configuration information from the system, run this command:# esxcli storage core device detached remove -d NAA_IDFor example:# esxcli storage core device detached remove -d naa.50060160c46036df50060160c46036df

Standard Windows Monitoring Threshold Parameters

Posted on Updated on


Confused with setting up Threshold Parameters on the Tools Server for Performance Monitoring?

Here are the typical parameters and the threshold limit with Warning, High, Alert levels with polling intervals. This will depend upon the SoW signed with the client.

Standard Windows Monitoring

RoD is nothing but Remedy on Demand.

VMware vCenter Release and Build Number History

Posted on Updated on


vCenter Build Numbers
Name Version Release Build Installer Version
vCenter Server 6.0.0 Update 1 6.0 U1 9/10/2015 3018524 3040890
vCenter Server 6.0.0b 6.0.0b 7/7/2015 2776511 2800571
vCenter Server 6.0 Express Patch 1 6.0.0a 4/16/2015 2656758 2656757
vCenter Server 6.0 6.0 GA 3/12/2015 2494585 2562643
vCenter Server 5.5 Update 3 5.5 U3 9/16/2015 3000241 3000346
vCenter Server 5.5 Update 2e 5.5 U2e 4/16/2015 2646482 2646481 5.5.0.44687
vCenter Server 5.5 Update 2d 5.5 U2d 1/27/2015 2442329 2442328 5.5.0.43769
vCenter Server 5.5 Update 2b 5.5 U2b 10/9/2014 2183111 2183112 5.5.0.43013
vCenter Server 5.5 Update 2 5.5 U2 9/9/2014 2001466 2105955 5.5.0.42389
vCenter Server 5.5 Update 1c 5.5 U1c 7/22/2014 1945274 1945270 5.5.0.42156
vCenter Server 5.5 Update 1b 5.5 U1b 6/12/2014 1891310 1891314 5.5.0.41927
vCenter Server 5.5 Update 1a 5.5 U1a 4/19/2014 1750795 1750787 5.5.0.41222
vCenter Server 5.5c 5.5c 4/19/2014 1750596 1750597 5.5.0.41218
vCenter Server 5.5 Update 1 5.5 U1 3/11/2014 1623101 1623099 5.5.0.40799
vCenter Server 5.5b 5.5b 12/22/2013 1476327 1476387 5.5.0.39885
vCenter Server 5.5a 5.5a 10/31/2013 1378903 1378901 5.5.0.38845
vCenter Server 5.5 5.5 GA 9/22/2013 1312298 1312299 5.5.0.38036
vCenter Server 5.1 Update 3b 5.1 U3b 10/1/2015 3070521 3072311
vCenter Server 5.1 Update 3a 5.1 U3a 4/30/2015 2669725 2670344
vCenter Server 5.1 Update 3 5.1 U3 12/4/2014 2306353 2308386 5.1.0.43263
vCenter Server 5.1 Update 2c 5.1 U2c 10/30/2014 2207772 2212977 5.1.0.43068
vCenter Server 5.1 Update 2a 5.1 U2a 7/7/2014 1882349 1917403 5.1.0.41903
vCenter Server 5.1 Update 2 5.1 U2 1/16/2014 1473063 1474365 5.1.0.39867
vCenter Server 5.1 U1c 5.1 U1c 10/30/2013 1364037 1364079 5.1.0.38659
vCenter Server 5.1 U1b 5.1 U1b 8/1/2013 1235232 1235309 5.1.0.37189
vCenter Server 5.1 U1a 5.1 U1a 5/22/2013 1123961 1123966 5.1.0.36098
vCenter Server 5.1 Update 1 5.1 U1 4/25/2013 1064983 1065152 5.1.0.35539
vCenter Server 5.1b 5.1b 12/20/2012 947673 947939 5.1.0.34460
vCenter Server 5.1a 5.1a 11/19/2012 880146 880471 5.1.0.33762
vCenter Server 5.1 5.1 GA 9/11/2012 799731 799735 5.1.0.32743
vCenter Server 5.0 Update 3e 5.0 U3e 10/1/2015 3073236 3073234
vCenter Server 5.0 Update 3d 5.0 U3d 4/30/2015 2656067 2692807
vCenter Server 5.0 Update 3c 5.0 U3c 11/20/2014 2210222 2215678 5.0.0.43079
vCenter Server 5.0 Update 3a 5.0 U3a 7/1/2014 1917469 1923446 5.0.0.42044
vCenter Server 5.0 Update 3 5.0 U3 10/17/2013 1300600 1343691 5.0.0.37933
vCenter Server 5.0 Update 2 5.0 U2 12/20/2012 913577 923238 5.0.0.34130
vCenter Server 5.0 U1b 5.0 U1b 8/16/2012 804277 804276 5.0.0.32829
vCenter Server 5.0 U1a 5.0 U1a 7/12/2012 755629 757163 5.0.0.31955
vCenter Server 5.0 Update 1 5.0 U1 3/15/2012 623373 639890 5.0.0.29542
vCenter Server 5.0 5.0 GA 8/24/2011 456005 456005 5.0.0.16964
vCenter Server 4.1 U3a 1/31/2013 925676 978694
vCenter Server 4.1 U3 8/30/2012 799345 816786
vCenter Server 4.1 U2 10/27/2011 491557 493063
vCenter Server 4.1 U1 2/10/2011 345043 345042
VUM-KB-1023962 7/19/2010 275390
vCenter Server 4.1 7/13/2010 259021 259021
vCenter Server 4.0 Update 4b 4.0 U4b 2/7/2013 934016
vCenter Server 4.0 U4 11/17/2011 496403
vCenter Server 4.0 U3 5/5/2011 385281
vCenter Server 4.0 U2 6/10/2010 258672
vCenter Server 4.0 U1 11/19/2010 208111
vCenter Server 4.0 Patch 1 2/25/2010 183347
vCenter Server 4.0 5/21/2009 162856
VirtualCenter 2.5.0 U6b 3/8/2012 598800
VirtualCenter 2.5.0 U6a 5/5/2011 341471
VirtualCenter 2.5.0 U6 Localized 1/29/2010 227640
VirtualCenter 2.5.0 U6 English 1/29/2010 227637
VirtualCenter 2.5.0 U5 Localized 7/10/2009 174835
VirtualCenter 2.5.0 U5 English 7/10/2009 174768
VirtualCenter 2.5.0 U4 Localized 2/23/2009 147704
VirtualCenter 2.5.0 U4 German 2/23/2009 147697
VirtualCenter 2.5.0 U4 English 2/23/2009 147633
VirtualCenter 2.5.0 U3 10/3/2008 119598
VirtualCenter 2.5.0 U2 7/25/2008 104215
VirtualCenter 2.5.0 U1 4/10/2008 84767
VirtualCenter 2.5.0 12/10/2007 64192