VMware

How to monitor the Disk Command Aborts on an ESXi host

Posted on Updated on


When storage is severely overloaded, commands are aborted because the storage subsystem is taking too long to respond to the commands. The storage subsystem has not responded within an acceptable amount of time, as defined by the guest operating system. Aborted commands are a sign that the storage hardware is overloaded and unable to handle the requests in line with the host’s expectations.

The number of aborted commands can be monitored by using either vsphere client or esxtop.

  1.  from vsphere client, monitor disk commands aborts

this one can be generated from host and clusters->Performance-> Advanced -> Switch to disk -> chart options-> commands aborted-> ok.

  1. from esxtop, monitor ABRTS/s

Open putty, login to the ESXi host, run esxtop, for the disk type u, type f to change the settings and type L to select Error stats. Press W to save it.

Once this is we can see the ABRTS/s field there which tracks the SCSI aborts, Aborts generally occur because the array takes long time to respond to commands.

Now if you are planning to deploy a monitoring tool to monitor this parameter, the threshold for ABRTS/s should be 1. This signifies number of SCSI commands aborted during the collection interval i.e. in 1 second.

DISK       ABRTS/s               1              Aborts issued by guest(VM) because storage is not responding. For Windows VMs this happens after 60 seconds by default. Can be caused for instance when paths failed or array is not accepting any IO for whatever reason.

However having said that the in ideal case the output of ABRTS/s should be 0, which may sometime not been observer during peak hours i.e. Backup may be running on the servers hosted on the ESXi host resulting in disk intensive workouts. This ABRTS/s will fluctuate 0 to 0.xx in real case scenario as the storage is always overloaded during these peak hours.

The Performance Overview tab fails to display with the error: Navigation to the webpage was cancelled (1014454)

Posted on Updated on


There are several possible causes for this issue. Attempt each of the troubleshooting steps below in sequence, without skipping any.

  1. Confirm that the vCenter Web Management Service is running.

    To resolve this issue, connect to vCenter locally with a vSphere Client located on the vCenter Server and followStopping, starting, or restarting vCenter services (1003895).

  2. Ensure that the correct DNS settings and IP address are being used:
    1. On the vCenter Server, navigate toC:\Program Files\VMware\Infrastructure\VirtualCenter Server\extensions\com.vmware.vim.stats.report\.
    2. Openxml in a text editor.
    3. Edit the line<url>https://hostname:8443/statsreport/vicr.do</url> to use an IP address instead of an FQDN to rule out issues with DNS.
    4. Restart vCenter Web Management Service and the vCenter Server Service after making any changes to the.xml  For moreinformation, see Stopping, starting, or restarting vCenter services (1003895).
  3. Disable any third party web services that may be interfering with the vCenter Web Management Services.

    To confirm that a third party web service is the cause:

    1. Stop the vCenter Web Management Service. For more information, seeStopping, starting, or restarting vCenter services (1003895).
    2. Try to connect to port 8443 (the port on which the Web Management Service runs) by executing:

      telnet IP8443

    3. If the port responds when the vCenter Web Management Service is stopped, there might be another service that is using the port. In this case, if you want to continue running the conflicting third party service, you may have to change the port that Performance Overview uses. To change the port used by Performance Overview, seeThe Performance Overview tab within vCenter Server reports the HTTP Status 404 error (1016160).
  4. Check if vCenter Server is using custom SSL certificates as a result of a recent upgrade to vCenter Server 4.0 Update 1. For more information, seeVMware vCenter Server plugins fail after adding custom SSL certificates (1017577).

Note: Additionally, you can perform these steps:

  • Disable the proxy settings from the browser.

    To disable the settings:

  1. LaunchInternet
  2. Navigate toTools > Internet options.
  3. Click theConnections
  4. ClickLAN settings.
  5. Select theUse automatic configuration script
  • If you experience this issue on workstations external to the vCenter Server, try to connect to port 8443 (the port on which the Web Management Service runs) as per Step 3b. If you are unable to connect to the port, disable the Windows Firewall on the vCenter Server system. For more information, see the Microsoft TechNet article I Need to Disable Windows Firewall.

How to change default snapshot location in VMware ESXi 5

Posted on Updated on


Defaulty the snapshots which are taken for any virtual machine are stored with their parent in the same directory or storage. Sometimes you may run out of space and you might not be able to take anymore snapshots so in that case you can always use some other location for the storage of snapshots.

snapshot
These are the required steps to be taken to change the default locations of all the snapshots .

NOTE: Please ensure that the vm you are working on is powered OFF.

Right Click the vm and select Edit Settings
Click on Options from the top TAB, select General and open the Configuration parameters

Add a new row with the following details

snapshot.redoNotWithParent

Save this parameter with a value “true” as shown below

Now open the CLI of the host where the vm is located

Go to the vm’s parent directory where all the vm files are stored and open the main .vmx file

As in my case

# cd /vmfs/volumes/53652b45-90f342h4-v3r3-s5dw676h5674/Windows2003
# vi Windows2003.vmx

Now add this line anywhere in the .vmx file with the path location where you want your snapshots to be stored

workingDir = “/vmfs/volumes/54332bf4-gd3bf353-g45b-g2ft353b5545/snapshots”

Save the file and exit

Now you need to reload this vm to make the changes take affect.
# vim-cmd vmsvc/getallvms | grep Windows2003
13 Windows2003 [iSCSI-Datastore15] Windows2003/Windows2003 win2003 vmx-07
Here 13 is the vm id which you can find out using the above command
# vim-cmd vmsvc/reload 13
Now when you take snapshots the snapshot files and vm swap files will be created in a different location.

How to redirect vm’s swap file

In case you do not want vm swap file to be redirected to another location and you want it to the same parent directory.
Add an extra parameter in the Configuration Parameter option shown above
sched.swap.dir=”<path_to_vm_directory>”
For example
/vmfs/volumes/54332bf4-gd3bf353-g45b-g2ft353b5545/vmswap

Save the settings and exit. Now each time you take snapshot the snapshot files and vm swap files will be saved at specified different location.

 

How to change the thick or thin provisioning of a virtual disk

Posted on Updated on


Caution: Before following these procedures, it is highly recommended that you have a valid backup of the virtual machine and enough space to convert the virtual machine’s disk(s) from thin to thick.

To change the provisioning of a virtual machine base disk from thin to thick from the Datastore Browser:

  1. Power off the virtual machine.
  2. In vSphere Client, right-click the virtual machine in the inventory.
  3. Click Edit Settings to display the Virtual Machine Properties dialog box.
  4. Click the Hardware tab and select the appropriate hard disk in the Hardware list.Note: The Disk Provisioning Type section on the right displays either Thin Provision or Thick Provision. If the disk provision type is Thick, disk provisioning has already taken place. In this case, the disk provisioning is Thin.
  5. Click Cancel to exit out of Virtual Machine Properties dialog box.
  6. Click the Summary tab of the virtual machine.
  7. Under Resources, right-click the datastore where the virtual machine resides and click Browse Datastore.
  8. Double-click the virtual machine folder to display the .vmdk file.
  9. Right-click the .vmdk file, and click Inflate. The Inflate option converts the disk to thick provisioned.
  10. Reload the .vmx file. For more information see Reloading a vmx file without removing the virtual machine from inventory (1026043).

Notes:

  • If the Inflate option is grayed out, this indicates that the virtual machine is not powered off or that it is not thin provisioned.
  • There should be no snapshots and the conversion is performed on the base disk.

To convert a virtual machine base disk from thick to thin provisioning by changing the datastore and using offline virtual machine migration:

Note: This process requires more than one datastore. If only a single datastore exists, you can clone the virtual machine to a destination machine with thin provisioned disks instead of migrating.

  1. Power off the virtual machine.
  2. Right-click the virtual machine, and click Migrate.
  3. Click Change datastore.
  4. Click Next, and select a datastore that is not the same as the current datastore.
  5. From the dropdown, select the Thin Provision virtual disk format.
  6. Click Next, then Finish. You can monitor the progress of the conversion in the Tasks and Events view in vCenter Server.

Follow the below steps to do Storage vMotion from vSphere Webclient for vSphere 5.5:

  1. Right-click the virtual machine and select Migrate.
    1. To locate a virtual machine, select a datacenter, folder, cluster, resource pool, host, or vApp.
    2. Click the Related Objects tab and click Virtual Machines.
  2. Select Change datastore and click Next.
  3. Select “Thin Provision” for the virtual machine’s disks and click Next
  4. Select a virtual machine storage policy from the VM Storage Policy drop-down menu, click Next
  5. Select the datastore location where you want to store the virtual machine files and click Next.
  6. Review the information on the Review Selections page and click Finish.

Keyboard or mouse do not work in a hosted virtual machine

Posted on Updated on


To troubleshoot a misbehaving mouse or keyboard:
  1. Confirm that the virtual machine guest window currently has focus by clicking into the virtual machine’s console window. Keyboard or mouse input is only directed to the virtual machine when it has focus or is in full-screen mode.
  2. If your mouse is slow to respond or if it is working erratically, change the optimization settings. For more information, see Mouse movement is not smooth in a hosted virtual machine (1033416).
  3. If you are using a wireless mouse or keyboard, ensure that you have not connected it to the virtual machine by removing and re-adding the USB Controller from the virtual machine. For more information, see the Workstation Help topic Add a USB Controller to a Virtual Machine.
  4. Restart your host computer.
  5. For Windows hosts, use the Windows System Configuration (msconfig) utility to eliminate software and processes as possible causes. For more information, see Using the Windows System Configuration utility (1004010).
  6. For Linux hosts, start Linux in a mode that disables start daemons (typically run level 2; if your host lacks this, you may need to start in single-user mode). For more information, see Changing Linux run levels (1004015) or your operating system documentation.
    Note: These next steps require a second mouse/keyboard. For more information, see Connecting a second mouse or keyboard directly to a hosted virtual machine (1033435).
  7. Look for non-Microsoft, non-VMware mouse/keyboard drivers or software installed in the virtual machine, and remove them.
  8. Re-install VMware Tools. For details, see Overview of VMware Tools (340).

Syslog Server storage logs size calculation

Posted on Updated on


Syslog Server storage calculation:
I want to modify the settings so that my logs size is upped from 2mb to 10mb and we rotate after 40 logs not 20.Here we need to do some planning to see if we have enough free space.

Count hosts: 100
Current size log max: 2
Current rotation count: 20
Total possible MB used: 100x2x20 = 4,000MB  (4gb)

Count hosts: 100
Desired size log max: 10
Desired  rotation count: 40
Total possible MB used: 100x10x40=40,000MB (40GB)

So the drive where your logs are stored would need 40gbfree in the above example to be able to service future demands.
How to modify the VMware Syslog Collector configuration after it is installed:

  1. Make a backup of the file:vCenter Server 5.5 and lower%PROGRAMDATA%\VMware\VMware Syslog Collector\vmconfig-syslog.xml
    vCenter Server 6.0: %PROGRAMDATA%\VMware\vCenterServer\cfg\vmsyslogcollector\config.xml
  2. Open the copied file using a text editor.
  3. Under <defaultValues>, change any of the options to the required values.For example, to increase the log file size to 10 MB and to decrease the number of files retained to 20, modify the attributes:<defaultValues>
<port>514</port>
<protocol>TCP,UDP</protocol>
<maxSize>10</maxSize>
<rotate>20</rotate>
<sslPort>1514</sslPort>

</defaultValues>Note: This configuration in vCenter Server overrides the ESXi host configuration file.

  1. Save and close the file.
  2. Stop the VMware Syslog Collector service.
  3. Remove the file:

    vCenter Server 5.5 and lower
    %PROGRAMDATA%\VMware\VMware Syslog Collector\vmconfig-syslog.xml
    vCenter Server 6.0%PROGRAMDATA%\VMware\vCenterServer\cfg\vmsyslogcollector\config.xml
  4. Rename the copy of the modified file to:vCenter Server 5.5 and lower%PROGRAMDATA%\VMware\VMware Syslog Collector\vmconfig-syslog.xml
    vCenter Server 6.0%PROGRAMDATA%\VMware\vCenterServer\cfg\vmsyslogcollector\config.xml
  5. Start the VMware Syslog Collector service. It may be required to restart the syslog service on the ESXi host if logs are no longer updating on the Syslog Server. To restart the syslog service, see VMware ESXi 5.x host stops sending syslogs to remote server (2003127).

 

The maximum supported number of hosts for use with each vSphere Syslog Collector instance is 30, however depending on the load generated by your environment, you may encounter issues below this number.

To work around this issue, you can deploy multiple instances of vSphere Syslog Collector on separate Windows machine which allows you to distribute the load.

If the customer wishes to continue using vSphere Syslog Collector with more than 30 hosts (current maximum for syslog collector). There are no plans to increase the supported number of hosts. The windows event log error does not apply to the appliance, in some instances we receive no error message but the service simply stops responding or collecting logs.
Alternatively, investigate a more scalable solution such a VMware vRealize Log Insight.

How to restart Management agents on ESXi host

Posted on Updated on


Restarting the Management agents on ESXi

To restart the management agents on ESXi:

From the Direct Console User Interface (DCUI):

  1. Connect to the console of your ESXi host.
  2. PressF2 to customize the system.
  3. Log in asroot.
  4. Use theUp/Down arrows to navigate to Restart Management Agents.

    Note: In ESXi 4.1 and ESXi 5.0, 5.1, 5.5 and 6.0 this option is available under Troubleshooting Options.

  5. PressEnter.
  6. PressF11 to restart the services.
  7. When the service has been restarted, pressEnter.
  8. PressEsc to log out of the system.

From the Local Console or SSH:

  1. Log in to SSH or Local console as root.
  2. Run these commands:

    /etc/init.d/hostd restart
    /etc/init.d/vpxa restart

    Note: In ESXi 4.x, run this command to restart thevpxa agent:

    service vmware-vpxa restart

    Alternatively:

  • To reset the management network on a specific VMkernel interface, by default vmk0, run the command:

    esxcli network ip interface set -e false -i vmk0; esxcli network ip interface set -e true -i vmk0

    Note: Using a semicolon (;) between the two commands ensures the VMkernel interface is disabled and then re-enabled in succession. If the management interface is not running on vmk0, change the above command according to the VMkernel interface used.

  • To restart all management agents on the host, run the command:

    services.sh restart

    Caution:

  • Check if LACP is enabled on DVS for version 5.x and above. For more information, see vSphere 5.0 Networking Guide

If LACP is not configured, the services.sh script can be safely executed.

How to get Serial number and System information of ESXi host remotely using putty

Posted on Updated on


Get hardware serial number using this command from putty

esxcfg-info | grep “Serial N”

Type the following command from the command line on the service console and you will get some Vendor details and serial number information.

/usr/sbin/dmidecode |grep -A4 “System Information”

Multipathing policies in ESXi 5.x and ESXi/ESX 4.x

Posted on Updated on


These are referred to as Path Selection Plug-ins (PSP), and are also called Path Selection Policies.

These pathing policies can be used with VMware ESXi 5.x and ESXi/ESX 4.x:

  • Most Recently Used (MRU): Selects the first working path, discovered at system boot time. If this path becomes unavailable, the ESXi/ESX host switches to an alternative path and continues to use the new path while it is available. This is the default policy for Logical Unit Numbers (LUNs) presented from an Active/Passive array. ESXi/ESX does not return to the previous path if, or when, it returns; it remains on the working path until it, for any reason, fails.

    Note: The preferred flag, while sometimes visible, is not applicable to the MRU pathing policy and can be disregarded.

  • Fixed (Fixed): Uses the designated preferred path flag, if it has been configured. Otherwise, it uses the first working path discovered at system boot time. If the ESXi/ESX host cannot use the preferred path or it becomes unavailable, the ESXi/ESX host selects an alternative available path. The host automatically returns to the previously defined preferred path as soon as it becomes available again. This is the default policy for LUNs presented from an Active/Active storage array.
  • Round Robin (RR): Uses an automatic path selection rotating through all available paths, enabling the distribution of load across the configured paths.
    • For Active/Passive storage arrays, only the paths to the active controller will be used in the Round Robin policy.
    • For Active/Active storage arrays, all paths will be used in the Round Robin policy.

    Note: For logical Units associated with Microsoft Cluster Service (MSCS) and Microsoft Failover Clustering virtual machines, the Round Robin pathing policy is supported only on ESXi 5.5 and later.

  • Fixed path with Array Preference: The VMW_PSP_FIXED_AP policy was introduced in ESXi/ESX 4.1. It works for both Active/Active and Active/Passive storage arrays that support Asymmetric Logical Unit Access (ALUA). This policy queries the storage array for the preferred path based on the array’s preference. If no preferred path is specified by the user, the storage array selects the preferred path based on specific criteria.

    Note: The VMW_PSP_FIXED_AP policy has been removed from ESXi 5.0. For ALUA arrays in ESXi 5.0, the MRU Path Selection Policy (PSP) is normally selected but some storage arrays need to use Fixed. To check which PSP is recommended for your storage array, see the Storage/SAN section in the VMware Compatibility Guide or contact your storage vendor.

Notes:

  • These pathing policies apply to VMware’s Native Multipathing (NMP) Path Selection Plug-ins (PSP). Third-party PSPs have their own restrictions.
  • Round Robin is not supported on all storage arrays. Please check with your array documentation or storage vendor to verify that Round Robin is supported and/or recommended for your array and configuration. Switching to a unsupported or undesirable pathing policy can result in connectivity issues to the LUNs (in a worst-case scenario, this can cause an outage).

Warning: VMware does not recommend changing the LUN policy from Fixed to MRU, as the automatic selection of the pathing policy is based on the array that has been detected by the NMP PSP.

RVTools 3.7

Posted on Updated on


RVTools

RVTools is a windows .NET 2.0 application which uses the VI SDK to display information about your virtual machines and ESX hosts. Interacting with VirtualCenter 2.5, ESX Server 3.5, ESX Server 3i, VirtualCenter 4.x, ESX Server 4.x, VirtualCenter 5.0, VirtualCenter Appliance, ESX Server 5.0, VirtualCenter 5.1, ESX Server 5.1, VirtualCenter 5.5, ESX Server 5.5. RVTools is able to list information about VMs, CPU, Memory, Disks, Partitions, Network, Floppy drives, CD drives, Snapshots, VMware tools, Resource pools, Clusters, ESX hosts, HBAs, Nics, Switches, Ports, Distributed Switches, Distributed Ports, Service consoles, VM Kernels, Datastores, Multipath info and health checks. With RVTools you can disconnect the cd-rom or floppy drives from the virtual machines and RVTools is able to update the VMware Tools installed inside each virtual machine to the latest version.

Version 3.7 (March, 2015)

VI SDK reference changed from 5.0 to 5.5
Extended the timeout value from 10 to 20 minutes for really big environments
New field VM Folder on vCPU, vMemory, vDisk, vPartition, vNetwork, vFloppy, vCD, vSnapshot and vTools tabpages
On vDisk tabpage new Storage IO Allocation Information
On vHost tabpage new fields: service tag (serial #) and OEM specific string
On vNic tabpage new field: Name of (distributed) virtual switch
On vMultipath tabpage added multipath info for path 5, 6, 7 and 8
On vHealth tabpage new health check: Multipath operational state
On vHealth tabpage new health check: Virtual machine consolidation needed check
On vInfo tabpage new fields: boot options, firmware and Scheduled Hardware Upgrade Info
On statusbar last refresh date time stamp
On vhealth tabpage: Search datastore errors are now visible as health messages
You can now export the csv files separately from the command line interface (just like the xls export)
You can now set a auto refresh data interval in the preferences dialog box
All datetime columns are now formatted as yyyy/mm/dd hh:mm:ss
The export dir / filenames now have a formated datetime stamp yyyy-mm-dd_hh:mm:ss
Bug fix: on dvPort tabpage not all networks are displayed
Overall improved debug information
Download link: http://robware.net/index.php/register

Documentation: http://robware.net/download/RVTools.pdf