ESXi

Nuclear Option: How to Force Power Off a Hung VM via SSH | Lazy Admin Blog

Posted on Updated on

We’ve all been there: a Windows Update goes sideways or a database lock freezes a guest OS, and suddenly the “Shut Down Guest” command is greyed out or simply times out. When the GUI fails you, the ESXi Command Line (esxcli) is your best friend.

Step 1: Identify the “World ID”

In ESXi terminology, every running process is assigned a World ID. To kill a VM, you first need to find this unique identifier.

  1. SSH into your ESXi host using Putty.
  2. Run the following command to see all active VM processes:Bashesxcli vm process list
  3. Locate your hung VM in the list. Look for the World ID (a long string of numbers). You will also see the Display Name and the path to the .vmx file to confirm you have the right one.

Step 2: Execute the Kill Command

ESXi offers three levels of “force” to stop a process. It is best practice to try them in order:

  1. Soft: The most graceful. It attempts to give the guest OS a chance to shut down cleanly.
  2. Hard: Equivalent to pulling the power cable. Immediate cessation of the VMX process.
  3. Force: The “last resort.” Use this only if ‘Hard’ fails to clear the process from the kernel.

The Syntax:

Bash

esxcli vm process kill --type=[soft,hard,force] --world-id=WorldNumber

Example (Hard Kill): esxcli vm process kill -t hard -w 5241852

Step 3: Verify the Result

After running the kill command, it may take a few seconds for the host to clean up the memory registration. Run the list command again to ensure it’s gone:

Bash

esxcli vm process list | grep "Your_VM_Name"

If the command returns nothing, the VM is officially offline, and you can attempt to power it back on via the vSphere Client.

Lazy Admin Tip 💡

If esxcli still won’t kill the VM, the process might be stuck in an “I/O Wait” state (usually due to a failed storage path). In that rare case, you might actually need to restart the Management Agents (services.sh restart) or, in extreme cases, reboot the entire host.

#VMware #vSphere #ESXi #SysAdmin #Troubleshooting #Virtualization #ITOps #LazyAdmin #ServerManagement #DataCenter

This entry was posted in VMware and tagged , , , , , , , , , .

Monitoring Disk Command Aborts on ESXi: Identifying Storage Overload | Lazy Admin Blog

Posted on Updated on

When your storage subsystem is severely overloaded, it cannot process commands within the acceptable timeframe defined by the Guest Operating System. The result? Disk Command Aborts. For Windows VMs, this usually triggers after 60 seconds of silence from the storage array.

Aborted commands are a critical red flag indicating that your storage hardware is overwhelmed and unable to meet the host’s performance expectations. Monitoring this parameter is essential for proactive datacenter management.

Here is how you can track these aborts using two primary methods: the vSphere Client and esxtop.

💻 Method 1: vSphere Client (Graphical Interface)

This method provides a visual, historical look at command aborts across your infrastructure.

  1. Navigate to Hosts and Clusters.
  2. Select the object you want to monitor (Host or Cluster).
  3. Click on the Monitor tab, then Performance, and select Advanced.
  4. Click Chart Options.
  5. Switch the metric grouping to Disk.
  6. Select Commands aborted from the list of measurements.
  7. Click OK.

🛠️ Method 2: esxtop (Command Line Interface)

For real-time, granular troubleshooting, esxtop is the definitive tool. It monitors the ABRTS/s (Aborts per Second) field, specifically tracking SCSI aborts.

Steps to Configure esxtop for Aborts:

  1. Open Putty and log in to your ESXi host via SSH.
  2. Type esxtop and press Enter.
  3. Type u to switch to the Disk Device view.
  4. Type f to change the field settings.
  5. Type L to select Error stats.
  6. Press Enter, then press W to save these settings for future sessions.

You will now see the ABRTS/s column. This number represents the SCSI commands aborted by the guest VM during the 1-second collection interval.

📈 Thresholds and Interpretation

If you are deploying a monitoring tool, the critical threshold for ABRTS/s is 1. A value of 1 or higher means SCSI commands are actively being aborted by the guest OS because the storage is not responding.

What is Ideal?

In an ideal scenario, ABRTS/s should always be 0.

What is Real-World?

In a busy production environment, you may see this value fluctuate between 0 and 0.xx. This occurs during “peak hours”—for instance, when multiple servers on the host are running disk-intensive backup operations simultaneously, leading to temporary storage saturation. However, any consistent spike above 1 requires immediate investigation into path failures, array congestion, or complete storage unresponsiveness.

This entry was posted in VMware and tagged , , , , , , , , , , , .

Top VMware ESXi & vSphere Interview Questions

Posted on Updated on

Preparing for a Virtualization role? This guide covers everything from legacy ESX vs. ESXi differences to advanced HA and DRS logic.

🔄 The Evolution: ESX vs. ESXi

  • Service Console: ESX had a Service Console (based on RHEL); ESXi is “thin” and has no console, leading to a smaller footprint and faster boots.
  • Hardware: ESXi can be purchased as an embedded hypervisor directly on hardware.
  • Health Checks: ESXi features built-in server health status monitoring.

🛡️ High Availability (HA) 5.0 Deep Dive

In vSphere 5.0, the HA architecture moved from a Primary/Secondary model to a Master/Slave concept using the FDM (Fault Domain Manager) agent.

RoleResponsibilities
MasterMonitors host/VM availability, manages restarts, communicates with vCenter.
SlaveMonitors local VMs, sends status to Master, participates in elections if Master fails.

Heartbeat Mechanisms:

  1. Network Heartbeat: Sent between Master and Slaves every second.
  2. Datastore Heartbeat: Used if the network heartbeat is lost to determine if a host is isolated or has actually failed.

🚀 vMotion & DRS (Distributed Resource Scheduler)

vMotion Prerequisites:

  • Shared storage (required prior to 5.1).
  • GigaBit Ethernet dedicated vMotion network (VMkernel port).
  • Processor compatibility (EVC – Enhanced vMotion Compatibility).
  • No active CD-ROM/ISO mounts or CPU affinity.

DRS Automation Levels:

  • Manual: vCenter suggests migrations; admin executes.
  • Partially Automated: vCenter handles initial VM placement; suggestions for migrations.
  • Fully Automated: vCenter moves VMs automatically based on load.

💾 Storage & Networking Quick Hits

  • vSAN: Aggregates local storage from ESXi hosts into a single shared datastore.
  • iSCSI Port Binding: Used when multiple VMkernel ports are in the same subnet to allow multiple paths to an array.
  • Path Selection Policies (PSP): Fixed, MRU (Most Recently Used), and Round Robin.
  • Key Command Line Tools:
    • esxtop: Live performance data.
    • vmkfstools: Virtual disk management.
    • vmware-cmd: VM management and info.

📊 Hardware Version Comparison

FeatureHW Version 4 (ESX 3.x)HW Version 7 (vSphere 4.x)HW Version 8 (vSphere 5.0)
Max vRAM64 GB256 GB1 TB
Max vCPU4832
USB SupportNoYesYes (incl. 3.0)
NICs per VM41010

🚀 Key Differences in Modern vSphere (7.0 & 8.0)

1. The Architecture Shift: Project Monterey & DPUs

Modern vSphere now supports DPUs (Data Processing Units). Instead of the CPU handling networking and security, these tasks are offloaded to the SmartNIC.

2. Tanzu (Kubernetes Integration)

The biggest change in vSphere 7/8 is that Kubernetes is built directly into the hypervisor. You no longer just manage VMs; you manage “Namespaces” and containers natively on ESXi.

3. vMotion Enhancements (vSphere 7+)

In version 5.0, vMotion would “stun” a VM briefly. Modern vMotion uses a “Claim” mechanism that makes migrating massive VMs (Monster VMs) almost instantaneous with zero performance impact.

4. Scalability Comparison (vSphere 5.0 vs. 8.0)

FeaturevSphere 5.0 (Legacy)vSphere 8.0 (Modern)
vCPUs per VM32768
RAM per VM1 TB24 TB
Hosts per Cluster3296
VMs per Cluster3,00010,000

🆕 2026 Interview Questions: Modern Edition

Q: What is the “vSphere Distributed Services Engine”?

A: It is the feature that allows vSphere to use DPUs (SmartNICs) to offload infrastructure services like NSX and vSAN, freeing up the host’s CPU for application workloads.

Q: What is a “Lifecycle Manager” (vLCM)?

A: In vSphere 7+, vLCM replaced the old Update Manager (VUM). It uses a declarative model (Desired State) where you define an image for a cluster, and the hosts automatically maintain that version/driver level.

Q: What is “vSAN Express Storage Architecture” (ESA)?

A: Introduced in vSphere 8, ESA is a new way of processing data optimized for high-performance NVMe drives, removing the old “Disk Group” (Cache/Capacity) requirement.

Q: How does vSphere 8 handle AI/ML workloads?

A: Through vGPU and Device Groups, allowing VMs to span multiple physical GPUs and utilizing High-Bandwidth Memory (HBM) for massive AI model training.

#VMware #vSphere #ESXi #Virtualization #SysAdmin #TechInterview #vMotion #CloudComputing #LazyAdmin #DataCenter

This entry was posted in Interview Questions, VMware and tagged , , , , , , , , , .

Syslog Server storage logs size calculation

Posted on Updated on

Upgrading your syslog retention is a great move for troubleshooting depth, but as your math shows, it comes with a significant increase in storage demands. Moving from 4GB to 40GB is a 10x jump, so ensuring your volume can handle the growth is critical.

Here is the breakdown of the calculation and the step-by-step guide to applying these changes.

📊 Syslog Storage Planning

Before modifying configuration files, verify your available disk space. Using your specific requirements for 100 hosts:

VariableCurrent SettingDesired Setting
Max Log Size2 MB10 MB
Rotation Count20 Files40 Files
Retention per Host40 MB400 MB
Total Storage (100 Hosts)4,000 MB (4GB)40,000 MB (40GB)

⚠️ A Note on Scalability

While you are planning for 100 hosts, keep in mind that the VMware Syslog Collector for Windows is officially supported for up to 30 hosts.

  • The Risk: Beyond 30 hosts, the service may stop responding or drop logs without an error message.
  • The Fix: If you need to support 100 hosts reliably, consider deploying multiple collectors or moving to a high-scale solution like VMware vRealize Log Insight.

🛠️ How to Modify Syslog Collector Configuration

To apply your new 10MB / 40 Rotate policy, you must manually edit the configuration XML.

1. Locate and Backup

Before editing, create a copy of the configuration file.

  • vCenter 6.0: %PROGRAMDATA%\VMware\vCenterServer\cfg\vmsyslogcollector\config.xml
  • vCenter 5.5 & older: %PROGRAMDATA%\VMware\VMware Syslog Collector\vmconfig-syslog.xml

2. Edit the XML

Open the copy in a text editor (like Notepad++) and locate the <defaultValues> section. Update the values as follows:

XML
<defaultValues>
    <port>514</port>
    <protocol>TCP,UDP</protocol>
    <maxSize>10</maxSize>
    <rotate>40</rotate>
    <sslPort>1514</sslPort>
</defaultValues>

3. Swap and Restart

  1. Stop the Service: Open services.msc and stop the VMware Syslog Collector.
  2. Replace File: Delete the original config.xml and rename your modified copy to the original filename.
  3. Start the Service: Restart the VMware Syslog Collector.

Lazy Admin Tip: If the logs don’t start flowing immediately, you may need to restart the syslog service on the ESXi hosts themselves to re-establish the connection to the server.

#VMware #vSphere #Syslog #DataCenter #Storage #SysAdmin #ITPro #Virtualization #LogManagement #LazyAdmin #TechGuide

This entry was posted in VMware, Windows and tagged , , , , , , , .

How to Restart Management Agents on an ESXi Host?

Posted on Updated on

When an ESXi host appears as “Not Responding” in vCenter, or the Web Client becomes sluggish, you don’t always need to reboot the entire server. Often, simply restarting the Management Agents (hostd and vpxa) will restore connectivity without interrupting your running Virtual Machines.

Method 1: Using the Direct Console User Interface (DCUI)

If you have physical access to the server or a remote management console (like iDRAC, ILO, or IPMI), this is the safest method.

  1. Connect to your ESXi host console and press F2 to customize the system.
  2. Log in as root.
  3. Navigate to Troubleshooting Options using the arrow keys.
  4. Select Restart Management Agents.
  5. Press Enter, then press F11 to confirm the restart.
  6. Once finished, press Esc to log out.

Method 2: Using SSH or Local Shell

If SSH is enabled, you can restart the specific agents individually. This is the preferred “Lazy Admin” method as it provides granular control.

For ESXi 5.x, 6.x, and 7.x/8.x: Run these two commands to restart the host agent and the vCenter agent:

Bash

/etc/init.d/hostd restart
/etc/init.d/vpxa restart

For legacy ESXi 4.x:

Bash

/etc/init.d/hostd restart
service vmware-vpxa restart

Method 3: The “Nuclear” Option (services.sh)

If the individual agent restarts don’t work, you can restart all management services on the host.

Command:

Bash

services.sh restart

⚠️ CRITICAL CAUTION

If LACP is enabled on your Distributed Virtual Switch (DVS), do not use services.sh restart. Doing so can disrupt network connectivity. Instead, restart independent services or schedule a maintenance window.

Method 4: Resetting the Management Network

Sometimes the issue isn’t the agent, but the VMkernel interface itself. You can bounce the management interface (usually vmk0) with this one-liner:

Bash

esxcli network ip interface set -e false -i vmk0; esxcli network ip interface set -e true -i vmk0

The semicolon ensures the interface comes back up immediately after being disabled.

#VMware #ESXi #vSphere #SysAdmin #ITPro #Virtualization #Troubleshooting #LazyAdmin #DataCenter #TechTips #CloudAdmin

This entry was posted in VMware and tagged , , , , , , , , , .

How to get Serial number and System information of ESXi host remotely using putty

Posted on Updated on

🛠️ Method 1: Using esxcfg-info

The esxcfg-info command is a comprehensive tool that dumps a massive amount of data regarding the host’s configuration. Filtering this with grep is the quickest way to find your serial number.

Command:

Bash

esxcfg-info | grep "Serial Number"
  • What it does: Searches the entire configuration dump for the specific “Serial Number” string.
  • LazyAdmin Tip: If you get too many results, try esxcfg-info -w | grep "Serial Number" to focus specifically on hardware information.

🛠️ Method 2: Using dmidecode (DMI Table Decoder)

If you need more than just the serial number—such as the Manufacturer, Product Name (Model), and UUID—dmidecode is the standard tool. It retrieves data directly from the system’s Desktop Management Interface (DMI) table.

Command:

Bash

/usr/sbin/dmidecode | grep -A4 "System Information"
  • What it does: The -A4 flag tells grep to show the 4 lines after the match.
  • The Result: You will typically see:
    1. Manufacturer (e.g., Dell Inc., HP, Cisco)
    2. Product Name (e.g., PowerEdge R740)
    3. Version
    4. Serial Number
    5. UUID

🛠️ Method 3: The Modern ESXCLI Way

If you are on ESXi 6.x or 7.x/8.x, VMware has standardized most commands under the esxcli framework. This is often faster and cleaner than the legacy scripts.

Command:

Bash

esxcli hardware platform get
  • Why use this? It provides a clean, organized output of the Vendor Name, Product Name, and Serial Number without needing to grep.

⚠️ Troubleshooting Access

  1. SSH is Disabled: By default, SSH is turned off for security. You must enable it via the DCUI (the yellow and grey monitor screen) under “Troubleshooting Options” or via the Host Client web interface.
  2. Permission Denied: Ensure you are logging in as root. Standard users generally do not have permission to query the hardware DMI tables.
  3. Shell Lockdown: If the host is in “Lockdown Mode,” you will be unable to SSH in even with the correct credentials. You’ll need to disable Lockdown Mode via vCenter first.

#VMware #ESXi #SysAdmin #ITPro #CommandLine #ServerHardware #TechTips #LazyAdmin #DataCenter #RemoteManagement #vSphere

This entry was posted in VMware and tagged , , , , , , , , , .

ESXi Multipathing Decoded: MRU, Fixed, and Round Robin

Posted on Updated on

When you present a LUN to an ESXi host, the Native Multipathing (NMP) engine automatically assigns a policy based on the type of storage array detected. However, as an admin, you need to understand why a policy was chosen—and when you should manually intervene.

1. Most Recently Used (MRU)

Best For: Active/Passive Arrays. MRU selects the first working path it finds at boot. If that path fails, it switches to a standby path.

  • Key Behavior: It does not fail back. Even if the original path becomes healthy again, the host stays on the current path. This prevents “path thrashing” on Active/Passive arrays where switching controllers is an expensive operation.

2. Fixed

Best For: Active/Active Arrays. The Fixed policy uses a specific “Preferred Path.” If the preferred path fails, it moves to an alternative.

  • Key Behavior: It does fail back. As soon as that designated preferred path is back online, the host immediately switches back to it.

3. Round Robin (RR)

Best For: Load Balancing (Active/Active or ALUA). Round Robin rotates through all available paths to distribute the I/O load.

  • Active/Active: Uses every available path.
  • Active/Passive: Only uses all paths leading to the active controller.

Note: For Microsoft Failover Clusters (MSCS), Round Robin is only supported on ESXi 5.5 and later.

4. Fixed with Array Preference (FIXED_AP)

Introduced in ESXi 4.1 for ALUA-capable arrays, this policy lets the storage array tell the host which path is the “optimal” one.

  • Note: This was removed in ESXi 5.0 in favor of letting the NMP automatically select MRU or Fixed based on the array’s ALUA response.

⚠️ Critical Warnings for Admins

  1. Don’t Fight the NMP: VMware generally warns against manually changing a LUN from Fixed to MRU. The host chooses the policy based on the hardware it detects; forcing a change can lead to instability.
  2. Verify Vendor Support: Round Robin is powerful but not supported by every array. Always check the VMware Compatibility Guide before making it your default.
  3. MSCS Limitations: If you are virtualizing SQL clusters or other failover clusters, double-check your ESXi version before toggling Round Robin, or you risk losing disk heartbeat connectivity.

#VMware #ESXi #StorageAdmin #vSphere #Multipathing #SysAdmin #ITPro #Virtualization #LazyAdmin #DataCenter #StorageTips

This entry was posted in VMware and tagged , , , , , , , , , .

Troubleshooting Storage Latency with esxtop: The Admin’s Guide

Posted on Updated on

When “the server is slow,” the storage subsystem is usually the first suspect. While vCenter performance charts are great for history, esxtop gives you real-time data from the heart of the hypervisor.

🛠️ How to Configure esxtop for Storage Monitoring

You can monitor performance at three different levels depending on where you suspect the issue lies.

1. Per-HBA (Host Bus Adapter) Mode

  • Command: Type esxtop, then press d.
  • Tip: Press Shift + L and enter 36 to see the full device names.
  • Fields: Press f and ensure b, c, d, e, h, and j are selected.

2. Per-LUN (Device) Mode

  • Command: Type esxtop, then press u.
  • Why use this? To see if a specific volume on your SAN is being hammered.

3. Per-VM (Virtual Machine) Mode

  • Command: Type esxtop, then press v.
  • Why use this? To identify the “noisy neighbor”—the specific VM that is consuming all the IOPS.

🔍 Analyzing the “Big Three” Latency Columns

To understand storage health, you must look at these three columns. They tell you exactly where the delay is happening.

ColumnNameWhat it representsThreshold
DAVGDevice LatencyTime spent at the hardware level (HBA + SAN).< 10ms
KAVGKernel LatencyTime spent inside the VMware VMkernel.< 1ms
GAVGGuest LatencyTotal latency perceived by the Guest OS (DAVG + KAVG).< 10ms

What the numbers are telling you:

  • High DAVG: The problem is external to ESXi. Check your SAN controllers, disk spindles, or fabric switches.
  • High KAVG: The problem is inside the host. This usually means the host is overloaded or there is a queueing issue (e.g., Disk.SchedNumReqOutstanding is too low).
  • High GAVG: Your users are feeling the pain. If this exceeds 10–15ms consistently, application performance will suffer.

⚠️ When to Panic: Timeouts and Logs

If latency hits 5000ms (5 seconds), ESXi will abort the command. If you see high numbers in esxtop, immediately check your logs for SCSI aborts:

  • ESXi 5.x/6.x/7.x/8.x: /var/log/vmkernel.log
  • Legacy ESX 3.5/4.x: /var/log/vmkernel

#VMware #ESXi #esxtop #StorageAdmin #SysAdmin #Virtualization #PerformanceMonitoring #ITPro #LazyAdmin #DataCenter #vSphere

This entry was posted in VMware and tagged , , , , , , , , , .

Fixed: The VMRC Console has Disconnected (Error 2050470)

Posted on Updated on

It’s a frustrating scenario: you go to check a virtual machine, and instead of a login screen, you get a black box with the message: “The VMRC Console has Disconnected… Trying to reconnect.” To make matters worse, the VM often appears unreachable on the network, leading you to believe the Guest OS has blue-screened or frozen. However, the issue is frequently just a hang-up in the VMware Remote Console (VMRC) process on your local management workstation.

The Quick Fix

You do not need to restart the VM or the ESXi host. Usually, the “stuck” process is living right on your own PC.

  1. Open Task Manager: Right-click your taskbar and select Task Manager (or press Ctrl + Shift + Esc).
  2. Find the Process: Go to the Processes or Details tab.
  3. Kill VMRC: Look for vmware-vmrc.exe (or vmware-vmrc.exe*32 on older systems).
  4. End Task: Right-click the process and select End Task.
  5. Relaunch: Go back to your vSphere Client and attempt to open the console again.

Why does this happen?

This error usually occurs when the VMRC process loses its handshake with the ESXi host but fails to terminate properly. By killing the process, you force a fresh authentication and network handshake, which typically restores the video feed immediately.

What if the VM is still “Black Screened”?

If killing the local process doesn’t work and the VM is still unreachable via ping/RDP, the issue might be on the host side:

  • Check the Hostd Service: Sometimes the management agent on the ESXi host needs a restart.
  • Video Memory: Ensure the VM has enough Video RAM allocated in its “Edit Settings” menu to support the resolution you are using.

#VMware #vSphere #VMRC #SysAdmin #ITPro #Virtualization #TechSupport #LazyAdmin #ServerAdmin #WindowsTroubleshooting

This entry was posted in VMware and tagged , , , , , , , , .