VMware
Troubleshooting Storage Latency with esxtop: The Admin’s Guide
When “the server is slow,” the storage subsystem is usually the first suspect. While vCenter performance charts are great for history, esxtop gives you real-time data from the heart of the hypervisor.
🛠️ How to Configure esxtop for Storage Monitoring
You can monitor performance at three different levels depending on where you suspect the issue lies.
1. Per-HBA (Host Bus Adapter) Mode
- Command: Type
esxtop, then press d. - Tip: Press Shift + L and enter
36to see the full device names. - Fields: Press f and ensure b, c, d, e, h, and j are selected.
2. Per-LUN (Device) Mode
- Command: Type
esxtop, then press u. - Why use this? To see if a specific volume on your SAN is being hammered.
3. Per-VM (Virtual Machine) Mode
- Command: Type
esxtop, then press v. - Why use this? To identify the “noisy neighbor”—the specific VM that is consuming all the IOPS.
🔍 Analyzing the “Big Three” Latency Columns
To understand storage health, you must look at these three columns. They tell you exactly where the delay is happening.
| Column | Name | What it represents | Threshold |
| DAVG | Device Latency | Time spent at the hardware level (HBA + SAN). | < 10ms |
| KAVG | Kernel Latency | Time spent inside the VMware VMkernel. | < 1ms |
| GAVG | Guest Latency | Total latency perceived by the Guest OS (DAVG + KAVG). | < 10ms |
What the numbers are telling you:
- High DAVG: The problem is external to ESXi. Check your SAN controllers, disk spindles, or fabric switches.
- High KAVG: The problem is inside the host. This usually means the host is overloaded or there is a queueing issue (e.g., Disk.SchedNumReqOutstanding is too low).
- High GAVG: Your users are feeling the pain. If this exceeds 10–15ms consistently, application performance will suffer.
⚠️ When to Panic: Timeouts and Logs
If latency hits 5000ms (5 seconds), ESXi will abort the command. If you see high numbers in esxtop, immediately check your logs for SCSI aborts:
- ESXi 5.x/6.x/7.x/8.x:
/var/log/vmkernel.log - Legacy ESX 3.5/4.x:
/var/log/vmkernel
#VMware #ESXi #esxtop #StorageAdmin #SysAdmin #Virtualization #PerformanceMonitoring #ITPro #LazyAdmin #DataCenter #vSphere
Fixed: The VMRC Console has Disconnected (Error 2050470)
It’s a frustrating scenario: you go to check a virtual machine, and instead of a login screen, you get a black box with the message: “The VMRC Console has Disconnected… Trying to reconnect.” To make matters worse, the VM often appears unreachable on the network, leading you to believe the Guest OS has blue-screened or frozen. However, the issue is frequently just a hang-up in the VMware Remote Console (VMRC) process on your local management workstation.
The Quick Fix
You do not need to restart the VM or the ESXi host. Usually, the “stuck” process is living right on your own PC.
- Open Task Manager: Right-click your taskbar and select Task Manager (or press
Ctrl + Shift + Esc). - Find the Process: Go to the Processes or Details tab.
- Kill VMRC: Look for
vmware-vmrc.exe(orvmware-vmrc.exe*32on older systems). - End Task: Right-click the process and select End Task.
- Relaunch: Go back to your vSphere Client and attempt to open the console again.
Why does this happen?
This error usually occurs when the VMRC process loses its handshake with the ESXi host but fails to terminate properly. By killing the process, you force a fresh authentication and network handshake, which typically restores the video feed immediately.
What if the VM is still “Black Screened”?
If killing the local process doesn’t work and the VM is still unreachable via ping/RDP, the issue might be on the host side:
- Check the Hostd Service: Sometimes the management agent on the ESXi host needs a restart.
- Video Memory: Ensure the VM has enough Video RAM allocated in its “Edit Settings” menu to support the resolution you are using.
#VMware #vSphere #VMRC #SysAdmin #ITPro #Virtualization #TechSupport #LazyAdmin #ServerAdmin #WindowsTroubleshooting
- ← Previous
- 1
- …
- 3
- 4