Troubleshooting Storage Latency with esxtop: The Admin’s Guide
When “the server is slow,” the storage subsystem is usually the first suspect. While vCenter performance charts are great for history, esxtop gives you real-time data from the heart of the hypervisor.
🛠️ How to Configure esxtop for Storage Monitoring
You can monitor performance at three different levels depending on where you suspect the issue lies.
1. Per-HBA (Host Bus Adapter) Mode
- Command: Type
esxtop, then press d. - Tip: Press Shift + L and enter
36to see the full device names. - Fields: Press f and ensure b, c, d, e, h, and j are selected.
2. Per-LUN (Device) Mode
- Command: Type
esxtop, then press u. - Why use this? To see if a specific volume on your SAN is being hammered.
3. Per-VM (Virtual Machine) Mode
- Command: Type
esxtop, then press v. - Why use this? To identify the “noisy neighbor”—the specific VM that is consuming all the IOPS.
🔍 Analyzing the “Big Three” Latency Columns
To understand storage health, you must look at these three columns. They tell you exactly where the delay is happening.
| Column | Name | What it represents | Threshold |
| DAVG | Device Latency | Time spent at the hardware level (HBA + SAN). | < 10ms |
| KAVG | Kernel Latency | Time spent inside the VMware VMkernel. | < 1ms |
| GAVG | Guest Latency | Total latency perceived by the Guest OS (DAVG + KAVG). | < 10ms |
What the numbers are telling you:
- High DAVG: The problem is external to ESXi. Check your SAN controllers, disk spindles, or fabric switches.
- High KAVG: The problem is inside the host. This usually means the host is overloaded or there is a queueing issue (e.g., Disk.SchedNumReqOutstanding is too low).
- High GAVG: Your users are feeling the pain. If this exceeds 10–15ms consistently, application performance will suffer.
⚠️ When to Panic: Timeouts and Logs
If latency hits 5000ms (5 seconds), ESXi will abort the command. If you see high numbers in esxtop, immediately check your logs for SCSI aborts:
- ESXi 5.x/6.x/7.x/8.x:
/var/log/vmkernel.log - Legacy ESX 3.5/4.x:
/var/log/vmkernel
#VMware #ESXi #esxtop #StorageAdmin #SysAdmin #Virtualization #PerformanceMonitoring #ITPro #LazyAdmin #DataCenter #vSphere