Get expert server management services to fix VMware datastore connectivity issues, reduce storage latency, and improve virtual machine performance.
VMware virtual machines are very sensitive to storage. Virtual machines can slow or stop functioning when ESXi hosts are unable to access a datastore or when there are issues with storage latency. These issues can additionally slow down other applications, affect the backup process and can even result in system outages.
There are many causes of these problems, such as failures in the storage paths, environment configuration issues in storage, network problems, storage problems, environment configuration issues, and issues with storage hardware.
This guide explains different storage latency and datastore connection issues and provides a step-by-step guide to resolve these issues.
An Overview
Common Causes of VMware Datastore Connectivity Issues
ESXi loses access to storage systems and therefore cannot access the datastore.
The following list enumerates some of the more common causes of this issue:
- Network issues with iSCSI and NFS datastores
- Failure of Fibre Channel paths
- Faulty storage controllers
- Incorrect presentation of LUNs
- Issues with mounting datastores
- Multipath issues
- Failures of SAN switches
- Changes in storage configuration
As a result, virtual machines may become inaccessible or unresponsive.
Common Causes of VMware Storage Latency
Storage latency occurs when the storage system takes longer to respond to I/O requests. High latency can affect both virtual machines and applications.
| Cause | Possible impact |
|---|---|
| High I/O activity | More storage loads |
| Excessive snapshots | Lower storage efficiency |
| Limited storage resources | Slows I/O processing |
| Network congestion | Delays storage requests |
| High datastore usage | Increases storage pressure |
| Backup or replication activity | Adds storage workload |
| Poor storage hardware performance | Increases response time |
Get Expert Server Support.
How to Troubleshoot VMware Datastore Connectivity
1. Check Datastore Availability
The first step is to verify the availability of all required ESXi hosts’ datastores.
Review:
- Datastore status in vCenter
- ESXi host connectivity
- Active alarms and events
- Recent environment changes
Not all hosts failing to access a datastore indicates an issue with the storage system.
2. Check Storage Paths
Check the storage paths for the ESXi hosts and the storage system. Multipathing provides multiple storage connections, so access may be lost to storage due to a path failure.
Check for:
- Active paths
- Dead or inactive paths
- Path failover events
- Storage adapter issues
A path failure may be caused by a storage adapter, switch, or storage controller.
3. Review ESXi Storage Logs
ESXi logs can help identify storage communication problems.
Review:
- vmkernel.log
- vobd.log
- hostd.log
Look for errors related to:
- APD (All Paths Down)
- PDL (Permanent Device Loss)
- SCSI reservations
- Storage timeouts
- Path failovers
It helps in determining the issue’s root cause.
4. Check Storage Network Connectivity
For iSCSI and NFS datastores, the storage network must remain stable.
Check:
- Network Adapter status
- VLAN configuration
- Storage network connectivity
- Jumbo frame settings
- Switch port status
Verify there is no packet loss or network issues accessing the datastore.
5. Check Storage Array Health
Finally, check the health of the underlying storage system.
Review:
- Controller status
- Disk status
- Storage processor load
- Hardware alerts
- Capacity levels
If several hosts or datastores have problems at the same time, the storage array may be the common cause.
How to Troubleshoot VMware Storage Latency
Once datastore connectivity is stable, check what causes the high latency.
1. Check Datastore Performance
Use vCenter performance data to review:
- Read latency
- Write latency
- Datastore throughput
- IOPS utilization
If a performance issue is affecting just one datastore or the entirety of the storage environment.
2. Find High-I/O Workloads
Some virtual machines place significant loads on storage.
Check for:
- I/O-intensive virtual machines
- Database transactions
- Backup jobs
- Snapshots
- Replication activity
Throttling storage activity can enhance performance.
3. Review Snapshot Usage
Shorter snapshot retention time allows for better use of storage.
Check for:
- Long snapshot chains
- Orphaned snapshots
- Long-running snapshots
Storage load can be reduced by removing snapshots.
4. Check Storage Queue Depth
Storage space congestion happens when the requests far exceed the storage system’s capacity to process the commands.
Review:
- Queue depth usage
- ESXi storage adapter statistics
- Storage processor workloads
High queue depth may show signs of storage congestion and contribute to latency.
5. Improve Storage Performance
Whichever the case, corrective measures should be taken.
This may include:
- Balance virtual machine workloads across datastores
- Move high utilization virtual machines
- Increase storage capacity
- Improve storage network
- Upgrade storage systems
This can improve virtual machine performance and reduce storage latency.
Best Practices to Prevent VMware Storage Issues
Regular storage management improves connectivity and reduces latency issues.
Follow these practices:
- Activate storage multipathing when necessary
- Monitor storage performance
- Manage storage capacity
- Remove unneeded snapshots
- Distribute I/O-intensive workloads across datastores
- Ensure ESXi hosts and storage firmware are up to date
- Regularly perform storage health checks
- Monitor storage latency
Conclusion
VMware datastores and storage latency issues cause problems not only to virtual machines, but also to applications, backups, and system availability. Storage path problems, network issues, storage configuration issues, and high I/O in combination with storage hardware issues can cause latency issues.
To begin with, check the availability of the datastores and in the order, check the storage paths, ESXi logs, network connection, and the health of the storage arrays. Review performance metrics, storage workloads, snapshots, and queue consume in order to get a deeper understanding of the cause of the storage latency.
Efficient storage management and regular checks of the infrastructure leads to a predictable VMware environment.
