File system errors are always a nightmare for server owners.

One such error is “Unexpected inconsistency: Run fsck manually” error in VMware servers.

At Bobcares, we help server owners resolve these critical errors as part of our Server Management Services.

Today, let’s discuss the top 2 reasons for this error and how we fix them.

‘Unexpected inconsistency: Run fsck manually’ error in VMware – What this means?

Customers usually face this error after powering on the virtual machine in VMware. This can be a newly deployed VM or an existing VM failing to boot.

This error means that the system fails to boot the operating system and has been stuck during file system check. Users see the complete error message as below.

UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
(i.e., without -a or -p options)
*** An error occurred during the file system check.
*** Dropping you to a shell; the system will reboot
*** when you leave the shell.
Give root password for maintenance
(or type CTRL-D to continue)

Here, the system prompts for the root password to repair the corrupted file system.

‘Unexpected inconsistency: Run fsck manually’ error in VMware – Causes & Fixes

Now, let’s see the main reasons for this error and how our Dedicated Engineers fix them.

1) Corrupted file system

One of the main reasons for this error is the corrupted file system on the VMware host. This prevents the server from booting automatically. File system may get corrupted due to improper shutdowns, power failures, hard disk errors, etc.

How we fix?

The solution is to run a file system check and recover the corrupted file system. Sometimes, the system itself prompts for the root password. Here, we enter the root password and continue with the file system check. Otherwise, our Support Experts access the VMware host in single user mode and do a file system check.

For example, we enter in single user mode on a VMware host using the below steps.

• Reboot the VMware host.
• Once the GRUB screen appears, press space bar to disable autoboot.
• Select the Service Console only(troubleshooting mode).
• Press a key to modify the boot options.
• Then, type a space and add the word single. This will boot the server in single user mode.

Further, to do a complete file system check, we use the following command.

fsck -f -c -y

Similarly, we use e2fsck command to do file system check on a single partition. In other words, we follow the below steps to identify and recover the affected partition.

• Verify the partition tables using fdisk command.

fdisk -l

• Unmount the affected partition. For example, we use the below command to unmount the partition /dev/sda5.

umount /dev/sda5

• Run e2fsck command to do a file system check.

e2fsck -p /dev/sda5

Finally, we reboot the system to update the changes. Most importantly, our Server Experts ensure to make a clone of the VMware host before running fsck.

2) Wrong data & time on VMware host

Similarly, another common reason for this error is that the date and time on the VMware Host is far off the current date and time.

This can mainly occur when the NTP server which provides clock synchronization isn’t installed/running on the VMware host. As a result, the system fetches the date and time from the local hardware which can be far off from the current date and time and throws this error.

/dev/sda5: Superblock last mount time (Tue Nov 30 13:15:25 2018, now = thu Nov 8 08:30:25 2012) is in the future.
/dev/sda5: UNEXPECTED INCONSISTENCY: RUN fsck MANUALLY. (i.e., without -a or -p options) [FAILED]

How we fix?

Firstly, our Support Experts verify the current date and time on the VMware host. We usually check it from the command line of VMware host or from the VMware VSphere client. If we find any mismatch in date and time, we’ll correct these settings in the VMware host.

Initially, we correct the date and time in the BIOS of the hardware where the VMware host is running. As the server start up, we edit the BIOS configuration and update the correct date and time.

Further, we update the correct date and time in VMware host configuration from the VSphere client. Here, we modify the date and time of the VMware host from Manage > Settings > Time Configuration. Also, we ensure that the NTP service is configured and started in the VMware host. Most importantly, we ensure that all the VMs on the VMware host are stopped before making these changes. Otherwise, it affects the working of all the running VMs.

Sometimes, we need to restart the VMware host to reflect these changes. In such cases, all the VMs on this host will undergo a graceful reboot. So, our Server Experts always plan this task with the customer and schedule it at off peak hours.

Finally, we verify the date and time from the VSphere client or after accessing the VMware host via SSH.

[Do you need an expert to resolve this error on your VMware server. Our Server Experts can help you here.]

Conclusion

In short, ‘unexpected inconsistency: run fsck manually’ error in VMware can occur mainly due to file system errors, wrong date and time on VMware host, etc. Today, we’ve discussed these reasons in detail and how our Dedicated Support Engineers fix them.