[EN] How to Fix VMware vCenter Server Appliance Emergency Mode and File System Errors

[EN] How to Fix VMware vCenter Server Appliance Emergency Mode and File System Errors

VMware vCenter Server Appliance (VCSA) may sometimes fail to boot normally and enter Emergency Mode. This usually happens when the appliance detects a problem with its file system during startup.

A typical console message looks like this:

[FAILED] Failed to start File System Check on /dev/vg_root_0/lv_root_0

[DEPEND] Dependency failed for /sysroot
[DEPEND] Dependency failed for Initrd Root File System

This indicates that vCenter could not successfully check or mount the root file system, so the operating system stopped the normal boot process and switched to Emergency Mode.

Why Does This Problem Occur?

This issue can appear after several different events, such as:

  • An unexpected shutdown of the vCenter VM
  • An ESXi host crash or sudden reboot
  • A power outage
  • Temporary storage or datastore connectivity loss
  • A file system inconsistency
  • A full disk or partition
  • Storage latency or I/O problems
  • An improperly terminated VCSA session

In many cases, the issue is caused by the appliance being shut down without a clean operating system shutdown.

What Does the Error Mean?

The key line is:

Failed to start File System Check on /dev/vg_root_0/lv_root_0

The device:

/dev/vg_root_0/lv_root_0

is the logical volume that contains the VCSA root file system. During startup, Linux checks the file system for consistency. If the check fails, the system cannot safely mount the root file system and therefore cannot continue booting normally.

Recovery Steps

1. Check Your Recovery Options First

Before modifying the file system, it is important to confirm that you have a recovery plan. In a production environment, check the following:

  • Is there a recent VCSA backup?
  • Is file-based backup configured?
  • Is the datastore healthy and accessible?
  • Are there any storage alarms?
  • Is there a usable snapshot or restore point?

If possible, make sure the current state of the appliance is protected before starting repair operations.

2. Try Automatic File System Repair from GRUB

The first option is to let the operating system attempt an automatic file system repair. Restart the vCenter appliance. When the GRUB boot screen appears, press:

e

Find the line that starts with linux and add the following parameter at the end:

fsck.repair=yes

Then press:

F10

to continue booting. The system will attempt to detect and repair file system inconsistencies automatically. Depending on the severity of the issue, the boot process may take longer than usual.

3. Use Emergency Mode for Manual Checks

If the automatic repair does not solve the issue, manual intervention may be required. You may see:

Give root password for maintenance:

Enter the VCSA root password. Once logged in, verify the disk and LVM structure. Check block devices:

lsblk

Check logical volumes:

lvs

Check volume groups:

vgs

Check physical volumes:

pvs

Confirm that the affected logical volume is:

/dev/vg_root_0/lv_root_0

4. Repair the File System

After confirming the correct device, you can run a file system check.

For example:

fsck -fy /dev/vg_root_0/lv_root_0

Alternatively:

e2fsck -y /dev/vg_root_0/lv_root_0

During the repair process, messages similar to the following may appear:

Free blocks count wrong
FIXED

Free inodes count wrong
FIXED

Filesystem was modified

These messages indicate that the file system contained inconsistencies and that repairs were made. Because file system repair changes metadata, it is recommended to have a valid backup or recovery option before running these commands on a production system.

5. Restart the Appliance

After the repair is completed, restart the appliance:

reboot

Or shut it down cleanly:

shutdown -h now

Then power it on again from the hypervisor.

If the repair was successful, VCSA should continue the normal boot process instead of returning to Emergency Mode.

Checks After vCenter Starts

Bringing vCenter back online is only part of the recovery process. You should also identify why the problem occurred.

Check vCenter Services

Run:

service-control --status --all

Most services should show a Running status. If required, you can start all services with:

service-control --start --all

Check Disk Usage

Check overall disk usage:

df -h

Also check inode usage:

df -i

Pay special attention to the following VCSA partitions:

/
/storage/log
/storage/db
/storage/archive
/storage/core
/storage/seat

Partitions reaching 90% to 100% utilization can cause vCenter services to become unstable and may lead to additional operational problems. You can check the root partition specifically with:

df -h /

Review System Logs

To understand the root cause, review the appliance logs after recovery. Check system errors:

journalctl -p err

Check errors from the previous boot:

journalctl -b -1 -p err

Check kernel errors:

dmesg | grep -i error

For disk, I/O, and file system related events:

dmesg | egrep -i "error|fail|I/O|filesystem|ext4"

These logs can help determine whether the problem was caused by the operating system, the storage layer, or an unexpected shutdown.

Do Not Forget the ESXi and Storage Side

A VCSA file system error does not always mean the problem exists only inside the appliance. You should also check the infrastructure hosting the VM.

Review:

  • Datastore accessibility
  • Storage path status
  • APD or PDL events
  • Storage latency
  • HBA or iSCSI connectivity
  • Multipathing configuration
  • Datastore free space
  • ESXi storage logs

In shared storage environments, even a short interruption in storage access can result in file system inconsistencies inside a virtual machine.

How to Prevent the Issue from Happening Again

The following practices can reduce the risk of similar incidents:

  • Configure VCSA file-based backup
  • Perform regular restore tests
  • Monitor VCSA partition utilization
  • Create alerts for /storage/log and other critical partitions
  • Monitor datastore capacity
  • Monitor storage latency
  • Avoid forced VCSA shutdowns
  • Avoid keeping snapshots for long periods
  • Keep vCenter and ESXi updated
  • Forward critical logs to a centralized logging or SIEM platform
  • Review storage health and path redundancy regularly

Quick Recovery Flow

A simple troubleshooting workflow is:

vCenter fails to boot
        ↓
Emergency Mode
        ↓
Check file system error
        ↓
Confirm /dev/vg_root_0/lv_root_0
        ↓
Verify backup and recovery options
        ↓
Try fsck.repair=yes from GRUB
        ↓
If the problem continues
        ↓
Run fsck or e2fsck manually
        ↓
Restart VCSA
        ↓
Check vCenter services
        ↓
Check disk usage
        ↓
Review ESXi and storage logs

Comments

No comments yet. Why don’t you start the discussion?

Leave a Reply

Your email address will not be published. Required fields are marked *