[EN] Network Rollback and Recovery in VMware vSphere 7.0 and Later

Central management of ESXi hosts in a vSphere environment rests on a single condition: a healthy management network. vCenter Server communicates with every host in its inventory over this network it pushes configuration changes through this channel and collects status information through it. On top of that, vSphere HA (High Availability) relies on the same network for heartbeat traffic between hosts.

This single point of dependency makes the management network one of the most sensitive components of the entire virtualization stack. A mistyped VLAN ID, an inadvertently changed MTU value, or the removal of an uplink operations that look trivial at first glance can sever a host’s connection to vCenter Server instantly. And here is the frustrating part: the moment connectivity drops, the very tool you would use to fix the mistake (vCenter Server) becomes unreachable as well. It is the classic case of sawing off the branch you are sitting on.

This is exactly the vicious circle that vSphere’s automatic rollback and DCUI-based recovery features were designed to break. This article walks through how these mechanisms work, which events trigger them, and how an administrator can put them to practical use.

Note: The definitive and authoritative reference on this topic is the vSphere Networking Guide. If you find any discrepancy between the guide and what is described here, treat the guide as correct.

Where the Problem Comes From: VSS vs. VDS

To understand why the rollback mechanism exists at all, you first need to appreciate the management difference between a standard virtual switch (VSS) and a distributed virtual switch (VDS).

The Standard Virtual Switch (VSS) Scenario

By definition, a VSS is local to the host its configuration lives on the ESXi server itself. So if the management network runs on a VSS and something goes wrong, the fix is relatively simple: connect to the server’s physical console (or an out-of-band interface such as IPMI, iLO, or iDRAC), reconfigure the management network through the DCUI (Direct Console User Interface), and you are done.

The blast radius stays confined to a single host.

The Distributed Virtual Switch (VDS) Scenario

With a VDS, configuration is centralized and managed by vCenter Server, with multiple hosts attached to the same distributed switch. That is a major advantage for operational consistency — but in a failure scenario the advantage turns into a liability very quickly.

Imagine an invalid change made to the distributed port group that carries the management VMkernel adapter. That change propagates to every host on the switch simultaneously and can knock all of them off vCenter Server at once.

The result vCenter Server can no longer make any correction to the VDS port group configuration, because there is no longer a channel through which to deliver it.

At that point the only classic remedy is a painful one:

1. Connect to each host individually via physical or remote console.
2. Build a VSS with a correct configuration on each one.
3. Move the management network onto that temporary VSS.
4. Once all hosts can communicate again, fix the VDS from vCenter Server.
5. Finally, migrate the management network back onto the VDS.

It is not hard to imagine what this means in a cluster with dozens of hosts.

An important detail: While the management network is on a VDS, the “Network Adapters” and “VLAN” options in the DCUI appear greyed out.

They remain unavailable until ESXi management networking is moved to a VSS.

This behavior catches many administrators off guard in the middle of an outage.

The Traditional Workaround: A Split Design

If physical NIC (pNIC) count is not a constraint on your hosts, there is a design pattern that has been recommended for years: keep the management network on a VSS and all other traffic (vMotion, vSAN, virtual machine traffic, and so on) on a VDS. This requires at least four network adapters per host — two connected to the VSS and two to the VDS.

The design is still valid and safe, but it comes at the cost of port density and cabling. vSphere’s automatic rollback feature was developed precisely to remove that trade-off: you can now comfortably keep the management network on a VDS.

How Automatic Rollback Works

The logic is simple but effective. When a network configuration change is applied, the host verifies whether it can still reach vCenter Server afterwards. If connectivity has been lost, the host does not let the change take effect and reverts to the last known valid configuration.

The corresponding task is marked as failed on the vCenter Server side.

In other words, the system protects you from your own mistake automatically: the wrong command runs, the outcome is measured, and if the outcome is bad, the operation is rewound.

Rollback is enabled by default. It can be enabled or disabled at the vCenter Server level if needed. Outside of very specific maintenance scenarios, disabling it is generally not advisable.

The events that can trigger a rollback fall into two main categories.

Category 1: Host Networking Rollbacks

These rollbacks fire when an invalid change is made to the host’s own networking configuration (at the virtual switch or network system level).

The governing rule is clear: every network change that disconnects a host triggers a rollback.

Typical triggers include:

Type of change	Description
Physical NIC speed/duplex	Setting a speed or duplex value incompatible with the physical switch port
DNS and routing settings	Defining an incorrect default gateway or an unreachable DNS server
Teaming/failover or traffic shaping policies	Policy changes on a standard port group that contains the management VMkernel adapter
VLAN change	Changing the VLAN ID of a standard port group containing the management VMkernel adapter
MTU increase	Raising the MTU of the management VMkernel adapter and its switch to a value the physical infrastructure does not support (e.g. jumping to 9000 without end-to-end jumbo frame configuration)
IP settings	Changing the IP address, subnet mask, and similar settings of the management VMkernel adapter
Removing the VMkernel adapter	Deleting the management VMkernel adapter from a standard or distributed switch
Removing a physical NIC	Pulling a pNIC (uplink) from a standard or distributed switch that carries the management VMkernel adapter

If any of these operations causes a loss of connectivity, the task fails and the host returns to its last valid configuration. From the user’s point of view the visible outcome is an “operation failed” error — but behind the scenes a serious outage has just been averted.

Category 2: Distributed Switch Rollbacks

The second category covers invalid updates made to distributed switches, distributed port groups, or individual distributed ports.

The main VDS changes that can trigger a rollback:

• Changing the MTU of the distributed switch
• Changing any of the following on the distributed port group carrying the management VMkernel adapter:
  • Teaming and failover
  • VLAN
  • Traffic shaping
• Blocking all ports in the distributed port group that contains the management VMkernel adapter
• Overriding the policies above at the level of the individual distributed port to which the management VMkernel adapter is connected

If one of these changes results in an invalid configuration, one or more hosts may end up out of sync with the distributed switch.

Fixing Hosts That Fall Out of Sync

There are two paths forward here, and which one you take depends on whether you know the source of the problem.

1. If you know where the conflict is – correct the setting manually.

A classic example: you accidentally migrated the management VMkernel adapter to a new VLAN, but that VLAN was never trunked to the relevant ports on the physical switch. The fix here is to correct the physical switch configuration. As soon as the physical side is right, the next distributed switch-to-host synchronization resolves the issue on its own.

This scenario is a useful reminder that the problem is not always inside vSphere. What looks like a fault in the virtual layer is very often a reflection of a gap in the physical one.

2. If you are not sure where the problem is – roll back to a previous configuration.

You can revert the distributed switch or the distributed port group to an earlier configuration. This process is not automatic — both steps are performed manually. Because vSphere retains previous versions of VDS configurations, returning to a known-good point is possible.

For the steps to revert to a previous configuration or to disable network rollback through the web client, refer to the “vSphere Networking Rollback” section of the vSphere Networking documentation.

Recovering from Network Misconfiguration Using the DCUI

Automatic rollback is a powerful safety net, but it cannot cover every scenario. If a change occurs in the physical infrastructure after the fact, or if connectivity loss surfaces with a delay, a host can still end up unreachable.

This is where vSphere 7.0 and later offers a second route to recovery: connecting directly to the host and fixing distributed switch properties or other networking misconfigurations through the DCUI. This capability was not available in VDS environments in the past, and it eliminates precisely the “build a VSS on every host” chore described above.

Prerequisites and Limitations

• Recovery is not supported on stateless ESXi instances. For hosts running stateless via Auto Deploy, you need a different recovery strategy.
• The management network must be configured on a distributed switch. This is the only way to fix distributed switch configuration errors from the DCUI.

Steps to Restore a VDS from the DCUI

1. Connect to the host’s DCUI (via the physical console or an out-of-band management interface).
2. From the Network Restore Options menu, select Restore vDS.
3. Enter the correct values in the relevant fields: VLAN, uplink, and, where appropriate, the blocked property.
4. Press Enter to confirm.

What Happens Behind the Scenes

Once these steps complete, a chain of operations takes place:

1. The DCUI clones a host local port from the existing misconfigured port.
2. It applies the VLAN and Blocked values you supplied to that new port.
3. It switches the management network over to this newly created local port — restoring connectivity to vCenter Server.
4. vCenter Server detects the new host local port and updates its database with the new information.
5. vCenter Server creates a standalone port connected to the management network.

The end result is that the host rejoins the vCenter Server inventory in a healthy state, and you can make permanent corrections from the central interface again.

Practical Recommendations

A few concrete suggestions for translating all of this into day-to-day operations:

• Do not disable rollback. There is a good reason it ships enabled by default. Unless you have a very specific scenario that demands otherwise, leave it alone.
• Apply changes incrementally. Settings that require end-to-end consistency — MTU and VLAN in particular — should be tested on a single host before being rolled out across the cluster.
• Prepare the physical layer first. VLAN trunks, jumbo frame support, and port channel settings should all be verified before you change anything on the virtual side. A significant share of “virtual” networking faults actually originate in the physical layer.
• Keep out-of-band access ready at all times. DCUI recovery only helps if you can actually reach the console. Make sure your iLO/iDRAC/IPMI access works and that credentials are current — test this in advance, not in the middle of an incident.
• Plan separately for stateless hosts. If you use Auto Deploy, DCUI recovery is not an option; define a fallback strategy at the host profile and image profile level instead.
• Export the configuration before major VDS changes. Having a clean reference point makes manual rollback dramatically easier when you need it.