In high-availability environments, the most misleading incidents are not always the ones where a server completely goes offline.

Sometimes every server is reachable, services are running, resource utilization looks normal, and basic connectivity checks show no obvious issue.

Yet application connections begin to fail.

We experienced exactly this type of incident in a SQL Server Always On Availability Group environment.

At first glance, everything looked stable:

• Cluster members were reachable.
• Windows Cluster services were running.
• SQL Server instances showed no unusual resource pressure.
• Network connectivity was available.
• Listener DNS and IP resources appeared to be online.

However, the application side told a very different story.

Connections to the database were being interrupted periodically, sessions established through the Listener were dropping, and the Availability Group would occasionally go offline without an obvious reason.

What made the issue especially difficult was that most standard health checks were passing.

Why Did the Initial Actions Fail?

When a production outage is in progress, the pressure to make quick changes increases rapidly.

In this case, the initial focus was placed on quorum behavior and the number of cluster nodes.

An additional server was introduced into the cluster and the witness configuration was changed.

The expectation was that these changes would make the cluster decision process more resilient.

But after the changes were completed, the connection drops continued exactly as before.

A new node had been added.

The cluster topology had changed.

But the actual problem was still there.

That was an important reminder:

Expanding the architecture does not automatically eliminate the source of a failure.

Especially in distributed systems, adding another component to an unresolved issue can simply increase the number of dependencies that need to be analyzed.

At that point, we stopped making architectural changes and started focusing on what was happening at the exact time of the failures.

The Turning Point: Cluster Communication Logs

When the WSFC logs were reviewed chronologically, a repeating pattern became visible.

At the NetFT layer, several cluster nodes were missing consecutive heartbeat packets.

Shortly afterward, the logs also showed that UDP communication between certain nodes had been interrupted.

A simplified example looked like this:

Microsoft Failover Cluster Virtual Adapter (NetFT)

Node-02 -> Node-03 : Multiple consecutive heartbeat packets missed
Node-02 -> Node-01 : Multiple consecutive heartbeat packets missed

Node-02 -> Node-03 : UDP communication lost
Node-02 -> Node-01 : UDP communication lost

These entries shifted our attention away from the SQL Server engine itself and toward communication between the cluster members.

The network was not completely unavailable.

There was no long-lasting outage either.

Instead, there were very short periods of instability that were difficult to capture with traditional network checks.

In other words:

The network was reachable, but it was not consistently stable enough from the cluster’s perspective.

Those are not the same thing.

Why Ping Was Not Enough

One of the common mistakes during troubleshooting is to evaluate network health only through ICMP tests.

A successful ping confirms that a packet reached the target at a particular moment.

It does not prove that communication between two cluster nodes is consistently stable from a timing perspective.

WSFC relies on cluster members exchanging health signals at specific intervals.

If enough of those signals are missed consecutively, the cluster must decide whether another node is still considered available.

This means that a short network disturbance may:

• go unnoticed by users,
• remain invisible during a manual ping test,
• have little impact on average latency metrics,

and still affect cluster heartbeat communication.

That was exactly what we were seeing.

How the Failure Chain Developed

After correlating the logs and event timestamps, the sequence became clearer:

Short-lived network instability

→

Several heartbeat packets missed in succession

→

NetFT communication disruption

→

Cluster health evaluation affected

→

Availability Group resource enters an unhealthy state

→

Client connections through the Listener are interrupted

From the application team’s perspective, only the final outcome was visible:

“The database connection dropped.”

But the actual chain of events had started much deeper in the infrastructure stack.

The Parameters That Matter

Three settings became especially relevant during the investigation:

• SameSubnetDelay
• SameSubnetThreshold
• LeaseTimeout

These parameters do not control the same mechanism.

Understanding that distinction is important before making any changes.

What Does SameSubnetDelay Control?

SameSubnetDelay defines how frequently cluster nodes on the same subnet send heartbeat traffic to one another.

For example, if the value is 1000 milliseconds, nodes exchange heartbeat signals roughly once every second.

A simplified representation is:

Node-A  ---- heartbeat ---->  Node-B
Node-A  <--- heartbeat -----  Node-B

The purpose is to confirm that another cluster member is still communicating.

However, missing a single heartbeat does not necessarily cause the node to be declared unavailable.

That is where the next parameter becomes important.

What Does SameSubnetThreshold Define?

SameSubnetThreshold determines how many consecutive heartbeat misses can occur before the cluster treats a node as unreachable.

Therefore, the combination of Delay and Threshold roughly defines the cluster’s heartbeat tolerance window.

For example:

SameSubnetDelay     : 1000 ms
SameSubnetThreshold : 5

gives an approximate tolerance of:

1000 × 5 = 5000 ms

or about 5 seconds.

That number should not be considered in isolation.

In real production environments, several factors can influence cluster communication, including:

• virtualization host contention,
• NIC behavior,
• network congestion,
• temporary switch latency,
• driver issues,
• CPU scheduling delays,
• security devices,
• transient packet loss.

Any combination of these can consume the available heartbeat window.

Why LeaseTimeout Must Be Evaluated Separately

Heartbeat and lease behavior are sometimes treated as if they were the same mechanism.

They are not.

Heartbeat communication primarily concerns cluster members detecting each other.

Lease behavior is part of the health relationship between SQL Server and the Windows Failover Cluster service.

A simplified view looks like this:

Cluster heartbeat

NODE-01  <---------->  NODE-02

Whereas the lease relationship is closer to:

SQL Server  <---------->  Windows Failover Cluster

If SQL Server cannot renew its lease within the expected period, the Availability Group resource can be marked unhealthy even if the operating system and SQL Server service are still running.

That is why reviewing only node-to-node heartbeat behavior may not provide the full picture.

Measure First, Change Later

The right starting point is not to immediately increase thresholds.

The first step should be documenting the current values.

For example:

Get-Cluster |
Select-Object SameSubnetDelay, SameSubnetThreshold

The Availability Group resource lease value can also be inspected with:

Get-ClusterResource -Name "AG_Name" |
Get-ClusterParameter -Name LeaseTimeout

The next step is to correlate these values with the actual timing of failures in the cluster logs and SQL Server logs.

The key question is:

How long are the communication interruptions, and how close are they to the existing tolerance limit?

Without that answer, any configuration change is based on assumption rather than evidence.

The Calibration Applied in Our Environment

Based on the timing patterns identified in the logs, we adjusted the cluster’s tolerance for short communication interruptions.

An example configuration might look like:

(Get-Cluster).SameSubnetDelay = 1000
(Get-Cluster).SameSubnetThreshold = 20

This gives an approximate heartbeat tolerance window of:

1000 ms × 20 = 20000 ms

or around 20 seconds.

On the Availability Group resource side, the lease value can also be adjusted where justified:

(Get-ClusterResource -Name "AG_Name").LeaseTimeout = 60000

After any change, the effective values should be verified:

Get-Cluster |
Select-Object SameSubnetDelay, SameSubnetThreshold

Get-ClusterResource -Name "AG_Name" |
Get-ClusterParameter -Name LeaseTimeout

In our environment, once these values were recalibrated based on the observed behavior, the recurring Availability Group interruptions stopped.

One Important Distinction

Increasing a threshold is not the same as fixing a network problem.

That distinction is critical.

If the infrastructure contains real packet loss, NIC issues, switch instability, virtualization latency, or network congestion, those issues still need to be investigated separately.

Increasing heartbeat tolerance only changes how the cluster reacts to temporary communication instability.

In other words:

You may be increasing the cluster’s tolerance to the problem rather than eliminating the problem itself.

For that reason, tuning should never replace root-cause analysis.

The two should complement each other.

The Trade-Off Between Tolerance and Failover Speed

Every high-availability tuning decision has a cost.

If the cluster waits longer before considering a node unavailable, it may become more tolerant of short-lived disturbances.

But during a real node failure, that same behavior can delay failover detection.

Therefore, the objective should never be:

“Use the highest threshold possible.”

The correct objective is:

Find the lowest safe tolerance that matches the actual behavior of the environment.

In practice, the design must balance two extremes.

Values that are too aggressive

→ unnecessary failovers during brief disturbances.

Values that are too tolerant

→ slower reaction during a genuine failure.

The correct setting depends on the system’s business and technical requirements.

Technical Lessons from the Incident

This case reinforced several important points.

First, adding another cluster node does not automatically make the environment more stable.

If the real issue exists in the communication layer, another node may simply introduce more communication paths that need to be maintained.

Second, default values should not automatically be treated as ideal values.

Defaults are starting points.

In critical environments with strict RTO requirements, high transaction volumes, and complex network paths, these settings should be evaluated against actual workload and infrastructure behavior.

Third, SQL Server logs alone are not enough for this type of incident.

An Always On architecture is not made up of SQL Server alone.

The following layers may all contribute to the final behavior:

• SQL Server
• Windows Server Failover Cluster
• NetFT
• Operating system
• Network infrastructure
• Virtualization platform
• Client connection behavior

A meaningful investigation requires these layers to be reviewed together.

Why “Is It Network or Database?” Is the Wrong Question

During production incidents, teams often try to determine ownership before determining behavior.

“The network looks fine.”

“SQL Server is running.”

“No Windows alarms are visible.”

“Monitoring shows everything green.”

Each statement may be individually correct.

And the overall system may still be failing.

In distributed architectures, the problem is often not that one component has completely failed.

The failure may exist in the interaction between components.

That is why the more useful question is not:

“Which team owns the problem?”

but:

“Which health mechanism failed, and what telemetry proves it?”

That change in perspective turns troubleshooting from assumption-driven investigation into evidence-driven analysis.

In a high-availability environment, the fact that a server is online is not enough.

A node may be running.

SQL Server may be started.

The NIC may still show a valid link.

Monitoring dashboards may remain green.

But if cluster members cannot communicate with one another within the expected timing window, the high-availability chain can still break.

For that reason, Always On troubleshooting should not focus only on whether services are running.

It should also focus on whether the components can maintain healthy communication within the time boundaries expected by the platform.

Because a resilient HA architecture is not defined only by servers that are powered on.

It is defined by components that can continue to recognize one another as healthy at the right time.