Sui Releases Post-Mortem Report Following Mainnet Delay That Halted Transactions for Six Hours

Sui publishes post-mortem after mainnet stall halted transactions for six hours

The Sui Foundation has released a comprehensive post-mortem report detailing the incident that caused a six-hour halt in transaction processing on January 14. The stoppage was a protective measure taken to prevent inconsistent finalization of network states.

Key Takeaways

A mainnet stall disrupted transactions for about six hours on January 14.
The issue arose from a bug in the consensus commit logic, resulting in validator discrepancies.
Transaction execution paused, but read operations maintained data consistency.
The Sui Foundation will implement improvements to enhance recovery time and checkpoint validation.

Understanding the Mainnet Stall: A Closer Look

The Sui Foundation's report clarifies that the mainnet stall occurred as a precaution to uphold network safety amidst divergent validator consensus processes.

The disruption lasted for roughly six hours, primarily due to an internal disagreement among validators about transaction confirmations. During this time, they could not certify new checkpoints, which caused transaction submissions to timeout while prioritizing network safety over immediate transaction processing.

What Caused the Consensus Divergence?

According to the Foundation, the root of the incident was an edge-case bug within the consensus commit logic. This issue impacted how conflicting transactions were dealt with, particularly in situations involving garbage collection. Different validators began deriving inconsistent consensus outputs and tried to execute incompatible candidate checkpoints.

When it became apparent that over a third of the stake was signing a different checkpoint digest, the process to certify those checkpoints was halted. This decision was made to prevent the finalization of an inconsistent state. The Foundation highlighted that this is the expected failure mode for these types of issues, emphasizing that the network is designed to safely stop rather than risk forks or create irreversible inconsistencies.

No Forks, Rollbacks, or Loss of Funds

The Sui Foundation reassured users that the stall was not a result of network congestion, excessive transaction volume, or external threats. Although transactions were unable to execute during the incident, read operations continued to function, providing users and applications with access to the last certified state, which ensured data consistency.

Future Improvements to Enhance Network Resilience

In light of the incident, the Sui Foundation aims to implement several enhancements to minimize recovery time in case of similar issues occurring in the future. These planned improvements include:

Quicker identification of checkpoint inconsistencies.
More automated tools for operators to rectify divergent states.
Expanded consensus-specific testing to reproduce issues and confirm solutions before they are deployed.

Despite the disruptions experienced, the Foundation affirmed that Sui’s safety-centric architecture responded as intended during this episode.

Conclusion

The Sui Foundation's response and analysis of the mainnet stall reinforce its commitment to maintaining a secure network environment. By learning from this incident and prioritizing safety, Sui aims to strengthen its infrastructure and enhance overall user experience in the future.