Ethereum 2.0’s final Medalla testnet, which was launched on August 4, experienced a crash last week where validators found themselves losing up to 25% of their holdings and validator participation on the network dropped straight from close to 80% to 0% in a matter of minutes, according to a diagram shared by u/vinL0r on Reddit.
A Cloudflare bug on the Prysm client, one out of four clients that were running at genesis for the testnet, was identified as the cause of the problem. Prysmatic Labs quickly launched an incident report detailing the issue and the steps that were being taken to diagnose the synchronisation problem and introduce a fix.
“The cloudflare roughtime servers all returned wrong information, and Prysm nodes did not properly fallback from this situation. This bug caused all Prysm nodes to exhibit clock skew. Because of this clock skew, validators incorrectly proposed blocks and attestations for future slots. Erroneous roughtime responses from cloudflare. Specifically, “ticktock” reported 24 hours into the future. This timestamp, averaged with all 6 servers, made for a +4 hour clock adjustment in all Prysm nodes,” the report reads.
After dealing with the problem swiftly to avoid a total collapse of the testnet, Prysmatic Labs released Alpha.22 on August 17 to target synch issues on the Medalla testnet. The team also called on the Ethereum community to maintain and update healthy nodes on the network to boost the overall participation rate by validators.
You may also want to read: High DeFi Rewards Won’t Last with New Tokens Continuously Flooding the Market, Says CZ
