The Sepolia hardfork was scheduled to go live on March 5th at 7:30 UTC. Shortly after the hardfork was activated we told Jim McDonald to send a deposit to test the execution triggered withdrawal functionality added in Pectra. Shortly after, we saw error messages on our geth node and started seeing a lot of empty blocks being mined.
The error message said “unable to parse deposit data: deposit wrong length: want 576, have 32”. The only explanation for this error message was, that the deposit contract emitted some event that was not the expected Deposited event that we were looking for.
We quickly realized that, because the deposit contract is token gated, an ERC-20 transfer event was emitted whenever a deposit was processed.
The logic in EIP-6110 demanded that all logs from the deposit contract would be processed in the same way and the client should error out, if the emitted event was incorrect. All clients followed this behavior. However because of this, the produced block would be invalidated. Therefore the nodes would only produce empty blocks, because as soon as they included the transaction to the deposit contract, the block production would produce an error.
Felix from the go-ethereum team came up with the following fix, which ignores all erroneous logs coming from the deposit contract:
The problem now was to coordinate the rollout. We knew that if we rolled this fix out in an uncoordinated fashion, the chain would split, since the un-updated nodes would not be able to sync the new chain. So we decided at roughly 10:30 UTC to do a coordinated rollout at 14 UTC. This gave teams the necessary time to cut releases and to get their machines ready for the update.
These three and a half hours were a pretty long time during which Sepolia would only produce empty blocks. So we decided to remove the transactions that were continuously triggering this edge case by replacing them. This was possible, because we knew that only Jum and Paritosh from the EthPandaOps team send transactions to the contract. Those transactions were replaced by higher paying ones, so the edge case was not triggered anymore and we started seeing full blocks again.
We checked the deposit contract and verified that no one could trigger the deposit functionality (because it is token gated and we only gave out tokens to trusted parties for Sepolia). We missed one edge case in the ERC20 spec though.
After a few minutes we saw a lot of empty blocks again, so we looked again into the transaction pools and found another offending transaction that triggered the same edge cases. First we thought that someone from the trusted validators has made a mistake, but we quickly realized that this transaction originated from a new account recently funded by the faucet. Someone found the edge case in the ERC20 contract that we missed.
The ERC20 standard does not forbid 0 token transfer, this allows anyone (even if they don’t own any token) to transfer 0 tokens to another address which will emit an event. The “attacker” realized this edge case that we missed and sent a transaction that triggered the behavior again.
The only way to stop the attack would be to filter out all transactions that interact with the deposit contract. So we made the following private fix which we deployed to a few of the devops nodes. We suspected that the attacker was reading some of our chats, so we decided not to publicize the fix, but only update a few nodes that we controlled in order to get more full blocks on the network.
The fix is only filtering out transactions that directly call the deposit contract. If we publicized the fix, the attacker would’ve been able to circumvent our mitigation by calling the contract from another contract. These internal calls would still trigger the event, but they wouldn’t be easy to filter out during block creation.
Once we updated all ef_devops nodes (which are roughly 10% of the network) they started proposing full blocks again. Doing so allowed users to still use the chain until we deployed the real fix in a coordinated fashion.
At 14:00 UTC all nodes updated to the new releases which contained the real fix. A few blocks later the attackers transaction was mined successfully which proved that all node operators had successfully updated. We never lost finalization during the incident and as previously said, the issue only happened on Sepolia, because we’re using a token gated deposit contract instead of the normal mainnet deposit contract.