In the advent of AI, responsible disclosure has become more important and critical. AI allows to reproduce and uncover issues quicker and shortens the time a bug can stay hidden. It also enables a motivated attacker to easily monitor releases of critical software and their dependencies for hidden bug fixes and frontrun deployments. Therefore we must adapt our disclosure and patch processes, especially for Ethereum mainnet.
Ethereum’s special challenges
The Ethereum ecosystem is special in that we have multiple implementations in many different languages that are all open source. We have libraries that are shared by multiple clients on one layer (All CL clients need a BLS library) and across layers (e.g. Prysm & Geth both written in golang, share stdlib). And Ethereum secures hundreds of billions of dollars of value, which creates a direct incentive for attackers.
There are many downstream projects that work on forks of clients. These projects also rely on client teams for security updates and coordination around disclosures. And there is no direct relationship with our users. Since its a permissionless system, we don’t know who needs to be reminded about releases.
Additionally clients are very complex pieces of software that depend on a lot of different libraries and dependencies that can be very critical.
Disclosure process in the past
Previously we often fixed bugs in private and hid the patches in unrelated refactors in order to throw attackers off. This worked quite well, we would smuggle in patches at the last second before we cut the release and suggest users to quickly update.
At some point we realized that we needed to share patches with our downstream dependents. L2s and other L1s that forked the go-ethereum code base were frequently subject to the same bugs as us. We didn’t want to be kingmakers and pull favorites, so pre-sharing the patches wouldn’t work for us. So we decided to adapt the process, so we would pre-announce to a trusted pool of downstream projects that there will be a release soon with some security critical patches. This allows downstream projects to prepare a release and reduce the time for them to release.
Previously we also assumed that a bug in a single client will not affect other clients with different libraries. So we usually fixed clients one by one. An attacker with AI can now feed all previous found bugs and thus will be able to find similar bugs in the other codebases as well, so we need higher coordination efforts.
New processes
We are currently working on new processes for responsible disclosure (which also sparked the idea for this post). There are multiple challenges with the past processes that we would like to address:
| Problem | Proposed Solution |
|---|---|
| There is little knowledge sharing between upstream and forks | Set up dedicated channels between upstream and forks |
| The turnaround time for users to update is too high | Setting up a dedicated mailing list for clients to share updates |
| No connection between clients and dependencies | Set up dedicated channels to maintainers of dependencies |
| Bugs in one client might affect another client | Set up a database of previously found bugs |
| Coordination between multiple client teams is hard | New security-team internal guidelines on coordination |
| Release coordination between dependencies and client teams | New security-team internal guidelines on coordination |
| Release coordination between clients and dependents (L2s, etc.) | New security-team internal guidelines on coordination |
| Enforcement of release embargos | Internal list of dependencies/players that have broken embargo (¹) |
| Coordination in the event of a live emergency | New security-team internal guidelines and playbooks |
| Regression tests not added to clients | Database for regression tests and when they can be added into client databases |
| Visibility into how many % of the network has updated | Promotion of things like https://lab.ethpandaops.io/xatu/contributors |
| Inconsistent CVE policy across clients | Standardized guidelines when and how a CVE should be applied for |
| Timelines for fixes differ greatly between clients | Clearer SLAs for how timelines look from submission via bug bounty to release and disclosure |
(¹) This has only happened twice in the last 6+ years as far as I’m aware. This needs to be paired with clear guidelines on how partners will handle issues.
Closing notes
Over the last few months it became quite clear that traditional security and disclosure processes are outdated in a world with AI. Even small motivated attackers can leverage the power of AI to find and exploit issues in a decentralized system and in open source software quite easily.
AI also speeds up security work and leads to a flood of new bugs and issues, which in turn can create a fatigue in fixing and disclosing bugs. For instance a recent release of Nethermind fixed 17(!) CVEs.
Similar to software development and security work, disclosure processes have to change as well and be much faster than before. We can no longer hide security fixes and have to assume that someone is letting their AI comb through every single commit on our open source repos. Additionally once a vulnerability is found by an LLM, the chance for it to be found by someone else increases significantly, which again shortens the time we have for releasing a fix.