On December 9, 2025, BuilderNet experienced a service degradation lasting approximately 55 minutes (UTC 14:20–15:15). During this period, BuilderNet landed only three blocks, all without blobs. Full service was restored at UTC 15:15.
Root Cause
The incident was triggered by the BPO1 fork activation at UTC 14:21. Our rbuilder dependency on reth was missing a commit required for correct blob gas calculation under the new parameters. This caused rbuilder to produce blocks with invalid excess blob gas values, which were rejected by the network.
Timeline (UTC)
- 14:21 - BPO01 fork activated on mainnet.
- 14:23 - Monitoring alerts triggered; rbuilder producing invalid blocks. Incident bridge started, including engineers from Flashbots, Beaver and Nethermind.
- 14:42 - Root cause identified, implementing fix.
- 14:51 - Testing new version in staging environment.
- 15:10 - Starting production rollout.
- 15:15 - Full service restored; BuilderNet blocks with blobs land normally again.
Impact
- ~55 minutes of degraded block production.
- 3 blocks landed during the incident window, all with 0 blobs.
- 2 missed slots caused by invalid optimistic block submissions to relays. We are refunding the lost value to the proposers.
Resolution
We upgraded the reth dependency in rbuilder to include the blob parameter fix and released rbuilder v1.2.30. The update was rolled out across all nodes, restoring normal operation.
Lessons Learned
What went well:
- Rapid incident detection via automated monitoring
- Quick identification of root cause and fix
- Strong team coordination and communication throughout
What we’re improving:
- Implementing automated dependency tracking for critical upstream changes (reth).
- Adding protocol activation dates to our operational calendar, with a single DRI.
- Reducing CI build times to enable faster emergency deployments.
We take BuilderNet reliability seriously and are committed to continuous improvement. If you have questions, reach out to us on the Forum or Telegram.