# Post-mortem for a relay vulnerability leading to proposers falling back to local block production (Nov. 10, 2022)
**URL:** https://collective.flashbots.net/t/post-mortem-for-a-relay-vulnerability-leading-to-proposers-falling-back-to-local-block-production-nov-10-2022/727
**Category:** Relays
**Tags:** mev-boost, post-mortem
**Created:** [November 11, 2022, 3:39pm UTC](https://collective.flashbots.net/t/post-mortem-for-a-relay-vulnerability-leading-to-proposers-falling-back-to-local-block-production-nov-10-2022/727 "2022-11-11T15:39:41Z")
**Posts on this page:** 2
**Page:** 1
### Author:  [@metachris](https://collective.flashbots.net/u/metachris)
#### Post date: [November 11, 2022, 3:39pm UTC](https://collective.flashbots.net/t/post-mortem-for-a-relay-vulnerability-leading-to-proposers-falling-back-to-local-block-production-nov-10-2022/727/1 "2022-11-11T15:39:41Z")
## Summary
Block builder submissions with incorrect `timestamp` and `prev_randao` values were sent in bids to proposers, resulting in the beacon node rejecting the bid and falling back to local block production.
By sending specially crafted payloads to exploit this, attackers could prevent any MEV-Boost block from landing on chain, forcing proposers to fallback to local block production. However, there was no risk of proposers missing slots.
The Manifold team responsibly disclosed the issue, with tests in goerli. In close collaboration with EF security, consensus client, and relay operators teams, we identified the root cause of the issue, an incomplete builder submission validation, and deployed a solution to all networks.
A malicious builder sent bids to the Flashbots Mainnet relay, leading to a drop in MEV-Boost blocks between 12:00 and 16:00 UTC on Nov. 10:
Once we noticed and identified the builder as [`0xab847b...91921b`](https://boost-relay.flashbots.net/relay/v1/data/bidtraces/builder_blocks_received?builder_pubkey=0xab847befe59b5effffa12f47acf44cbf8ef875e7c891a4ee9e9c483254cf9a55f5ed688e43ff5bc6cd9276e99091921b), we briefly blacklisted him at 15:30 UTC while implementing a solution, and unblocked him again afterwards. At the time of writing, there are no blacklisted builders on the Flashbots relay.
## Timeline
(all times in UTC)
**Thursday, Nov. 10, 2022**
1. We’ve received a [high-level report](https://gist.github.com/sambacha/43d2a09de63ec21deaa12259fcc82d3b) from Manifold about a possible relay DoS vector at 7:39.
2. Started the investigation.
3. At 12:18, engineers from Manifold provided an example payload that triggers the issue, and we asked them to continuously cause the DoS vector on the Goerli relay.
4. Tried to reproduce the issue by running the payloads through MEV-Boost, which did not show any errors. What was needed to diagnose the behavior further were logs from MEV-Boost and Beacon nodes on Goerli.
5. At 14:21, we asked engineers from the Teku and Prysm whether they run Goerli validators that may have received bids where local fallback was triggered.
6. At 14:34 [Enrico](https://twitter.com/tbenr) and [Stefan](https://github.com/StefanBratanov) from Teku found logs showing that an incorrect timestamp was causing the local fallback.
7. A solution for the timestamp validation was implemented in [https://github.com/flashbots/mev-boost-relay/pull/241](https://github.com/flashbots/mev-boost-relay/pull/241) and deployed at around 15:19.
8. The cross-org engineering team worked on figuring out the best way to check the `prev_randao` field in the relay API. We arrived at [https://github.com/flashbots/mev-boost-relay/pull/242](https://github.com/flashbots/mev-boost-relay/pull/242) using a new (and not yet documented) beacon-node API (`/eth/v1/beacon/states//randao`).
9. The `prev_randao` validation solution was tested, and deployed to mainnet at 22:00.
**Friday, Nov 11, 2022**
1. The BlockSec security team, contracted by Manifold, notified us at 7:44 that there may also be a missing validation of `gas_limit`.
2. After confirming the report, we fixed the `gas_limit` validation logic in the validation nodes [[1]](https://github.com/flashbots/mev-boost-relay/commit/4d223fd06000db697c3279ea4ff51ee27cc8f940) [[2]](https://github.com/flashbots/block-validation-geth/pull/8).
3. The update was deployed to Mainnet at 9:35.
## Impact
Around 350 locally built blocks were proposed instead of mev-boost blocks between 12:00 UTC and 16:00 UTC on Nov. 10. There was no risk of proposers missing slots.
* * *
Big thanks to [@justintraglia](https://twitter.com/JustinTraglia), [@tbenr](https://twitter.com/tbenr), [@stefanbratanov](https://github.com/StefanBratanov), [@terencechain](https://twitter.com/terencechain), [@dickmanben](https://github.com/dickmanben), [@realbigsean](https://github.com/realbigsean) and several others who helped investigating, diagnosing and resolving! ⭐
We thank [Manifold](https://twitter.com/foldfinance) in particular for the responsible disclosure.
---
### Author:  [@sambacha](https://collective.flashbots.net/u/sambacha)
#### Post date: [November 12, 2022, 5:48am UTC](https://collective.flashbots.net/t/post-mortem-for-a-relay-vulnerability-leading-to-proposers-falling-back-to-local-block-production-nov-10-2022/727/2 "2022-11-12T05:48:40Z")
To clarify, this issue was discovered by Manifold Finance (last week. November 4th), an initial assessment was written and provided to Flashbots - [Mev-relay-dos-vector.md · GitHub](https://gist.github.com/sambacha/43d2a09de63ec21deaa12259fcc82d3b)
a bug bounty reward (I think the first for flashbots, I am unsure of this) is being discussed (as was told to me by the flashbots team).
Cheers!