Toward an open research and development process for MEV-Boost

[metachris]

In the three weeks since The Merge, MEV-Boost has become a critical part of the Ethereum block-production infrastructure, with almost 50% validator adoption.

Yet it is not a part of the Ethereum core development, and we still have a long way to go until in-protocol PBS or alternatives to it. For the foreseeable future, MEV-Boost / out-of-protocol PBS will remain as a key coordination mechanism along the transaction supply chain, both in and outside of Ethereum.

The systemic importance of MEV-Boost makes it necessary to be thoughtful about how ongoing research and development of MEV-Boost should be organized. To that end, we wish to open up to the community the discussion about what the process of agreeing on future changes to MEV-Boost should look like.

There are critical design decisions to be made, such as partial block auctions and inclusion lists, that affect market equilibriums and the fundamental principles of the infrastructure. There are also possible improvements related to security, performance and quality-of-life (i.e. logging, operations). Many of these require thorough evaluation of trade-offs from a research as well as a philosophical angle.

Some broader philosophical issues include:

• How do we balance the interests of different stakeholders?
• How do we prioritize functionalities and features?
• How do we coordinate releases?

With these open questions in mind, we would like to invite the key stakeholders, core devs, cat herders, researchers, ecosystem builders, node operators, block builders, searchers and other creators of the Ethereum ecosystem to co-design an open, collaborative, and effective R&D process for MEV-Boost.

Please share your thoughts and ideas to collectively navigate the path forward.

[vbuterin]

MEV-Boost is interesting because it doesn’t quite fit into any of the buckets that we’re used to thinking about:

• It’s not part of “the Ethereum protocol”, so it doesn’t fit well under AllCoreDevs. AllCoreDevs historically is very cautious about expanding its role, and has focused almost exclusively on consensus-layer changes. Hence, MEV-Boost needs some separate structure.
• It is a protocol, not a centralized application. MEV-Boost gets contributions from many organizations. Flashbots has no way to shut down MEV-Boost. It’s logically decentralized, though there are benefits to most of the ecosystem using the same protocol. Hence, it doesn’t fit well to just have Flashbots-the-company run it.
• It’s not like a regular ERC (eg. ERC-20), because it’s not “finished”. There are many ongoing changes being considered to MEV-Boost, including really important ones like inclusion lists to protect against builder censorship. Hence, “sit back and do nothing” isn’t a solution.
• It’s already systemically important, and greatly affects people’s economic interests (unlike eg. ERC-4337). Hence, it can’t progress effectively with small informal chat groups, because it needs the ability to handle conflict.

Another thing to keep in mind is that in the medium term (realistically, 2-8 years? Not sure), MEV-Boost will be replaced by in-protocol PBS, so the structure does not need to last forever, but we can’t assume PBS will come quickly.

What does all this imply? Some kind of parallel version of AllCoreDevs? Definitely not a simple question.

[stonecoldpat]

I think projects like MEV-Boost are more like “Plugins” or “Opt-ins” for node software. Perhaps a bit like MetaMask Snaps.

I don’t think there should be a body that governs whether it should be encouraged or not. I suspect what might be more useful is a Plugin Ethics Committee that lists the benefits and harms of a node plugin. So it can help validators make an informed decision about doing something and potentially how they should configure it.

[souptacular]

I put my thought on Twitter and will also put them here:

Very difficult problem indeed!

Drawing on my experience as one of the longest serving EIP editors (now retired) and ACD coordinator for about 5 years (also retired) + a stint at Flashbots I have the following perspective.

MEV-boost is an early technology & should optimize for a combination of maximal community involvement with an organic, trusted leadership element that pushes things along to avoid decision deadlocks or slow development. This is something Ethereum protocol development used to succeed at, but has not been as good at as we’ve grown. That isn’t the fault of the actors involved (at least not entirely), but growing pains as a community and platform growth is hard to plan for and adjust to.

My main example is in the early days of Ethereum, there were only about 20 core devs, many who cohabitated & coded daily together. The bulk of Ethereum clients and the protocol were built under the direction of people like Vitalik and Gavin Wood and others. Things progressed rapidly because there wasn’t a community of thousands of devs having input. It’s important early in the life of these decentralized, open source initiatives that a leadership group form to steer the ship before things get more complicated and stagnate because of coordination.

Another example is the EIP repository. It is a core part of Ethereum now, but for the first year it started, it floundered. I was asked by the then ED of the Ethereum Foundation to become an EIP editor to help revive it and because the other editors had stopped editing. I re-did EIP-1 in 2016/2017 and just pushed the changes to the repo with no push back because everyone else was busy with other things. Since then, other EIP-1 changes have taken months of debate. The Ethereum Cat Herders set up EIPIP (Ethereum Improvement Proposal Improvement Proposals) meetings a few years back which have been good to help recruit editors and keep up with EIP repo changes.

I say all this to say that naturally elements of a project like Ethereum or MEV-Boost will stagnate due to a growing community with more input and that’s okay. As long as you anticipate those changes, have the proper leadership in place (and replacements if they leave), and promote openness in decision making things should be minimally disruptive.

Specifically for MEV-boost, I would love for there to be a bi-weekly call, probably recorded and with notes like the ACD calls, where interested parties discuss MEV-boost. Although Flashbots is going to be the obvious majority participant in the calls at first, I encourage them to adopt a policy of minimizing their influence by encouraging others to speak up about their opinions and ideas. The geth team does an excellent job of this during the ACD calls. The calls should have a consistent person leading them, an agenda set ahead of time, and a Discord server/room and forum to discuss things async. The Eth R&D Discord and Flashbots Discourse forum work for this. Every quarter there should be a survey of participants to see if the calls are working and what can be adjusted.

MEV-boost is at a layer of Ethereum that is somewhere in between protocol and dapp, which is new, exciting, and unexplored for the most part. Let’s embrace that and adjust our expectations to actively promote and build this service into something best for the ecosystem

[Yazanator]

I do see a few approaches for an open-process, each with their pros and cons:

1. MEV-Boost Improvement Proposal process. I totally resonate with all the points shared by @souptacular that things can stagnate there unless properly energized. Otherwise it’ll just be the Flashbots team driving the specification and participation process, and without community involvement, it would also slow down the process for Flashbots to continue updating MEV-Boost.
2. Governance process for MEV-Boost. This can take the shape of a DAO with proposals submitted via Snapshot. The way to steward governance proposals would be the maintain a repo like the EIP for governance of MEV-boost. The drawbacks here is that every governance proposal will have it’s own unique ID on-chain when submitted, and it would probably be different than the ID it was given as a draft stage. When anyone would be able to submit a governance proposal and not follow the process on say Github, you’ll have a divergence between draft proposal IDs and on-chain proposal IDs (useful for when people are voting on a proposal)
3. ADR - Architectural Design Records: Basically have a system were Flashbots creates ADRs for any design decision it makes on MEV-Boost that’s large. More info on ADRs can be here. Folks can comment on different ADRs and their pull requests.

No matter the decision made, I also agree with Hudson that having bi-weekly calls with notes taken and published in a Github README somewhere would be so valuable.

[metachris]

Made a proposal (and breakout topic) about MEV-Boost core development tenets, which could help guide future decision making and R&D process: MEV-Boost Development Philosophy

[lightclient]

This is a surprising post to me, because I felt we already had an open research and development process for mev-boost. Many researchers, core developers, and security testers have already contributed to mev-boost over the last year. I feel things are working as intended and there is not a need for much more process.

A source of confusion may be the naming collision of mev-boost the protocol and mev-boost the software. @vbuterin’s sentiment perfectly describes mev-boost the protocol.

However, mev-boost the software is very much not a protocol and does not benefit from same level of openness as protocol specifications do. Ultimately, the software needs a maintainer. Beyond standard OSS courtesies, the maintainer(s) should not be concerned with an “open development” process. Ethereum clients don’t let the community dictate their architectural decisions. That is up to the engineers tasked with building and maintaining the software (provided they implement the protocol correctly).

Because this confers a lot of power to client developers, it is important that these individuals respect the protocol and are aligned with the ultimate philosophy of democratizing MEV and eliminating censorship on Ethereum. I think developing this type of software under flashbots can be difficult, because there are conflicts of interest. They are the largest block builder on Ethereum (profit), stewards of the mev-boost protocol (public good), and maintainers of the only mev-boost implementation. Although I feel the process is currently open enough to rebuke any attempts of capture, it is something to keep in mind.

In closing, a few concrete recommendations:

• Separate the research and non-commercial development work from product offerings. These things should be housed in a non-profit organization. It’s not possible to be considered credibly neutral by the community as a for-profit company. If flashbots is unwilling to do this, we as a community should erect such an institution.
• Rename either mev-boost the software or mev-boost the protocol to avoid confusion.
• Retain 1-3 full time engineers in the non-profit org to work on mev-boost the software. There are a lot of new features we can add to improve mev-boost and we need full time humans building and testing those things.
• Under the non-profit, create a repository for the protocol specification of mev-boost and relays. This can be the main hub for communication and coordination.
• If necessary, begin a monthly call to discuss mev-boost the protocol openly with the community. I say if necessary because I think the majority of the coordination can be done asynchronously over the repo and occasionally on the CL call.
• Don’t be too prescriptive on the governance process. Please don’t make MBIPs. Please don’t make a DAO. For inspiration, see how the beacon chain has dealt with coordination up until the merge.

[metachris]

Thanks, Matt, for sharing your thoughts! A lot of good points in your post.

@lightclient

A source of confusion may be the naming collision of mev-boost the protocol and mev-boost the software.

I agree that there’s some confusion between mev-boost the software and the protocol. The term mev-boost seems widely used to describe either or both. When writing the initial post, I thought mostly about mev-boost-the-software.

@lightclient

• Rename either mev-boost the software or mev-boost the protocol to avoid confusion.

Isn’t mev-boost the protocol already the builder specs?

Curious what else you’d see as part of the protocol, either now or possibly in the future. I guess it could describe an additional set of interactions between mev-boost-the-software and the relays/builders, but it’s unclear what of that could be done without CL interaction. Would love to hear what specifically you were thinking about.

[lightclient]

@metachris

Isn’t mev-boost the protocol already the builder specs?

mev-boost is an overlay on top of the builder api specs. The builder api specs should aim at being as simple and general as possible. In contrast, mev-boost can be extremely featureful.

For example, there have been proposals for introducing fraud proofs into mev-boost so that validators can recover from relays who send them an invalid block. The builder api does not need to be aware of this construction. That feature sits uniquely at the mev-boost layer.

There are even constructions for inclusion lists which do not modify the builder api. Doing so would be a bit convoluted. However, it could be made simpler by adding an additional field to ValidatorRegistrationV1 such as “extPubkey”. This would allow the validator to declare an external key which can be used to authorize messages in layered protocols like mev-boost.

[timbeiko]

Posting this much later than I was hoping, but at least now I benefit from lightclient’s insights FWIW, my perspective is much more around “mev-boost the protocol”, to use lightclient’s phrasing than the software itself.

@lightclient

• Don’t be too prescriptive on the governance process. Please don’t make MBIPs. Please don’t make a DAO. For inspiration, see how the beacon chain has dealt with coordination up until the merge.

Big +1 here. “If it ain’t broke, don’t fix it”. People tend to look at AllCoreDevs as a “default” when thinking about coordination and while the calls work for Ethereum L1’s purposes, it’s worth highlighting some issues with them:

• Sync calls optimize for people who live in a specific timezone
• Live calls to debate changes optimize for fluent/eloquent English speakers who “think on their feet”
  • Perhaps even more basic: calls require you to show up and speak, which isn’t a big problem in the Ethereum L1 development community, but in a space like MEV where anonymity is a bigger concern, it might be a blocker
• Calls with a lot of people on them have a high time cost - they probably only make sense in cases where async stuff has piled up so much that you need to “unblock” it with live discussion

That said, one thing that does work well with Ethereum L1 governance is that we have a strong “separation of powers” within the community. AllCoreDevs isn’t perfect, but it doesn’t get a full say on what gets deployed on-chain. Client developers write the software, but don’t have any “hard” power to get the community to run it. This is a very valuable property, and I’d encourage you all to think of ways you can get this property as part of any governance process.

[oisin]

(Also posting this much later than I was hoping to, great minds @timbeiko )

Hi all,

For those of you that haven’t met us, we (myself and Collin Myers) are the Founders of Obol. We have been researching and working on different pieces of PoS Ethereum since 2018. Today, we are focused on scaling and protecting the consensus layer with DVT.

Obol is building a middleware implementation of Distributed Validator Technology, the primitive that enables multi-operator Ethereum validators. For the past 18 months, we have been working with the client teams and flashbots to support all validator clients and mev-boost with our middleware. We are also a member of the Flashbots Eth2 working group.

We are surfacing in this thread to both weigh in with thoughts, but more importantly to make an offer of support.

The tl;dr is; we are a team of 15, with venture capital, and the means to support and continue the development and progressive decentralization of mev-boost for the foreseeable future, and this post is us throwing our proverbial hat in the ring to take on an accountable role in the stewardship and further development of the mev-boost client.

The Next Frontier
Ethereum is entering its next chapter of growth, which comes with intense growth efforts intertwined with decentralization efforts.

Over the past ~24 months the Ethereum infrastructure industry has been preparing for a market of middleware protocols. Through experimentation and collective design decisions we have enabled middlewares to be additive to the core client options and not competitive. However, this has resulted in a separate coordination layer (or lack thereof) detached from the work of the core client teams/ACD. This is difficult given the overlap of responsibilities.

These additive middlewares enable advanced features for core network participants (validators, builders, relayers etc.) while providing advanced protection for users of the Ethereum mainnet. A mutually beneficial relationship between core software and ancillary software is what is needed to pave our next chapter of scaling.

In today’s context we have two things happening:

1. We are collectively searching for a decentralization framework for widely used middleware protocols on PoS Ethereum (builder-api)
2. We are attempting to decentralize the development of our first majority run middleware client (mev-boost).

It is critical as a community that we discuss the on-going design, management, coordination, and incentive models of this new layer in a broad context as well as in the context of Flashbots.

Middlewares Today

Today the middleware market can be divided up quite simply:

1. Execution Middlewares

• mev-boost
  • Exists to offer you extra rewards
  • On Mainnet
  • GoLang

2. Consensus Middlewares

• Obol

  • Exists to protect the largest network reward (consensus)
  • On Testnet
  • GoLang

• Eigenlayer

  • Exists to allow other middlewares to add crypto-economic security without bootstrapping that security from 0.
  • In development
  • I’m not sure what language it’s being written in.

Today these middlewares are operating mostly in isolation, and have only begun to operate together on testnets (please see below).

Middlewares Tomorrow

Today there is only one middleware that matters (mev-boost), however tomorrow we envision a collection of different middlewares that run in the majority of network stake in one way or another. In addition to an increase in the total number of middlewares available to network participants, we will also see a number of cross-middleware combinations being run on mainnet.

Eventually we will reach a state of standardization such that users can run multiple middlewares at the same time, in a secure manner. From our experience, in this middleware value chain, core protocols such as mev-boost and Obol will be required to build closely as the overlap in user groups is substantial (hence this post). On the Obol side we have already begun to test distributed validators that can propose blinded beacon blocks. We expect to see a new field of research develop focused on MEV and its impact on multi-operator validators.

Today is a great time to get in front of this on-going problem and establish some sort of framework before the middleware market explodes with growth.

To touch on some of the suggestions highlighted in this thread, here are my thoughts:

• I would also discourage moving towards an EIP style process for feature addition, I think that can make sense when there are multiple builder-api clients and specs are more ossified, but in the near term, I think encouraging an involved group of core devs fleshing out the client collaboratively is a more fluid and flexible paradigm for the near-term. This ties directly to my following point on where a client ends and a spec begins.

• As for the debate between building a client or building a spec, this is somewhere I probably lack perspective, so I will defer to the likes of @lightclient. I will say though at Obol we are currently walking a similar road, of wondering when is too soon to formalize the protocol and direct our focus primarily on a canonical spec, and when is too late. My gut feeling/bias is that you shouldn’t move to a spec until the development of a reference implementation starts to stabilize and plateau. I think this bias is based on my belief that DVT performance will continue to improve over the near term, so I have been reticent to formalize charon’s operation into a spec until we approach some of the theoretical lower-bounds of communication rounds to form distributed consensus, rather than ossifying a relatively naïve implementation.

• I’m strongly aligned with the general consensus that things don’t/shouldn’t progress up the systemic importance levels until ACD has to step in to steward it and protect the base protocol. That is neither decentralized nor scalable. I believe ACD and the EL and CL teams should continue to strengthen and broaden their APIs, and that a plethora of software will flourish on top of these APIs. I believe this is consistent with the “middleware renaissance” described by the likes of Obol, Flashbots, and Eigenlayer. If Ethereum development is to be alive and thriving 20 years from now, in my eyes the development at that point will be on top of a foundation of standardized APIs, implemented by a number of client teams.

Finally to conclude with some concrete suggestions:

• We as Obol would be open to stepping into a stewardship role for the mev-boost repo, sharing accountability for reviewing PRs and standing over what gets merged to main.
• We as Obol are keen to develop and improve mev-boost, particularly with respect to enabling distributed validators to extract MEV. Our first potential feature addition would be to make it easier for distributed validators to register with relayers effectively. Currently, due to the inclusion of a timestamp in the signature scheme, it is quite difficult to make many independent validator clients sign the exact same registration hash/digest. We hope to add a standard API endpoint to the beacon-api, where validators can ask for something to sign to register with a relay. This, if adopted, will allow for multi-client distributed validators to smoothly register with MEV relays. Without it, the workarounds are rather involved and ugly and manual and different for each client team.
• We as Obol are strongly incentivised to solve/mitigate MEV in a game-theoretic stable manner. When designing DVT, before mev-boost, I was deeply concerned that MEV present in block construction was an externality that would have compromised the stable consensus of a DV. As a greedy operator, it would be worth your while declining to approve a block proposed by someone else, such that you could propose a block where you have swept the MEV to your own side address. The pivot to blinded beacon blocks, although introducing different censoring, trusted relayer compromises, allows for a DV to come to consensus on a block hash instead of a full block, which does not have the same capacity for abuse. We believe with the right builder constraints, like crLists, this model of ‘dumb proposers’ is the right path forward for Ethereum, validating in a maximally profitable manner needs to be doable by everyone.

[metachris]

@oisin

• We as Obol would be open to stepping into a stewardship role for the mev-boost repo, sharing accountability for reviewing PRs and standing over what gets merged to main.

Thanks for stepping up with your offer for support!

A good next step could be for you to start participating in the discussions on various issues in the mev-boost repository, and please also post a few Github handles to loop in to PR reviews.

[oisin]

Perfect, will reach out to you this week to get some handles sent over and some chats set up.