Below you will find is rough transcript of the call. Due to the character limit the transcript is broken into two posts. Thank you to @ralexstokes for organizing and all of the participants. Please call out any omissions or mistakes.
Pt. 1
-
Stokes: Okay sounds good. I think that’s probably it for Sepolia and we’ll just keep moving right into Goerli. So this Hardfork on the 14th in just a few days here. Any relay operators on the call? We had said we could use this call to do any coordination required for Goerli, any relays here who want to chime in? Do you feel ready? Do you know what to do? It sounds like Chris has notes on what releases to use and how to do the upgrade, If your using the flashbots relay?
-
Austonst (Aestus): - It sounds like the mev-boost relay code is good to go. I have been following that pretty closely. I am a little uncertain about the block validation geth status. I saw there is a shapella fork or branch. So hearing from Chris its not necessarily necessary to have Geth updated for that, just wondering, a little more details there?
-
Chris Hager: On test-nets we are using builder code base as validation node and a builder at the same time. So you can enable the validation logic in the builder project itself. I will double check if the validation gets repository is also updated. But what is for sure working is the builder repository, in particular the Capella tree. So you can use the Capella tree of builder to run it as a block validation gets.
-
Austonst: Will this be the main source for validation going ahead? Should I consider block validation geth to be kind of deprecated at this point?
-
Chris Hager: Do we want to maintain two repositories that are largely overlapping but not quite? Or if we are fine with having the additional builder overhead in the code also running as a part of the validation geth which would simplify the maintenance and upgrading, but It’s not decided yet. I will certainly post an update going forward. But for Goerli I would recommend running the builder project.
-
Austonst: Got it thank you.
-
Stokes: Chris you said you would put together a document with what exactly relays need to do?
-
Chris Hager: I will try to write this up a bit more concisely for relay operators specifically. Keep in mind that you also need the FB Prysm fork because we implement to getWithdrawals endpoint that you kind of want the relay to validate that withdrawals are correct. This is a temporary solution until the SSE events are implemented in hopefully a few weeks. Until then our relay needs this Prysm fork to provide the withdrawals.
-
Stokes: To be clear, Will anyone need to supply a Shanghai override? Or if they used the right branches and releases everything will be good to go?
-
Chris Hager: On Prysm it’s good to go.
-
Stokes: I thought you mentioned something with the builder, but that might not be an issue.
-
Chris Hager: Yeah, I will post more updates. We are currently not yet on the latest Geth release that wouldn’t require the Shanghai override.
-
Stokes: I see
-
Chris Hager: I will certainly post an update tomorrow about the details.
-
Stokes: Just to elaborate, Basically the Goerli configuration is hard coded in later geth releases, the builder code it sounds like is not using that yet. Sounds like code is ready to go, again if you are relay operator and hopefully you feel prepared, if you don’t please reach out to me or FB or someone you trust. It should be pretty exciting. One question I could ask is How will we think about testing this? We will just Look at the usual mev-boost metrics and see if it works? I don’t know if anyone has thoughts on that? I guess relays will know quickly if everything stops working at the fork.
-
Chris Hager: Yeah the relays internally will throw errors if there is something not working or builder submissions not accepted. I think Paritosh runs a large number of goerli proposers. W/e relays are configured there we should be quick to know any issues that arise through Paritosh.
-
Stokes: Right that’s a good point. I don’t think Pari is on this call is he? No that’s okay, but I can follow up with him. Anything else for Goerli is everyone else ready it sounds like there might be a few more details on the exact software releases and how to run them that Chris will work on, but otherwise sounds like we are ready to go.
-
Terence: I have a question. What is the best communication channel for relayers these days. If I have an alert system that’s going on and I want to look in a relay channel to see if they are recording anything, where is the best place for that, where are relays hanging out?
-
Stokes: That’s a good question does somewhere here want to chime in? My understanding is that most relays have a web interface from their website. Perhaps there is a status page link there. I think It’s up to relays how they communicate. Separately if you see something and need to push to everyone, I suggest using the ETH R&D discord channel or maybe the flashbots channel. But that being said do any relays on the call have anything more specific here?
-
Justin Drake: There is a relay chanel on the FB discord that I constantly monitor.
-
Max Birge: Yes I monitor that as well
-
Chris Hager: Block construction channel on Ethereum R&D channel (discord) is a place all the relays are aware of.
-
Stokes: Okay FB relay channel. Do we know exactly the name of that one?
-
Justin Drake: Relays, under general.
-
Stokes: Great I can grab a link to that in the notes after and the ETH R&D channel Block construction. One thing to consider, we con consider a mailing list of sorts for critical issues. I know this is something a lot of the client teams do both at both the EL and CL layer. This could be one quick way to get very urgent things in front of everyone as we need to.
-
Anything else on Shapella? Alex asked in the chat about this e-mail list. We could just make one and people could subscribe as they want to. I’ll let you come up with the name.
-
Stokes: I think the next point we will move to is the optimistic relay. I don’t know if Justin or Mike if you want to give an update on where things are at there. There has been a lot of progress since last call. Maybe something is on the way to being rolled out
-
Mike: Yeah I can give a quick update here. Just so everyone is on the same page, Let me link to the PR here.
-
This is the context. For anyone who was not on the last call this is a change we are proposing for relays where instead of validating the block submissions as they come in before marking those bids as eligible to win the auction, we defer that block validation till later. This should hopefully improve block submission latency. We have in terms of status we are getting close to launching in prob. We have been running in Goerli quite smoothly, just working out last details of getting code rebased on capella changes. We Have been in contact with a few builders who are interested in piloting for us and have an initial interest there.
-
One other link I want to plug. Justin Linked the builder on-boarding docs. Which can show what the process is on the builder side as far as posting collateral and signing up for this roll of optimistic relaying. And the last thing I wanted to share is the broader roadmap are have been working on.
-
This hopefully should help contextualize what we think of Optimistic relaying and how it fits into the broader enshrined PBS landscape. We are thinking hopefully next week we will have some optimistic relaying begin. That may shift depending on if we can make sure everything is working on the software side. That is the main update, do you have anything to add Justin?
-
Justin Drake: Yeah I guess one thing we have been doing, we have been doing is dramatically reduce the amount of simulation errors we are seeing on the relay. There are two things we have done.
-
We have found various edge cases in the relay infra that lead to simulation errors that shouldn’t happen. For example there was an artificial constraint on the block size. Valid blocks considered to be invalid because of their size. That should improve the liveness of Ethereum. It means these bigger blocks will make it on chain more easily going forward. There is a long tail of such edge cases we found, maybe 10 or so that were in the process of fixing or have already fixed. We are pushing some PRs upstream to FB repo as well to reduce these edge cases.
-
Another set of edge cases is on the builders side. It turns out that various builders have bugs in the sense that when they create their blocks there are just some issues. We have very meticulously gone through every error on the builder side and told them about these errors. I’m in contact with most builders. If you are a builder that I have not approached yet please contact me because its more likely than not you are producing some invalid blocks This should help the whole ecosystem because it means there is less spam on the relays and things run smoother. Generally speaking the signal to noise is higher.
-
All of this work is Part of the legwork to optimistic relay up and running, we don’t want any simulation errors.
-
One thing I do want to highlight again; If there is a builder bug, more likely than not it will be caught extremely early and lead to an early demotion without a bad block ever winning the auction. I am still not expecting bad bids to lead to on-chain incidents with missed slots. If that were to happen as described in the doc I linked were going to write a post mortem which will be public. Basically we are trying to be as transparent as possible about these on chain events, and they should happen exceedingly rarely.
-
Stokes: From experiments on Goerli it sounds like you haven’t run into too many issues yet?
-
Mike: Right so basically we have been addressing simulation errors on the relay end or the builder end. Now we have several days worth of blocks, thousands of submissions without simulation errors. We are confident that the fixes are working as intended. Up-streaming which ones seem relevant, making tweaks on the relay on our end and, communicating with builders if it seems to be on the builder end.
-
Stokes: That sounds good, at a high level if one of you can call out these PRs to the relay code at least this would be really good so we can get more eyes on it. I understand there can be edge cases but we should just make sure there are reasons to remove these edge cases and they weren’t there in the first place.
-
Mike: I’ll follow up with a link to PRs.
-
Stokes: Great. I think another good question is up-streaming the whole optimistic relay into the mainstream FB relay code. Last time there was some discussion around is this a Good idea, not good idea? I don’t know if anyone here on the call wants to chime in further on that? I believe this is still the plan of the ultrasound team is to upstream this code.
-
Justin Drake: So one data point I want to put forward. If you go on relay.network/builders its a knew dashboard that’s been built by the rated.network guys. Basically what they do is for every builder they show the relay distribution of winning blocks. It’s very interesting, One data point that stands out to me is the fourth row, bloXroute. The builder is connected to many diff relays. As you can see the blue portion which is BloXroute is 96.6%. Most of their blocks are being won by their own relay.
-
Now I don’t know 100% for sure, but my suspicion is there is some vertical integration going on. The relay is not re-simulating the blocks that come from the builder. They are running in optimistic mode essentially and that’s The power of Optimistic relaying. It dramatically changes the equation. What I think will happen is that once we set up optimistic relaying for some of these builders, The Ultrasound relay will be winning most of the auctions. It will almost be a forcing function for the whole ecosystem to move onto optimistic relaying. So that might be a good reason eventually at least to merge in the optimistic relaying logic in the FB repo.
-
Phil: So at least from my perspective we are definitely open to having the conversation about up-streaming. I think it’s an early stage conversation. I think the work and experimentation going on is super interesting. Our worry or concerns would be from market integrity and market structure side. We view our role in the mev-boost ecosystem and MEV in general as standard setting in some way. So we need to think through what are all the possible affects on market structure. The key vector I would be thinking about that myself is centralization. What are the additional centralization pressures of capital requirements and also possibly opening new games by making certain changes to the relay. We do need to do that analysis and have those conversations. We just started thinking about this stuff, putting out content and engaging with the people who are working on it so I see it as a starting point. We are definitely not against up-streaming it especially if some of these concerns can be mitigated or resolved.
-
A few practical concerns I have that are low hanging fruit. One is I would still very strongly advocate for a path to win a full block without needing collateral. I think Preserving the one off permissionless nature of block construction and allowing for re-simulation is an important market property to allow for new entrants, people who have edges that are not latency related to enter the market. That’s a low hanging fruit and there are some deeper questions Collateral, latency, up-streaming to L1 and I think those are still early research questions. I do appreciate everyone who has been providing content. I am reading it and thinking about this now.
-
My long term dream this is somewhat relevant for mev-boost is to move towards a relay-less system. Obviously there will always be relays from a network perspective, but I don’t think we should have trusted party that exert any centralization pressure in a network role called quote-un-quote relay and that includes flashbots. Eventually the goal would be to deprecate that functionality.
-
The other angle other than decentralization pressure that I would want to reason about; Does this require any changes that would obviate our ability to later upstream this role or w/e else we want to do with it? Does it remove any flexibility for mev-boost development. Those are my angles and I wish I had a better answer, but I’m thinking about it very hard and like I’m sure the next month will be interesting in exploring these things.
-
Justin Drake: So a couple points on what you brought up. I think no1 is the low hanging fruit that you highlighted has already been picked. So by default builders are in non-optimistic mode meaning that they don’t need any collateral and that will always be the case. You can think of the optimistic mode as being an ultra high priority Que. SO we already have the low priority, high priority, and there is this optimistic priority which is a new priority almost. It’s opt in and right now we are capping collateral requirements at 1 eth, to keep the collateral requirements low. 1 eth is sufficient to capture the vast majority of blocks from a quantitative standpoint.
-
Phil: I think that’s great, and the low hanging fruit approach is great. We do also need to take into account that mev is super spiky and many black swan blocks for which the validator really does want to check with high assurance because there is significant incentive to cheat them. I do want to preserve that just for network robustness, but it sounds like that’s already in there. I do still worry about the centralizing affect of having this more privileged set that’s my vague fear but Id on’t think its a total deal breaker, just something to reason through.
-
Justin Drake: On the point of the spikiness, whenever there is a block with more than 1 eth of value in the bid it gets simulated the current way its not optimistic relayed. The thing we are trying to optimize for is more blocks being relayed by non-censoring relays. One of the reasons why changes in market structure, Optimistic relaying is pretty bad for censoring relays. The reason is that In order to know whether or not you want to censor a block you need to simulate it. You can do some string matching and look at sender addresses and recipient addresses without simulation but that won’t give you the deep call stack inspection which is currently happening on censoring relays. Optimistic version V2 we will optimistically relay bids without even downloading the data. It turns out it takes another 100 ms just to download the block, just the bits and bytes. With Optimistic relay V2 which is already implemented you are saving 100 ms and that really completely removes the censorship opportunity even for the easy string matching on the sender and recipient addresses.
-
Another big market structure we want to push for and it’s the one you have highlighted is this dream of removing the relays all together. From an EF research perspective, the way that we achieve this dream is through Enshrined PBS. Actually we made a hire recently, we hired Mike to join the Ethereum foundation. His title is Enshrined PBS coordinator. His sole mission is in a way to remove the need for relays. We see optimistic relaying as steps towards enshrined PBS. What is ePBS? It’s basically just the removal of the relays. Optimistic relaying is the partial removal of relays. The removal of the relay from simulation standpoint in V1, the removal of the relay in downloading the data and verifying the availability of the payload is V2, and then we have this V3 version of op relaying where the critical path from a networking perspective doesn’t even involve the relay. There can be a direct TCP connection from the builder to the proposer and the relay only acts as a DA oracle as a mediator and custodian of the collateral in no way is it in the critical path. That’s essentially what enshrined PBS is except The custodian is the Beacon chain itself and the DA oracle is a crustless committee instead of being a trusted relay that acts as the oracle. We kind of see optimistic relaying as being an accelerationism for enshrined PBS and the removal of relays.
-
Phil: Yeah, I fully buy into the narrative I just worry more about practical details. For example, I agree its great to make changes that make censorship more difficult, I think if we were actually to entrench ourselves and reaches sum sub-optimal Moloch like Bitcoin has or ETH L1 did for a while and we are stuck with mev-boost status quo and ePBS takes a long time, we still do need to consider the current market structure. Im not sure that if we freeze that game theory this change actually does necessarily penalize… is good for market structure viz a viz censorship. I think maybe it is maybe it’s not, I am less strongly convinced than you are. For example let’s say censoring actors say they cannot comply with anything other than censorship, this is not FB opinion just gaming out game theory with you, you may in this change require more of a permission less or side channel validation of transactions, different forms of liability on the builder or etc. Regulators may look at diff are of the stack.
-
From a technical perspective it does make censorship harder in some ways. From my analysis I am very hesitant to project that out into we end up in economic game with less actual censorship. The affect seems less clear to me than it would to you. I do think we will analyze this. I will commit to the next month and work on this and come up with some opinion. Do I feel there are any clear centralization risks for the market structure or anything like that. If there aren’t and I don’t see any, I would personally support up-streaming the change. My North Star there would be let make sure we don’t remove any optionality from enshrined PBS. We all agree ePBS is the most important thing we are iterating towards as a mev-boost community and protecting that is the most important goal at least for me. I will support this if it does play out in the game theory I see as this path to ePBS and does make censorship m ore difficult. I think those are all great things.
-
Justin Drake: On the topic of censorship the ultrasound team has recently deployed new dashboards that provide insights. If you go to relay.ultrasound.money and scroll down a bit there is relay censorship dashboard, Lido operator censorship dashboard (we started with Lido because they had a grant but we will expand to more operators), and builder censorship. For example On the builder side of things we tell you how many pubkeys each builder has and also if any pub keys are censoring and we tell you the dominance and things like that. We also look at the impact of censorship on the network. In the last seven days 326 txs suffered a delay from censorship and the average delay was 48.8 seconds. In the grand scheme of things the impact of censorship is very low nowadays. But least now we have the data.
-
Phil: How do you calculate delay?
-
Justin Drake: we have mempool data which looks at when a tx is first seen in mempool in 3 diff geographies; Asia, US and EU. Then we look at the timestamp of when the tx was confirmed and take the delta between these timestamps.
-
Phil: I would say that’s not an entirely accurate statistic and the delay is probably much better than that. What you really need to look at is when otherwise it would have been mined. I would suggest refining the statistic when is there a different bid for that relay slot that would have had that tx that got beaten by a censored block. What is the delta from that block being mined to the block where the censored tx eventually gets mined.
-
Justin Drake: That is exactly right, so what you are saying is that the delay should be computed in blocks, basically where the tx could have been included but for some reason was not included.
-
Phil: I also like doing it in seconds but just at the block level.
-
Justin Drake: Yeah for sure we actually prioritized in one of the widgets if you search for tx delay we measure delay in blocks and seconds. If you scroll towards the bottom of the website you can see the delay in blocks if you unselect unsanctioned and only have the sanctioned ones you will see the delay txs there, and for each one you have delay in blocks.
-
Phil: I think that’s super interesting thanks for building that.
-
Justin Drake: Now at least from a data standpoint we have a database which has all of the metadata we think could be relevant. For each block who was the proposer, who was the builder, which relays relayed that block and we also know if it contains OFAC or other censored txs. We also have all the mempool data. If you want to do some data analysis we have done heavy lifting of building database and we can give you access to the database to run your own queries.
-
Jolene D: I remember on the last call we had some relay operators who expressed concerns about the additional responsibilities that up-streaming this change would incur for them. Things like holding collateral and managing that, potential legal implications because of that. I was just wondering if any of those people are here today and would like to talk about what this change for them. That seems like something that should be taken into consideration weather up-streaming this is going to affect who is able to run relays at the moment
-
Max Birge (Aestus) - we have been thinking about OR stuff very carefully, we have had a couple chats with Justin about it. I think Some of this is inevitable because latency wants to be optimized. In many ways we are supportive because it will reduce relays operating cost much of which is the simulation stuff. On the other hand there are some dynamics at play here which Phil pointed out around centralization. Some of them are pretty difficult to reason about at the moment. Certainly on the relay side we have effectively become a financial entity which is responsible for some type of collateral and are there KYC/AML requirements? We are not clear about that, we are not lawyers. So on the one hand you can see It does reduce censorship but on the other hand it puts relay in a bind of a position because it’s not really clear who the builders are. So what we think likely happen here is you will form quite tight coupling relationships between relays and builders to navigate these additional responsibilities and that does have a tendency to create barriers to entry on both sides of this. It may reduce some of the network resilience we’ve seen where builders are shooting out blocks to multiple relays at once. We are still thinking about this, have not gotten to the end of this. We will publish something soon.
-
Phil: I think those are great points would love to point out Potential collaborations we’d love on our side. One of them is the data you pointed out Justin, can we help analyze refine crate more dashboards, collaborate somehow on that? We are doing many things on the data side, so happy to see if that makes sense. Also, I think Max’s points are amazing on market structure and different requirements for the roles. I was speaking more from a technologist network POV but I think those things need to be carefully thought through and would be happy to have more conversations about that too.
-
Justin Drake: Full disclosure we have been talking to a lot of builders about this. On the builder side of things there is a lot of excitement. There was one exception, one of the smaller builders in a more conservative jurisdiction told us from their perspective it may may create more accounting headaches or there are some legal questions precisely around sending collateral to us.
-
They suggested in order to send money to us there would need to be some sort of legal contract. The ultra sound relay operators is not incorporated, we are just a bunch of random people on the internet so in terms of setting up contracts its not straightforward and it’s not necessarily something that we want to do. Just wanted to point that out that the concern on the relay side is a concern for at least one of the builders
-
Phil: We wouldn’t want to make it harder for a bunch of random dudes on the internet to start a new relay which is what you all have done and I think its super valuable and I want to protect that on both relay and builder side.
-
Austonst: Along these line, in the current Optimistic relay builder on boarding guide it seems like it’s a bit of a manual process right now which involves the builder posting 1 eth of collateral to relay.ultrasound.eth. I haven’t checked if that’s a regular address or if there is some contract there. What are the thoughts on the long-term future of this? If enshrined PBS is long ways out are there some kind of solutions that involves some more complicated system like a sort of multi-sig that’s accessible to multiple relays or a contract of some sort? Can we off load the collateral responsibility to another group of entities? Could there be reduction in legal concerns or make things simpler for technically on-boarding more relays into the process?
-
Justin Drake: right now relay.ultrasound.eth doesn’t point to anything. It will point to simple EOA, not a contract. In terms of fancy things, yes you can have a multi-sig. You can also have simply a standard address and no multisig and the custodian is some third party custodian. If someone wants to step up, someone with reputation like etherscan or rocket pool, I have no idea just trusted entities that want to step up as relay custodians I think that could be a new role created which could help reduce some of the concerns for relay operators who don’t want to touch financial side at all.
-
On that note, we are also happy to be custodian for other relays. We’re happy to hold custody of funds for your relay. I think what I also told you is that we are happy to be a guarantor for most of the builders. What do I mean by guarantor? I mean that we would actually provide collateral for the builder using our own funds. We wouldn’t be receiving funds externally we would just be putting up our own collateral. We trust most of the builders. We believe that reputation of the builder is worth orders of magnitude more than one eth. If there is an event , a builder bug, the builder would be more than happy to directly refund the proposer as opposed to buying their reputation and running away with this one bad block. We are happy to be a guarantor which means that if you trust us then you don’t need to receive collateral from the builders. If and when a bad block happens you don’t need to touch any money because the builder can directly refund the proposer fee recipient. It’s a direct transfer from the builder to the proposer and the relay does not need to be a financial intermediary in that transaction.
-
Stokes: I think it’s worth calling out that there are smart contract designs, different things we can do on chain that move a lot of this responsibilities away from centralized parties like the ultrasound team. That being said I think it comes down to jurisdiction as to what it actually means to custody funds here or there and anyway its definitely an interesting question.
-
Alex Obadia: A question for Justin and Mike, I’m looking at the Towards enshrined PBS - an optimistic roadmap from Mike, and I’m trying to ascertain at which point would you not have this centralized escrow anymore? It sounds like it exists in multiple steps right?
-
Mike: We would need escrow in all three V1, V2 and V3 as described. Even in V3 where the optimistic relay is behaving as an oracle for signed header appearing on-time and the block body appearing on time. If the builder commits a header that wind but doesn’t publish the block on time, then the proposer is still missing that slot and needs to be refunded. So in that case where the messages are happening over the p2p layer and the relay is just observing the p2p layer, the relay still needs to hold that collateral in case the block is not published on time. All three of the iterations require that collateral still being there.
-
Alex Obadia: Maybe a stupid question but a follow-up is, Once you go to ePBS and have a committee do you expect committee to be holding collateral?
-
Mike: So in ePBS I guess this behaves slightly differently in that there will need to be some form of unconditional payment. If the builder submits a bid that gets signed and not produced then that’s only going to damage the builder. So whatever mechanism we end up doing in ePBS, the proposer won’t necessarily be refunded but it will be and unconditional payment from the builder. Does that answer your question?
-
Justin Drake: In other words the beacon chain is holding the collateral, the beacon chain itself is trust-less custodian.
-
Alex Obadia: Okay, I have more questions but there is others on the call, I will ask after, thank you both.
-
Stokes: I think we can have a follow-up call and get updated on some of these details, specifically there.
”
and ty Alex for organizing!
aestus (www.aestus.live)