Is it an issue for the builder to filter out the withdrawal transactions from their block?
Why is this an issue? There is no way to prevent a builder from subsidizing their blocks regardless of the scoring method used. I don’t think Flashbots want’s to be in the position of deciding which payments are “legitimate” and which are “illegitimate”. This undermines the credible neutrality of the system and requires differentiating between “ethical and unethical” mev.
The CL provides mev-boost with the parent hash as part of getHeader call, so the block header on the parent can be validated trustlessly by mev-boost regardless of where the data comes from. The options are: bring from relay, bring from CL, bring from EL. Requesting the parent header from the relay as part of the payment proof seems like the most natural approach to me.
Sure builders could filter it out, but would really need to be forced to, for not lower their bid value. The question is whether the algorithm is actually sound if such a workaround is needed to arrive at a comparative scoring.
I’d like to understand why you feel block balance diff is preferable over payment transaction value?
Is there then no way for an outside observer to account for MEV? The feeRecipient is on-chain, but the “fee recipient” is private. Hence, by observing the chain, we cannot distinguish between the builder paying the proposer and the builder paying themself (or even versus no builder, just a proposer transferring out). As such, there seems to be no secure way to mirror a staking pool payout inclusive of MEV, without private information about the staking pool’s beneficiary addresses.
Inclusion of withdrawal amounts in bid values are currently causing incorrect bids to be selected.
Example is slot 7914991: the bid was 0.466032380614827 ETH but the actual payment outside of withdrawals was 0.0268737436148266 ETH. A rival bid that didn’t include withdrawals but came in around the same time offered 0.0777436185616146 ETH, so the proposer lost out on 0.050869874946788 ETH by the relay giving it an incorrect bid.
I think that the arguments for using balance delta ultimately fail because they aren’t providing the expected number, which is the value of the execution transactions in the payload to the proposer. Transaction-based value is the only solution here that provides a useful number for proposers.
Tracing the block and working through each transaction to obtain the actual value to the proposer isn’t very difficult, and if it takes too long to do this during the slot it could be done after the fact if relays are prepared to ban builders that report incorrect bids. But at the moment the field is distorted, relays are providing non-optimal bids, and this should be addressed as soon as possible so that bids provided by relays can start to be trusted again.
While balance delta is helpful in having a robust way for relays to technically make sure bids are “accurate” in a relatively efficient manner, it’s probably not intended (in spirit) and definitely not practical or useful for bids to include in the bid score (and thus how bids are ordered in terms of the value that they provide) things such as CL->EL withdrawals (which @thegostep astutely pointed out as a possible side effect earlier, I think we just didn’t realize that builders would utilize this on purpose).
If we want to keep the balance delta approach, I think we should try to agree that there are certain kinds of things which should be excluded when composing the bid score for bids when utilizing this approach (withdrawals for starters, but there may be other things that make sense in the future, too). Basically, it should be accounted for to make sure the bid is correct, but IMO it should not be incorporated into the bid value itself.
Otherwise, a concerted effort by relays to work through each Tx for block scoring as Jim says may end up being better in terms of avoiding these bid value irregularities.
Do we have a way to estimate how difficult it would be to timely simulate payloads to the extent necessary to fully construct the bid value from payload transactions and if it would meaningfully affect current timings on the relay side? Given that a lot of relays are using an optimistic model now, this can help to offset this as the simulation can happen later.
Here is some statistics about that on flashbots relay. We started to calculate bid proposal value this way 3 month ago and this data covert that range.
There are 196463 blocks proposed by our relay in that period.
Out of them:
had withdrawals to the fee recipient in the same block: 498 (0.25%)
had direct transfers to fee recipient: 95 (0.05%)
had transfest out of the fee recipient: 510 (0.26%)
Note that proposer is not loosing anything because this does not give any builder advantage. They all can include these values into the bid.
Removing withdrawals from the calculation is very easy, others is a bit harder but still possible. But note that this still affects only 0.5% of the blocks and it does not give any builder advantage at least if builder know about it.
and if it takes too long to do this during the slot it could be done after the fact
builder should obey any bid calculation rule that relay enforce so this does not work. Especially so if you propose to ban them for incorrect reporting
The proposer is definitely losing funds at current, I gave an example above where the flashbots relay provided a non-optimal bid.
The percentage is largely irrelevant, as long as it is >0: it generates a number of false positives that make it harder to check for real discrepancies (although I would argue that these are also real discrepancies; a block with 0 transactions would still return the withdrawals so it shouldn’t be considered builder-provided value).
And every builder adding in the withdrawal value would impact the current min-bid functionality within mev-boost, so I would hope that this wouldn’t be considered as a solution.
@jgm I think that Flashbots is sympathetic to your point of view that withdrawals should not be accounted. It’s why we originally went with a payment-tx validation model. That’s been eroded over time not because it’s what we wanted, but because it is what the market has wanted. We’d be happy to change how our relay works to subtract withdrawals, but unfortunately that will need to be coordinated with other relays in order to be effective. One relay accounting for withdrawals a different way would strictly dominate the others. Moreover, we’d prefer that validation was more uniform across the market so that there aren’t slightly different rules across the relay market. So, we are sympathetic, but we’d like to move in a coordinated way with others and that will require their buy-in too.
At current I believe there is only a single builder that is generating bids with withdrawal values included, so it seems like a relatively simple thing to contact them and ask them to stop doing so (or block them if they refuse). You could then fix this in your relay codebase and approach maintainers of other codebases to do the same.
I understand that co-ordination is important, but so is taking the lead. Relays are meant to be trusted entities, and doing the right thing in situations like this helps to maintain that trust.
It appears that the flashbots builder is still including withdrawals in their bid value. For example, slots 8375765, 8373799 and 8373202. Please could this be addressed as a matter of urgency?
We added withdrawals to our builder profit calculation and since we send all the value to the proposer all other builder must outbid us for those slots and this can no longer be abused. Unless for some reason our builder was not working for those slots.
For the slots that you mentioned all blocks were won by flashbots builder (0xa1f1a5a49) that provided the highest bid that included withdrawal value + all the mev tip.
@vitaliy are you saying that your builder purposefully sets the bid value to include withdrawals?
If so then it’s totally the wrong way round to do this. The approach should be that your relay refuses payloads from builders that include withdrawals in their bid value. Otherwise proposers cannot use the bid value for comparison purposes, as they don’t know if the bid includes withdrawals or not. They also cannot use min bid functionality.
It has been over two months since my post that showed that proposers are losing ETH due to relays not excluding payloads that artificially inflate the bid value. Today it appears that there are only two builders that do this, and flashbots is one of them. Indeed, flashbots builders are generating over 90% of these payloads with artificially inflated bid values.
So this is not some case of levelling the playing field, it is gaining an advantage over the other builders to the detriment of proposers. Please address this in your builder and relay code, so that validators can have trust in the information supplied to them by relays.
Do you know @jgm what these withdrawals would typically be for pool node operators as a percentage of a normal block? I mean for high-paying blocks these should be irrelevant essentially. For low paying blocks it could be quite a bit, and since most blocks are not high-paying, I’d like to know a reasonable number here. At the very least we can offset this attack by having reasonable defaults on the CL client. The simplest solution that I can implement right now is to simply add the withdrawals also to the local value when we compare against the bid.
If I take a recent slot I have recorded for this behaviour, slot 8377811, we have the following:
bid value was 0.370 ETH
withdrawals to the fee recipient were 0.320 ETH
proposer received 0.050 ETH MEV
So the vast majority of the bid value was in fact withdrawals and not MEV.
The problem with adding withdrawal values to local value is that it mucks up min bid. A proposer may well decide that an amount of 0.37 ETH is worth using a relay for, and that a value of 0.05 ETH is not. In this case they would use the relay even though the actual benefit of using the relay over not using the relay is <0.05 ETH.
I’m generally a fan of the status quo because it catches edge cases, but I do think that people underestimate the magnitude of blocks with withdrawals to the mev recipient. It might be time to carve out an exception.
I threw together a quick Dune dashboard in case anyone wants access to the raw data. There was a bit of a lull in December/January, but generally about 10 blocks per day are affected. Most of them are partial withdrawals and go relatively unnoticed.
Does anyone understand what’s happening in 19259217? No withdrawals are involved here.
Reported mev rewards by flashbots include two separate transfers to the mev rewards recipient. Unless I’m mistaken, only the last transfer from the fee recipient should be considered.
Withdrawals aren’t the only issue, anything that changes the account balance for the mev reward recipient counts toward bid values. In 19,259,217 you correctly identify that there are two payments to the recipient, even though the second one isn’t related to the mev auction it still counts for bid scoring in the current regime.
In an extreme case, you could have a transfer out from the mev reward recipient and potentially a negative auction value.
This is what I’m wondering as well. Last example 19,259,217 I shared summed two transfers that were both at the very end of the block which is characteristic of mev payouts.
Found a more blatant case in block 19,239,576. Reported payout 5.09 ETH is the sum of last TX 6d2cf (actually seems to go through a contract which is obfuscating) and be828 which is somewhere in the middle of the block.
I’m trying to understand if only the last one alone should be considered as mev rewards in all cases or if others could be legit as well.
