Engineering Update - 2025-09-19

BuilderNet
1

Welcome back to another issue of BuilderNet engineering updates.

This post gives an overview of what happened in the weeks leading up to September 19, 2025, and what’s up next!

Engineering

  • Forced Azure Gen6 upgrade

    • Azure mandated a deprecation of all Gen5 TDX instances and migration to Gen6 TDX instances, on very short notice (only four weeks).
    • BuilderNet used Gen5 instances, which were the latest and fastest generation of Azure TDX instances.
    • When the deprecation notice was sent out, Gen6 instances were not fully working. Notably, the SEAM loader of new Gen6 instances was outdated, which led to TEE proof validation failures.
    • We initiated communication with Azure support and explained the issues we were seeing.
    • Over the next few weeks, Azure gradually updated their Gen6 instances with the latest bootloader versions, which made attestations work and allowed BuilderNet to seamlessly upgrade to Gen6 instances and deprecate all Gen5 instances.

  • Azure Gen6 TDX instance performance issues

    • We are observing some performance degredation on Gen6 instances. and are investigating further, as well as communicating with Azure support.
    • This came as a surprise since all the hardware components are slighly improved and newer-generation, of which we expected at least slight performance improvements.
    • Our initial - and incomplete - investigation shows significant performance degredations around CPU and Disk IO, under both light and heavy mixed load:
      image
      image816×836 103 KB
    • These numbers are not authoritative yet and we are investigating this further. We plan to post a more comprehensive upgrade with more data in the near future.
    • We used the stress-ng benchmarking tool to simulate the mixed load on multiple components of the system simultaneously. We ran the benchmarks on multiple different VMs, all of them demonstrated similar degradation in performance.
    • To simulate light load we used
    stress-ng --cpu 4 --io 2 --vm 2 --vm-bytes 1G --timeout 300s --metrics-brief
    • To simulate heavy load we used
    stress-ng --cpu 0 --cpu-method all --io 4 --vm 4 --vm-bytes 75% --vm-keep --timeout 600s --aggressive --metrics-brief

  • Reth root hash issue causing ecosystem-wide service disruptions

  • Refunds API

    • BuilderNet will improve it’s API to give better insights into pending refunds for a given bundle or signer.
    • We are working on both a long-term API as well as a short-term interface.

  • Heavy Node Egress

    • Builder nodes with blobs: 700MB/min/node to each relay = 42GB/h = 1TB/day
    • Builder nodes without blobs: it’s only 1/10th; 70MB/min = 4GB/h = 100GB/day
    • This pain point should mostly go away with Optimistic v3.
    • We are investigatign VPC peering between Azure networks as well as links to other clouds, to reduce cost and improve latency.

  • Optimistic v3

  • Heavy ongoing work on mkosi

    • We’re migrating from Yocto to mkosi as a tool for building reproducible OS images for BuilderNet VMs.
    • Instead of building the kernel from scratch as in Yocto, mkosi is based on a Debian kernel, and can use Debian packages, which alone drastically improves developer experience.
    • It also includes a switch from sysvinit to Systemd, which comes with several additional useful capabilities, for instance running units with podman.
    • You can track the ongoing work in the trunk/buildernet branch of flashbots-images repository.
    • With the switch from fully custom OS with sysvinit as init system to a Debian-based system with systemd it involves a heavy refactoring of basic building blocks comprising the image.
    • A Debian-based system enables us to access a wide-range of pre-built binary packages that are verifiably reproducible.

  • rbuilder updates

    • Several performance improvements
    • Updates to the building algorithm

  • BuilderHub updates

  • Fusaka Ethereum Upgrade Ready

Upcoming

  • mkosi end-to-end testing.
  • Orderflow proxy
    • Return bundle hash + uuid as well as errors.
    • Networking and multiplexing performance improvements
  • Automatic transfer of coinbase funds to multisig from inside rbuilder.
  • Document refundIdentity and replacementNonce
  • Move rbuilder-operator as a crate to rbuilder repo
  • Remove non-optimistic key
  • Remove more dead code and clean up unnecessary feature flags
  • Refund API
  • Architecture exploration document about scaling BuilderNet to 100x the nodes.
  • We are exploring:
    • Bare-metal hosting: explore running builder nodes on OpenMetal instances.
    • Allowing eth_sendBundle without mandatory signature
    • Pull-based refunds
    • Smarter rate limiting with quality scoring
    • Multiplexing in BuilderNet
    • Reducing the amount of outgoing data
    • L2 compatibility.

Other Highlights

  • New forum post: Combined refunds
  • Second BuilderNet Community Call
    • Happened last week, September 10th at 4pm UTC.
    • Key topics included an (a) engineering status update, (b) engineering roadmap, and (c) a deeper conversation about redistribution and refunds.
    • Forum post, Recording
3 Likes