[Idea] Devnet environment for end-to-end testing BuilderNet images

BuilderNet
1

It would be useful to have a local devnet environment for e2e testing BuilderNet images.

An initial setup could look like this, with these local components:
Builder Playground Provides the L1 network (EL + CL + MEV-Boost-Relay). No changes needed.
BuilderHub service The builder image gets configuration and secrets from it, and won’t start without it. The only change to this service is to allow all measurements (could be a simple env var DEV_ALLOW_ALL_MEASUREMENTS)
BuilderNet Yocto Image The TDX VM image for the builder node. Needed changes might include additional configuration for Reth and Rbuilder to enable a custom chain, and to skip the chain download step.

Additional steps (for later phases) might include:

  • contender for transactions generation
  • orderflow-proxy to send transactions to the BuilderNet instance
  • vTPM for disk encryption (optional)

See also the documentation:

Production Architecture Overview

image
image2735×2057 368 KB

Production Boot Sequence

image
image1920×1563 125 KB

Initial (incomplete) recommended steps:

  • Check out Builder Playground and run it (to get familiar with it)
  • Run BuilderHub (read the full README) and play with it’s API (see also API code here)
  • Build a BuilderNet Yocto image, with the following modifications:
    • Bake in your SSH key (and possibly SSH enabled by default)
    • Change the BuilderHub URL to the new local one
    • Disable Reth snapshot sync
    • Update EL (Reth) and CL (Lighthouse) settings to with with those of the Devnet
  • Start the BuilderNet VM image locally with QEMU, and log in with SSH
    • It will fail at download secrets+config
  • Update BuilderHub
    • Allow all measurements (just needs to expect an empty measurements json)
    • Enable that builder and add configuration for it
  • At this point, downloading secrets + config should already work
3 Likes
2

I think adding such a change to the central component (BuilderHub) might lead to a secrets/configuration leakage if not safe-guarded enough (e.g. that flag allows only our VPN or something like that). Because otherwise, someone could deployment any image and send its measurement to fetch the deployment secrets/configs.

If this is only regarding devnet env purposes and not later repurposed for production env, then it won’t matter if the disk encryption key coming from a vTPM or BuilderHub.
But if we want to test the functionality of the vTPM itself as a viable option for the deployment, then I agree that it should be there.
Also, we could add to it later Loose Seal as an option for bare-metal deployments.
Ideally, we would have soon a secure KMS solution (e.g. this PR) that we could utilize for such use-cases

3

In favor of the idea, I would ideally also consider what changes are needed to do a shadow mainnet (call it staging) deployment for things like performance profiling. Maybe the same changes would apply as for devnet, maybe we would take slightly different ones if we consider both devnet and staging mainnet deployment.

On allowing measurements, I wonder if it wouldn’t be cleaner to add a simple make target that takes the measurement of the image (built or already deployed) and whitelists it along with configuration. Something like this would really be quite simple, and we would be avoiding a possible headache with propagating the allowed measurements for checking builder-to-builder connectivity for example.