As detailed previously, Flashbots is transitioning from Yocto to a Mkosi for building our system images. This move was driven by our desire to leverage Debian’s extensive ecosystem of reproducible packages and streamline our development workflow.
Our approach, which is currently available on GitHub as “flashbots-images”, recognizes that different users have varying levels of trust requirements and available time for verification. Rather than forcing a one-size-fits-all solution, we’ve created a tiered system that allows users to choose their preferred balance between verification depth and practicality.
Mkosi’s Reproducibility Guarantees
The mkosi build system itself provides reproducible image generation. Given the same inputs and configuration, mkosi will produce bit-for-bit identical images. This reproducibility is essential for verification, as it ensures that anyone can rebuild our images and compare them against our published versions.
Additionally, when mkosi builds an image, it generates a detailed manifest of every Debian package included in the final system. This transparency forms the foundation of our verification approach. Since Debian has invested heavily in bit-for-bit reproducibility across their package ecosystem, we can leverage this work to provide strong guarantees about the integrity of our images. Of course, at some fundamental level, users still need to trust their compiler, source code, and hardware, but that’s a consideration that applies to essentially all computing systems. Beyond that foundational layer, our verification framework provides complete transparency and verifiability for every other component in the stack.
Tier 1: Complete Source Verification
For users who require the highest level of assurance, we provide a script that rebuilds every Debian package used in our images from source. This process works by taking the package list generated during the mkosi build and systematically recompiling each package that was included in it.
The script downloads each package’s source code along with its corresponding buildinfo file. This buildinfo file is crucial as it contains detailed metadata about the exact environment used to compile the official Debian package. The script then uses a program called debrebuild, which recreates this precise environment and compiles the source code within it. By using identical source code and build environment, we achieve bit-for-bit identical binaries.
After compilation, the script compares these freshly built packages against the ones mkosi downloaded from the official Debian repositories. Any discrepancy would immediately signal a potential issue. This verification method works reliably for the vast majority of packages in the Debian ecosystem, including all of the packages we use in our images.
The main drawback of this approach is time. Rebuilding many packages takes multiple hours, making it impractical for frequent verification.
Tier 2: Leveraging Debian’s Reproducibility Infrastructure
Recognizing that full recompilation isn’t always practical, we offer an alternative script that provides nearly identical trust guarantees while running in seconds rather than hours.
The Debian project maintains reproduce.debian.net, a continuous integration system that regularly rebuilds all Debian packages and publishes the results. This infrastructure runs the same debrebuild process our full verification script uses.
Our quick verification script checks each package in the mkosi manifest against the reports from this CI system, verifying that no reproducibility issues have been flagged for the packages we’re using. This gives us confidence that the packages can be independently rebuilt from source to produce identical binaries.
Currently, while we fix our package versions to specific snapshots, our verification compares against the latest versions in the reproduce.debian CI. To strengthen this verification, Flashbots will soon begin archiving the Debian reproducibility CI results, allowing us to match the exact package versions used in each build against the corresponding CI reports. Since the Debian CI build servers cryptographically sign all published information, this archival process doesn’t introduce Flashbots into the trust model. We’re simply preserving signed attestations that can be independently verified.
This approach maintains strong security properties because our infrastructure and the reproduce.debian.net infrastructure would both need to be compromised. Plus, the transparency of the system provides a strong foundation for trust. The fact that this CI system runs the exact same program we use in our full reproduction script means any attempts to hide issues would be immediately apparent when cross-referenced.
Tier 3: Local Verification Against Published Artifacts
Most users can simply clone the Flashbots mkosi images repository, run the build script, and verify that the resulting image matches our published version. This is because multiple independent parties are separately verifying the images we publish.
This baseline approach already exceeds industry standards. Many of the biggest technology companies routinely deploy the exact same Debian packages in critical infrastructure without the additional verification layers we provide. This trust is well-founded given Debian’s rigorous package maintenance processes and security track record.
Understanding the Threat Model
The full verification script protects against a very specific and highly complex attack scenario. For malicious binaries that don’t match the audited source code to make it into our images, an adversary would need to execute an extraordinarily sophisticated attack:
First, they would need to compromise the Debian build infrastructure, either through insider access or by breaching the build servers themselves. This alone would be challenging given Debian’s security practices and the scrutiny their infrastructure receives.
Next, they would need to separately infiltrate the Debian reproducibility team and compromise their entirely separate pool of build servers. This is necessary because the reproducibility CI would otherwise detect and flag the discrepancy between the source code and the backdoored binary.
Then, when Flashbots inevitably noticed the discrepancy between the official package and the version produced by our full verification script, the attackers would need to have also compromised our team to prevent us from raising the alarm.
All of this would need to happen while avoiding detection by the countless companies that carefully monitor this exact chain of trust for their own critical infrastructure. These organizations have security teams watching for precisely these types of supply chain attacks.
Finally, this entire house of cards would collapse the moment any independent party decided to run the full verification script themselves and compare results. The public nature of all the components involved makes secret collusion at this scale virtually impossible to maintain.
Broader Context
While we provide these verification tools for those who want them, it’s important to recognize that the Debian ecosystem’s commitment to reproducibility, combined with the transparency of our build process, creates multiple layers of protection against supply chain attacks.
The verification scripts serve as both practical tools and demonstrations of the security properties our system provides. They make explicit the trust relationships that many organizations implicitly rely upon, while providing mechanisms to verify those relationships for those who require additional assurance.
Conclusion
Our multi-tiered approach to image reproducibility provides flexibility without compromising security. By building on Debian’s reproducibility infrastructure and adding our own verification layers, we’ve created a system that can meet diverse trust requirements while remaining practical for day-to-day use.
The tools and scripts described here are available in our public repositories, and we encourage users to explore them and understand the security properties they provide. As the ecosystem of reproducible builds continues to mature, we remain committed to leveraging these advances to provide the most trustworthy images possible for our users.