Running Geth within SGX: Our Experience, Learnings and Code | Flashbots

At Flashbots, we are exploring trusted execution environments (TEEs) such as SGX, as well as other privacy technologies such as Multi-Party Computation and Homomorphic Encryption, as important building blocks for trustless collaboration along the transaction supply chain. This is particularly relevant for applications such as decentralized block building and sharing private orderflow.

In collaboration with konVera, a software development company with strong expertise in confidential computing, we have achieved a fully working prototype of Geth running inside SGX and syncing with Ethereum mainnet.

You can find the code at GitHub - flashbots/geth-sgx-gramine: Geth-in-SGX provides an example of running go-ethereum in SGX.

We hope this is useful to others, and want to invite open research and collaboration! :star:

10 Likes

Could you say more about the IO leakage you explored? Did you quantify it or demo it on some transactions/applications?

Iā€™m amazed it was possible to run Geth as a whole i side Gramineā€¦ thatā€™s a generic approach that swaps out all the system calls.

If the goal is just to assemble blocks without being able to peek at the transactions, why not carve out a much smaller portion of Geth?

Also Iā€™m curious if you looked at the Oasis Sapphire EVM ported to SGX. oasis-sdk/runtime-sdk/modules/evm at main Ā· oasisprotocol/oasis-sdk Ā· GitHub

3 Likes

Agree with the broader point that a subset of geth tailored for a given purpose would be simpler to run inside TEEs. For instance, we run a flashbots relay for Polygon and are exploring stateless clients and witnesses to simulate bundles inside TEEs without having to sync and store a ton of state. The same could work for block builders who simply assemble bundles and transactions into blocks.

The place where full nodes would most likely be necessary is to generate bundles and witnesses in the first place. However, even here, I suspect ā€œstate-liteā€ clients that store only a subset of state that is relevant would be feasible.

Iā€™m also very curious about feasibility of eclipse attacks on the node. Seems like a hardened discovery mechanism would be needed that is a lot more fail-closed than the current mechanism.

We are exploring a range of privacy technologies, and running geth inside SGX is just one of these explorations. Indeed another one is running only EVM inside geth, as well as using MPC and combinations of SGX and MPC.

Fowarded internally, will report back with details.

3 Likes

@ra and Frieder could say a bit more about the IO leakage details.

Related to this are covert-channel attacks, which potentially allow attackers to extract confidential information by observing resource usage on the host machine. In particular if user supplied inputs to the program running in the enclave can help leak information, for instance through CPU usage or memory access patterns.

More broadly speaking, our general understanding is that if an attacker has hardware access to the server running the SGX enclaves, itā€™s only a matter of time until private data is extracted through one of multiple ways (vulnerabilities, side-channel attack, covert-channel attacks, programming errors, etc). If run in such a setup, itā€™s important to prepare for the eventual breach. There are some ways to mitigate, including include rotating keys, TCB recovery plans, compartmentalization, smart software upgrade processes, etc.

3 Likes

Awesome! The idea of running Geth in SGX is impressive. I am especially interested in how to verify attestations on-chain. Is there some progress or reference material on this topic?

1 Like

Iā€™m also super curious about this, are there anything I can read?