Open Sourcing Sync Proxy

The Flashbots Ship Flashbots Status Updates
1

Sync Proxy

We are happy to publish sync-proxy, a key component in our infrastructure that may be useful to others.

sync-proxy is lightweight proxy between beacon nodes and execution-layer clients to improve reliability and simplicity of execution client synchronization. It enables a group of beacon nodes to drive a group of execution-layer clients in a redundant and performant way. By using a group of beacon nodes instead of one, it mitigates issues from any single beacon node. Moreover, it removes the need for adding a beacon node to every execution-layer client.

The sync proxy accepts connections from multiple beacon nodes. It attempts to use the beacon node with the highest slot number from a custom json rpc request in our prysm fork. If a beacon node goes offline, stops sending requests, or falls out of sync, the sync proxy will attempt to switch to another beacon node to forward requests to instead. The sync proxy will then proxy fire-and-forget RPC requests to one or multiple execution clients to drive the execution client sync. The net result is a high degree of reliability and redundancy, which is critical in operating important infrastructure such as a builder or relay.

As beacon nodes do not natively support multiple execution clients, we use nginx to proxy requests and responses to a main (local) execution client that responds to the beacon node. We then configure nginx to mirror requests to all the sync proxies connected to the execution clients we want to sync, as seen in the architecture diagram below:

Nginx Setup
Nginx Setup1358×1122 182 KB

An example nginx config is shown below for a beacon node client pointing to localhost on port 8552, mirroring to 3 other sync proxies. (Note: the short mirror proxy timeouts is necessary because mirrored requests delay the main request)

$ cat /etc/nginx/conf.d/sync_proxy.conf

server {
        listen 8552;
        listen [::]:8552;

        server_name _;

        location / {
                mirror /sync_proxy_1;
                mirror /sync_proxy_2;
                mirror /sync_proxy_3;

                proxy_pass http://localhost:8551;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header Host $host;
                proxy_set_header Referer $http_referer;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }

        #
        # execution nodes
        #
        location = /sync_proxy_1 {
                internal;
                proxy_pass http://sync-proxy-1.local:8552$request_uri;
                proxy_connect_timeout 100ms;
                proxy_read_timeout 100ms;
        }

        location = /sync_proxy_2 {
                internal;
                proxy_pass http://sync-proxy-2.local:8552$request_uri;
                proxy_connect_timeout 100ms;
                proxy_read_timeout 100ms;
        }

        location = /sync_proxy_3 {
                internal;
                proxy_pass http://sync-proxy-3.local:8552$request_uri;
                proxy_connect_timeout 100ms;
                proxy_read_timeout 100ms;
        }
}

The sync-proxy code is published under the MIT license on Github.

12 Likes