pnj

Cross-account Prometheus replication with end-to-end mTLS via Envoy

· 11 min read · prometheus · envoy · mtls · aws · observability · security

There are easier ways to ship metrics across AWS accounts than the one I’m about to describe. You could push to Amazon Managed Prometheus. You could peer the VPCs. You could use PrivateLink. We didn’t — we wanted both ends to be self-hosted Prometheus, separated by an account boundary, and we wanted the link to be the kind of thing a compliance reviewer can audit end-to-end. So the wire ended up going across the public internet with mutual TLS as the only thing standing between any random host on the internet and our metrics ingest endpoint.

This post walks through the design: how the two Prometheus servers talk to each other, what Envoy does on each side, how the certificates fit together, and the networking and security considerations that ended up shaping the final shape of the system.

The setup at a glance

┌────────────── Account A (sender) ──────────────┐         ┌──────────── Account B (receiver) ────────────┐
│                                                 │         │                                              │
│  ┌──────────┐   ┌─────────┐   ┌────────────┐   │   TCP   │  ┌─────┐   ┌─────────┐   ┌─────────────┐    │
│  │Prometheus│──>│  Envoy  │──>│ NAT (EIP)  │───┼──443───┼─>│ NLB │──>│  Envoy  │──>│ Prometheus  │    │
│  │ (scraper)│   │ (mTLS   │   │ stable     │   │ public  │  │     │   │ (mTLS   │   │ (receiver,  │    │
│  │          │   │ client) │   │ egress IP  │   │ internet│  │     │   │ server) │   │  remote_w)  │    │
│  └──────────┘   └─────────┘   └────────────┘   │         │  └─────┘   └─────────┘   └─────────────┘    │
│                                                 │         │                                              │
└─────────────────────────────────────────────────┘         └──────────────────────────────────────────────┘
  • Account A runs Prometheus that scrapes everything it cares about and uses remote_write to ship samples elsewhere. The elsewhere is a local Envoy on the same host (or sidecar), which terminates outbound TLS, adds its client certificate, and dials the public endpoint in Account B.
  • The traffic leaves Account A through a NAT gateway with a pinned Elastic IP, so the egress IP is stable.
  • It lands in Account B on a public Network Load Balancer that fronts another Envoy. Account B’s security group allows port 443 only from Account A’s NAT EIP. So even a perfectly forged TLS handshake from a random host on the internet hits a closed socket.
  • Envoy in Account B terminates TLS, validates the client certificate against a trusted intermediate CA, and forwards the request to a receiver-mode Prometheus running locally.

Three independent layers of access control, in order of how early they reject a bad request:

  1. Network ACL (Account B’s SG) — only Account A’s egress IP can even open a connection.
  2. mTLS server validation — Envoy rejects clients that don’t present a certificate signed by the intermediate CA we trust.
  3. mTLS client validation — Account A’s Envoy refuses to send to a server whose cert isn’t signed by the same intermediate.

You need all three to fail for an exfiltration or injection to land.

Prometheus side

The sender’s prometheus.yml is boring on purpose — all of the security logic is delegated to the local Envoy.

# Account A: prometheus.yml (excerpt)
remote_write:
  - url: http://127.0.0.1:9091/api/v1/write
    queue_config:
      capacity: 10000
      max_samples_per_send: 2000
      max_shards: 50
      min_backoff: 100ms
      max_backoff: 30s
    metadata_config:
      send: true
      send_interval: 1m

A few things worth noting here:

  • The URL is http:// (plain) and 127.0.0.1 (loopback). The crypto happens one hop later, in the Envoy sidecar.
  • queue_config is tuned higher than the defaults because cross-account links have higher latency and more variance than intra-cluster remote_write. Bumping max_shards lets Prometheus parallelize when the link is healthy; the WAL acts as the backpressure buffer when it isn’t.
  • metadata_config.send: true ships sample metadata so the receiver side can render correct help text in /api/v1/metadata. It’s free bytes and worth it.

The receiver is also boring:

# Account B: prometheus startup flags
prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --web.listen-address=127.0.0.1:9090 \
  --web.enable-remote-write-receiver \
  --enable-feature=memory-snapshot-on-shutdown

--web.enable-remote-write-receiver is the magic flag — it turns Prometheus into a receiver for the remote-write protocol on /api/v1/write. Without it, the endpoint 404s. --web.listen-address binds to loopback only; nothing on the network can reach Prometheus directly. Everything has to come through Envoy.

Envoy on the receiver (Account B)

This is the more interesting half of the configuration. Envoy is the public face of the metrics ingest endpoint — it terminates TLS, validates the client certificate, and acts as a reverse proxy to Prometheus.

# Account B: envoy.yaml (excerpt)
static_resources:
  listeners:
    - name: metrics_ingest
      address:
        socket_address: { address: 0.0.0.0, port_value: 8443 }
      filter_chains:
        - transport_socket:
            name: envoy.transport_sockets.tls
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
              require_client_certificate: true
              common_tls_context:
                tls_certificates:
                  - certificate_chain: { filename: /etc/envoy/certs/server.crt }
                    private_key:       { filename: /etc/envoy/certs/server.key }
                validation_context:
                  # Trust the intermediate CA only — not any random
                  # root the OS happens to trust.
                  trusted_ca: { filename: /etc/envoy/certs/intermediate-ca.crt }
                  match_typed_subject_alt_names:
                    - san_type: DNS
                      matcher:
                        exact: "metrics-sender.account-a.internal"
          filters:
            - name: envoy.filters.network.http_connection_manager
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                stat_prefix: ingress_http
                route_config:
                  virtual_hosts:
                    - name: prom_ingest
                      domains: ["*"]
                      routes:
                        - match: { prefix: "/api/v1/write" }
                          route: { cluster: prometheus_local }
                http_filters:
                  - name: envoy.filters.http.router
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
                access_log:
                  - name: envoy.access_loggers.stdout
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
                      log_format:
                        text_format_source:
                          inline_string: |
                            %START_TIME% %DOWNSTREAM_PEER_SUBJECT% %REQ(:METHOD)% %REQ(:PATH%)% %RESPONSE_CODE% %BYTES_RECEIVED%

  clusters:
    - name: prometheus_local
      connect_timeout: 1s
      type: STATIC
      load_assignment:
        cluster_name: prometheus_local
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address: { address: 127.0.0.1, port_value: 9090 }

The bits that actually matter, called out:

  • require_client_certificate: true — the handshake fails if the client doesn’t present a cert. No “optional” mTLS, no falling back to one-way TLS for clients that “haven’t migrated yet.”
  • trusted_ca points at the intermediate CA, not the system root store. This is critical. If you point it at the OS truststore you’ve effectively said “anything Let’s Encrypt or DigiCert signs is fine,” which is the opposite of what mTLS is supposed to give you. The trust anchor must be the small internal CA you control.
  • match_typed_subject_alt_names pins the expected SAN on the client cert. Without this, any cert signed by your intermediate would be accepted, including ones for other services that happen to be in the same trust domain. Pinning the SAN means a stolen-but-different cert from inside the trust domain still gets rejected.
  • access_log includes DOWNSTREAM_PEER_SUBJECT — the client cert’s subject DN is logged on every request. This is the audit trail that says “this metric came from a system holding cert X.” When the compliance reviewer asks “who can write to this Prometheus?” the answer is in the logs.
  • The route is restricted to /api/v1/write. The receiver Prometheus has plenty of other endpoints (/api/v1/query, /metrics, the UI), and none of them are reachable through Envoy. The sender can write, and only write.

Envoy on the sender (Account A)

The sender’s Envoy is the mirror image: it terminates the upstream TLS, presents the client cert, and validates the server side.

# Account A: envoy.yaml (excerpt)
static_resources:
  listeners:
    - name: prom_remote_write_egress
      address:
        socket_address: { address: 127.0.0.1, port_value: 9091 }
      filter_chains:
        - filters:
            - name: envoy.filters.network.http_connection_manager
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                stat_prefix: egress_http
                route_config:
                  virtual_hosts:
                    - name: remote_write
                      domains: ["*"]
                      routes:
                        - match: { prefix: "/api/v1/write" }
                          route: { cluster: remote_prometheus }
                http_filters:
                  - name: envoy.filters.http.router
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router

  clusters:
    - name: remote_prometheus
      connect_timeout: 5s
      type: STRICT_DNS
      lb_policy: ROUND_ROBIN
      load_assignment:
        cluster_name: remote_prometheus
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: metrics-ingest.example.com
                      port_value: 443
      transport_socket:
        name: envoy.transport_sockets.tls
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
          sni: metrics-ingest.example.com
          common_tls_context:
            tls_certificates:
              - certificate_chain: { filename: /etc/envoy/certs/client.crt }
                private_key:       { filename: /etc/envoy/certs/client.key }
            validation_context:
              trusted_ca: { filename: /etc/envoy/certs/intermediate-ca.crt }
              match_typed_subject_alt_names:
                - san_type: DNS
                  matcher:
                    exact: "metrics-ingest.account-b.internal"

Same trust anchor, same pinning discipline — just from the other side.

A subtle detail: the upstream TlsContext’s validation_context.trusted_ca must contain the intermediate that signed the server cert. The server cert’s chain (sent during handshake) chains up to that intermediate. If you instead put only the root CA in the trust store, it works — but you lose the ability to revoke at the intermediate level, which is usually the whole point of running a two-tier PKI.

The networking layer

The transport choices were the trickiest part to justify on paper.

VPC peering would have worked, but it requires both sides’ networking teams to coordinate on CIDR blocks, route tables, and security group references — and that coordination needs to be redone any time either side re-IPs. PrivateLink is cleaner but is a single-vendor (AWS) story: the moment one side moves out of AWS or runs from a different cloud for DR, you’re rebuilding the link. The public internet path is portable; the mTLS + IP-allow-list combination provides the security envelope without binding to a specific AWS networking primitive.

The trade-off is throughput and latency: this path crosses internet hops, so a regional outage in either direction is a hard cut. Prometheus’s local WAL absorbs that — samples accumulate and replay when the link comes back, up to the WAL’s retention window (typically 2 hours).

Why a NAT with a pinned EIP?

The Account B security group needs a stable source IP to allow-list. The default behavior of an AWS NAT Gateway with no explicit EIP can survive fine, but if it ever gets re-created (Terraform destroy/apply, AZ failover, region migration), the IP changes and Account B starts silently rejecting traffic. Pinning the NAT to a pre-allocated EIP makes the egress IP part of state you explicitly version.

In Terraform that’s a four-line pattern:

resource "aws_eip" "metrics_egress" {
  domain = "vpc"
  tags   = { Name = "metrics-egress" }
}

resource "aws_nat_gateway" "metrics_egress" {
  allocation_id = aws_eip.metrics_egress.id
  subnet_id     = aws_subnet.public_a.id
}

The receiver side allow-lists aws_eip.metrics_egress.public_ip and is done. EIPs are sticky and survive NAT recreation.

Why an NLB, not an ALB?

An ALB terminates TLS itself and would either bypass the Envoy behind it or require a second TLS hop. The point of Envoy is to be the single TLS termination point; an NLB lets the TLS connection pass through untouched at L4. The NLB exists only for DNS stability and public IP attachment.

The certificate chain

A clean PKI layout helped a lot more than I expected when I was setting this up. The structure was:

                    Internal Root CA              (offline, in HSM)
                          │
                          │ signs
                          ▼
              Internal Intermediate CA            (online, short-lived)
                  │                  │
                  │ signs            │ signs
                  ▼                  ▼
         Server cert                Client cert
       (metrics-ingest.…)          (metrics-sender.…)
       Account B Envoy             Account A Envoy

Both Envoys trust the intermediate, not the root, in their trusted_ca. Why:

  • Revocation is cheap at the intermediate level. If a leaf cert is compromised, you don’t need to wait for the leaf to expire — you re-issue the intermediate and the old leaf is automatically untrusted on the next handshake. Both sides pull the new intermediate at the next certificate-rotation cycle.
  • The root stays offline. It only ever signs intermediates. It isn’t loaded on any production box, which means a compromise of any running Envoy host cannot extract material that re-signs new intermediates.
  • The intermediate is short-lived (think days, not years). Cert rotation is a routine cron, not a fire drill.

The most underrated thing about this design is what it lets you do operationally: when something is broken, the question “is the cert expired, on either side, or is something else broken?” can be answered by running a five-line openssl s_client against the public endpoint and checking the dates. The trust chain is short enough that a human can mentally model the failure modes.

What we learned to watch

A few things broke in instructive ways:

  • prometheus_remote_storage_failed_samples_total — this is the alert you actually want. It catches everything from “cert expired” to “receiver out of disk” to “we tripped the rate limit on the receiver.” Without it, the failure mode is silent: Prometheus keeps trying, keeps failing, WAL grows, eventually the sender’s disk fills up and scrape failures start happening too.
  • prometheus_wal_segment_current — if this is climbing without bound on the sender, the link is congested or down. The WAL is doing its job, but you have a finite amount of time before it stops doing its job.
  • Envoy’s ssl.fail_verify_cert counter on the receiver — non-zero values mean someone (probably your own infrastructure mid-rotation, occasionally something actually wrong) is trying to connect with a cert that doesn’t pass validation. Alert on it.
  • The NAT EIP — sounds dumb, but: alert if the EIP allocation disappears from state. We had a single incident where a Terraform refactor moved the EIP into a different module without re-using the same allocation ID, the new EIP wasn’t on the allow list, and Account B started rejecting writes within seconds.

The summary

The shape of the system, distilled:

  • Prometheus on both sides, both self-hosted, separated by an AWS account boundary.
  • Envoy sidecars on each side handle all the cryptography and certificate lifecycle. Prometheus stays cleartext, local, simple.
  • mTLS with a private intermediate CA, with SAN pinning on both sides, is the inner fence.
  • A pinned-EIP NAT and a single-IP allow-list on the receiver SG is the outer fence.
  • An NLB sits between the internet and the receiver Envoy so the public DNS name and IP are stable independent of how Envoy is deployed.
  • The WAL on the sender absorbs link outages; alerts on failed_samples_total and wal_segment_current catch real problems.

None of this is novel — every component is documented somewhere — but the combination is what gave us a metrics replication link we’d be happy to put in front of an auditor and confident we’d hear about it before it failed silently.