pnj

Regional EventBridge, singleton OpenSearch, Grafana annotations

· 10 min read · observability · eventbridge · opensearch · grafana · aws

Most of the useful “why did that metric move?” questions get answered faster when the events that changed the system are drawn directly on the metric you’re staring at. A deploy, a feature-flag flip, a config rollout, a scale event, an alert firing — if you can see the vertical line on the same panel as the latency spike, you don’t need to correlate timestamps across four tools.

This is the setup I ended up with:

  • Pods and other workloads emit events to AWS EventBridge in their own region (one bus per region — data-locality and latency reasons).
  • Every regional EventBridge forwards its events to a single OpenSearch cluster in one designated region. That cluster is a singleton by design: the whole point is to aggregate everything in one place so Grafana has one datasource, one index pattern, one query surface.
  • Grafana is configured with an OpenSearch datasource pointed at that cluster, and each panel that wants event overlays declares an annotation query against a specific index pattern.

The result is that a deploy in eu-west-1 and a config push in us-east-1 show up as annotations on the same dashboard panel, side by side, in timestamp order.

Why not just query the regional bus?

The tempting design is to leave events where they land — one EventBridge bus per region, one OpenSearch cluster per region, and let Grafana query the closest one. That falls over the moment you want a cross-region view.

Real questions people ask a dashboard:

  • “Did the global config rollout at 14:07 UTC correspond to the p99 blip in EU?”
  • “We shipped a new model to two regions on Friday. Which one degraded first?”
  • “Was the tenant-migration event before or after the retry storm?”

If your events are sharded per region, every one of those questions requires either a federated OpenSearch query (painful) or a client-side merge in Grafana across multiple datasources (also painful, and Grafana’s annotation UI doesn’t really do a clean N-way merge). Aggregation up front in a singleton cluster is worth the coupling.

You pay for it with:

  • One cluster is a single point of failure for the event overlay, not for the emit path. If the singleton is down, EventBridge in each region keeps accepting events; only the visualization degrades. That’s usually a fine tradeoff — annotations are diagnostic, not operational.
  • Cross-region egress cost from the regional buses to the singleton. In practice event volume is tiny compared to metrics or logs, so this is noise on the AWS bill.
  • One region’s blast radius includes “no more event annotations for anyone.” Pick the region you’re most confident in.

The emit path

Workloads emit an PutEvents call to their local EventBridge bus. The event envelope is the standard AWS shape:

{
  "Source": "svc.checkout",
  "DetailType": "deploy",
  "Detail": {
    "service": "checkout",
    "version": "v1.42.0",
    "actor": "ci",
    "env": "prod",
    "region": "us-east-1",
    "started_at": "2026-07-06T14:07:33Z"
  },
  "EventBusName": "obs-events"
}

A few things worth pinning down at the emit boundary, because you’re going to grep them later and you don’t want inconsistency:

  • Source is the event emitter (a service, a system, a human tool). Keep it hierarchical and stable — svc.checkout, platform.deploy, oncall.human.
  • DetailType is the event type — deploy, config-change, flag-flip, incident-open, scale-event. This is what Grafana will filter on for a given panel.
  • Detail is your free-form payload. Always include region, env, and the identifier of the thing the event is about (service, tenant, component, whatever) so you can filter panels down to just the relevant subset.
  • time is set by the emitter, not by AWS. If you let EventBridge assign it, you’ll see fan-out latency (usually sub-second, but it exists) as a timestamp skew, and your overlay will drift from the actual event.

Publishing is a few lines in each service. I’ve seen this done as:

  • A tiny in-process client (Go/Python) that owns the connection to EventBridge and adds the region/env/host tags automatically.
  • A sidecar or Lambda that consumes a Kafka topic and republishes to EventBridge — useful if your services already emit to Kafka and you don’t want to touch them.

The important part isn’t the emit code; it’s that every event carries region, env, and enough identifiers for Grafana to filter.

Fan-in to the singleton

Each regional EventBridge bus has a rule that matches “everything I care about routing centrally” and a target pointing at the singleton cluster’s region. Two common ways to do the target:

Option A: EventBridge → EventBridge (cross-region)

EventBridge supports “another EventBridge bus in another region” as a target. The rule looks like:

resource "aws_cloudwatch_event_rule" "forward_all" {
  name           = "forward-all-to-central"
  event_bus_name = "obs-events"
  event_pattern  = jsonencode({
    source = [{ "prefix": "" }]  # everything
  })
}

resource "aws_cloudwatch_event_target" "central" {
  rule           = aws_cloudwatch_event_rule.forward_all.name
  event_bus_name = "obs-events"
  target_id      = "central-bus"
  arn            = "arn:aws:events:us-east-1:${var.central_account}:event-bus/obs-events-central"
  role_arn       = aws_iam_role.forwarder.arn
}

Then in the central region, a second rule on the central bus routes to whatever finally lands the event in OpenSearch (Firehose, Lambda, whatever you use).

Pros: pure EventBridge, retries and DLQ are built in, no code. Cons: two hops of EventBridge charges, and the central bus has to be sized for the sum of all regions.

Option B: EventBridge → Firehose → OpenSearch

Each regional bus has a rule whose target is a Kinesis Firehose delivery stream in the singleton region, and Firehose writes directly to OpenSearch with a rolling index name.

Pros: fewer moving parts, Firehose handles buffering/retries/backpressure, and OpenSearch gets writes in nice batches instead of per-event. Cons: Firehose adds a bit of end-to-end latency (usually 60s buffer), so the annotation lags the event by that much. For annotations, that’s fine — people don’t stare at a dashboard at t+5s.

Option B is what I’d default to. The batching alone is worth it — OpenSearch does not enjoy per-event indexing at scale, and Firehose’s buffer knobs let you tune “how fresh the annotations need to be” vs “how hot OpenSearch runs.”

The OpenSearch index

Keep the schema explicit; do not let dynamic mapping run wild on a free-form detail payload. A minimal template:

{
  "index_patterns": ["events-*"],
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 1,
      "refresh_interval": "30s"
    },
    "mappings": {
      "properties": {
        "@timestamp":  { "type": "date" },
        "source":      { "type": "keyword" },
        "detail_type": { "type": "keyword" },
        "region":      { "type": "keyword" },
        "env":         { "type": "keyword" },
        "service":     { "type": "keyword" },
        "actor":       { "type": "keyword" },
        "message":     { "type": "text" },
        "detail":      { "type": "object", "enabled": false }
      }
    }
  }
}

Two calls I made and would make again:

  • detail is stored but not indexed (enabled: false). You get the raw payload back when you click the annotation, but you don’t pay for mapping every ad-hoc field a random emitter throws in. Grafana filters on the promoted top-level keywords (source, detail_type, region, env, service).
  • refresh_interval: 30s. Faster refresh burns CPU on a cluster that’s effectively write-heavy and read-sparse (people load dashboards, then don’t reload for minutes). 30s is imperceptible in the annotation UX.

Roll indices daily (events-YYYY.MM.DD) via ISM policy, delete after your retention window (I run 90 days for annotations — plenty to correlate against any dashboard you’d load).

Wiring Grafana’s OpenSearch datasource

The OpenSearch datasource in Grafana is stock — URL, auth, index pattern. Where it gets interesting is the annotation configuration on individual panels or on the dashboard.

At the dashboard level, define a set of reusable annotation queries:

{
  "annotations": {
    "list": [
      {
        "datasource": "opensearch-events",
        "name": "Deploys",
        "enable": true,
        "iconColor": "green",
        "target": {
          "query": "detail_type:deploy AND env:prod",
          "timeField": "@timestamp",
          "textField": "message",
          "tagsField": "service"
        }
      },
      {
        "datasource": "opensearch-events",
        "name": "Config changes",
        "enable": true,
        "iconColor": "purple",
        "target": {
          "query": "detail_type:config-change AND env:prod",
          "timeField": "@timestamp",
          "textField": "message",
          "tagsField": "component"
        }
      },
      {
        "datasource": "opensearch-events",
        "name": "Incidents",
        "enable": true,
        "iconColor": "red",
        "target": {
          "query": "detail_type:incident-open",
          "timeField": "@timestamp",
          "textField": "message"
        }
      }
    ]
  }
}

The user toggles each layer on/off from the dashboard toolbar. A deploy line and a config-change line and an incident line can co-exist on the same panel without visual noise — different colors, different tags.

For per-panel annotations (only draw deploys for this service on this panel), the query uses dashboard variables:

detail_type:deploy AND service:$service AND region:$region

$service and $region come from the panel’s own variables, so a dashboard filtered to service=checkout, region=us-east-1 only shows deploy lines for that scope.

Making the annotation click useful

The default click behavior in Grafana is to show the textField in a tooltip. That’s fine, but you’ll want more. Two upgrades:

  • Set textField to something meaningful — I use a pre-computed message field on ingest (Firehose Lambda transform), e.g. "deploy of checkout v1.42.0 by ci in us-east-1". That’s what shows in the tooltip.
  • Set tagsField to a field like service or component so annotations get filterable tags.

For “I want to go from an annotation to the actual event / PR / runbook,” attach a URL to the event on ingest. The transform Lambda can compute a canonical link — deploy events point to the CI build page, incident events point to the PagerDuty incident, config-change events point to the PR. Store the link on the document; render it in the tooltip via a Grafana panel description or a manual “click for details” convention.

The failure modes worth thinking about

  • EventBridge silent drops. Cross-region EventBridge targets have a retry policy but not an obvious DLQ by default. Configure a DLQ on the target and alert on non-zero depth. If forwarding is broken, the emit side succeeds and everyone assumes annotations will show up — they won’t, and you’ll only notice when someone asks “why don’t I see the deploy?”
  • Time skew. If an emitter has a wrong clock (containers sometimes do), its events land at the wrong timestamp and float away from the metric they should annotate. Have the ingest Lambda drop events with a timestamp more than a few minutes off from ingest-time, or clamp them to a “received_at” field and use that on the annotation.
  • Cardinality explosion in detail. Even with enabled: false on the object, if someone starts emitting a million-token payload per event you’ll blow disk and lose retention. Set a max event size at the emit boundary (EventBridge caps at 256KB per event; enforce smaller in your client).
  • The singleton cluster’s region goes down. Emit path keeps working — events sit in the regional bus with retry — but for the duration of the outage, no dashboards show annotations, and once the cluster recovers you get a burst of backfill. Size the ingest side (Firehose buffer, cluster indexing capacity) for that burst, not just the steady-state rate.
  • Overlay noise. It’s very tempting to add every possible event to every panel. Don’t. Panels that show latency want deploys and config changes; panels that show error rates want deploys and incidents; utility panels want nothing. Curate annotation queries per panel or per row, not blanket at the dashboard level.

Why this ended up worth the plumbing

A dashboard that draws deploys and config pushes on the same panel as latency turned “root cause the p99 spike” from a 20-minute cross-tool timestamp hunt into a five-second visual scan. That’s the whole return on the setup — everything else is scaffolding.

If you’re starting fresh and only need one region, skip the fan-in and put OpenSearch next to your one EventBridge bus. The design above is what you graduate to the moment you have multi-region workloads and someone asks a cross-region “when did what happen” question.