Streaming CloudWatch metrics to Chronosphere via Firehose + Lambda

If you’re running anything serious on AWS and want CloudWatch metrics in the same place as the rest of your observability stack, you’ve probably gone through this evolution at some point:

1. Stand up prometheus/cloudwatch_exporter (or YACE, which is just a Go rewrite of the same idea). It polls CloudWatch’s APIs and exposes metrics in Prometheus format.
2. Notice that dashboards are inexplicably 5+ minutes behind reality.
3. Wonder if there’s a knob.
4. Discover the knob isn’t really a knob — it’s a property of how CloudWatch returns metric data.

This is a write-up of what we replaced that pattern with: AWS CloudWatch Metric Streams pushing into Kinesis Firehose, with a small Lambda function in the middle to fix up labels, delivering to Chronosphere as the metrics backend. End-to-end latency dropped from “5+ minutes” to “about 1 minute,” and the operational story got noticeably simpler.

Why polling exporters lag

prometheus/cloudwatch_exporter and YACE work the same way at the core: they call CloudWatch’s GetMetricData API on a schedule, ask for a range of recent timestamps, and translate the result into Prometheus metrics.

The issue isn’t the exporter — it’s CloudWatch. CloudWatch publishes metrics with a timestamp aligned to when the data was generated, but the data takes some time to actually appear in the API. The recommended behavior, per AWS’s own documentation for GetMetricData, is to query for data points that are at least 3–5 minutes old for standard resolution metrics, and even longer for some namespaces. If you query “now,” you get holes; if you query “5 minutes ago,” you get stable numbers.

So both exporters effectively buffer the present by 5+ minutes to get clean data. That delay shows up everywhere downstream:

• Dashboards are stale, which is annoying but tolerable.
• Alerts fire late, which is the actually-bad part. A spike at 12:00 becomes an alert at 12:07.
• Auto-remediation has a doubled feedback loop — by the time the alert fires and the auto-action triggers, the underlying state is already five minutes past whatever the alert was about.

Tuning the exporter’s polling window forward (closer to “now”) trades freshness for correctness: you start getting null values, gaps, and sometimes silently-wrong dashboards. There’s no good place to sit on that curve.

The streaming alternative

CloudWatch Metric Streams flip the model. Instead of you polling CloudWatch, CloudWatch pushes metrics to a Kinesis Firehose as soon as they’re aggregated. The publication latency from “metric was generated” to “metric is in the stream” is about a minute, not five.

Firehose then handles batching, retries, and delivery to a downstream destination. Chronosphere accepts the OpenTelemetry-formatted payload that Metric Streams produces, so the path is direct — with one stop in the middle for label normalization.

A few things worth calling out about the diagram:

• Lambda is invoked by Firehose, not the metric stream. The metric stream produces a stream of records; Firehose buffers a batch and then synchronously invokes Lambda on the buffered records. Lambda returns the transformed records, and Firehose forwards them to Chronosphere.
• S3 is the backup, not the primary destination. Firehose can be configured to drop failed records (and optionally the originals before transform) into an S3 bucket so nothing is lost when the downstream is unhappy. Chronosphere is the only primary destination.
• Everything past the metric stream is a Firehose-managed flow. You don’t manage delivery to Chronosphere directly; Firehose handles buffering, retries, backoff, and IAM-signed HTTP delivery.

Setting up the stream

The simplest possible Terraform for the stream + Firehose looks roughly like this:

resource "aws_cloudwatch_metric_stream" "to_chronosphere" {
  name          = "metrics-to-chronosphere"
  role_arn      = aws_iam_role.metric_stream.arn
  firehose_arn  = aws_kinesis_firehose_delivery_stream.metrics.arn
  output_format = "opentelemetry1.0"

  # Filter to the namespaces you actually care about — every metric
  # update sent costs money.
  include_filter {
    namespace = "AWS/EC2"
  }
  include_filter {
    namespace = "AWS/RDS"
  }
  include_filter {
    namespace = "AWS/Lambda"
  }
}

resource "aws_kinesis_firehose_delivery_stream" "metrics" {
  name        = "metrics-to-chronosphere"
  destination = "http_endpoint"

  http_endpoint_configuration {
    url                = var.chronosphere_ingest_url
    name               = "chronosphere"
    access_key         = var.chronosphere_api_key   # stored in SSM/Secrets Manager
    buffering_size     = 4   # MB
    buffering_interval = 60  # seconds

    s3_backup_mode = "FailedDataOnly"

    processing_configuration {
      enabled = true
      processors {
        type = "Lambda"
        parameters {
          parameter_name  = "LambdaArn"
          parameter_value = "${aws_lambda_function.metric_label_rewriter.arn}:$LATEST"
        }
      }
    }
  }

  s3_configuration {
    role_arn   = aws_iam_role.firehose.arn
    bucket_arn = aws_s3_bucket.metric_failures.arn
    prefix     = "failed/"
  }
}

Two things worth slowing down on:

• output_format = "opentelemetry1.0" — the metric stream emits records in OTel protobuf. Chronosphere accepts this natively. JSON output is also supported, but it’s larger on the wire and slower to decode in the Lambda.
• include_filter — every metric update sent is billed by AWS. By default a metric stream would send literally everything in CloudWatch, which is both very expensive and full of metrics nobody asked for. Explicit inclusion is cheaper and intentional.

What the Lambda actually does

A metric stream record looks roughly like this when decoded (simplified JSON view, the wire is protobuf):

{
  "metric_name": "CPUUtilization",
  "namespace": "AWS/EC2",
  "timestamp": "2026-06-30T14:23:01Z",
  "value": { "average": 17.4, "sum": 87.0, "max": 21.1, "min": 11.0, "count": 5 },
  "unit": "Percent",
  "dimensions": {
    "InstanceId": "i-0abc1234567890def",
    "AutoScalingGroupName": "asg-frontend"
  },
  "stream_name": "metrics-to-chronosphere"
}

That’s not directly useful as a Prometheus metric. A few things need to happen before it goes into Chronosphere:

1. Normalize naming. AWS/EC2.CPUUtilization → aws_ec2_cpu_utilization. This is just a string transform but it makes downstream queries way nicer ({__name__=~"aws_ec2_.*"} is a useful shape).
2. Normalize label names. InstanceId → instance_id, AutoScalingGroupName → asg_name. Same idea, applied to dimensions.
3. Drop noisy labels. Some AWS-emitted dimensions are not what you want in your TSDB — high-cardinality identifiers that explode the series count without adding value.
4. Add enrichment labels. You usually want a team or service label sourced from resource tags or an internal lookup. The Lambda is where that join happens.

A skeleton Lambda (Python) looks like:

import base64
import gzip
import json
from typing import Iterable

# Maps AWS dimension names → desired label names.
LABEL_MAP = {
    "InstanceId": "instance_id",
    "AutoScalingGroupName": "asg_name",
    "DBInstanceIdentifier": "db_instance",
    "FunctionName": "function_name",
}

# Dimensions we never want in the TSDB.
DROP_LABELS = {"AccountId"}

def normalize_metric_name(namespace: str, name: str) -> str:
    # "AWS/EC2", "CPUUtilization" → "aws_ec2_cpu_utilization"
    ns = namespace.lower().replace("/", "_")
    m = "".join(["_" + c.lower() if c.isupper() else c for c in name]).lstrip("_")
    return f"{ns}_{m}"

def transform(record: dict) -> dict | None:
    record["metric_name"] = normalize_metric_name(
        record["namespace"], record["metric_name"]
    )
    new_dims = {}
    for k, v in record.get("dimensions", {}).items():
        if k in DROP_LABELS:
            continue
        new_dims[LABEL_MAP.get(k, k.lower())] = v
    record["dimensions"] = new_dims
    return record

def handler(event, _ctx):
    output = []
    for r in event["records"]:
        try:
            raw = base64.b64decode(r["data"])
            # In real life, decode OTel protobuf here. JSON shown for clarity.
            records = (json.loads(line) for line in raw.decode().splitlines() if line)
            transformed = [transform(rec) for rec in records]
            output_payload = "\n".join(json.dumps(t) for t in transformed if t)
            output.append({
                "recordId": r["recordId"],
                "result":   "Ok",
                "data":     base64.b64encode(output_payload.encode()).decode(),
            })
        except Exception:
            # Mark unparseable records as failed so Firehose routes them
            # to the S3 backup bucket. Do NOT raise — that would fail the
            # entire batch.
            output.append({
                "recordId": r["recordId"],
                "result":   "ProcessingFailed",
                "data":     r["data"],
            })
    return {"records": output}

Two things in this code that are easy to get wrong:

• The return contract is strict. Every input record needs an output record with the same recordId and a result of Ok, Dropped, or ProcessingFailed. Missing records cause Firehose to error the batch. Mismatched IDs cause silent corruption.
• Catch broadly inside the loop, never raise. A single bad record crashing the function fails the entire batch, which gets retried, which (depending on the metric set) might fail again, which can grind the pipeline to a halt. Catch per-record and surface the failure as a status; let Firehose’s backup do its job.

Staleness and the timeout web

This is the part that took the longest to get right. There are at least four independent timeout/buffer knobs interacting:

Knob	Owner	Default	Effect
Firehose buffering_interval	Firehose	60s	Max delay between record arrival and Lambda invocation
Firehose buffering_size	Firehose	5 MB	If batch fills first, invoked earlier
Firehose-Lambda timeout	Firehose	60s	If Lambda hasn’t returned within this, batch is failed
Lambda function timeout	Lambda	configurable, up to 15 min	If Lambda exceeds this, the function is killed

The relationship between these matters a lot:

• Firehose’s Lambda invocation timeout is 60 seconds and not configurable. If your Lambda runs longer than that, Firehose treats the batch as failed regardless of what Lambda itself thinks. So setting the Lambda function timeout to 15 minutes doesn’t buy you anything — Firehose will have already given up at 60s.
• Lambda’s own timeout should be set somewhere below 60s — we use 30s — so that you get a clean Lambda-level error (visible in the Lambda metrics) instead of a Firehose-level “no response” failure (which is harder to attribute).
• buffering_interval is the main staleness lever. Setting it to 60s means metrics are at worst 60s + Lambda time + delivery time behind real time. Setting it lower than 60s isn’t supported by Firehose. Setting it higher trades freshness for fewer Lambda invocations.
• buffering_size is the cost lever. A small buffer triggers more Lambda invocations (and Lambda costs money per invocation). A large buffer means bigger payloads, more memory in Lambda, and potentially hitting Firehose’s per-record-size limits if the batch decompresses to something huge.

Practical tuning: we landed at buffering_interval=60, buffering_size=4MB, Lambda timeout 30s, Lambda memory 512 MB. That’s a ~90-second worst-case end-to-end (metric generation → Chronosphere) at the budget where Lambda costs are dominated by metric-stream costs rather than the other way around.

What you give up by switching

It would be dishonest to say streaming is unambiguously better. A few things you lose, or have to handle differently:

• Re-ingestion of historical data. Metric stream is a stream — you can’t ask it “give me yesterday’s metrics again.” If your downstream is down for an hour, you’ve lost that hour (well — almost: the S3 backup catches failed records, and you can replay from there).
• Per-metric polling control. The polling exporters let you choose which metric to fetch at what resolution at what aggregation. Metric Streams give you everything for the namespaces you include. For some teams that’s the wrong default.
• Cost shape. Polling exporters cost roughly $0 per metric in CW fees (API calls, but those are usually free at low rates). Metric Streams charge per metric update sent. At low scale streams are cheaper; at very high scale streams can be more expensive than carefully-filtered polling. Worth modeling against your actual metric volume.
• Operational complexity. Polling exporter = one process. Streaming = metric stream + Firehose + Lambda + S3 + IAM + Chronosphere ingest config. More moving parts, more to monitor, more to debug.

When polling is still the right answer

Three cases where I’d stick with prometheus/cloudwatch_exporter or YACE:

1. You have a small set of metrics from a small set of services. If the total namespace surface is manageable and you don’t need sub-five-minute reaction times, the operational simplicity of a single exporter pod wins.
2. You need on-demand re-queries. Some teams use the exporter for debugging — “show me CPU on these 10 instances over the last 6 hours, right now.” Streaming-into-a-TSDB is great for the long-term record but worse for ad-hoc backfill questions.
3. You can tolerate the latency. Cost dashboards, capacity-planning metrics, finance-team views — none of these care about 5-minute freshness. Don’t over-engineer something that doesn’t need it.

The summary

CloudWatch Metric Streams + Firehose + Lambda + Chronosphere is a strictly fresher pipeline than the polling-exporter pattern, at the cost of some operational complexity:

• Latency: ~1 minute streaming vs. 5+ minutes polling. Alerts fire closer to real time, which is the only really important thing in this whole post.
• The Lambda step exists to bridge AWS’s metric model with Prometheus conventions. Renames, drops, and tag enrichment all happen there. It must be defensive about per-record errors and stay well inside Firehose’s 60s invocation budget.
• Staleness tuning is mostly the buffering_interval knob, with Lambda timeout sitting underneath the Firehose invocation timeout to surface failures cleanly.
• S3 backup is your safety net for transient downstream failures. Set it; it’s free until you need it.
• Don’t switch if you don’t need the freshness. The polling exporters still earn their place when the answer to “how fast” is “not very.”