Blog
Notes on observability, distributed systems, and software engineering. RSS.
2026
- Regional EventBridge, singleton OpenSearch, Grafana annotations 07-06
How to fan events from workloads in every region into a single OpenSearch cluster and overlay them as annotations on Grafana panels — so a deploy, a config push, or an incident shows up as a vertical line next to the metric it moved.
- Three Terraform escape hatches: the `external` provider, parallelism control, and beating the argv limit 07-01
Three patterns that come up over and over when Terraform doesn't quite fit the situation you're in — shelling out for lookups your provider can't do, controlling concurrency to avoid hammering upstream APIs, and routing large inputs around the OS argument-length limit.
- Streaming CloudWatch metrics to Chronosphere via Firehose + Lambda 06-30
Why we replaced the prometheus_cloudwatch_exporter / YACE polling pattern with CloudWatch Metric Streams, what the Lambda transform step actually does, and how the staleness and timeout knobs interact.
- Cross-account Prometheus replication with end-to-end mTLS via Envoy 06-29
How to stream Prometheus metrics from one AWS account to another over the public internet, with Envoy doing mTLS termination on both sides and a NAT-pinned IP allow-list as the outer fence.
- How ServiceMonitor can quietly overload your kube-apiserver 06-28
Adding a ServiceMonitor looks innocent — at scale it sets off a chain reaction through EndpointSlices, Istio XDS, and kube-proxy that can take down your control plane.