Three Terraform escape hatches: the `external` provider, parallelism control, and beating the argv limit

Terraform is mostly fine until it isn’t. The three patterns below are ones I’ve reached for more than once when the stock model didn’t fit:

The external data source for one-off reads against a system that has no provider, or whose provider doesn’t expose the lookup you need.
Parallelism control when your plan or apply hammers an external API hard enough to trip rate limits.
Routing large inputs around the OS argv limit when your module’s input map gets big enough to overflow the command line.

They’re not really features so much as escape hatches around features. Each is easy to misuse, so the post leans hard on the failure modes.

1. The `external` data source

The external provider is a single data source that shells out to a script you write and reads JSON back. The contract is small:

You give it a program to run.
You optionally pass a query — a flat string-to-string map.
The program reads JSON (the query) from stdin.
The program writes JSON (a flat string-to-string map) to stdout.
A non-zero exit code = data source failure = plan failure.

That’s it. The whole shape fits in a paragraph and that’s deliberate — it’s an escape hatch, not a programming model. If you need richer types or lifecycle hooks, you’re supposed to write a real provider.

A concrete example: soft-checking PagerDuty for a service

Suppose you have a module that wires services into a runtime service-graph representation. One of the optional fields is a reference to a PagerDuty service ID. Some services have a PD entry, many don’t. You want the module to include the PD ID if the service exists, and silently fall back to nothing otherwise.

The PagerDuty provider’s pagerduty_service data source is strict — if the service doesn’t exist, the data source errors and your plan dies. That’s the correct behavior for a provider; it’s the wrong behavior for “find X if it exists.” external is good at that gap.

data "external" "pd_service" {
  program = ["${path.module}/scripts/check_pd_service.sh"]
  query = {
    service_name = var.service_name
    token_env    = "PAGERDUTY_API_TOKEN"
  }
}

locals {
  pd_exists     = data.external.pd_service.result.exists == "true"
  pd_service_id = data.external.pd_service.result.service_id

  dependencies = concat(
    var.base_dependencies,
    local.pd_exists ? [local.pd_service_id] : [],
  )
}

resource "service_graph_node" "this" {
  name         = var.service_name
  dependencies = local.dependencies
}

The script:

#!/usr/bin/env bash
set -euo pipefail

input=$(cat)
service_name=$(jq -r '.service_name' <<<"$input")
token_env=$(jq -r '.token_env'    <<<"$input")
token=$(printenv "$token_env" || true)

[[ -n "$token" ]] || { echo "token env '$token_env' not set" >&2; exit 1; }

response=$(curl -sS \
  -H "Authorization: Token token=$token" \
  -H "Accept: application/vnd.pagerduty+json;version=2" \
  "https://api.pagerduty.com/services?query=$(jq -rn --arg n "$service_name" '$n|@uri')")

match_id=$(jq -r --arg name "$service_name" \
  '.services[] | select(.name == $name) | .id' <<<"$response" \
  | head -n1)

if [[ -n "$match_id" ]]; then
  jq -n --arg id "$match_id" '{exists: "true",  service_id: $id}'
else
  jq -n                       '{exists: "false", service_id: ""}'
fi

A few things in there that are easy to skip on the first draft:

set -euo pipefail. Without it, a partial failure inside the script silently propagates. The whole point of external is that exit codes mean something.
Exact-name filtering in jq, not in the URL ?query=. PD’s query parameter does substring matching, which will match every service whose name contains your needle. Filter for exact equality client-side.
Consistent output schema across both branches. Both the “exists” and “doesn’t exist” branches return the same keys. If they don’t, Terraform throws an “inconsistent attribute set” on the branch that’s missing a key.
Secret passed by env-var name, not value. Anything in query is logged at TRACE and can show up in error messages. Passing the name and reading the value inside the script keeps the token out of Terraform’s surface.

The failure modes that catch people

In rough order of pain:

Runs on every plan. No caching. A 5-second script adds 5 seconds to every plan, in every environment, every time. Multiply by module instance count.
Output drift breaks idempotency. If your script returns different output for the same input (because, e.g., the upstream API returns results in non-deterministic order), every plan shows a diff in the downstream resources. Make the script deterministic: sort, pick the lowest ID on ties, hash on stable keys.
Conflating “found nothing” with “couldn’t ask.” Empty result is a valid output ({exists: "false"}); auth failure or network error is an exit-non-zero. First-draft scripts collapse those and end up catastrophic on upstream outages.
Secrets in query get logged. Pass env-var names, not values.
State cache surprise. The data source result is cached in state. If the upstream changes without your inputs changing, the next plan uses the stale cached belief. Handle the “doesn’t exist anymore” case at the consumer.

The rule of thumb: external is for reads, with deterministic output, that are cheap enough to run on every plan. If any of those three is shaky, it’s the wrong tool.

2. Parallelism to avoid rate-limiting your upstream

By default, terraform apply and terraform plan work the dependency graph with -parallelism=10: up to ten resources or data sources in flight at once. That’s a reasonable default for most graphs because most resources don’t share an external dependency.

The trouble starts when many resources or data sources all do share one. Three common shapes:

A module that’s instantiated dozens of times, each with an external data source that calls the same internal API.
A provider that doesn’t internally rate-limit, paired with a resource type whose upstream does. The classic one: cloud APIs that throttle aggressively per-account.
A data source that does an expensive query (a database, a slow API gateway, a service mesh discovery endpoint) where ten concurrent queries are enough to push tail latency through the roof.

You’ll see one of two failure shapes:

Hard failures — 429 Too Many Requests, ThrottlingException, RateLimitExceeded, an explicit refusal. These at least tell you what’s going on.
Soft failures — timeouts, intermittent 5xx, partial responses, weird “the resource was created but a follow-up read returned 404” errors. These are worse because they look like a flaky upstream instead of a self-inflicted thundering herd.

The two levers

The graph-wide lever is the -parallelism flag:

terraform apply -parallelism=3
terraform plan  -parallelism=3

That caps all concurrent operations at three. For a graph where the bottleneck resource is the only thing throttling, that’s correct but coarse — you’ve also slowed down every unrelated resource that would have been fine at the default.

The per-resource lever is provider-specific. Most providers expose some flavor of:

max_retries — how many times to retry on transient errors, typically with exponential backoff.
retry_mode or retry_delay — control the backoff curve.
request_timeout or similar — fail fast vs. wait it out.

provider "aws" {
  region      = "us-east-1"
  max_retries = 25   # default is much lower; bump for noisy graphs
}

If you’re using external data sources as the bottleneck, the most useful lever lives inside your script: implement client-side rate limiting and exponential backoff there, so even at -parallelism=10, each invocation respects the upstream’s limits.

A real example

A module that fans out to ~60 services, each running an external data source that calls an internal services API. At the default parallelism the API hits ~6 RPS in concentrated bursts, the upstream starts returning 429s on roughly a quarter of calls, and plans start failing nondeterministically — sometimes ten in a row succeed, sometimes three in a row fail.

There are three reasonable fixes, in increasing order of effort:

-parallelism=3 on the wrapper that runs Terraform. Trivial; slows the whole graph; works.
Per-script rate limiting in the external program — sleep between calls, respect Retry-After headers, exponential backoff on 429s. The right answer if the upstream is genuinely slow and you don’t want to slow the rest of the graph.
Cache outside Terraform. If the lookup is rarely-changing, run the lookups once on a schedule, commit results as a .tfvars file, and have Terraform read the file instead of calling the API each plan. The right answer if the per-plan cost is actually the problem.

We ended up doing (1) for the noisy module while we built (2) for the new modules going forward. Heuristic: if it’ll save more than a week of plan time across the team in the next month, it’s worth doing (2).

3. “argument list too long” — bypassing the OS argv limit

This one always comes as a surprise. Your wrapper around Terraform (Terragrunt, a custom script, an in-house Atlantis fork) tries to invoke terraform with a large set of inputs, and the OS refuses:

fork/exec /usr/local/bin/terraform: argument list too long

The cause is the kernel’s ARG_MAX limit on the combined size of a process’s arguments and environment. On Linux it’s typically ~128 KB; on macOS it’s higher but still finite. Both -var CLI flags and environment variables (the ones the wrapper sets to feed Terraform, like TF_VAR_…) count against it.

For most module inputs this is irrelevant — a handful of small values, maybe a few hundred bytes of env. But large inputs — service catalogs, host inventories, generated dependency graphs, anything from a spreadsheet or a service registry — can easily push past the limit. A few thousand entries of {name, owner, tags} is enough.

The fix is to stop passing large inputs through CLI flags or env vars at all. Terraform reads variables from several places, in increasing order of “doesn’t count against ARG_MAX”:

-var CLI flag — counts. (Worst for this problem.)
TF_VAR_<name> env variable — counts.
*.auto.tfvars / *.auto.tfvars.json file in the working directory — doesn’t count. Terraform reads it from disk.
-var-file=path CLI flag — the path counts but the file content doesn’t. Effectively unlimited.

The pattern, then: split your input map into small values you pass the normal way and large values you write to a JSON file the process reads from disk.

Here’s the shape, from a wrapper that takes a single big input map and prepares it for the underlying Terraform invocation:

# In whatever layer is computing the inputs for the wrapped module:
inputs = {
  for k, v in local.base.input : k => v
  if k != "services" && k != "service_dependencies"
}

tf_vars = {
  services             = local.base.input.services
  service_dependencies = try(local.base.input.service_dependencies, [])
}

What’s happening:

inputs is everything except the two known-large keys. These get passed to Terraform the normal way — through TF_VAR_* env vars or -var flags, depending on the wrapper. The remaining values are small, so this is fine.
tf_vars holds just the large values. The wrapper takes this map and serializes it to a file alongside the module — typically terraform.auto.tfvars.json or a -var-file-loaded file. Terraform reads it from disk, so it never touches ARG_MAX.

The filter is the load-bearing part. Without the if k != "services" && k != "service_dependencies", the two large keys would also flow through the env-var path and overflow it.

Generalizing the pattern

It’s the same shape in Terragrunt, in a hand-rolled wrapper, or in plain Terraform with a CI step that generates a .tfvars.json:

big input map
   │
   ├── small keys ──────────► env vars / CLI flags (subject to ARG_MAX)
   │
   └── large keys ──────────► file on disk (not subject to ARG_MAX)
                              └─► Terraform reads via -var-file or
                                  *.auto.tfvars.json discovery

Three things worth knowing:

The split needs to be by known large keys, not “everything over N bytes.” You don’t want the failure mode where a normally-small key occasionally crosses the threshold and your inputs land in a different place. Pick the keys that are categorically large (collections, generated graphs, registries) and route them to the file path always.
Generated .tfvars.json files should be .gitignored if the generation lives outside Terraform, and regenerated before every plan if the source might change. Stale file-based inputs are a common cause of “I changed the input but the plan didn’t notice.”
Don’t try to pass huge structures through -var '...' with JSON encoding as a workaround. You’ll still hit ARG_MAX, and now your command line is also unreadable in process listings.

The general lesson: any time a Terraform input could grow without bound (because it comes from a registry, a spreadsheet, or a generated graph), assume it will, and route it through a file from day one.

All three, in one breath

external when you need a lookup that no provider exposes — but only if the lookup is read-only, deterministic, and cheap to run on every plan.
-parallelism when the bottleneck is an external API and your graph fan-out is more than the upstream wants to take. Per-script backoff is the more surgical version of the same fix.
File-based variable inputs the moment a single input value could grow unbounded. The OS ARG_MAX limit is small enough that this comes up sooner than you’d expect.

None of these are exotic, but each one is the kind of thing you discover when something breaks at 2 a.m. and you’d rather have already known.

1. The external data source