Prometheus Behind NAT: Scraping Without Open Ports

Prometheus pulls. It sends an HTTP GET to your machine and reads the response. Simple — until your machines sit in private subnets across three clouds, with no public IPs and security groups you’d rather not touch.

The usual answers are bad: give every box a public IP, or build a VPN mesh, or run a Prometheus per cluster and federate.

There’s a fourth option. The machine dials out and holds the line open. Here’s the whole path, step by step, and the one mental model that makes it click.

The setup

Three pieces:

1. node_exporter on the machine, bound to 127.0.0.1:9100 — loopback only, unreachable from outside

2. A tunnel client (Cloudflare Tunnel, ngrok, frp — same idea) that dials out and stays connected

3. Prometheus, somewhere else entirely

The tunnel provider stores one rule: requests for metrics-host.example.com go to http://127.0.0.1:9100 on this machine.

When nobody is looking

1
2
3

machine ═══════════════════════════> tunnel edge
        one idle outbound connection
        traffic: ~0 (heartbeats only)

This is the part people get wrong. The machine is not sending metrics anywhere. It has dialed out, and it is holding an empty line open. That’s all.

There is no buffer, no queue, no upload. If nobody scrapes for an hour, nothing is transmitted for an hour.

The scrape, step by step

Every 30 seconds:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

① Prometheus ──GET https://metrics-host.example.com/metrics──► tunnel edge
                                                                    │
② edge checks auth (service token / mTLS / whatever you configured) │
                                                                    ▼
③ edge pushes the request INTO the connection the machine already opened
                                                                    │
                                                                    ▼
④ tunnel client ──GET http://127.0.0.1:9100/metrics──► node_exporter
                                                                    │
⑤ node_exporter opens /proc/meminfo, /proc/stat …                  │
   computes the numbers RIGHT NOW, returns plain text               │
                                                                    │
⑥ text travels back the same way ◄─────────────────────────────────┘

⑦ Prometheus stores the values with a timestamp

Every arrow from ① to ⑥ is driven by that one request. Nothing moves on its own.

The mental model: phone line, not mailbox

The edge holds configuration — where to route, who’s allowed. It holds zero metrics. It’s a switchboard, not a warehouse.

Proof: the numbers don’t exist until you ask

Scrape the same metric twice, one second apart:

1
2

node_memory_MemFree_bytes 2.419135619072e+12
node_memory_MemFree_bytes 2.419087740928e+12    ← different

If node_exporter were serving a stored file, these would match. They don’t, because each request triggers a fresh read of /proc/meminfo:

1
2

/proc/meminfo →  MemFree: 2362365080 kB
                 2362365080 × 1024 = 2.41906e12 bytes   ✓

node_exporter opens the file, reads a line, converts kB to bytes, responds, closes. It holds no state and keeps no history.

Metrics are not logs

Worth separating, because the words get mixed up constantly:

node_exporter only ever knows this instant. The curve you see in Grafana is assembled by Prometheus, which asks every 30 seconds and keeps the answers. Nothing on the machine remembers yesterday.

And it isn’t reading “files” in any normal sense. /proc and /sys are pseudo-filesystems — kernel state dressed up as files. The paths are Linux standards, hardcoded in the exporter. No configuration, no discovery.

Why this beats the alternatives

The machine needs exactly one thing: outbound HTTPS. That’s it.

It works identically on AWS, on GCP, on a rented GPU box, on a machine in a restricted region behind aggressive egress filtering. The cloud provider becomes irrelevant, which is the real win when your fleet spans several.

It also inverts the security story. There is no listening port to find, so there is no port to scan. Access control moves to the edge, where you configure it once instead of maintaining security-group rules per machine.

The trade-offs

Not free:

• The edge is now a dependency. Edge down, or tunnel client crashed, means the target looks down — indistinguishable from an actually-down machine. Alert on the tunnel process itself.

• Latency is higher. You’re routing through a third party. Fine at a 30s scrape interval, wrong for sub-second polling.

• The connector token is a credential. Anyone holding it can run a connector for your tunnel. Treat it like an SSH key — and don’t let it end up in a ps output you paste somewhere.

• You’re trusting the middle. Traffic is decrypted and re-encrypted at the edge. For host metrics, usually acceptable. Decide deliberately.

Takeaway

Pull-based monitoring and locked-down networks aren’t actually in conflict. You just have to stop thinking of the connection direction as fixed to the data direction.

The machine dials out. The data flows back in — but only when asked, and only after being measured on the spot.

Once that clicks, the rest follows: nothing is buffered at the edge, nothing is stored on the machine, and every number you see in a dashboard was computed during a phone call that lasted a few milliseconds.

If you’re sizing the other half of this — how often to scrape, how much a queue backs up — Little’s Law is the arithmetic worth knowing.

Suggested image: two panels — an idle phone line between a house and a switchboard (nobody talking), then the same line mid-call with a speech bubble reading “how much free memory right now?”. Alt text: “A reverse tunnel is a phone line, not a mailbox.”

• Next Post →