Self-Hosting Observability on a Raspberry Pi — Part 1: The Architecture

Before any tools, one idea. The single most useful thing I internalised about observability is that almost everything you'd ever want to watch falls into one of two buckets.

Metrics are numbers over time. CPU at 41%, temperature at 58°C, 1.2 GB of RAM in use, 340 requests in the last minute. A metric is a measurement you sample again and again, and its natural shape is a line on a graph.

Logs are text lines emitted by programs. container nginx started, connection refused, OOM killed pid 4821. A log is an event with words in it, and its natural shape is a scrolling stream you search through.

That's the whole mental model:

metrics  →  numbers over time   →  "is it healthy right now?"
logs     →  text lines          →  "what actually happened?"

They answer different questions, so they want different storage. Numbers compress beautifully when you store them as time series; text does not. This single distinction is why the stack has the shape it does — one path for numbers, one path for text, and one place to look at both.

Nothing on the Pi advertises its own vitals by default. Something has to stand at the door and report them. These reporters are called exporters, and each one has exactly one job.

node_exporter exposes the *host's* vitals — CPU, RAM, disk, network, and (the one I actually care about on a Pi) temperature. It serves them as plain text on port :9100. Hit that endpoint and you get a wall of numbers like node_cpu_seconds_total and node_hwmon_temp_celsius.

cAdvisor does the same trick one level up, for *containers*. It watches the Docker runtime and reports CPU and memory per container, so I can tell which one is eating the box.

So now the Pi has two endpoints quietly publishing numbers. But publishing isn't storing — refresh the page and the old values are gone. Something needs to come along, read these endpoints on a schedule, and remember.

That something is Prometheus.

Prometheus is a time-series database, but the part worth understanding is *how it gets its data*. It pulls.

Most databases sit and wait for you to push writes at them. Prometheus is the opposite: you hand it a list of endpoints, and every 15 seconds it goes out and *scrapes* each one — reads the current numbers and files them away with a timestamp. The exporters are passive; Prometheus does the walking.

every 15s:
  Prometheus → GET node_exporter:9100/metrics
  Prometheus → GET cadvisor:8080/metrics
  (store each sample with a timestamp)

This pull model is why adding a new thing to monitor is just adding one line to Prometheus's config — the new target doesn't need to know Prometheus exists. It just has to leave its front door open.

Once the data is stored, you ask questions in PromQL, Prometheus's query language. Something like rate(node_network_receive_bytes_total[5m]) means "how fast are bytes arriving, averaged over the last five minutes." PromQL gets its own post later — here it's enough to know it's the language for turning stored numbers into answers.

That's the numbers half of the system, end to end. Now for the text.

The obvious way to handle logs is a full-text search engine — index every word so you can search anything. The problem is that a full-text index of every log line is *enormous*, and a Raspberry Pi has a few gigs of RAM and an SD card with feelings.

Loki takes a cheaper bet. Instead of indexing the *content* of every log line, it only indexes a few small labels — things like container=nginx or host=pi5. The actual log text is just compressed and stored in chunks.

heavyweight engine:  index every word of every line   → huge, RAM-hungry
Loki:                index only labels, compress text  → cheap, Pi-friendly

The trade-off is that you can't instantly search arbitrary words across all of history — you first narrow down by label, *then* grep within. For a homelab that's exactly the right shape: "show me nginx's logs from the last hour" is fast and tiny.

Loki doesn't go collect logs itself, though. That's Promtail's job — the agent that tails the container logs and the system journal (journald), attaches those labels, and ships the lines to Loki.

Loki's query language is LogQL, which deliberately rhymes with PromQL. It too gets its own deep-dive later.

Notice the symmetry: exporters feed Prometheus, Promtail feeds Loki. Two parallel pipelines, one for numbers, one for text.

Now I have two databases — Prometheus full of numbers, Loki full of text. I don't want to learn two web UIs and stitch them together in my head.

Grafana is the single pane of glass. The key thing to understand about Grafana is what it *doesn't* do: it stores no data of its own. It is purely a viewer.

You point Grafana at Prometheus and Loki as data sources. When you open a dashboard, Grafana fires PromQL at Prometheus and LogQL at Loki, gets the results back, and renders them — graphs of temperature next to a live tail of logs, on the same screen.

exporters ─┐
           ├─► Prometheus ─┐
Promtail ──┘               ├─► Grafana (renders, stores nothing)
           ┌─► Loki ───────┘

So the data flows one way and pools in two databases; Grafana reaches into both only when you're looking. Every box so far — node_exporter, cAdvisor, Prometheus, Loki, Promtail, Grafana — runs as a Docker container, all defined together in a single docker-compose.yml. One file describes the whole observability stack, and docker compose up brings it to life.

The Pi can now see itself. The last problem is letting *me* see it — safely.

The naive way to view Grafana from outside my flat is port forwarding: tell the home router "send anything on port 443 to the Pi." It works, and it's also how you end up with your home IP address published and a permanently open port for the whole internet to knock on. No thanks.

Instead I run a Cloudflare Tunnel on the Pi. The move here is sneaky and lovely: the tunnel dials *outbound* to Cloudflare and holds that connection open. Nothing ever connects *in*.

Router with open port (port-forward):   internet ──► [OPEN :443] ──► Pi
Cloudflare Tunnel:                       Pi ──► (outbound, held open) ──► Cloudflare

When you visit https://grafana.imashish.dev, you hit Cloudflare, not my house. Cloudflare terminates TLS (so HTTPS is automatic and I never touch a certificate) and pushes your request *back down* that already-open outbound tunnel to the Pi's local Grafana.

The wins stack up:

• No open inbound ports on my router.
• My home IP stays hidden behind Cloudflare.
• HTTPS handled for me, for free.

The Cloudflare Tunnel — how it's configured and what it's actually doing under the hood — is a post of its own later in the series.

Exposing a dashboard to the internet means deciding, carefully, what a stranger is allowed to do.

First, Grafana is bound to localhost on the Pi. It doesn't even listen on the local network — only the Cloudflare Tunnel, running on the same machine, can reach it. A laptop on my home Wi-Fi can't hit Grafana directly; everything goes through the front door I control.

Second, the public never logs in. What I expose are Grafana's Public Dashboards — render-only links with:

• no login,
• no ad-hoc queries (nobody can type their own PromQL or LogQL),
• no access to the logs at all.

The admin panel — where you *can* run queries, edit dashboards, and read logs — stays behind a password and is only reachable through the tunnel.

public visitor   →  read-only dashboard  (numbers only, no login, no queries)
me (admin)       →  full Grafana          (password, queries, logs)

So the surface area facing the internet is tiny on purpose: a handful of pre-baked graphs, and nothing else.

Here's everything at once. Read it top to bottom: data is collected, stored, viewed, and tunnelled out.

            ┌──────────────────────── Raspberry Pi 5 ────────────────────────┐
            │                                                                │
            │   HOST VITALS            CONTAINER VITALS         LOGS         │
            │   node_exporter:9100     cAdvisor:8080            Promtail     │
            │        │                      │                      │         │
            │        └───────┬──────────────┘                      │         │
            │                ▼                                     ▼         │
            │         ┌─────────────┐                       ┌───────────┐    │
            │         │ Prometheus  │  (scrapes every 15s)  │   Loki    │    │
            │         │ time series │                       │   logs    │    │
            │         └──────┬──────┘                       └─────┬─────┘    │
            │          PromQL│                              LogQL │          │
            │                └───────────────┬─────────────────  ┘          │
            │                                ▼                               │
            │                         ┌─────────────┐                       │
            │                         │   Grafana   │  (stores nothing)     │
            │                         │ localhost   │                       │
            │                         └──────┬──────┘                       │
            │                                ▼                               │
            │                      ┌───────────────────┐                    │
            │                      │ Cloudflare Tunnel │  (dials outbound)  │
            │                      └─────────┬─────────┘                    │
            └────────────────────────────────│──────────────────────────────┘
                                             ▼
                                       ☁  Cloudflare  (TLS, hides home IP)
                                             ▼
                                   https://grafana.imashish.dev
                                             ▼
                                      you, on your phone

Every box is a single-purpose container. The numbers go up the left, the logs up the right, they meet in Grafana, and the whole view leaves the building through one outbound tunnel.

That's the architecture — the map, not the territory. Each box above is a small story of its own, and the interesting details live *inside* them.

So this is Part 1 of a series. Coming up:

• Prometheus & PromQL — the pull model up close, and how to actually ask useful questions of your numbers.
• Loki & LogQL — label-based indexing, why it's cheap, and querying logs without a full-text engine.
• The Cloudflare Tunnel — configuring it end to end, and what's really happening on that outbound connection.
• A war story: the nasty Docker IPv6 bug — the one where containers could reach the internet but not each other, and the evening I spent staring at docker compose wondering why.

If you take one thing from Part 1, make it the two-bucket model: numbers over time, and text lines. Everything else in the stack is just plumbing around that one distinction. See you in Part 2.