I Built a NAS on a Raspberry Pi (And the exFAT Drive Tried to Stop Me)

Before Nextcloud, a word on how anything gets onto the Pi, because it shapes everything else.

I didn't want Kubernetes on a Raspberry Pi. I didn't want a big PaaS. I wanted the smallest thing that would stop me from SSHing in and editing YAML by hand at midnight. So I wrote a tiny convention I call pi-apps, and it's barely a framework at all.

Each service is just a folder. Inside: a docker-compose.yml, a gitignored .env for secrets, and that's mostly it.

pi-apps/
├── nextcloud/
│   ├── docker-compose.yml
│   ├── .env            (gitignored — db passwords, etc.)
│   └── deploy.sh
├── monitoring/
│   └── docker-compose.yml
└── learning-board/
    └── docker-compose.yml

A deploy.sh rsyncs the folder up to the Pi and runs docker compose up -d. That's the entire deploy pipeline. No CI, no registry, no Helm. rsync and one SSH command.

The one piece of glue that matters: every service joins a shared Docker network called piapps. That lets containers reach each other by name — Nextcloud talks to mariadb and redis as hostnames, no IP addresses anywhere.

networks:
  piapps:
    external: true

And the rule I'm strictest about: internal-only services publish no ports. MariaDB, Redis, Mongo — none of them map a port to the host. They're reachable on the piapps network and nowhere else. If a container doesn't need to be on the internet, it isn't even on the LAN. The only things with published ports are the handful that the Cloudflare Tunnel actually points at.

Here's the full picture before we zoom into the interesting parts. The hardware is a Raspberry Pi 5, 8GB RAM, arm64, running Debian 13 (trixie), with the 3TB USB HDD hanging off it.

Here's the whole thing at a glance:

        you & family  (phones, laptops)
                 |  HTTPS
                 v
        Cloudflare edge   --  TLS, WAF, rate-limit
                 |  (the Pi dials OUT; the tunnel stays open)
                 v
        cloudflared  --  on the Raspberry Pi 5, Debian 13, docker net "piapps"
                 |
        +--------+---------------------+
        v        v                     v
   Nextcloud   Grafana          Learning Board (kanban)
      |  |                             |
      v  v                             v
   MariaDB  Redis                    MongoDB
   (the DB) (file locks)

   nightly backup  (keeps the last 7 days):
        mariadb-dump  ->  MariaDB  -+
        mongodump     ->  MongoDB  -+--->  archived onto the 3TB drive

   monitoring:  Prometheus  <-  cAdvisor, node_exporter, blackbox
                Loki        <-  Promtail            both  ->  Grafana

   user files:  Nextcloud  ->  External Storage (Local)  ->  3TB USB HDD (exFAT)

Read it as three jobs running on one box: the NAS (Nextcloud + MariaDB + Redis + the 3TB drive), a kanban app (Learning Board on MongoDB, with its own backup), and the monitoring stack watching all of it. Everything reaches the outside world through one outbound Cloudflare Tunnel.

Here's the trap, and I walked straight into it.

Nextcloud wants to put its data directory on disk — the place where it stores user files, app data, and its own bookkeeping. The instinct is: I have a 3TB drive, point Nextcloud's data directory at the 3TB drive, done.

That instinct is wrong on exFAT, and it'll bite you quietly.

The one-line version: ext4 is Linux's native filesystem — real ownership, permissions, and flock locking baked in; exFAT is a Microsoft format for SD cards and USB sticks that any OS can read, but with none of that underneath. That portability is exactly what costs you the two things Nextcloud leans on hard:

• Unix permissions. Every file on an exFAT drive shows up as mode 0777, owned by whoever mounted it. There's no real chmod. You cannot make a file private at the filesystem level — the bits just don't exist.
• `flock`. No POSIX file locking. Nextcloud uses locking constantly to stop two processes from mangling the same file. On exFAT, those locks are unreliable or absent.

So the design I landed on splits the data across two filesystems on purpose:

SD card (ext4)                 3TB USB HDD (exFAT)
──────────────                 ───────────────────
Nextcloud data dir   ◄── here  user-uploaded files  ◄── here
MariaDB database              (mounted as External
Nextcloud config               Storage, not the
                               primary data dir)

The primary data directory and the database live on the SD card, which is ext4 — real permissions, real locking, everything Nextcloud expects. Only the actual user files live on the 3TB drive, and they get there through a Nextcloud feature called External Storage (Local): you mount a local path *into* a user's Nextcloud as if it were a remote share.

And the locking problem? I sidestepped it entirely. Nextcloud supports Redis-based file locking instead of filesystem flock. I already had Redis running — it's the same redis:7-alpine container my monitoring stack pokes at — so I pointed Nextcloud's locking at Redis. Locking now happens in memory, over the network, and never touches exfat.

'memcache.locking' => '\OC\Memcache\Redis',
'redis' => [ 'host' => 'redis', 'port' => 6379 ],

That 'host' => 'redis' is the piapps network paying off — Nextcloud finds Redis by container name.

So: metadata and locks on a real filesystem, bulk bytes on the big cheap drive. exFAT never has to do anything it's bad at.

Now the part I'm quietly proud of.

There are three people on this NAS: user1, user2, and user3 (family). Each should see their own files and absolutely none of anyone else's. The obvious way to enforce that is filesystem permissions — give each user a folder only they can read.

Except remember: the 3TB drive is exFAT. Every file is 0777. There are *no* per-user permissions on that disk. At the filesystem layer, everyone can read everything. I can't fix that without reformatting, which I won't do.

So isolation happens one layer up — in Nextcloud, not in the filesystem.

The mechanism is how you scope an External Storage mount. When you add a Local external mount, Nextcloud lets you say *who it applies to*. The critical command is occ files_external:applicable --add-user, scoped to exactly one user:

# each user gets ONE mount, applicable to ONLY them
occ files_external:applicable <mount_id> --add-user user1
occ files_external:applicable <mount_id> --add-user user2
occ files_external:applicable <mount_id> --add-user user3

The thing you must never do here is make a mount global (applicable to all users). One global mount and everyone sees the same folder. The whole isolation rests on every mount being bound to a single identity.

Here's why it actually works. When a user logs in, Nextcloud assembles *only* the mounts that apply to that identity. user1's session literally never has user2's mount constructed in it. It's not hidden by a permission check that could be bypassed — the folder simply isn't part of user1's filesystem view at all.

user1 logs in  ->  Nextcloud builds [user1's mount]  ->  sees only user1/
user2 logs in  ->  Nextcloud builds [user2's mount]  ->  sees only user2/
user3 logs in  ->  Nextcloud builds [user3's mount]  ->  sees only user3/

Now the honest caveat, because this kind of isolation has a hard edge and I'd rather say it out loud:

This is app-layer isolation, not OS-layer isolation. Anyone with SSH or Docker access to the Pi can mount the exFAT drive and read every file on it, because on disk it's all 0777 with no separation. The isolation only holds *through Nextcloud's front door*.

For my threat model — family photos, three trusted users, one physical box in my flat that only I have keys to — that's the right trade. If this were multi-tenant for strangers, 0777-on-disk would be unacceptable and I'd need a real filesystem with ACLs. Know which situation you're in.

The thing that made my family stop using Google Photos wasn't the architecture. It was that the Nextcloud mobile app does automatic photo upload, and it just works.

You install the app, log in, toggle "auto upload," and every photo your phone takes gets pushed to the NAS in the background. Under the hood it's WebDAV — the app talks to Nextcloud over the same HTTP-based file protocol the web UI uses.

The nice part, given everything above: because each user's files live on an External Storage mount pointed at the 3TB drive, the photos land straight on the big drive. They don't fill up the SD card. The SD card only ever holds metadata and the database; the 30GB of someone's camera roll goes where the space actually is.

phone camera roll
      │  Nextcloud app, auto-upload (WebDAV over HTTPS)
      ▼
   Nextcloud  ──(External Storage)──►  3TB drive

It's the closest thing to "Google Photos but it's mine," and it's the feature that justified the whole project to the people who didn't care about Docker networks.

This one cost me an evening and it's the kind of bug that doesn't announce itself.

Picture the boot sequence. The Pi powers on. systemd starts bringing services up. Docker starts. Docker starts my containers, including Nextcloud, which immediately wants to write to /mnt/3tb — the External Storage path.

But mounting a USB drive takes a moment. If Docker and Nextcloud start up *before* the 3TB drive has finished mounting, then /mnt/3tb is just an empty directory on the SD card. Nextcloud sees a valid path, writes happily, and now you've got files silently landing on the SD card instead of the drive. No error. Everything looks fine. The drive mounts a few seconds later, *on top of* the files you just wrote, and they vanish from view.

That's a data-loss-shaped bug wearing a "working fine" costume.

The fix is two pieces. First, make the mount itself robust in /etc/fstab with systemd's automount:

UUID=xxxx  /mnt/3tb  exfat  defaults,x-systemd.automount,nofail  0 0

x-systemd.automount means the drive is mounted on first access rather than racing at boot, and nofail means a missing drive doesn't wedge the whole boot.

Second — and this is the load-bearing bit — tell Docker it is not allowed to start until that mount exists, via a drop-in:

# /etc/systemd/system/docker.service.d/wait-for-mount.conf
[Unit]
RequiresMountsFor=/mnt/3tb

RequiresMountsFor makes systemd treat the mount as a hard dependency of docker.service. Docker — and therefore every container — physically cannot start before /mnt/3tb is really mounted. The race is gone by construction, not by hoping the timing works out.

Every container also runs with restart: unless-stopped, so a crash or an OOM brings it back without me.

And I didn't trust any of this on paper. I actually rebooted the Pi and watched: drive mounts, *then* Docker starts, Nextcloud comes up pointing at the real drive, files intact. If you build this, reboot it before you believe it.

I already had a monitoring stack on this Pi — Prometheus, Grafana, Loki, Promtail, cAdvisor, node_exporter — from an earlier project. (I wrote up that architecture separately, in the observability post.) Adding the NAS gave it something more interesting to measure than CPU temperature.

The new piece is blackbox_exporter. Where node_exporter reports the host's vitals from the inside, blackbox_exporter probes endpoints from the *outside* — it makes an actual HTTP request to a URL and reports whether it answered, how fast, and with what status code.

blackbox_exporter  ──HTTP probe──►  https://cloud.imashish.dev
                   ──HTTP probe──►  https://grafana.imashish.dev
   (is it up? how slow? what status?)  →  Prometheus  →  Grafana

That turns "is the NAS up?" into a number I can graph. And once you have that number, you can do the thing that's usually reserved for production systems with a straight face: set an SLO and track an error budget.

I picked a 99.5% availability target. That sounds strict until you do the math — 99.5% over 30 days allows about 3.6 hours of downtime a month. For a Raspberry Pi in a flat that occasionally gets rebooted, that's a sane, honest target, not aspirational nonsense.

99.5% over 30 days  →  ~0.5% × 30 days  →  ~3.6 hours/month of allowed downtime

The Grafana dashboard shows three things: availability (am I hitting 99.5%?), error budget (how much of that 3.6 hours have I already burned this month?), and latency (how slow are the probes?). When the error budget runs low, it's a signal to stop tinkering and let the thing be stable.

The dashboard is surfaced on a public Grafana dashboard — render-only, no login, no ad-hoc queries — so anyone can see whether my NAS is currently keeping its promise. There's something clarifying about putting your home server's SLA where strangers can check it.

Everything above is reachable from anywhere, and my home router doesn't have a single port forwarded. That's a Cloudflare Tunnel doing the work.

The naive way to expose a home server is port forwarding: tell the router "send port 443 to the Pi." That publishes your home IP and leaves a permanent open door on the internet. Hard no — especially for a box holding family photos.

Instead, cloudflared runs as a container on the Pi and dials outbound to Cloudflare, holding that connection open. Nothing ever connects *in*. The tunnel is token/dashboard-managed — I map subdomains to local ports in Cloudflare's UI:

cloud.imashish.dev    →  localhost:<nextcloud>
grafana.imashish.dev  →  localhost:<grafana>
tasks.imashish.dev    →  localhost:<learning-board>

When someone visits cloud.imashish.dev, they hit Cloudflare's edge, not my house. The wins, basically for free:

• TLS terminates at Cloudflare's edge — HTTPS is automatic, I never touch a certificate.
• My home IP is never exposed — visitors see Cloudflare, not my flat.
• DDoS protection, a WAF, and rate limiting come along for the ride, which matters a lot more once there's a login form facing the internet.

The Learning Board kanban app (tasks.imashish.dev, backed by MongoDB) rides the same tunnel — it's a separate app from the NAS but shares the exposure model and the piapps network.

I'd be lying if I called this finished. Here's what's genuinely not good enough yet, because pretending otherwise is how you lose data.

The "backup" isn't a real backup. I run a daily backup service — an alpine container with mongodb-tools and a cron entry that runs mongodump for the Learning Board's MongoDB and keeps the last 7 days:

nightly cron:
  mongodump  →  /mnt/3tb/backups/mongo-YYYY-MM-DD.gz
  delete dumps older than 7 days

Read that path carefully. The backup lands on the same 3TB drive that holds the original data. If that one drive dies — and drives die — I lose the data *and* the backup in the same instant. That is not disaster recovery. That's a copy on the same disk, which protects against "I deleted a file" and nothing worse.

This violates the 3-2-1 rule (three copies, two different media, one off-site) on every count. Right now it's basically 1-1-0.

The plan to fix it, honestly stated:

• Add a single 4TB HDD as a second drive, and rsync the user files + the database dumps to it off-device. Two physical drives means one can fail without taking everything.
• Also back up MariaDB, not just Mongo. Right now the Nextcloud database has *no* automated dump, which is a gap I noticed writing this post. The NAS's own metadata is the least-backed-up thing in the whole system, which is embarrassing.

No encryption at rest — and this one's deliberate. The drives aren't encrypted. For a physical box in my home, full-disk encryption mostly protects against someone stealing the drive, and I judged that overkill versus the hassle of managing keys on a headless Pi. The plan is to encrypt the *off-device backups* instead — those are the copies that might one day leave the building, and that's where encryption earns its keep.

So: it works, my family uses it daily, and it would survive a fat-fingered deletion. It would not yet survive a dead drive. I know. The 4TB is on the list.

If you're looking at your own dusty Pi and a spare drive, here's the map of what this project forces you to pick up. Not a syllabus — more "here's the terrain I had to cross." It's a genuinely broad slice of infrastructure, which is the real reason a project like this is worth doing.

Containers and how they talk:

• Docker & Compose — every service as a folder with a docker-compose.yml; restart: unless-stopped; sidecar containers (the Nextcloud cron).
• Container networking — a shared user-defined network (piapps), service discovery by container name, and the discipline of *not* publishing ports for internal services.

Getting it on the internet, safely:

• Reverse proxy / Cloudflare Tunnel / DNS — outbound-only tunnels, mapping subdomains to local ports, TLS termination at the edge, and getting a WAF + rate limiting for free.
• Security hardening — SSH keys over passwords, internal-only services, app-layer vs OS-layer isolation, rate limiting on the login surface.

Seeing what's happening:

• Prometheus + PromQL — the pull model, scraping exporters, querying time series.
• Grafana — dashboards as the single pane of glass; public render-only dashboards.
• Loki / LogQL — cheap label-indexed logs that fit on a Pi.
• blackbox_exporter + SLO thinking — probing public endpoints, then turning "is it up?" into an availability target and an error budget (99.5% ≈ 3.6h/month).

The Linux and filesystem layer (where this project gets spicy):

• fstab / systemd mounts & automount — x-systemd.automount, nofail, and RequiresMountsFor to stop Docker racing the drive.
• Filesystem trade-offs — exFAT vs ext4, why 0777-everywhere and missing flock change your whole design, and where permissions/ACLs actually live.

The application stack:

• Nextcloud External Storage — mounting a local path into a *specific* user, and using that as your isolation primitive.
• Redis as a lock/cache — offloading file locking off a filesystem that can't do it.
• Mongo & Maria basics — running databases as internal-only containers, and dumping them.

And the lesson I'm still learning:

• Backups & the 3-2-1 rule — three copies, two media, one off-site — and noticing, honestly, when your setup is really 1-1-0 and saying so out loud.

If you build it, reboot it before you trust it, and back it up to a *different drive* before you trust it with anything you'd cry about losing. I'm most of the way there. The 4TB is on the list.