politics-tracking/docs/deployment-unraid.md

# Deployment on Unraid

## Prerequisites

* Unraid 6.12+ with the **Docker Compose Manager** plugin (Community Applications).
* An appdata share, for example `/mnt/user/appdata/politrack`.
* Network access from the server to PyPI/npm for the image builds (`files.pythonhosted.org`,
  `registry.npmjs.org`) – see "Troubleshooting".
* Optional: a reverse proxy (SWAG, Nginx Proxy Manager, Traefik or Caddy) and SSO (Authentik or Authelia)
  if the site is exposed; an Ollama container for semantic search.

## Steps

1. Clone the repository onto the server (for example `/mnt/user/appdata/politrack/src`) or add it as a
   stack in Compose Manager pointing at this folder.
2. `cp .env.example .env` and set at least:
   * `DATA_DIR=/mnt/user/appdata/politrack` – database (`postgres/`) and raw archive (`raw/`) live here
   * `POSTGRES_PASSWORD`, `POLITRACK_READER_PASSWORD`
   * `POLITRACK_USER_AGENT` with a real contact address (publishers must be able to reach you)
   * optional: `POLITRACK_ADMIN_TOKEN`, `POLITRACK_OLLAMA_URL`, API keys (YouTube, Meta Ad Library, X)
3. Build and start:
   ```bash
   docker compose build
   docker compose up -d
   ```
   Docker creates `${DATA_DIR}/raw` as root, but the backend runs as a non-root user, and every fetch
   then fails with "Permission denied". Hand it over once after the first `up`:
   `chown -R "$(docker run --rm --entrypoint id politrack-backend -u):$(docker run --rm --entrypoint id politrack-backend -g)" "$DATA_DIR/raw"`.

   `migrate` runs once (schema, seed data, source registry, job-queue tables), then `api`, `web`, `mcp`
   and three workers start:

   | service | queues | what runs there |
   |---|---|---|
   | `worker-scrape` | scrape, social | most adapters (hourly to daily) |
   | `worker-bulk` | bulk | large or slow downloads: procurement, RTR, ad libraries, EC meetings, GDELT |
   | `worker-enrich` | enrich | linking, events, legal detection, topics, corroboration, embeddings |

4. Initial load (otherwise the schedules pick everything up over the next day):
   ```bash
   docker compose exec api politrack sources list            # what exists, when it runs
   docker compose exec api politrack run parlament.persons   # members first: other adapters resolve against them
   docker compose exec api politrack run wikidata.politicians
   docker compose exec api politrack run europarl.meps
   docker compose exec api politrack run parlament.items     # votes etc. – the longest run (30-60 min)
   docker compose exec api politrack run config.social_accounts
   docker compose exec api politrack enrich
   ```
5. Open `http://<unraid-ip>:3000` (web), `http://<unraid-ip>:8000/api/docs` (API) and
   `http://<unraid-ip>:8001/mcp` (MCP).

**Upgrading from the first version (v1):** the schema was rewritten (`0001_schema.sql`). Start with a
fresh database: stop the stack, move `${DATA_DIR}/postgres` away, `docker compose up -d`, re-run the
initial load. Nothing is lost that the sources can't provide again.

## Scaling and separate deployments

* More throughput: `docker compose up -d --scale worker-scrape=3`. Queueing locks ensure one adapter
  never runs twice at the same time; per-host throttling is per process, so keep one replica per queue
  if you are close to a publisher's rate limit.
* Another machine (different egress IP, GPU): run the same image with `POLITRACK_DATABASE_URL` pointing at
  this database and `procrastinate --app=politrack.worker.app worker --queues=<queues>`. Mount a raw
  archive volume there too (or set `POLITRACK_RAW_DIR` to shared storage).

## Semantic search (optional)

```bash
docker compose --profile ai up -d ollama
docker compose exec ollama ollama pull bge-m3
# .env: POLITRACK_OLLAMA_URL=http://ollama:11434
docker compose up -d
```

bge-m3 runs fine on CPU for this data volume. With an NVIDIA GPU, add a device reservation
(`deploy.resources.reservations.devices`) to the ollama service.

## Backups

* Database: a nightly `pg_dump`, e.g. with the **User Scripts** plugin:
  ```bash
  docker exec politrack-db-1 pg_dump -U politrack -Fc politrack > /mnt/user/backups/politrack/$(date +%F).dump
  find /mnt/user/backups/politrack -mtime +30 -delete
  ```
* Raw archive (`${DATA_DIR}/raw`): optional to back up – it saves refetching after parser fixes, and it can
  be pruned (`find raw -mtime +365 -delete`); missing blobs are simply refetched.

## Exposing publicly

* Expose only `web` (and optionally `api`) through the reverse proxy. Keep `mcp` and the DB port LAN- or
  VPN-only, or protect them with SSO or an API key at the proxy.
* Read [legal.md](legal.md) before going public (Impressum, privacy policy, §7b MedienG).

## Monitoring

* `docker compose exec api politrack status` – recent runs with stats and errors (table `run`).
* `GET /api/v1/admin/runs` with `Authorization: Bearer $POLITRACK_ADMIN_TOKEN`.
* Fetch log: `SELECT adapter, status, count(*) FROM fetch_log WHERE fetched_at > now() - interval '1 day' GROUP BY 1, 2;`
* Job queue: `SELECT task_name, status, count(*) FROM procrastinate_jobs GROUP BY 1, 2;`
* Uptime Kuma can poll `http://api:8000/healthz`.

## Troubleshooting

**`pip install` / `npm install` fail during the build with TLS errors, while other sites work.**
Something between the server and the internet blocks the package CDNs by host name (TLS SNI): typically
a DNS filter (Pi-hole/AdGuard, router "parental control"/"web protection"), a firewall with TLS
inspection, or antivirus web protection on the build machine. Check:

```bash
nslookup files.pythonhosted.org                 # resolves to a Fastly address, not 0.0.0.0 / your router?
curl -v https://pypi.org/simple/ -o /dev/null   # works?
curl -v https://files.pythonhosted.org/ -o /dev/null
curl -v https://registry.npmjs.org/ -o /dev/null
```

If pypi.org works but files.pythonhosted.org / registry.npmjs.org fail during the TLS handshake, allow
these host names in the filter/firewall/AV (also `*.pythonhosted.org`, `registry.yarnpkg.com` if used). On
Windows, the System event log (source *Schannel*) and the AV's web-protection log show which component
reset the connection.