politics-tracking/docs/deployment-unraid.md

Deployment on Unraid

Prerequisites

• Unraid 6.12+ with the Docker Compose Manager plugin (Community Applications).
• An appdata share, for example /mnt/user/appdata/politrack.
• Network access from the server to PyPI/npm for the image builds (files.pythonhosted.org, registry.npmjs.org) – see "Troubleshooting".
• Optional: a reverse proxy (SWAG, Nginx Proxy Manager, Traefik or Caddy) and SSO (Authentik or Authelia) if the site is exposed; an Ollama container for semantic search.

Steps

1. Clone the repository onto the server (for example /mnt/user/appdata/politrack/src) or add it as a stack in Compose Manager pointing at this folder.

2. cp .env.example .env and set at least:

   • DATA_DIR=/mnt/user/appdata/politrack – database (postgres/) and raw archive (raw/) live here
   • POSTGRES_PASSWORD, POLITRACK_READER_PASSWORD
   • POLITRACK_USER_AGENT with a real contact address (publishers must be able to reach you)
   • optional: POLITRACK_ADMIN_TOKEN, POLITRACK_OLLAMA_URL, API keys (YouTube, Meta Ad Library, X)

3. Build and start:

   docker compose build
   docker compose up -d
   

   migrate runs once (schema, seed data, source registry, job-queue tables), then api, web, mcp and three workers start:

   service	queues	what runs there
   worker-scrape	scrape, social	most adapters (hourly to daily)
   worker-bulk	bulk	large or slow downloads: procurement, RTR, ad libraries, EC meetings, GDELT
   worker-enrich	enrich	linking, events, legal detection, topics, corroboration, embeddings

4. Initial load (otherwise the schedules pick everything up over the next day):

   docker compose exec api politrack sources list            # what exists, when it runs
   docker compose exec api politrack run parlament.persons   # members first: other adapters resolve against them
   docker compose exec api politrack run wikidata.politicians
   docker compose exec api politrack run europarl.meps
   docker compose exec api politrack run parlament.items     # votes etc. – the longest run (30-60 min)
   docker compose exec api politrack run config.social_accounts
   docker compose exec api politrack enrich
   

5. Open http://<unraid-ip>:3000 (web), http://<unraid-ip>:8000/api/docs (API) and http://<unraid-ip>:8001/mcp (MCP).

Upgrading from the first version (v1): the schema was rewritten (0001_schema.sql). Start with a fresh database: stop the stack, move ${DATA_DIR}/postgres away, docker compose up -d, re-run the initial load. Nothing is lost that the sources can't provide again.

Scaling and separate deployments

• More throughput: docker compose up -d --scale worker-scrape=3. Queueing locks ensure one adapter never runs twice at the same time; per-host throttling is per process, so keep one replica per queue if you are close to a publisher's rate limit.
• Another machine (different egress IP, GPU): run the same image with POLITRACK_DATABASE_URL pointing at this database and procrastinate --app=politrack.worker.app worker --queues=<queues>. Mount a raw archive volume there too (or set POLITRACK_RAW_DIR to shared storage).

Semantic search (optional)

docker compose --profile ai up -d ollama
docker compose exec ollama ollama pull bge-m3
# .env: POLITRACK_OLLAMA_URL=http://ollama:11434
docker compose up -d

bge-m3 runs fine on CPU for this data volume. With an NVIDIA GPU, add a device reservation (deploy.resources.reservations.devices) to the ollama service.

Backups

• Database: a nightly pg_dump, e.g. with the User Scripts plugin:

  docker exec politrack-db-1 pg_dump -U politrack -Fc politrack > /mnt/user/backups/politrack/$(date +%F).dump
  find /mnt/user/backups/politrack -mtime +30 -delete
  

• Raw archive (${DATA_DIR}/raw): optional to back up – it saves refetching after parser fixes, and it can be pruned (find raw -mtime +365 -delete); missing blobs are simply refetched.

Exposing publicly

• Expose only web (and optionally api) through the reverse proxy. Keep mcp and the DB port LAN- or VPN-only, or protect them with SSO or an API key at the proxy.
• Read legal.md before going public (Impressum, privacy policy, §7b MedienG).

Monitoring

• docker compose exec api politrack status – recent runs with stats and errors (table run).
• GET /api/v1/admin/runs with Authorization: Bearer $POLITRACK_ADMIN_TOKEN.
• Fetch log: SELECT adapter, status, count(*) FROM fetch_log WHERE fetched_at > now() - interval '1 day' GROUP BY 1, 2;
• Job queue: SELECT task_name, status, count(*) FROM procrastinate_jobs GROUP BY 1, 2;
• Uptime Kuma can poll http://api:8000/healthz.

Troubleshooting

pip install / npm install fail during the build with TLS errors, while other sites work. Something between the server and the internet blocks the package CDNs by host name (TLS SNI): typically a DNS filter (Pi-hole/AdGuard, router "parental control"/"web protection"), a firewall with TLS inspection, or antivirus web protection on the build machine. Check:

nslookup files.pythonhosted.org                 # resolves to a Fastly address, not 0.0.0.0 / your router?
curl -v https://pypi.org/simple/ -o /dev/null   # works?
curl -v https://files.pythonhosted.org/ -o /dev/null
curl -v https://registry.npmjs.org/ -o /dev/null

If pypi.org works but files.pythonhosted.org / registry.npmjs.org fail during the TLS handshake, allow these host names in the filter/firewall/AV (also *.pythonhosted.org, registry.yarnpkg.com if used). On Windows, the System event log (source Schannel) and the AV's web-protection log show which component reset the connection.