From 45514624f40708bc42677c707ec070ee1594825e Mon Sep 17 00:00:00 2001 From: Simon Diesenreiter Date: Wed, 28 Oct 2026 20:02:33 +0100 Subject: [PATCH] Add AGENTS.md for agents picking up work in new sessions - AGENTS.md: ground rules (legal access, evidence, fairness), invariants that are easy to break, commands, and notes on the owner's dev machine; CLAUDE.md imports it - backend/tests/Dockerfile.debian: test image from Debian packages for networks where PyPI is unreachable, documented in testing.md - roadmap: list what v2 has not been verified against yet Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 83 +++++++++++++++++++++++++++++++++ CLAUDE.md | 1 + backend/tests/Dockerfile.debian | 11 +++++ docs/contributing/testing.md | 14 ++++++ docs/roadmap.md | 12 +++++ 5 files changed, 121 insertions(+) create mode 100644 AGENTS.md create mode 100644 CLAUDE.md create mode 100644 backend/tests/Dockerfile.debian diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..0843b56 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,83 @@ +# AGENTS.md + +Guidance for AI coding agents working on this repository. Humans: start with [README.md](README.md) +and [CONTRIBUTING.md](CONTRIBUTING.md); everything below links there instead of repeating it. + +Politrack tracks Austrian politicians from EU to municipal level (roles, votes, money, statements, +meetings, social media, legal proceedings). Every fact is a claim backed by evidence links. Backend: +Python (FastAPI, Postgres 16 + pgvector, Procrastinate workers, MCP server). Frontend: Next.js 15 with +next-intl, German by default, English at `/en`. Deployed with Docker Compose on the owner's Unraid server. + +## Before you start + +| Task | Read | +|---|---| +| Pick up open work | [docs/roadmap.md](docs/roadmap.md): *Next*, *Not yet verified*, *Known limitations* | +| Add a data source | [docs/contributing/adapters.md](docs/contributing/adapters.md), legal checklist in §0 first | +| Derive facts from stored data | [docs/contributing/enrichers.md](docs/contributing/enrichers.md) | +| API endpoint or MCP tool | [docs/contributing/api.md](docs/contributing/api.md) | +| UI page or new claim type in the UI | [docs/contributing/frontend.md](docs/contributing/frontend.md) | +| Tests and fixtures | [docs/contributing/testing.md](docs/contributing/testing.md) | +| Why it is built this way | [docs/architecture/02-comparison-and-decisions.md](docs/architecture/02-comparison-and-decisions.md) | +| Tables, claim types, identifier schemes | [docs/data-model.md](docs/data-model.md) | +| What each source allows | [docs/sources.md](docs/sources.md), [docs/legal.md](docs/legal.md) | + +## Ground rules + +1. **Legal access only.** Collect as much as possible, but only through official APIs, open data and + pages whose terms and robots.txt allow it. Never use X proxies (Nitter and the like), paywall + bypasses, logged-in scraping, or WhatsApp/Signal. Telegram: public channels via the `t.me/s` web + preview only. A robots.txt exemption needs the publisher's documented permission, recorded in the + adapter's `SourceSpec.tos_notes`; the RTR API is the only one so far. If a source can't be accessed + cleanly, say so and propose legal alternatives instead of working around it. +2. **Every claim has evidence.** No claim without a document link. +3. **Fairness.** Neutral wording. Legal matters carry the procedural stage and "Es gilt die + Unschuldsvermutung". Automatically detected legal claims stay in review (`review_status = pending`). + Connections are not accusations. Personal data only in the person's public role. +4. **Not public yet.** The instance is private as of October 2026. The X deletion sync (roadmap) is + deferred until launch, but must exist before the instance goes public or an X token is configured. +5. **Commits:** only when the owner asks, as `Simon Diesenreiter `, not the global + (work) address: `git -c user.email=simon@disi.dev commit …`, unless the repo-local config sets it. + +## Invariants that are easy to break + +- **Four stages.** Adapters fetch only through `Request` objects handled by the framework's `Fetcher` + (robots, throttling, TDM, archive), never with their own HTTP client. `parse()` is pure: no database, + no network. Only the `Loader` writes claims. Enrichers derive from stored data. +- **Claim identity.** Claims with a facet get their fingerprint from it. Claims without a facet need an + explicit `key`, or duplicates and lost updates follow. +- **Parser changes:** bump `parser_version`. Otherwise unchanged responses are skipped and the fix + never applies. `politrack reparse ` rebuilds from the raw archive and deletes claims the new + parser no longer emits. +- **Golden fixtures:** new or changed parsers come with a fixture (`politrack fixture …`). Review the + `.expected.json` diff; don't blindly regenerate with `UPDATE_GOLDEN=1`. +- **Schema:** `0001_schema.sql` and `0002_seed.sql` are applied in existing databases. Change the + schema only with a new numbered migration, with `COMMENT`s on new tables and columns. +- **API types:** after changing a response model, regenerate the frontend types: + `cd backend && politrack openapi --ts ../web/src/lib/api-types.ts`. +- **UI text:** every new key goes into both `web/messages/de.json` and `en.json`. + +## Commands + +```bash +cd backend && pytest # unit + golden tests, no DB or network needed +politrack sources list # adapters, enrichers, schedules, queues +politrack run --max-requests 20 -p key=val # limit requests when testing against live sources +politrack reparse # re-parse archived responses +politrack enrich | status | review | migrate # enrichers, recent runs, legal review, schema +``` + +## The owner's development machine + +These notes apply to the owner's Windows PC (Docker Desktop, Git Bash, PowerShell): + +- **Git** reports "dubious ownership" for this folder. Don't change the global git config; prefix git + commands with `git -c safe.directory=C:/Users/disi0/source/repos/politics-tracking`. +- **PyPI and npm** downloads failed there in October 2026 (a network filter blocks + `files.pythonhosted.org` and `registry.npmjs.org`). Until that is fixed, run the tests with the Debian + test image ([testing.md, "Without PyPI access"](docs/contributing/testing.md#without-pypi-access)). + The real images, `npm install` and the workers can't be built or run there. +- **Local database:** `docker compose up -d db` (port 5433, passwords in the gitignored `.env`, data in + `data/`). It has the schema and sources but no scraped data yet, and no job-queue schema until the + real `migrate` image runs. Postgres on a Windows bind mount is slow: the first start takes minutes. +- **`DATA_DIR`, not `APPDATA`:** Windows already defines `APPDATA`, so the compose file uses `DATA_DIR`. diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..43c994c --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1 @@ +@AGENTS.md diff --git a/backend/tests/Dockerfile.debian b/backend/tests/Dockerfile.debian new file mode 100644 index 0000000..ded7877 --- /dev/null +++ b/backend/tests/Dockerfile.debian @@ -0,0 +1,11 @@ +# Test image with the Python dependencies from Debian packages, for networks where PyPI is unreachable. +# Lacks procrastinate and mcp (not packaged in Debian), so it runs the test suite and adapters, not the +# workers or the MCP server. Usage: docs/contributing/testing.md ("Without PyPI access"). +FROM debian:trixie-slim +RUN apt-get update -qq && apt-get install -y -qq --no-install-recommends \ + python3 python3-psycopg python3-psycopg-pool python3-pydantic-settings python3-fastapi python3-httpx \ + python3-protego python3-feedparser python3-bs4 python3-slugify python3-rapidfuzz python3-typer \ + python3-yaml python3-uvicorn python3-pytest python3-openpyxl ca-certificates >/dev/null \ + && rm -rf /var/lib/apt/lists/* +WORKDIR /app +ENV PYTHONPATH=/app PYTHONUNBUFFERED=1 PYTHONDONTWRITEBYTECODE=1 diff --git a/docs/contributing/testing.md b/docs/contributing/testing.md index 7119afa..bad84c9 100644 --- a/docs/contributing/testing.md +++ b/docs/contributing/testing.md @@ -16,6 +16,20 @@ UPDATE_GOLDEN=1 pytest tests/test_golden.py # regenerate expected outputs (rev Parsers are pure, so none of these needs a database or network. +### Without PyPI access + +`tests/Dockerfile.debian` installs the dependencies from Debian packages instead (everything except +`procrastinate` and `mcp`, which the tests don't need): + +```bash +docker build -t politrack-test -f backend/tests/Dockerfile.debian backend/tests +docker run --rm -v "$PWD/backend:/app" politrack-test python3 -m pytest -q -p no:cacheprovider +``` + +The same image can run adapters against the compose database: add `--network politics-tracking_default` +and `-e POLITRACK_DATABASE_URL=postgresql://politrack:@db:5432/politrack`, then +`python3 -m politrack.cli run --max-requests 20`. + ## Golden fixtures ``` diff --git a/docs/roadmap.md b/docs/roadmap.md index fd5889b..1394058 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -59,6 +59,18 @@ Next.js frontend (de/en), Docker Compose for Unraid. - [ ] Partnerships: MeineAbgeordneten, Parteispenden.at, APA (OTS full-text licence) - [ ] Map view of trips and appearances +## Not yet verified (v2) + +Tested so far: unit and golden tests, a live pipeline run of every adapter with `--max-requests` +against a test database, every API endpoint, the frontend type check against the generated types. +Not run yet, because PyPI/npm were unreachable on the development network (October 2026): + +- [ ] Building the `backend` and `web` images (`docker compose build`), `npm run build` +- [ ] Procrastinate workers and the periodic schedule; the job-queue schema (`politrack migrate` installs it) +- [ ] The MCP server at runtime +- [ ] Mastodon (no account in the test sample), YouTube, X and Meta Ad Library (no credentials) +- [ ] GDELT (rate-limited during the test), EP declarations for MEPs whose API calls returned HTTP 500 + ## Known limitations - Votes in the Nationalrat are faction-level except roll calls (an Austrian parliamentary practice).