Add AGENTS.md for agents picking up work in new sessions

- AGENTS.md: ground rules (legal access, evidence, fairness), invariants
  that are easy to break, commands, and notes on the owner's dev machine;
  CLAUDE.md imports it
- backend/tests/Dockerfile.debian: test image from Debian packages for
  networks where PyPI is unreachable, documented in testing.md
- roadmap: list what v2 has not been verified against yet

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
simonandClaude Opus 5.5 committed 2026-10-28 20:02:33 +01:00
1 parent 583ccba31c
commit 45514624f4
5 files changed
+121

No files matched your search

+83
View File
@@ -0,0 +1,83 @@
# AGENTS.md
Guidance for AI coding agents working on this repository. Humans: start with [README.md](README.md)
and [CONTRIBUTING.md](CONTRIBUTING.md); everything below links there instead of repeating it.
Politrack tracks Austrian politicians from EU to municipal level (roles, votes, money, statements,
meetings, social media, legal proceedings). Every fact is a claim backed by evidence links. Backend:
Python (FastAPI, Postgres 16 + pgvector, Procrastinate workers, MCP server). Frontend: Next.js 15 with
next-intl, German by default, English at `/en`. Deployed with Docker Compose on the owner's Unraid server.
## Before you start
| Task | Read |
|---|---|
| Pick up open work | [docs/roadmap.md](docs/roadmap.md): *Next*, *Not yet verified*, *Known limitations* |
| Add a data source | [docs/contributing/adapters.md](docs/contributing/adapters.md), legal checklist in §0 first |
| Derive facts from stored data | [docs/contributing/enrichers.md](docs/contributing/enrichers.md) |
| API endpoint or MCP tool | [docs/contributing/api.md](docs/contributing/api.md) |
| UI page or new claim type in the UI | [docs/contributing/frontend.md](docs/contributing/frontend.md) |
| Tests and fixtures | [docs/contributing/testing.md](docs/contributing/testing.md) |
| Why it is built this way | [docs/architecture/02-comparison-and-decisions.md](docs/architecture/02-comparison-and-decisions.md) |
| Tables, claim types, identifier schemes | [docs/data-model.md](docs/data-model.md) |
| What each source allows | [docs/sources.md](docs/sources.md), [docs/legal.md](docs/legal.md) |
## Ground rules
1. **Legal access only.** Collect as much as possible, but only through official APIs, open data and
pages whose terms and robots.txt allow it. Never use X proxies (Nitter and the like), paywall
bypasses, logged-in scraping, or WhatsApp/Signal. Telegram: public channels via the `t.me/s` web
preview only. A robots.txt exemption needs the publisher's documented permission, recorded in the
adapter's `SourceSpec.tos_notes`; the RTR API is the only one so far. If a source can't be accessed
cleanly, say so and propose legal alternatives instead of working around it.
2. **Every claim has evidence.** No claim without a document link.
3. **Fairness.** Neutral wording. Legal matters carry the procedural stage and "Es gilt die
Unschuldsvermutung". Automatically detected legal claims stay in review (`review_status = pending`).
Connections are not accusations. Personal data only in the person's public role.
4. **Not public yet.** The instance is private as of October 2026. The X deletion sync (roadmap) is
deferred until launch, but must exist before the instance goes public or an X token is configured.
5. **Commits:** only when the owner asks, as `Simon Diesenreiter <simon@disi.dev>`, not the global
(work) address: `git -c user.email=simon@disi.dev commit …`, unless the repo-local config sets it.
## Invariants that are easy to break
- **Four stages.** Adapters fetch only through `Request` objects handled by the framework's `Fetcher`
(robots, throttling, TDM, archive), never with their own HTTP client. `parse()` is pure: no database,
no network. Only the `Loader` writes claims. Enrichers derive from stored data.
- **Claim identity.** Claims with a facet get their fingerprint from it. Claims without a facet need an
explicit `key`, or duplicates and lost updates follow.
- **Parser changes:** bump `parser_version`. Otherwise unchanged responses are skipped and the fix
never applies. `politrack reparse <adapter>` rebuilds from the raw archive and deletes claims the new
parser no longer emits.
- **Golden fixtures:** new or changed parsers come with a fixture (`politrack fixture …`). Review the
`.expected.json` diff; don't blindly regenerate with `UPDATE_GOLDEN=1`.
- **Schema:** `0001_schema.sql` and `0002_seed.sql` are applied in existing databases. Change the
schema only with a new numbered migration, with `COMMENT`s on new tables and columns.
- **API types:** after changing a response model, regenerate the frontend types:
`cd backend && politrack openapi --ts ../web/src/lib/api-types.ts`.
- **UI text:** every new key goes into both `web/messages/de.json` and `en.json`.
## Commands
```bash
cd backend && pytest # unit + golden tests, no DB or network needed
politrack sources list # adapters, enrichers, schedules, queues
politrack run <adapter> --max-requests 20 -p key=val # limit requests when testing against live sources
politrack reparse <adapter> # re-parse archived responses
politrack enrich | status | review | migrate # enrichers, recent runs, legal review, schema
```
## The owner's development machine
These notes apply to the owner's Windows PC (Docker Desktop, Git Bash, PowerShell):
- **Git** reports "dubious ownership" for this folder. Don't change the global git config; prefix git
commands with `git -c safe.directory=C:/Users/disi0/source/repos/politics-tracking`.
- **PyPI and npm** downloads failed there in October 2026 (a network filter blocks
`files.pythonhosted.org` and `registry.npmjs.org`). Until that is fixed, run the tests with the Debian
test image ([testing.md, "Without PyPI access"](docs/contributing/testing.md#without-pypi-access)).
The real images, `npm install` and the workers can't be built or run there.
- **Local database:** `docker compose up -d db` (port 5433, passwords in the gitignored `.env`, data in
`data/`). It has the schema and sources but no scraped data yet, and no job-queue schema until the
real `migrate` image runs. Postgres on a Windows bind mount is slow: the first start takes minutes.
- **`DATA_DIR`, not `APPDATA`:** Windows already defines `APPDATA`, so the compose file uses `DATA_DIR`.
+1
View File
@@ -0,0 +1 @@
@AGENTS.md
+11
View File
@@ -0,0 +1,11 @@
# Test image with the Python dependencies from Debian packages, for networks where PyPI is unreachable.
# Lacks procrastinate and mcp (not packaged in Debian), so it runs the test suite and adapters, not the
# workers or the MCP server. Usage: docs/contributing/testing.md ("Without PyPI access").
FROM debian:trixie-slim
RUN apt-get update -qq && apt-get install -y -qq --no-install-recommends \
python3 python3-psycopg python3-psycopg-pool python3-pydantic-settings python3-fastapi python3-httpx \
python3-protego python3-feedparser python3-bs4 python3-slugify python3-rapidfuzz python3-typer \
python3-yaml python3-uvicorn python3-pytest python3-openpyxl ca-certificates >/dev/null \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /app
ENV PYTHONPATH=/app PYTHONUNBUFFERED=1 PYTHONDONTWRITEBYTECODE=1
+14
View File
@@ -16,6 +16,20 @@ UPDATE_GOLDEN=1 pytest tests/test_golden.py # regenerate expected outputs (rev
Parsers are pure, so none of these needs a database or network.
### Without PyPI access
`tests/Dockerfile.debian` installs the dependencies from Debian packages instead (everything except
`procrastinate` and `mcp`, which the tests don't need):
```bash
docker build -t politrack-test -f backend/tests/Dockerfile.debian backend/tests
docker run --rm -v "$PWD/backend:/app" politrack-test python3 -m pytest -q -p no:cacheprovider
```
The same image can run adapters against the compose database: add `--network politics-tracking_default`
and `-e POLITRACK_DATABASE_URL=postgresql://politrack:<password>@db:5432/politrack`, then
`python3 -m politrack.cli run <adapter> --max-requests 20`.
## Golden fixtures
```
+12
View File
@@ -59,6 +59,18 @@ Next.js frontend (de/en), Docker Compose for Unraid.
- [ ] Partnerships: MeineAbgeordneten, Parteispenden.at, APA (OTS full-text licence)
- [ ] Map view of trips and appearances
## Not yet verified (v2)
Tested so far: unit and golden tests, a live pipeline run of every adapter with `--max-requests`
against a test database, every API endpoint, the frontend type check against the generated types.
Not run yet, because PyPI/npm were unreachable on the development network (October 2026):
- [ ] Building the `backend` and `web` images (`docker compose build`), `npm run build`
- [ ] Procrastinate workers and the periodic schedule; the job-queue schema (`politrack migrate` installs it)
- [ ] The MCP server at runtime
- [ ] Mastodon (no account in the test sample), YouTube, X and Meta Ad Library (no credentials)
- [ ] GDELT (rate-limited during the test), EP declarations for MEPs whose API calls returned HTTP 500
## Known limitations
- Votes in the Nationalrat are faction-level except roll calls (an Austrian parliamentary practice).