Add AGENTS.md for agents picking up work in new sessions
- AGENTS.md: ground rules (legal access, evidence, fairness), invariants that are easy to break, commands, and notes on the owner's dev machine; CLAUDE.md imports it - backend/tests/Dockerfile.debian: test image from Debian packages for networks where PyPI is unreachable, documented in testing.md - roadmap: list what v2 has not been verified against yet Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
583ccba31c
commit
45514624f4
5 files changed
+121
No files matched your search
@@ -0,0 +1,83 @@
|
|||||||
|
# AGENTS.md
|
||||||
|
|
||||||
|
Guidance for AI coding agents working on this repository. Humans: start with [README.md](README.md)
|
||||||
|
and [CONTRIBUTING.md](CONTRIBUTING.md); everything below links there instead of repeating it.
|
||||||
|
|
||||||
|
Politrack tracks Austrian politicians from EU to municipal level (roles, votes, money, statements,
|
||||||
|
meetings, social media, legal proceedings). Every fact is a claim backed by evidence links. Backend:
|
||||||
|
Python (FastAPI, Postgres 16 + pgvector, Procrastinate workers, MCP server). Frontend: Next.js 15 with
|
||||||
|
next-intl, German by default, English at `/en`. Deployed with Docker Compose on the owner's Unraid server.
|
||||||
|
|
||||||
|
## Before you start
|
||||||
|
|
||||||
|
| Task | Read |
|
||||||
|
|---|---|
|
||||||
|
| Pick up open work | [docs/roadmap.md](docs/roadmap.md): *Next*, *Not yet verified*, *Known limitations* |
|
||||||
|
| Add a data source | [docs/contributing/adapters.md](docs/contributing/adapters.md), legal checklist in §0 first |
|
||||||
|
| Derive facts from stored data | [docs/contributing/enrichers.md](docs/contributing/enrichers.md) |
|
||||||
|
| API endpoint or MCP tool | [docs/contributing/api.md](docs/contributing/api.md) |
|
||||||
|
| UI page or new claim type in the UI | [docs/contributing/frontend.md](docs/contributing/frontend.md) |
|
||||||
|
| Tests and fixtures | [docs/contributing/testing.md](docs/contributing/testing.md) |
|
||||||
|
| Why it is built this way | [docs/architecture/02-comparison-and-decisions.md](docs/architecture/02-comparison-and-decisions.md) |
|
||||||
|
| Tables, claim types, identifier schemes | [docs/data-model.md](docs/data-model.md) |
|
||||||
|
| What each source allows | [docs/sources.md](docs/sources.md), [docs/legal.md](docs/legal.md) |
|
||||||
|
|
||||||
|
## Ground rules
|
||||||
|
|
||||||
|
1. **Legal access only.** Collect as much as possible, but only through official APIs, open data and
|
||||||
|
pages whose terms and robots.txt allow it. Never use X proxies (Nitter and the like), paywall
|
||||||
|
bypasses, logged-in scraping, or WhatsApp/Signal. Telegram: public channels via the `t.me/s` web
|
||||||
|
preview only. A robots.txt exemption needs the publisher's documented permission, recorded in the
|
||||||
|
adapter's `SourceSpec.tos_notes`; the RTR API is the only one so far. If a source can't be accessed
|
||||||
|
cleanly, say so and propose legal alternatives instead of working around it.
|
||||||
|
2. **Every claim has evidence.** No claim without a document link.
|
||||||
|
3. **Fairness.** Neutral wording. Legal matters carry the procedural stage and "Es gilt die
|
||||||
|
Unschuldsvermutung". Automatically detected legal claims stay in review (`review_status = pending`).
|
||||||
|
Connections are not accusations. Personal data only in the person's public role.
|
||||||
|
4. **Not public yet.** The instance is private as of October 2026. The X deletion sync (roadmap) is
|
||||||
|
deferred until launch, but must exist before the instance goes public or an X token is configured.
|
||||||
|
5. **Commits:** only when the owner asks, as `Simon Diesenreiter <simon@disi.dev>`, not the global
|
||||||
|
(work) address: `git -c user.email=simon@disi.dev commit …`, unless the repo-local config sets it.
|
||||||
|
|
||||||
|
## Invariants that are easy to break
|
||||||
|
|
||||||
|
- **Four stages.** Adapters fetch only through `Request` objects handled by the framework's `Fetcher`
|
||||||
|
(robots, throttling, TDM, archive), never with their own HTTP client. `parse()` is pure: no database,
|
||||||
|
no network. Only the `Loader` writes claims. Enrichers derive from stored data.
|
||||||
|
- **Claim identity.** Claims with a facet get their fingerprint from it. Claims without a facet need an
|
||||||
|
explicit `key`, or duplicates and lost updates follow.
|
||||||
|
- **Parser changes:** bump `parser_version`. Otherwise unchanged responses are skipped and the fix
|
||||||
|
never applies. `politrack reparse <adapter>` rebuilds from the raw archive and deletes claims the new
|
||||||
|
parser no longer emits.
|
||||||
|
- **Golden fixtures:** new or changed parsers come with a fixture (`politrack fixture …`). Review the
|
||||||
|
`.expected.json` diff; don't blindly regenerate with `UPDATE_GOLDEN=1`.
|
||||||
|
- **Schema:** `0001_schema.sql` and `0002_seed.sql` are applied in existing databases. Change the
|
||||||
|
schema only with a new numbered migration, with `COMMENT`s on new tables and columns.
|
||||||
|
- **API types:** after changing a response model, regenerate the frontend types:
|
||||||
|
`cd backend && politrack openapi --ts ../web/src/lib/api-types.ts`.
|
||||||
|
- **UI text:** every new key goes into both `web/messages/de.json` and `en.json`.
|
||||||
|
|
||||||
|
## Commands
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd backend && pytest # unit + golden tests, no DB or network needed
|
||||||
|
politrack sources list # adapters, enrichers, schedules, queues
|
||||||
|
politrack run <adapter> --max-requests 20 -p key=val # limit requests when testing against live sources
|
||||||
|
politrack reparse <adapter> # re-parse archived responses
|
||||||
|
politrack enrich | status | review | migrate # enrichers, recent runs, legal review, schema
|
||||||
|
```
|
||||||
|
|
||||||
|
## The owner's development machine
|
||||||
|
|
||||||
|
These notes apply to the owner's Windows PC (Docker Desktop, Git Bash, PowerShell):
|
||||||
|
|
||||||
|
- **Git** reports "dubious ownership" for this folder. Don't change the global git config; prefix git
|
||||||
|
commands with `git -c safe.directory=C:/Users/disi0/source/repos/politics-tracking`.
|
||||||
|
- **PyPI and npm** downloads failed there in October 2026 (a network filter blocks
|
||||||
|
`files.pythonhosted.org` and `registry.npmjs.org`). Until that is fixed, run the tests with the Debian
|
||||||
|
test image ([testing.md, "Without PyPI access"](docs/contributing/testing.md#without-pypi-access)).
|
||||||
|
The real images, `npm install` and the workers can't be built or run there.
|
||||||
|
- **Local database:** `docker compose up -d db` (port 5433, passwords in the gitignored `.env`, data in
|
||||||
|
`data/`). It has the schema and sources but no scraped data yet, and no job-queue schema until the
|
||||||
|
real `migrate` image runs. Postgres on a Windows bind mount is slow: the first start takes minutes.
|
||||||
|
- **`DATA_DIR`, not `APPDATA`:** Windows already defines `APPDATA`, so the compose file uses `DATA_DIR`.
|
||||||
@@ -0,0 +1,11 @@
|
|||||||
|
# Test image with the Python dependencies from Debian packages, for networks where PyPI is unreachable.
|
||||||
|
# Lacks procrastinate and mcp (not packaged in Debian), so it runs the test suite and adapters, not the
|
||||||
|
# workers or the MCP server. Usage: docs/contributing/testing.md ("Without PyPI access").
|
||||||
|
FROM debian:trixie-slim
|
||||||
|
RUN apt-get update -qq && apt-get install -y -qq --no-install-recommends \
|
||||||
|
python3 python3-psycopg python3-psycopg-pool python3-pydantic-settings python3-fastapi python3-httpx \
|
||||||
|
python3-protego python3-feedparser python3-bs4 python3-slugify python3-rapidfuzz python3-typer \
|
||||||
|
python3-yaml python3-uvicorn python3-pytest python3-openpyxl ca-certificates >/dev/null \
|
||||||
|
&& rm -rf /var/lib/apt/lists/*
|
||||||
|
WORKDIR /app
|
||||||
|
ENV PYTHONPATH=/app PYTHONUNBUFFERED=1 PYTHONDONTWRITEBYTECODE=1
|
||||||
@@ -16,6 +16,20 @@ UPDATE_GOLDEN=1 pytest tests/test_golden.py # regenerate expected outputs (rev
|
|||||||
|
|
||||||
Parsers are pure, so none of these needs a database or network.
|
Parsers are pure, so none of these needs a database or network.
|
||||||
|
|
||||||
|
### Without PyPI access
|
||||||
|
|
||||||
|
`tests/Dockerfile.debian` installs the dependencies from Debian packages instead (everything except
|
||||||
|
`procrastinate` and `mcp`, which the tests don't need):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker build -t politrack-test -f backend/tests/Dockerfile.debian backend/tests
|
||||||
|
docker run --rm -v "$PWD/backend:/app" politrack-test python3 -m pytest -q -p no:cacheprovider
|
||||||
|
```
|
||||||
|
|
||||||
|
The same image can run adapters against the compose database: add `--network politics-tracking_default`
|
||||||
|
and `-e POLITRACK_DATABASE_URL=postgresql://politrack:<password>@db:5432/politrack`, then
|
||||||
|
`python3 -m politrack.cli run <adapter> --max-requests 20`.
|
||||||
|
|
||||||
## Golden fixtures
|
## Golden fixtures
|
||||||
|
|
||||||
```
|
```
|
||||||
|
|||||||
@@ -59,6 +59,18 @@ Next.js frontend (de/en), Docker Compose for Unraid.
|
|||||||
- [ ] Partnerships: MeineAbgeordneten, Parteispenden.at, APA (OTS full-text licence)
|
- [ ] Partnerships: MeineAbgeordneten, Parteispenden.at, APA (OTS full-text licence)
|
||||||
- [ ] Map view of trips and appearances
|
- [ ] Map view of trips and appearances
|
||||||
|
|
||||||
|
## Not yet verified (v2)
|
||||||
|
|
||||||
|
Tested so far: unit and golden tests, a live pipeline run of every adapter with `--max-requests`
|
||||||
|
against a test database, every API endpoint, the frontend type check against the generated types.
|
||||||
|
Not run yet, because PyPI/npm were unreachable on the development network (October 2026):
|
||||||
|
|
||||||
|
- [ ] Building the `backend` and `web` images (`docker compose build`), `npm run build`
|
||||||
|
- [ ] Procrastinate workers and the periodic schedule; the job-queue schema (`politrack migrate` installs it)
|
||||||
|
- [ ] The MCP server at runtime
|
||||||
|
- [ ] Mastodon (no account in the test sample), YouTube, X and Meta Ad Library (no credentials)
|
||||||
|
- [ ] GDELT (rate-limited during the test), EP declarations for MEPs whose API calls returned HTTP 500
|
||||||
|
|
||||||
## Known limitations
|
## Known limitations
|
||||||
|
|
||||||
- Votes in the Nationalrat are faction-level except roll calls (an Austrian parliamentary practice).
|
- Votes in the Nationalrat are faction-level except roll calls (an Austrian parliamentary practice).
|
||||||
|
|||||||
Reference in new issue
Block a user