|
|
|
@@ -0,0 +1,83 @@
|
|
|
|
|
# AGENTS.md
|
|
|
|
|
|
|
|
|
|
Guidance for AI coding agents working on this repository. Humans: start with [README.md](README.md)
|
|
|
|
|
and [CONTRIBUTING.md](CONTRIBUTING.md); everything below links there instead of repeating it.
|
|
|
|
|
|
|
|
|
|
Politrack tracks Austrian politicians from EU to municipal level (roles, votes, money, statements,
|
|
|
|
|
meetings, social media, legal proceedings). Every fact is a claim backed by evidence links. Backend:
|
|
|
|
|
Python (FastAPI, Postgres 16 + pgvector, Procrastinate workers, MCP server). Frontend: Next.js 15 with
|
|
|
|
|
next-intl, German by default, English at `/en`. Deployed with Docker Compose on the owner's Unraid server.
|
|
|
|
|
|
|
|
|
|
## Before you start
|
|
|
|
|
|
|
|
|
|
| Task | Read |
|
|
|
|
|
|---|---|
|
|
|
|
|
| Pick up open work | [docs/roadmap.md](docs/roadmap.md): *Next*, *Not yet verified*, *Known limitations* |
|
|
|
|
|
| Add a data source | [docs/contributing/adapters.md](docs/contributing/adapters.md), legal checklist in §0 first |
|
|
|
|
|
| Derive facts from stored data | [docs/contributing/enrichers.md](docs/contributing/enrichers.md) |
|
|
|
|
|
| API endpoint or MCP tool | [docs/contributing/api.md](docs/contributing/api.md) |
|
|
|
|
|
| UI page or new claim type in the UI | [docs/contributing/frontend.md](docs/contributing/frontend.md) |
|
|
|
|
|
| Tests and fixtures | [docs/contributing/testing.md](docs/contributing/testing.md) |
|
|
|
|
|
| Why it is built this way | [docs/architecture/02-comparison-and-decisions.md](docs/architecture/02-comparison-and-decisions.md) |
|
|
|
|
|
| Tables, claim types, identifier schemes | [docs/data-model.md](docs/data-model.md) |
|
|
|
|
|
| What each source allows | [docs/sources.md](docs/sources.md), [docs/legal.md](docs/legal.md) |
|
|
|
|
|
|
|
|
|
|
## Ground rules
|
|
|
|
|
|
|
|
|
|
1. **Legal access only.** Collect as much as possible, but only through official APIs, open data and
|
|
|
|
|
pages whose terms and robots.txt allow it. Never use X proxies (Nitter and the like), paywall
|
|
|
|
|
bypasses, logged-in scraping, or WhatsApp/Signal. Telegram: public channels via the `t.me/s` web
|
|
|
|
|
preview only. A robots.txt exemption needs the publisher's documented permission, recorded in the
|
|
|
|
|
adapter's `SourceSpec.tos_notes`; the RTR API is the only one so far. If a source can't be accessed
|
|
|
|
|
cleanly, say so and propose legal alternatives instead of working around it.
|
|
|
|
|
2. **Every claim has evidence.** No claim without a document link.
|
|
|
|
|
3. **Fairness.** Neutral wording. Legal matters carry the procedural stage and "Es gilt die
|
|
|
|
|
Unschuldsvermutung". Automatically detected legal claims stay in review (`review_status = pending`).
|
|
|
|
|
Connections are not accusations. Personal data only in the person's public role.
|
|
|
|
|
4. **Not public yet.** The instance is private as of October 2026. The X deletion sync (roadmap) is
|
|
|
|
|
deferred until launch, but must exist before the instance goes public or an X token is configured.
|
|
|
|
|
5. **Commits:** only when the owner asks, as `Simon Diesenreiter <simon@disi.dev>`, not the global
|
|
|
|
|
(work) address: `git -c user.email=simon@disi.dev commit …`, unless the repo-local config sets it.
|
|
|
|
|
|
|
|
|
|
## Invariants that are easy to break
|
|
|
|
|
|
|
|
|
|
- **Four stages.** Adapters fetch only through `Request` objects handled by the framework's `Fetcher`
|
|
|
|
|
(robots, throttling, TDM, archive), never with their own HTTP client. `parse()` is pure: no database,
|
|
|
|
|
no network. Only the `Loader` writes claims. Enrichers derive from stored data.
|
|
|
|
|
- **Claim identity.** Claims with a facet get their fingerprint from it. Claims without a facet need an
|
|
|
|
|
explicit `key`, or duplicates and lost updates follow.
|
|
|
|
|
- **Parser changes:** bump `parser_version`. Otherwise unchanged responses are skipped and the fix
|
|
|
|
|
never applies. `politrack reparse <adapter>` rebuilds from the raw archive and deletes claims the new
|
|
|
|
|
parser no longer emits.
|
|
|
|
|
- **Golden fixtures:** new or changed parsers come with a fixture (`politrack fixture …`). Review the
|
|
|
|
|
`.expected.json` diff; don't blindly regenerate with `UPDATE_GOLDEN=1`.
|
|
|
|
|
- **Schema:** `0001_schema.sql` and `0002_seed.sql` are applied in existing databases. Change the
|
|
|
|
|
schema only with a new numbered migration, with `COMMENT`s on new tables and columns.
|
|
|
|
|
- **API types:** after changing a response model, regenerate the frontend types:
|
|
|
|
|
`cd backend && politrack openapi --ts ../web/src/lib/api-types.ts`.
|
|
|
|
|
- **UI text:** every new key goes into both `web/messages/de.json` and `en.json`.
|
|
|
|
|
|
|
|
|
|
## Commands
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
cd backend && pytest # unit + golden tests, no DB or network needed
|
|
|
|
|
politrack sources list # adapters, enrichers, schedules, queues
|
|
|
|
|
politrack run <adapter> --max-requests 20 -p key=val # limit requests when testing against live sources
|
|
|
|
|
politrack reparse <adapter> # re-parse archived responses
|
|
|
|
|
politrack enrich | status | review | migrate # enrichers, recent runs, legal review, schema
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
## The owner's development machine
|
|
|
|
|
|
|
|
|
|
These notes apply to the owner's Windows PC (Docker Desktop, Git Bash, PowerShell):
|
|
|
|
|
|
|
|
|
|
- **Git** reports "dubious ownership" for this folder. Don't change the global git config; prefix git
|
|
|
|
|
commands with `git -c safe.directory=C:/Users/disi0/source/repos/politics-tracking`.
|
|
|
|
|
- **PyPI and npm** downloads failed there in October 2026 (a network filter blocks
|
|
|
|
|
`files.pythonhosted.org` and `registry.npmjs.org`). Until that is fixed, run the tests with the Debian
|
|
|
|
|
test image ([testing.md, "Without PyPI access"](docs/contributing/testing.md#without-pypi-access)).
|
|
|
|
|
The real images, `npm install` and the workers can't be built or run there.
|
|
|
|
|
- **Local database:** `docker compose up -d db` (port 5433, passwords in the gitignored `.env`, data in
|
|
|
|
|
`data/`). It has the schema and sources but no scraped data yet, and no job-queue schema until the
|
|
|
|
|
real `migrate` image runs. Postgres on a Windows bind mount is slow: the first start takes minutes.
|
|
|
|
|
- **`DATA_DIR`, not `APPDATA`:** Windows already defines `APPDATA`, so the compose file uses `DATA_DIR`.
|