simonandClaude Opus 5.5
693c32a5e9
social.x_deletions checks stored posts through the official API. A post X reports as not found keeps a marked record (author, link, dates, Internet Archive link if a copy exists) while every copy of its text is removed, as the Developer Agreement requires. A post that shows up in the timeline again is restored by the normal upsert. mcp 2.x removed FastMCP, which crash-looped the MCP server on first deploy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
106 lines
7.4 KiB
Markdown
106 lines
7.4 KiB
Markdown
# AGENTS.md
|
|
|
|
Guidance for AI coding agents working on this repository. Humans: start with [README.md](README.md)
|
|
and [CONTRIBUTING.md](CONTRIBUTING.md); everything below links there instead of repeating it.
|
|
|
|
Politrack tracks Austrian politicians from EU to municipal level (roles, votes, money, statements,
|
|
meetings, social media, legal proceedings). Every fact is a claim backed by evidence links. Backend:
|
|
Python (FastAPI, Postgres 16 + pgvector, Procrastinate workers, MCP server). Frontend: Next.js 15 with
|
|
next-intl, German by default, English at `/en`. Deployed with Docker Compose on the owner's Unraid server.
|
|
|
|
## Before you start
|
|
|
|
| Task | Read |
|
|
|---|---|
|
|
| Pick up open work | [docs/roadmap.md](docs/roadmap.md): *Next*, *Not yet verified*, *Known limitations* |
|
|
| Add a data source | [docs/contributing/adapters.md](docs/contributing/adapters.md), legal checklist in §0 first |
|
|
| Derive facts from stored data | [docs/contributing/enrichers.md](docs/contributing/enrichers.md) |
|
|
| API endpoint or MCP tool | [docs/contributing/api.md](docs/contributing/api.md) |
|
|
| UI page or new claim type in the UI | [docs/contributing/frontend.md](docs/contributing/frontend.md) |
|
|
| How the UI should look and read | [docs/design/README.md](docs/design/README.md), [design-system.md](docs/design/design-system.md) |
|
|
| Tests and fixtures | [docs/contributing/testing.md](docs/contributing/testing.md) |
|
|
| Agent sandbox: setup, what it can reach | [docs/contributing/agents.md](docs/contributing/agents.md) |
|
|
| Why it is built this way | [docs/architecture/02-comparison-and-decisions.md](docs/architecture/02-comparison-and-decisions.md) |
|
|
| Tables, claim types, identifier schemes | [docs/data-model.md](docs/data-model.md) |
|
|
| What each source allows | [docs/sources.md](docs/sources.md), [docs/legal.md](docs/legal.md) |
|
|
|
|
## Ground rules
|
|
|
|
1. **Legal access only.** Collect as much as possible, but only through official APIs, open data and
|
|
pages whose terms and robots.txt allow it. Never use X proxies (Nitter and the like), paywall
|
|
bypasses, logged-in scraping, or WhatsApp/Signal. Telegram: public channels via the `t.me/s` web
|
|
preview only. A robots.txt exemption needs the publisher's documented permission, recorded in the
|
|
adapter's `SourceSpec.tos_notes`; the RTR API is the only one so far. If a source can't be accessed
|
|
cleanly, say so and propose legal alternatives instead of working around it.
|
|
2. **Every claim has evidence.** No claim without a document link.
|
|
3. **Fairness.** Neutral wording. Legal matters carry the procedural stage and "Es gilt die
|
|
Unschuldsvermutung". Automatically detected legal claims stay in review (`review_status = pending`).
|
|
Connections are not accusations. Personal data only in the person's public role.
|
|
4. **Not public yet.** The instance is private (LAN only) as of November 2026. Posts that disappear from
|
|
X are marked, not erased, but their text must go (`social.x_deletions`, docs/legal.md).
|
|
5. **Commits:** only when the owner asks, as `Simon Diesenreiter <simon@disi.dev>`, not the global
|
|
(work) address: `git -c user.email=simon@disi.dev commit …`, unless the repo-local config sets it.
|
|
|
|
## Invariants that are easy to break
|
|
|
|
- **Four stages.** Adapters fetch only through `Request` objects handled by the framework's `Fetcher`
|
|
(robots, throttling, TDM, archive), never with their own HTTP client. `parse()` is pure: no database,
|
|
no network. Only the `Loader` writes claims. Enrichers derive from stored data.
|
|
- **Claim identity.** Claims with a facet get their fingerprint from it. Claims without a facet need an
|
|
explicit `key`, or duplicates and lost updates follow.
|
|
- **Parser changes:** bump `parser_version`. Otherwise unchanged responses are skipped and the fix
|
|
never applies. `politrack reparse <adapter>` rebuilds from the raw archive and deletes claims the new
|
|
parser no longer emits.
|
|
- **Golden fixtures:** new or changed parsers come with a fixture (`politrack fixture …`). Review the
|
|
`.expected.json` diff; don't blindly regenerate with `UPDATE_GOLDEN=1`.
|
|
- **Schema:** `0001_schema.sql` and `0002_seed.sql` are applied in existing databases. Change the
|
|
schema only with a new numbered migration, with `COMMENT`s on new tables and columns.
|
|
- **API types:** after changing a response model, regenerate the frontend types:
|
|
`cd backend && politrack openapi --ts ../web/src/lib/api-types.ts`.
|
|
- **UI text:** every new key goes into both `web/messages/de.json` and `en.json`.
|
|
- **Backend dependencies:** a dependency added to or changed in `backend/pyproject.toml` also goes
|
|
into the Debian package list in `backend/tests/Dockerfile.debian` (usually `python3-<name>`). That
|
|
list builds both the PyPI-free test image and the agent sandbox image; without the entry, tests
|
|
there fail on the import. If Debian doesn't package it, add it to the exceptions in the file's header.
|
|
|
|
## Commands
|
|
|
|
```bash
|
|
cd backend && pytest # unit + golden tests, no DB or network needed
|
|
politrack sources list # adapters, enrichers, schedules, queues
|
|
politrack run <adapter> --max-requests 20 -p key=val # limit requests when testing against live sources
|
|
politrack reparse <adapter> # re-parse archived responses
|
|
politrack enrich | status | review | migrate # enrichers, recent runs, legal review, schema
|
|
```
|
|
|
|
## In the agent-envs sandbox
|
|
|
|
You are in the sandbox if the repository is at `/workspace` and `HTTPS_PROXY` is `http://egress:8080`
|
|
([details](docs/contributing/agents.md)). There:
|
|
|
|
- **Network:** only allowlisted hosts. HTTP 403 "Blocked by egress policy" means the host isn't
|
|
allowed: tell the owner which host you need instead of working around it.
|
|
- **Git:** pushing and Gitea API calls over HTTPS work without credentials; the proxy adds them.
|
|
Don't set up tokens or credential helpers. The commit identity comes from the repository's config.
|
|
- **Tests:** the sandbox image has the backend's dependencies from Debian packages, so
|
|
`cd backend && python3 -m pytest -q` works without `pip install`. It comes from
|
|
`backend/tests/Dockerfile.debian` (see "Backend dependencies" above).
|
|
- **No Docker, no database.** Leave adapter runs against the database and the compose stack to the
|
|
owner. The notes below on Git's `safe.directory` don't apply in the sandbox; the PyPI and npm block
|
|
does when the session runs on the owner's PC.
|
|
|
|
## The owner's development machine
|
|
|
|
These notes apply to the owner's Windows PC (Docker Desktop, Git Bash, PowerShell):
|
|
|
|
- **Git** reports "dubious ownership" for this folder. Don't change the global git config; prefix git
|
|
commands with `git -c safe.directory=C:/Users/disi0/source/repos/politics-tracking`.
|
|
- **PyPI and npm** downloads failed there in October 2026 (a network filter blocks
|
|
`files.pythonhosted.org` and `registry.npmjs.org`). Until that is fixed, run the tests with the Debian
|
|
test image ([testing.md, "Without PyPI access"](docs/contributing/testing.md#without-pypi-access)).
|
|
The real images, `npm install` and the workers can't be built or run there.
|
|
- **Local database:** `docker compose up -d db` (port 5433, passwords in the gitignored `.env`, data in
|
|
`data/`). It has the schema and sources but no scraped data yet, and no job-queue schema until the
|
|
real `migrate` image runs. Postgres on a Windows bind mount is slow: the first start takes minutes.
|
|
- **`DATA_DIR`, not `APPDATA`:** Windows already defines `APPDATA`, so the compose file uses `DATA_DIR`.
|