simonandClaude Opus 5.5
583ccba31c
Initial import of Politrack, a self-hosted service that tracks Austrian politicians from federal to municipal level - roles, votes, money flows, statements, meetings, social media and legal proceedings - with every claim linked to its sources. - backend: FastAPI + Postgres/pgvector; adapter framework (fetch -> parse -> load -> enrich) with robots.txt/TDM-aware fetching, a raw-response archive for reparsing, and a claim/evidence model with corroboration - sources: Parliament, Wikidata, EU Parliament/Commission, Council of Ministers, Rechnungshof party reports and donations, procurement, RTR media transparency, political ad libraries, WKStA, news feeds, Wikipedia edits, Bluesky, Mastodon, public Telegram channels, YouTube and X (official API) - web: Next.js frontend in German and English with typed API client - mcp: MCP server for AI agents and RAG - docs: architecture, data model, sources, legal notes, Unraid deployment and contributor guides; Docker Compose setup for Unraid Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
14 KiB
Source catalogue & access policy
Rule #1: we only collect data in ways the publisher permits. We prefer official APIs and open-data downloads, then feeds and sitemaps that publishers offer for automated discovery, then polite crawling of public government pages. We never log in, never bypass paywalls or bot protection, and never scrape platforms whose ToS forbid it. The supermarket price-tracker case from the brief failed for exactly that reason: it scraped commercial sites against their ToS.
Enforcement is technical, not a convention – every request goes through the Fetcher
(backend/politrack/core/fetch.py):
- an honest User-Agent with a contact address (
POLITRACK_USER_AGENT), - robots.txt is obeyed (cached per origin),
- per-host throttling (
SourceSpec.min_interval, raised by a robots.txtCrawl-delay),Retry-Afterhonoured, - TDM reservations are detected (W3C TDMRep
/.well-known/tdmrep.json,/tdmrep.json,tdm-reservationheader). The text-and-data-mining exception (Art. 4 DSM Directive, §42h UrhG) does not apply once a site reserves its rights in machine-readable form; we then keep headline and link only, - per-publisher storage rules from the
SourceSpec:store_fulltext(may we keep text?) andkeep_raw(may we archive the raw response?).
robots.txt exemptions (SourceSpec.robots_exempt) exist only for documented APIs whose operators allow
programmatic use but list the endpoint in robots.txt to keep search-engine crawlers away:
the Wikidata Query Service (/sparql; Wikimedia User-Agent policy),
the MediaWiki API of de/en Wikipedia (/w/api.php; API etiquette),
and the authenticated X API (governed by the developer agreement, not by crawler rules).
Status: ✅ implemented (adapter name in code) · 🟡 planned · ⛔ deliberately not used.
Official sources (high trust)
| Source | Data | Access | Licence / terms | Status |
|---|---|---|---|---|
| Parlament Österreich | MPs since 1918, mandates, functions, side jobs & income category (§6 Unv-Transparenz-G), social-media links, plenary speeches, faction votes per item and reading | JSON API (/Filter/api/filter/data/409,101,251, ?json=TRUE detail pages) |
CC BY 4.0 | ✅ parlament.persons, parlament.items |
| – stenographic protocols | Roll-call votes: every member's yes/no from the "Mit „Ja“/„Nein“ stimmten …" lists | HTML fragments | CC BY 4.0 | ✅ parlament.items |
| – immunity waivers | prosecutors' requests to lift immunity: authority, offence, outcome | items + committee report | CC BY 4.0 | ✅ parlament.items |
| – written questions and answers | who asked which minister what | item list (J / AB) | CC BY 4.0 | ✅ parlament.items |
| – investigative committees (U-Ausschüsse) | hearings: who testified when | item list (KOMM) | CC BY 4.0 | ✅ parlament.items |
| Parlamentskorrespondenz | official press releases (full text); people linked by parliament id where the release links them, otherwise by the entity linker | list 110 + JSON | CC BY 4.0 | ✅ parlament.press |
| Parliament calendar | sessions, committees, hearings, events | list 600 | CC BY 4.0 | ✅ parlament.calendar |
| Rechnungshof – donations | party donations (CSV per year/election) | CSV downloads | official works (§7 UrhG) | ✅ rechnungshof.donations |
| Rechnungshof – party annual reports | income/expense totals, public funding, donations, sponsoring, ads, loans, holdings, affiliated organisations | XLSX + PDF per party and year | official works | ✅ rechnungshof.party_reports |
| Bundeskanzleramt – government news | summits, trips abroad, bilateral meetings | HTML archive per year (robots.txt only blocks /401) | official works | ✅ bundesregierung.news |
| Bundeskanzleramt – Council of Ministers minutes | every agenda item with the reporting minister(s), file number and subject | HTML per session | official works | ✅ bundesregierung.ministerrat |
| WKStA press releases | official statements on corruption proceedings (stored as documents; legal claims only after review) | HTML | official works | ✅ justiz.wksta |
| Public procurement – data.gv.at "Vergabe-Kerndaten" | contract awards: buyer, winner, amount, CPV (eForms) | OGD hub catalogue → per-publisher XML | open data (CC BY 4.0) | ✅ vergabe.awards |
| RTR / KommAustria Medientransparenz | government advertising and media subsidies per payer → medium | open-data API | free re-use, attribution | ✅ rtr.medientransparenz |
| European Parliament | Austrian MEPs, roles, declarations of interests | open-data/website XML | CC BY 4.0 | ✅ europarl.meps |
| European Commission – meetings | meetings of Commissioners/cabinets with interest representatives (Austrians and Austrian organisations) | XLSX export | EU re-use policy | ✅ eu.commission_meetings |
| Wikipedia (de/en) | anonymous edits from parliament/government IP ranges (config/ip_ranges.yaml) |
MediaWiki API usercontribs |
CC BY-SA (edit metadata) | ✅ wikipedia.institution_edits |
| Google Political Ads Transparency | Austrian political ads: advertiser, spend bracket, impressions | public bundle (zip, CSV) | Google's transparency report terms | ✅ ads.google |
| Meta Ad Library API | political ads of configured pages (config/meta_pages.yaml) |
Graph API, token after identity confirmation | Meta platform terms | ✅ ads.meta (opt-in) |
| RIS (data.bka.gv.at) | laws, court decisions | OGD API | CC BY 4.0 | 🟡 decisions are anonymised; only linkable via a case number cited elsewhere |
| Lobbying- und Interessenvertretungs-Register (BMJ) | lobbying firms & clients | Lotus Notes web application without stable URLs (deep links return 404) | official work | ⛔ technically not reliably accessible; revisit if an export appears |
| State parliaments & governments (9 Länder) | Landtag members, votes | varies; several publish OGD | mostly CC BY 4.0 | 🟡 one adapter per state; until then via Wikidata |
| Municipalities | mayors, councils | fragmented | – | 🟡 via Wikidata |
| EU Council participant lists | which minister attended which Council | PDF per meeting | EU re-use policy | 🟡 not implemented yet |
| AMA farm subsidies, EU state-aid register (TAM) | subsidy recipients | search interfaces | – | 🟡 not implemented yet (bulk access and terms still to be checked) |
| Election results, candidate lists | results per municipality, candidates | BMI / data.gv.at, official gazettes | open data / official | 🟡 not implemented yet |
| Insolvency notices (Ediktsdatei) | companies tied to politicians | Justiz web application | terms to be checked before automating | 🟡 |
| Firmenbuch, Grundbuch, beneficial owners (WiEReG) | company roles, property | paid clearing houses; WiEReG public access restricted since the 2022 CJEU ruling | – | ⛔ cost and legal limits |
| Criminal record (Strafregister) | – | not public | – | ⛔ does not exist publicly |
NGOs & aggregators
| Source | Data | Access | Terms | Status |
|---|---|---|---|---|
| Wikidata | positions incl. Landtage and mayors, party membership, careers, birth dates, social-media handles, ids | SPARQL | CC0 | ✅ wikidata.politicians |
| GDELT DOC 2.0 | article metadata for politicians' names, attributed to the original outlets | API, ≤ 1 request / 5 s | free | ✅ gdelt.coverage (supplement; thin for Austrian outlets) |
| HowTheyVote.eu | EP roll-call votes | CSV dump | check licence | 🟡 |
| OpenSanctions | PEP cross-references | bulk JSON | CC BY-NC 4.0 | 🟡 non-commercial only |
| MeineAbgeordneten.at | curated dossiers | – | sui-generis database right (§76c UrhG) | ⛔ scraping → 🟡 ask for a partnership |
| Parteispenden.at | party-finance analysis | – | ask | 🟡 partnership |
Media
The large Austrian outlets do not allow systematic extraction of their articles, and several declare TDM
reservations. We therefore (news.feeds, config/feeds.yaml):
- only read the RSS feeds and Google-News sitemaps they publish for automated discovery,
- store headline, teaser (≤ 400 characters) / sitemap keywords and link only (
store_fulltext=False,keep_raw=Falsefor every mediaSourceSpec), never the article body, - for sites with a TDM reservation (e.g. ots.at) keep only headline and link,
- always send readers to the original article.
| Outlet | Feed / sitemap |
|---|---|
| ORF | rss.orf.at/news.xml |
| DER STANDARD | RSS inland + news sitemap (robots.txt blocks AI crawlers by name; we only use RSS/sitemap) |
| Die Presse, KURIER | RSS + news sitemap |
| profil, Salzburger Nachrichten, Kleine Zeitung | news sitemap listed in robots.txt |
| FALTER, Krone | RSS |
| APA-OTS | RSS; TDM reserved → headline + link only. Full-text licence available from APA. |
Legal caveat: whether automated name-matching on headlines counts as TDM under §42h UrhG is not settled. The conservative implementation keeps the risk low; for more, licence content (APA) or ask outlets.
Social media
Accounts come from parliament profiles (verified), Wikidata (unverified until confirmed) and the curated
config/social_accounts.yaml (party channels and accounts you checked; config.social_accounts).
| Platform | Access | Status |
|---|---|---|
| Bluesky | public AppView API (public.api.bsky.app), open protocol |
✅ social.bluesky |
| Mastodon / Fediverse | public REST API of the account's instance | ✅ social.mastodon |
| Telegram – public broadcast channels | public web preview t.me/s/<channel> (no login, no robots.txt restriction); latest posts |
✅ social.telegram |
| YouTube | Data API v3 (free quota, POLITRACK_YOUTUBE_API_KEY) |
✅ social.youtube (opt-in) |
| X (Twitter) | official API v2 only (paid read access, POLITRACK_X_BEARER_TOKEN). No raw archive or full text kept; posts deleted on X must be removed (deletion sync still open, see roadmap) |
✅ social.x (opt-in) |
| Facebook / Instagram | ToS forbid scraping; Meta Content Library is for researchers only | ⛔ posts · ✅ political ads via the Ad Library API |
| TikTok | Research API (academic only) | ⛔ |
Messaging platforms: Telegram, WhatsApp, Signal
Some Austrian politicians and parties run broadcast channels on messengers. What we found and how we treat it:
- Telegram – the most used one in Austrian politics, mainly by the FPÖ and the MFG. Reported
examples: Herbert Kickl's channel had about 31,700 subscribers in October 2023
(profil), later about 33,000; the FPÖ channel "Festung Österreich" about
19,000–21,000 (kontrast.at); FPÖ TV has been on Telegram since 2019
(DER STANDARD). Public channels we could verify with a web preview:
t.me/s/MFGOesterreich(MFG, ≈ 11,000 subscribers, linked from mfg-oe.at) andt.me/s/dominikNepp(Dominik Nepp, FPÖ Wien). Channels that disable the web preview (e.g. Kickl's) cannot be read without the Telegram client/API and an account – we don't do that. Beware of look-alike channels (herbert_kicklwith 2 subscribers is not his). Policy: public broadcast channels via the web preview only, low frequency; never private groups or chats. - WhatsApp – Channels exist (e.g. campaign channels such as Van der Bellen's in the 2022 presidential
campaign), but WhatsApp offers no public web view or API for reading channels, and its terms forbid
automated collection. ⛔ Not collected. Link a channel in
social_accounts.yamlonly as information. - Signal – private by design; no public channels or web views exist. ⛔ Nothing to collect.
Context: Democracy Reporting International describes Telegram as a way for parties to reach supporters outside the content rules of mainstream platforms – one reason to include it.
Approaches we deliberately do not use
Scraping full news articles: even where robots.txt allows crawling, storing or mining article text conflicts with publishers' terms and, where declared, their TDM reservation (§42h UrhG). Headlines, teasers and sitemap keywords plus a link give us who/what/when. Legal ways to get more: licences (APA-OTS full text, APA-DeFacto), library access (Austrian National Library), agreements with outlets.
X/Twitter proxies, mirrors or "viewer" services (Nitter, xcancel and similar): ⛔. They work by
scraping X against its terms, so using them means relying on a ToS violation – the exact risk to avoid.
They also break constantly. Legal options: the official API (social.x), X's oEmbed for displaying a
specific post someone links to, the politicians' cross-posts on Bluesky/Mastodon/websites, and Internet
Archive copies as evidence links for individual deleted posts.
Paywall bypasses (archive.ph/12ft-style services, cookie or user-agent tricks, "reader" proxies): ⛔. Circumventing technical protection measures can violate §90c UrhG and the publishers' terms. Instead: link to the paywalled article, prefer free and public-service sources (ORF, Parlamentskorrespondenz, OTS headlines, Dossier, Kobuk, Moment), and get legitimate access for in-depth work. ANNO (Austrian National Library) offers digitised historical newspapers for free.
Telegram client API / user accounts, WhatsApp, Signal, private groups: ⛔ (see above).
Community submissions
Users can submit a link backing a claim (for example a regional newspaper reporting on a mayor). Submissions are moderated and get a low default trust (30) until verified. 🟡