simonandClaude Opus 5.5
b06f17e587
They come from the same responses as final votes. A sitting has over a hundred, so they are flagged part_vote and only appear on request: the vote lists have an option for them, feeds, counts and search chunks leave them out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
161 lines
14 KiB
Markdown
161 lines
14 KiB
Markdown
# Source catalogue & access policy
|
||
|
||
**Rule #1: we only collect data in ways the publisher permits.** We prefer official APIs and open-data
|
||
downloads, then feeds and sitemaps that publishers offer for automated discovery, then polite crawling of
|
||
public government pages. We never log in, never bypass paywalls or bot protection, and never scrape
|
||
platforms whose ToS forbid it. The supermarket price-tracker case from the brief failed for exactly that
|
||
reason: it scraped commercial sites against their ToS.
|
||
|
||
Enforcement is technical, not a convention – every request goes through the Fetcher
|
||
([`backend/politrack/core/fetch.py`](../backend/politrack/core/fetch.py)):
|
||
|
||
* an honest User-Agent with a contact address (`POLITRACK_USER_AGENT`),
|
||
* robots.txt is obeyed (cached per origin),
|
||
* per-host throttling (`SourceSpec.min_interval`, raised by a robots.txt `Crawl-delay`), `Retry-After` honoured,
|
||
* TDM reservations are detected (W3C TDMRep `/.well-known/tdmrep.json`, `/tdmrep.json`, `tdm-reservation`
|
||
header). The text-and-data-mining exception (Art. 4 DSM Directive, §42h UrhG) does not apply once a site
|
||
reserves its rights in machine-readable form; we then keep headline and link only,
|
||
* per-publisher storage rules from the `SourceSpec`: `store_fulltext` (may we keep text?) and `keep_raw`
|
||
(may we archive the raw response?).
|
||
|
||
**robots.txt exemptions** (`SourceSpec.robots_exempt`) exist only for documented APIs whose operators allow
|
||
programmatic use but list the endpoint in robots.txt to keep search-engine crawlers away:
|
||
the Wikidata Query Service (`/sparql`; [Wikimedia User-Agent policy](https://meta.wikimedia.org/wiki/User-Agent_policy)),
|
||
the MediaWiki API of de/en Wikipedia (`/w/api.php`; [API etiquette](https://www.mediawiki.org/wiki/API:Etiquette)),
|
||
and the authenticated X API (governed by the developer agreement, not by crawler rules).
|
||
|
||
Status: ✅ implemented (adapter name in `code`) · 🟡 planned · ⛔ deliberately not used.
|
||
|
||
## Official sources (high trust)
|
||
|
||
| Source | Data | Access | Licence / terms | Status |
|
||
|---|---|---|---|---|
|
||
| Parlament Österreich | MPs since 1918, mandates, functions, **side jobs & income category** (§6 Unv-Transparenz-G), social-media links, plenary speeches, **faction votes** per item and reading | JSON API (`/Filter/api/filter/data/409,101,251`, `?json=TRUE` detail pages) | [CC BY 4.0](https://www.parlament.gv.at/recherchieren/open-data/daten-und-lizenz/) | ✅ `parlament.persons`, `parlament.items` |
|
||
| – committee memberships | who sat on which committee, with function and dates | JSON API (`/Filter/api/filter/data/250`) | CC BY 4.0 | ✅ `parlament.persons` |
|
||
| – stenographic protocols | **Roll-call votes**: every member's yes/no from the "Mit „Ja“/„Nein“ stimmten …" lists | HTML fragments | CC BY 4.0 | ✅ `parlament.items` |
|
||
| – immunity waivers | prosecutors' requests to lift immunity: authority, offence, outcome | items + committee report | CC BY 4.0 | ✅ `parlament.items` |
|
||
| – written questions and answers | who asked which minister what | item list (J / AB) | CC BY 4.0 | ✅ `parlament.items` |
|
||
| – investigative committees (U-Ausschüsse) | **hearings**: who testified when | item list (KOMM) | CC BY 4.0 | ✅ `parlament.items` |
|
||
| Parlamentskorrespondenz | official press releases (full text); people linked by parliament id where the release links them, otherwise by the entity linker | list 110 + JSON | CC BY 4.0 | ✅ `parlament.press` |
|
||
| Parliament calendar | sessions, committees, hearings, events | list 600 | CC BY 4.0 | ✅ `parlament.calendar` |
|
||
| Rechnungshof – donations | **party donations** (CSV per year/election) | CSV downloads | official works (§7 UrhG) | ✅ `rechnungshof.donations` |
|
||
| Rechnungshof – party annual reports | income/expense totals, **public funding**, donations, sponsoring, ads, **loans**, **holdings**, **affiliated organisations** | XLSX + PDF per party and year | official works | ✅ `rechnungshof.party_reports` |
|
||
| Bundeskanzleramt – government news | summits, trips abroad, bilateral meetings | HTML archive per year (robots.txt only blocks /401) | official works | ✅ `bundesregierung.news` |
|
||
| Bundeskanzleramt – Council of Ministers minutes | every agenda item with the **reporting minister(s)**, file number and subject | HTML per session | official works | ✅ `bundesregierung.ministerrat` |
|
||
| WKStA press releases | official statements on corruption proceedings (stored as documents; legal claims only after review) | HTML | official works | ✅ `justiz.wksta` |
|
||
| Public procurement – data.gv.at "Vergabe-Kerndaten" | **contract awards**: buyer, winner, amount, CPV (eForms) | OGD hub catalogue → per-publisher XML | open data (CC BY 4.0) | ✅ `vergabe.awards` |
|
||
| RTR / KommAustria Medientransparenz | **government advertising** and **media subsidies** per payer → medium | open-data API | free re-use, attribution | ✅ `rtr.medientransparenz` |
|
||
| European Parliament | Austrian MEPs, roles, **declarations of interests** | open-data/website XML | CC BY 4.0 | ✅ `europarl.meps` |
|
||
| – plenary roll calls | **votes of Austrian MEPs** on whole texts and on amendments or parts (flagged `part_vote`), with totals and later corrections | Open Data API v2 (`/meetings/{id}/decisions`) | CC BY 4.0 | ✅ `europarl.votes` |
|
||
| European Commission – meetings | meetings of Commissioners/cabinets with **interest representatives** (Austrians and Austrian organisations) | XLSX export | EU re-use policy | ✅ `eu.commission_meetings` |
|
||
| Wikipedia (de/en) | anonymous edits from **parliament/government IP ranges** (`config/ip_ranges.yaml`) | MediaWiki API `usercontribs` | CC BY-SA (edit metadata) | ✅ `wikipedia.institution_edits` |
|
||
| Google Political Ads Transparency | Austrian political ads: advertiser, spend bracket, impressions | public bundle (zip, CSV) | Google's transparency report terms | ✅ `ads.google` |
|
||
| Meta Ad Library API | political ads of configured pages (`config/meta_pages.yaml`) | Graph API, token after identity confirmation | Meta platform terms | ✅ `ads.meta` (opt-in) |
|
||
| RIS (data.bka.gv.at) | laws, court decisions | OGD API | CC BY 4.0 | 🟡 decisions are anonymised; only linkable via a case number cited elsewhere |
|
||
| Lobbying- und Interessenvertretungs-Register (BMJ) | lobbying firms & clients | Lotus Notes web application without stable URLs (deep links return 404) | official work | ⛔ technically not reliably accessible; revisit if an export appears |
|
||
| State parliaments & governments (9 Länder) | Landtag members, votes | varies; several publish OGD | mostly CC BY 4.0 | 🟡 one adapter per state; until then via Wikidata |
|
||
| Municipalities | mayors, councils | fragmented | – | 🟡 via Wikidata |
|
||
| EU Council participant lists | which minister attended which Council | PDF per meeting | EU re-use policy | 🟡 not implemented yet |
|
||
| AMA farm subsidies, EU state-aid register (TAM) | subsidy recipients | search interfaces | – | 🟡 not implemented yet (bulk access and terms still to be checked) |
|
||
| Election results, candidate lists | results per municipality, candidates | BMI / data.gv.at, official gazettes | open data / official | 🟡 not implemented yet |
|
||
| Insolvency notices (Ediktsdatei) | companies tied to politicians | Justiz web application | terms to be checked before automating | 🟡 |
|
||
| Firmenbuch, Grundbuch, beneficial owners (WiEReG) | company roles, property | paid clearing houses; WiEReG public access restricted since the 2022 CJEU ruling | – | ⛔ cost and legal limits |
|
||
| Criminal record (*Strafregister*) | – | not public | – | ⛔ does not exist publicly |
|
||
|
||
## NGOs & aggregators
|
||
|
||
| Source | Data | Access | Terms | Status |
|
||
|---|---|---|---|---|
|
||
| Wikidata | positions incl. Landtage and mayors, party membership, careers, birth dates, **social-media handles**, ids | SPARQL | CC0 | ✅ `wikidata.politicians` |
|
||
| GDELT DOC 2.0 | article metadata for politicians' names, attributed to the original outlets | API, ≤ 1 request / 5 s | free | ✅ `gdelt.coverage` (supplement; thin for Austrian outlets) |
|
||
| HowTheyVote.eu | EP roll-call votes | CSV dump on GitHub (robots.txt blocks its download path), API | ODbL 1.0 (share-alike) | ⛔ not needed: `europarl.votes` takes the same votes from the EP under CC BY 4.0 |
|
||
| OpenSanctions | PEP cross-references | bulk JSON | CC BY-NC 4.0 | 🟡 non-commercial only |
|
||
| MeineAbgeordneten.at | curated dossiers | – | sui-generis database right (§76c UrhG) | ⛔ scraping → 🟡 ask for a partnership |
|
||
| Parteispenden.at | party-finance analysis | – | ask | 🟡 partnership |
|
||
|
||
## Media
|
||
|
||
The large Austrian outlets do not allow systematic extraction of their articles, and several declare TDM
|
||
reservations. We therefore (`news.feeds`, `config/feeds.yaml`):
|
||
|
||
* only read the **RSS feeds** and **Google-News sitemaps** they publish for automated discovery,
|
||
* store **headline, teaser (≤ 400 characters) / sitemap keywords and link** only (`store_fulltext=False`,
|
||
`keep_raw=False` for every media `SourceSpec`), never the article body,
|
||
* for sites with a TDM reservation (e.g. ots.at) keep only headline and link,
|
||
* always send readers to the original article.
|
||
|
||
| Outlet | Feed / sitemap |
|
||
|---|---|
|
||
| ORF | `rss.orf.at/news.xml` |
|
||
| DER STANDARD | RSS `inland` + news sitemap (robots.txt blocks AI crawlers by name; we only use RSS/sitemap) |
|
||
| Die Presse, KURIER | RSS + news sitemap |
|
||
| profil, Salzburger Nachrichten, Kleine Zeitung | news sitemap listed in robots.txt |
|
||
| FALTER, Krone | RSS |
|
||
| APA-OTS | RSS; TDM reserved → headline + link only. Full-text licence available from APA. |
|
||
|
||
**Legal caveat:** whether automated name-matching on headlines counts as TDM under §42h UrhG is not
|
||
settled. The conservative implementation keeps the risk low; for more, licence content (APA) or ask outlets.
|
||
|
||
## Social media
|
||
|
||
Accounts come from parliament profiles (verified), Wikidata (unverified until confirmed) and the curated
|
||
`config/social_accounts.yaml` (party channels and accounts you checked; `config.social_accounts`).
|
||
|
||
| Platform | Access | Status |
|
||
|---|---|---|
|
||
| Bluesky | public AppView API (`public.api.bsky.app`), open protocol | ✅ `social.bluesky` |
|
||
| Mastodon / Fediverse | public REST API of the account's instance | ✅ `social.mastodon` |
|
||
| Telegram – public broadcast channels | public web preview `t.me/s/<channel>` (no login, no robots.txt restriction); latest posts | ✅ `social.telegram` |
|
||
| YouTube | Data API v3 (free quota, `POLITRACK_YOUTUBE_API_KEY`) | ✅ `social.youtube` (opt-in) |
|
||
| X (Twitter) | **official API v2 only** (paid read access, `POLITRACK_X_BEARER_TOKEN`). No raw archive or full text kept; posts gone from X keep a marked record without their text (`social.x_deletions`) | ✅ `social.x` (opt-in) |
|
||
| Facebook / Instagram | ToS forbid scraping; Meta Content Library is for researchers only | ⛔ posts · ✅ political ads via the Ad Library API |
|
||
| TikTok | Research API (academic only) | ⛔ |
|
||
|
||
### Messaging platforms: Telegram, WhatsApp, Signal
|
||
|
||
Some Austrian politicians and parties run broadcast channels on messengers. What we found and how we treat it:
|
||
|
||
* **Telegram** – the most used one in Austrian politics, mainly by the FPÖ and the MFG. Reported
|
||
examples: Herbert Kickl's channel had about 31,700 subscribers in October 2023
|
||
([profil](https://www.profil.at)), later about 33,000; the FPÖ channel "Festung Österreich" about
|
||
19,000–21,000 ([kontrast.at](https://kontrast.at)); FPÖ TV has been on Telegram since 2019
|
||
([DER STANDARD](https://www.derstandard.at)). Public channels we could verify with a web preview:
|
||
`t.me/s/MFGOesterreich` (MFG, ≈ 11,000 subscribers, linked from mfg-oe.at) and `t.me/s/dominikNepp`
|
||
(Dominik Nepp, FPÖ Wien). Channels that disable the web preview (e.g. Kickl's) cannot be read without the
|
||
Telegram client/API and an account – we don't do that. Beware of look-alike channels (`herbert_kickl` with
|
||
2 subscribers is not his).
|
||
*Policy:* public broadcast channels via the web preview only, low frequency; never private groups or chats.
|
||
* **WhatsApp** – Channels exist (e.g. campaign channels such as Van der Bellen's in the 2022 presidential
|
||
campaign), but WhatsApp offers no public web view or API for reading channels, and its terms forbid
|
||
automated collection. ⛔ Not collected. Link a channel in `social_accounts.yaml` only as information.
|
||
* **Signal** – private by design; no public channels or web views exist. ⛔ Nothing to collect.
|
||
|
||
Context: [Democracy Reporting International](https://democracy-reporting.org) describes Telegram as a way
|
||
for parties to reach supporters outside the content rules of mainstream platforms – one reason to include it.
|
||
|
||
## Approaches we deliberately do not use
|
||
|
||
**Scraping full news articles:** even where robots.txt allows crawling, storing or mining article text
|
||
conflicts with publishers' terms and, where declared, their TDM reservation (§42h UrhG). Headlines,
|
||
teasers and sitemap keywords plus a link give us *who/what/when*. Legal ways to get more: licences
|
||
(APA-OTS full text, **APA-DeFacto**), library access (Austrian National Library), agreements with outlets.
|
||
|
||
**X/Twitter proxies, mirrors or "viewer" services (Nitter, xcancel and similar):** ⛔. They work by
|
||
scraping X against its terms, so using them means relying on a ToS violation – the exact risk to avoid.
|
||
They also break constantly. Legal options: the official API (`social.x`), X's oEmbed for displaying a
|
||
specific post someone links to, the politicians' cross-posts on Bluesky/Mastodon/websites, and Internet
|
||
Archive copies as evidence links for individual deleted posts.
|
||
|
||
**Paywall bypasses** (archive.ph/12ft-style services, cookie or user-agent tricks, "reader" proxies): ⛔.
|
||
Circumventing technical protection measures can violate §90c UrhG and the publishers' terms. Instead:
|
||
link to the paywalled article, prefer free and public-service sources (ORF, Parlamentskorrespondenz, OTS
|
||
headlines, Dossier, Kobuk, Moment), and get legitimate access for in-depth work. **ANNO** (Austrian
|
||
National Library) offers digitised historical newspapers for free.
|
||
|
||
**Telegram client API / user accounts, WhatsApp, Signal, private groups:** ⛔ (see above).
|
||
|
||
## Community submissions
|
||
|
||
Users can submit a link backing a claim (for example a regional newspaper reporting on a mayor).
|
||
Submissions are moderated and get a low default trust (30) until verified. 🟡
|