landtage: always reparse member lists, drop Wien (robots.txt disallows /advuew/)
enrich.landtage judges coverage by the role claims a run touched, so an unchanged page skipped by the fetcher made a state look incomplete (Burgenland). Wien's politician search is only allowed for the city's own search appliance. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
3e6b1328b4
commit
faf9572b9f
4 files changed
+19
-33
No files matched your search
@@ -12,9 +12,11 @@ which states are covered, and the person_status view then trusts the official li
|
||||
vbg vorarlberg.at/web/landtag/abgeordnete-alle
|
||||
sbg salzburg.gv.at, one page per Landtag club
|
||||
bgld bgld-landtag.at/organe/abgeordnete (club sections; the postal addresses printed there are not kept)
|
||||
wien wien.gv.at politician search (allowed in robots.txt), Landtag = Gemeinderat
|
||||
|
||||
Kärnten is missing on purpose: ktn.gv.at disallows all crawlers in robots.txt, so its members stay Wikidata-only.
|
||||
Kärnten and Wien are missing on purpose, so their members stay Wikidata-only: ktn.gv.at disallows all crawlers in
|
||||
robots.txt, and wien.gv.at disallows /advuew/ (the politician search) for every crawler except its own search
|
||||
appliance. Every request sets skip_unchanged=False: enrich.landtage judges coverage by which role claims a run
|
||||
touched, so an unchanged page must still be parsed and loaded.
|
||||
Only names, party, district/region and mandate dates are kept (public role); no contact details or addresses.
|
||||
"""
|
||||
|
||||
@@ -24,7 +26,7 @@ import re
|
||||
from collections.abc import Iterator
|
||||
from datetime import date
|
||||
from typing import Any
|
||||
from urllib.parse import parse_qs, urljoin, urlsplit
|
||||
from urllib.parse import urljoin
|
||||
|
||||
from bs4 import BeautifulSoup
|
||||
|
||||
@@ -44,9 +46,6 @@ LANDTAGE = {
|
||||
"vbg": ("Vorarlberg", "Vorarlberger Landtag", 36, "https://vorarlberg.at/web/landtag/abgeordnete-alle"),
|
||||
"sbg": ("Salzburg", "Salzburger Landtag", 36, "https://www.salzburg.gv.at/pol/landtag/lt-klubs"),
|
||||
"bgld": ("Burgenland", "Burgenländischer Landtag", 36, "https://www.bgld-landtag.at/organe/abgeordnete/"),
|
||||
"wien": ("Wien", "Wiener Landtag", 100,
|
||||
"https://www.wien.gv.at/advuew/internet/AdvPrSrv.asp?Layout=polstelle_at&Type=K"
|
||||
"&stellecd=1997031808394662&Hlayout=politikersuche&AUSSEN"),
|
||||
}
|
||||
SBG_CLUBS = {"lt-oevp": "ÖVP", "lt-fpoe": "FPÖ", "lt-spoe": "SPÖ", "lt-kpoeplus": "KPÖ", "lt-diegruenen": "GRÜNE"}
|
||||
PARTY_ALIASES = {"KEIN KLUB": None, "VP TIROL": "ÖVP", "VP": "ÖVP", "VOLKSPARTEI": "ÖVP", "GRÜNE": "GRÜNE", "DIE GRÜNEN": "GRÜNE",
|
||||
@@ -218,18 +217,6 @@ def parse_bgld(html: str, base: str) -> list[dict[str, Any]]:
|
||||
return out
|
||||
|
||||
|
||||
def parse_wien(body: bytes, base: str) -> list[dict[str, Any]]:
|
||||
soup = BeautifulSoup(body, "html.parser") # the page declares its (legacy) charset itself
|
||||
out = []
|
||||
for a in soup.find_all("a", href=re.compile(r"Layout=politiker_at")):
|
||||
cell = a.find_parent("td")
|
||||
after = [t for t in (cell.stripped_strings if cell else []) if t not in a.stripped_strings]
|
||||
party = after[-1].split()[-1] if after else None # "SPÖ" or "Klubvorsitzende NEOS"
|
||||
person = parse_qs(urlsplit(a["href"]).query).get("personcd", [None])[0]
|
||||
out.append(member(surname_first(a.get_text(" ", strip=True)), person, party, url=urljoin(base, a["href"])))
|
||||
return out
|
||||
|
||||
|
||||
# -- adapters -------------------------------------------------------------------------------------------------
|
||||
|
||||
def landtag_spec(key: str) -> SourceSpec:
|
||||
@@ -248,7 +235,7 @@ class LandtagMembers(Adapter):
|
||||
creates_persons = True
|
||||
|
||||
def start(self, ctx: RunContext) -> Iterator[Request]:
|
||||
yield Request(url=LANDTAGE[self.key][3], kind="list")
|
||||
yield Request(url=LANDTAGE[self.key][3], kind="list", skip_unchanged=False)
|
||||
|
||||
def members(self, ctx: RunContext, response: Response) -> Iterator[Any]:
|
||||
raise NotImplementedError
|
||||
@@ -294,9 +281,9 @@ def _noe(self, ctx, response):
|
||||
profiles, pages = parse_noe_list(response.text, response.url)
|
||||
if not response.meta.get("page"):
|
||||
for page in pages:
|
||||
yield Request(url=page, kind="list", meta={"page": True})
|
||||
yield Request(url=page, kind="list", meta={"page": True}, skip_unchanged=False)
|
||||
for url in profiles:
|
||||
yield Request(url=url, kind="profile")
|
||||
yield Request(url=url, kind="profile", skip_unchanged=False)
|
||||
else:
|
||||
yield parse_noe_profile(response.text, response.request.url)
|
||||
|
||||
@@ -305,7 +292,7 @@ def _sbg(self, ctx, response):
|
||||
if response.request.kind == "list":
|
||||
for club in SBG_CLUBS:
|
||||
yield Request(url=f"https://www.salzburg.gv.at/pol/landtag/lt-klubs/{club}", kind="club",
|
||||
meta={"party": SBG_CLUBS[club]})
|
||||
meta={"party": SBG_CLUBS[club]}, skip_unchanged=False)
|
||||
else:
|
||||
yield from parse_sbg_club(response.text, response.url, response.meta["party"])
|
||||
|
||||
@@ -317,7 +304,6 @@ _register("tirol", lambda self, ctx, r: iter(parse_tirol(r.text, r.url)))
|
||||
_register("vbg", lambda self, ctx, r: iter(parse_vbg(r.text, r.url)))
|
||||
_register("sbg", _sbg)
|
||||
_register("bgld", lambda self, ctx, r: iter(parse_bgld(r.text, r.url)))
|
||||
_register("wien", lambda self, ctx, r: iter(parse_wien(r.body, r.url)))
|
||||
|
||||
|
||||
@register
|
||||
|
||||
@@ -48,7 +48,7 @@ def test_stmk_tirol_vbg():
|
||||
("Manuela Auer", "auer-manuela", "SPÖ")]
|
||||
|
||||
|
||||
def test_sbg_bgld_wien():
|
||||
def test_sbg_bgld():
|
||||
sbg = """<ul><li><a href="/pol/landtag/lt-klubs/politiker-detail-view/pallauf-brigitta">
|
||||
<img alt="Brigitta Pallauf"></a></li></ul>"""
|
||||
assert [(m["name"], m["id"], m["party"]) for m in L.parse_sbg_club(sbg, "https://x", "ÖVP")] == [
|
||||
@@ -60,9 +60,9 @@ def test_sbg_bgld_wien():
|
||||
assert [(m["name"], m["party"]) for m in members] == [
|
||||
("Gerhard Bachmann", "SPÖ"), ("Astrid Eisenkopf", "SPÖ"), ("Claudia Schlager", "SPÖ")]
|
||||
assert all(m["district"] is None for m in members) # postal addresses are never kept
|
||||
wien = """<table><tr><td><a href="/advuew/internet/AdvPrSrv.asp?Layout=politiker_at&personcd=2015111618241279">
|
||||
Abrahamczik Nina, Mag<sup>a</sup></a><br>SPÖ</td><td>+43 1 4000 81925</td></tr>
|
||||
<tr><td><a href="/advuew/internet/AdvPrSrv.asp?Layout=politiker_at&personcd=1">Arapovic Selma, Dipl.-Ing in</a>
|
||||
<br>Klubvorsitzende NEOS</td><td>+43 1</td></tr></table>""".encode("utf-8")
|
||||
assert [(m["name"], m["id"], m["party"]) for m in L.parse_wien(wien, "https://www.wien.gv.at/")] == [
|
||||
("Nina Abrahamczik", "2015111618241279", "SPÖ"), ("Selma Arapovic", "1", "NEOS")]
|
||||
|
||||
|
||||
def test_robots_excluded_states_have_no_adapter():
|
||||
from politrack.core.registry import ADAPTERS
|
||||
assert "landtag.wien" not in ADAPTERS and "landtag.ktn" not in ADAPTERS
|
||||
assert "landtag.bgld" in ADAPTERS
|
||||
+2
-2
@@ -35,7 +35,7 @@ Next.js frontend (de/en), Docker Compose for Unraid.
|
||||
- [x] EP roll-call votes of Austrian MEPs, including amendments and part votes (`europarl.votes`)
|
||||
- [x] Committee memberships of the configured legislative periods (list 250, `parlament.persons`)
|
||||
- [x] Full speech texts and interjections from the protocols (`parlament.speeches`)
|
||||
- [x] Current members of eight Landtage from their official lists (`landtag.*`); status of state MPs from those lists
|
||||
- [x] Current members of seven Landtage (not Kärnten and Wien, whose robots.txt forbids it) from their official lists (`landtag.*`); status of state MPs from those lists
|
||||
- [x] Event dates from the text (gestern, am Montag, 12. November) instead of publication dates
|
||||
- [x] Election results and preference votes: NR 2024, NR 2019, NR 2017, EU 2024, EU 2019 preference votes (`bmi.elections`)
|
||||
- [x] Former Austrian MEPs of the 2019-2024 term; EC meetings of the 2014-2019 Commission; Bluesky/Mastodon history paging
|
||||
@@ -96,7 +96,7 @@ Not run yet, because PyPI/npm were unreachable on the development network (Octob
|
||||
- Votes in the Nationalrat are faction-level except roll calls (an Austrian parliamentary practice).
|
||||
- Split votes (factions voting differently on parts of a bill) keep the official text but no faction
|
||||
positions, because the summary cannot be attributed reliably.
|
||||
- Activity status: parliament, EP and the official Landtag lists are authoritative (`status_verified`). Kärnten
|
||||
- Activity status: parliament, EP and the official Landtag lists are authoritative (`status_verified`). Kärnten, Wien
|
||||
and local offices still rely on Wikidata, whose open-ended roles count as current for 15 years after their start.
|
||||
- Roll-call surnames that are ambiguous without a link in the protocol are skipped, not guessed.
|
||||
- Travel/meeting extraction is rule-based; dates are publication dates of the source.
|
||||
|
||||
+1
-1
@@ -54,7 +54,7 @@ Status: ✅ implemented (adapter name in `code`) · 🟡 planned · ⛔ delibera
|
||||
| Meta Ad Library API | political ads of configured pages (`config/meta_pages.yaml`) | Graph API, token after identity confirmation | Meta platform terms | ✅ `ads.meta` (opt-in) |
|
||||
| RIS (data.bka.gv.at) | laws, court decisions | OGD API | CC BY 4.0 | 🟡 decisions are anonymised; only linkable via a case number cited elsewhere |
|
||||
| Lobbying- und Interessenvertretungs-Register (BMJ) | lobbying firms & clients | Lotus Notes web application without stable URLs (deep links return 404) | official work | ⛔ technically not reliably accessible; revisit if an export appears |
|
||||
| State parliaments (Landtage) | **current members** with party, district and mandate start where published; roles of members who leave a list are closed (`enrich.landtage`) | official member pages of eight Landtage (robots.txt checked per site) | official information | ✅ `landtag.noe`, `landtag.ooe`, `landtag.stmk`, `landtag.tirol`, `landtag.vbg`, `landtag.sbg`, `landtag.bgld`, `landtag.wien`; ⛔ Kärnten (ktn.gv.at disallows all crawlers); 🟡 Landtag votes and state election results |
|
||||
| State parliaments (Landtage) | **current members** with party, district and mandate start where published; roles of members who leave a list are closed (`enrich.landtage`) | official member pages of seven Landtage (robots.txt checked per site) | official information | ✅ `landtag.noe`, `landtag.ooe`, `landtag.stmk`, `landtag.tirol`, `landtag.vbg`, `landtag.sbg`, `landtag.bgld`; ⛔ Kärnten (ktn.gv.at disallows all crawlers) and Wien (wien.gv.at disallows its politician search `/advuew/` for all but its own search appliance); 🟡 Landtag votes and state election results |
|
||||
| Municipalities | mayors, councils | fragmented | – | 🟡 via Wikidata |
|
||||
| EU Council participant lists | which minister attended which Council | PDF per meeting | EU re-use policy | 🟡 not implemented yet |
|
||||
| AMA farm subsidies, EU state-aid register (TAM) | subsidy recipients | search interfaces | – | 🟡 not implemented yet (bulk access and terms still to be checked) |
|
||||
|
||||
Reference in new issue
Block a user