Current members of eight Landtage from their official lists
One adapter per state (landtag.noe, .ooe, .stmk, .tirol, .vbg, .sbg, .bgld, .wien) reads the official member list: name, party, district and mandate start where published; no contact details or addresses. enrich.landtage closes the roles of members who leave a list (skipping incomplete runs) and records which states are covered; migration 0007 makes person_status trust those lists instead of open-ended Wikidata Landtag roles. Kärnten is left out because ktn.gv.at disallows all crawlers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
d9c6c3649e
commit
3e6b1328b4
5 files changed
+482
-5
No files matched your search
@@ -0,0 +1,50 @@
|
||||
-- State MPs: trust the official Landtag member lists (adapters landtag.*) where they exist. Wikidata often has no
|
||||
-- end date for state mandates, so former members looked current; for states that enrich.landtage marks as covered,
|
||||
-- open-ended Wikidata Landtag roles no longer count, and official Landtag roles count as verified.
|
||||
CREATE OR REPLACE FUNCTION landtag_key(role_title TEXT) RETURNS TEXT LANGUAGE sql IMMUTABLE AS $$
|
||||
SELECT CASE
|
||||
WHEN role_title ILIKE '%%landtag%%' AND role_title ILIKE '%%niederösterreich%%' THEN 'noe'
|
||||
WHEN role_title ILIKE '%%landtag%%' AND role_title ILIKE '%%oberösterreich%%' THEN 'ooe'
|
||||
WHEN role_title ILIKE '%%landtag%%' AND (role_title ILIKE '%%steir%%' OR role_title ILIKE '%%steiermark%%') THEN 'stmk'
|
||||
WHEN role_title ILIKE '%%landtag%%' AND role_title ILIKE '%%tirol%%' THEN 'tirol'
|
||||
WHEN role_title ILIKE '%%landtag%%' AND role_title ILIKE '%%vorarlberg%%' THEN 'vbg'
|
||||
WHEN role_title ILIKE '%%landtag%%' AND role_title ILIKE '%%salzburg%%' THEN 'sbg'
|
||||
WHEN role_title ILIKE '%%landtag%%' AND role_title ILIKE '%%burgenl%%' THEN 'bgld'
|
||||
WHEN role_title ILIKE '%%landtag%%' AND role_title ILIKE '%%wien%%' THEN 'wien'
|
||||
WHEN role_title ILIKE '%%landtag%%' AND role_title ILIKE '%%kärnt%%' THEN 'ktn'
|
||||
END
|
||||
$$;
|
||||
COMMENT ON FUNCTION landtag_key(TEXT) IS 'State key (noe, ooe, ...) of a Landtag role title from Wikidata, NULL for other roles.';
|
||||
|
||||
CREATE OR REPLACE VIEW person_status AS
|
||||
WITH covered AS (
|
||||
SELECT jsonb_array_elements_text(value) AS key FROM state WHERE adapter = 'enrich.landtage' AND key = 'covered'
|
||||
),
|
||||
current_roles AS (
|
||||
SELECT r.person_id, r.level, r.role, r.party_id, r.start_date, c.created_by
|
||||
FROM role r JOIN claim c ON c.id = r.claim_id
|
||||
WHERE c.merged_into IS NULL AND (r.end_date IS NULL OR r.end_date >= current_date)
|
||||
AND (c.created_by LIKE 'parlament%%' OR c.created_by LIKE 'europarl%%' OR c.created_by LIKE 'landtag.%%'
|
||||
OR r.end_date IS NOT NULL OR r.level = 'party'
|
||||
OR (r.start_date IS NOT NULL AND r.start_date >= current_date - interval '15 years'))
|
||||
AND NOT (c.created_by = 'wikidata.politicians' AND r.end_date IS NULL AND r.level = 'state'
|
||||
AND landtag_key(r.role) IN (SELECT key FROM covered))
|
||||
)
|
||||
SELECT p.id AS person_id,
|
||||
CASE
|
||||
WHEN p.death_date IS NOT NULL THEN 'deceased'
|
||||
WHEN EXISTS (SELECT 1 FROM current_roles cr WHERE cr.person_id = p.id AND cr.level <> 'party') THEN 'active'
|
||||
WHEN EXISTS (SELECT 1 FROM current_roles cr WHERE cr.person_id = p.id AND cr.level = 'party') THEN 'party_member'
|
||||
ELSE 'former'
|
||||
END AS status,
|
||||
EXISTS (SELECT 1 FROM current_roles cr WHERE cr.person_id = p.id AND cr.level <> 'party'
|
||||
AND (cr.created_by LIKE 'parlament%%' OR cr.created_by LIKE 'europarl%%' OR cr.created_by LIKE 'landtag.%%'))
|
||||
AS verified,
|
||||
(SELECT max(r.end_date) FROM role r JOIN claim c ON c.id = r.claim_id
|
||||
WHERE r.person_id = p.id AND c.merged_into IS NULL AND r.level <> 'party') AS last_office_end,
|
||||
(SELECT array_agg(DISTINCT cr.role) FROM current_roles cr WHERE cr.person_id = p.id AND cr.level <> 'party')
|
||||
AS current_roles,
|
||||
(SELECT array_agg(DISTINCT cr.party_id) FROM current_roles cr WHERE cr.person_id = p.id AND cr.party_id IS NOT NULL)
|
||||
AS current_party_ids
|
||||
FROM person p;
|
||||
COMMENT ON VIEW person_status IS 'active = current office (verified = confirmed by Parliament/EP data or an official Landtag list); party_member = only party membership; former; deceased. For states covered by an official Landtag list (enrich.landtage), open-ended Wikidata Landtag roles are ignored.';
|
||||
@@ -0,0 +1,359 @@
|
||||
"""Current members of the state parliaments (Landtage), from each Landtag's official member list.
|
||||
|
||||
Wikidata knows most state MPs but often without an end date, so former members looked current. These adapters
|
||||
read the official lists; `enrich.landtage` closes the official role of anyone who has left a list and records
|
||||
which states are covered, and the person_status view then trusts the official lists for those states (migration
|
||||
0007). One adapter per state, because every Landtag publishes its list differently:
|
||||
|
||||
noe noe-landtag.gv.at/personen (3 pages) + one profile per member (birth date, start of mandate)
|
||||
ooe land-oberoesterreich.gv.at/12748.htm (party, district, "Mitglied des Landtags seit ...")
|
||||
stmk landtag.steiermark.at member table
|
||||
tirol portrait gallery ("Name (Party)")
|
||||
vbg vorarlberg.at/web/landtag/abgeordnete-alle
|
||||
sbg salzburg.gv.at, one page per Landtag club
|
||||
bgld bgld-landtag.at/organe/abgeordnete (club sections; the postal addresses printed there are not kept)
|
||||
wien wien.gv.at politician search (allowed in robots.txt), Landtag = Gemeinderat
|
||||
|
||||
Kärnten is missing on purpose: ktn.gv.at disallows all crawlers in robots.txt, so its members stay Wikidata-only.
|
||||
Only names, party, district/region and mandate dates are kept (public role); no contact details or addresses.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from collections.abc import Iterator
|
||||
from datetime import date
|
||||
from typing import Any
|
||||
from urllib.parse import parse_qs, urljoin, urlsplit
|
||||
|
||||
from bs4 import BeautifulSoup
|
||||
|
||||
from politrack.core import Adapter, Enricher, Request, Response, RunContext, SourceSpec, register, register_spec
|
||||
from politrack.core.records import Claim, Evidence, OrgRef, PersonRecord, PersonRef, Role
|
||||
from politrack.core.spec import DERIVED
|
||||
from politrack.sources.bundesregierung import german_date
|
||||
from politrack.text import make_slug, strip_titles
|
||||
|
||||
LANDTAGE = {
|
||||
# key: (region, name of the parliament, seats, member list URL)
|
||||
"noe": ("Niederösterreich", "Landtag Niederösterreich", 56, "https://noe-landtag.gv.at/personen"),
|
||||
"ooe": ("Oberösterreich", "Oberösterreichischer Landtag", 56, "https://www.land-oberoesterreich.gv.at/12748.htm"),
|
||||
"stmk": ("Steiermark", "Landtag Steiermark", 48, "https://www.landtag.steiermark.at/cms/ziel/31866695/DE/"),
|
||||
"tirol": ("Tirol", "Tiroler Landtag", 36,
|
||||
"https://www.tirol.gv.at/landtag/unser-parlament/die-abgeordneten/bildergalerie/"),
|
||||
"vbg": ("Vorarlberg", "Vorarlberger Landtag", 36, "https://vorarlberg.at/web/landtag/abgeordnete-alle"),
|
||||
"sbg": ("Salzburg", "Salzburger Landtag", 36, "https://www.salzburg.gv.at/pol/landtag/lt-klubs"),
|
||||
"bgld": ("Burgenland", "Burgenländischer Landtag", 36, "https://www.bgld-landtag.at/organe/abgeordnete/"),
|
||||
"wien": ("Wien", "Wiener Landtag", 100,
|
||||
"https://www.wien.gv.at/advuew/internet/AdvPrSrv.asp?Layout=polstelle_at&Type=K"
|
||||
"&stellecd=1997031808394662&Hlayout=politikersuche&AUSSEN"),
|
||||
}
|
||||
SBG_CLUBS = {"lt-oevp": "ÖVP", "lt-fpoe": "FPÖ", "lt-spoe": "SPÖ", "lt-kpoeplus": "KPÖ", "lt-diegruenen": "GRÜNE"}
|
||||
PARTY_ALIASES = {"KEIN KLUB": None, "VP TIROL": "ÖVP", "VP": "ÖVP", "VOLKSPARTEI": "ÖVP", "GRÜNE": "GRÜNE", "DIE GRÜNEN": "GRÜNE",
|
||||
"KPÖ PLUS": "KPÖ", "KPÖ": "KPÖ", "FRITZ": "FRITZ", "LISTE FRITZ": "FRITZ"}
|
||||
# Functions written in front of names: Klubobmann, Landtagspräsidentin, KO-Stellvertreter, ...
|
||||
ROLE_PREFIX = re.compile(
|
||||
r"^(?:(?:Landtagspräsident(?:in)?|Präsident(?:in)?|(?:Zweite|Dritte)[rn]? Präsident(?:in)?|Klubobmann|Klubobfrau|"
|
||||
r"Klubobleute|KO-Stellvertreter(?:in)?|KO|VP(?:in)?|LTPräs(?:in)?|LAbg\.|Abg\.|Bgm\.?(?:in)?|KommR|HR|Prof\.)\s+)+")
|
||||
TITLE_TOKENS = {"Mag", "Maga", "MMag", "Dr", "DI", "DIin", "Dipl", "Ing", "Prof", "HR", "BA", "MA", "BSc", "MSc", "MBA",
|
||||
"BEd", "B.Ed", "B.A", "M.A", "LL.M", "LLM", "MPA", "PhD", "KommR", "Bgm", "Päd", "Univ", "Mag.a",
|
||||
"Dr.in", "Mag.(FH)", "(FH)", "MAS", "MSc.", "M.A.I.S", "Dipl.-Ing", "Dipl.Soz.-Wiss.in", "Ing.in"}
|
||||
SUFFIX_TOKENS = {"a", "in"} # "Mag. a", "Dr. in"
|
||||
|
||||
|
||||
def _is_title(token: str) -> bool:
|
||||
t = token.strip(",.")
|
||||
return t in TITLE_TOKENS or re.fullmatch(r"(?:Dipl|Univ|Mag|Dr)\.[\w.\-]*", token.strip(",")) is not None
|
||||
|
||||
|
||||
def clean_name(name: str) -> str:
|
||||
"""'KO Mag. Markus Abwerzger' -> 'Markus Abwerzger'; drops functions, academic titles and stray punctuation.
|
||||
Titles are matched as whole tokens, so a first name like "Ingrid" is never taken for "Ing."."""
|
||||
name = ROLE_PREFIX.sub("", re.sub(r"\s+", " ", name).strip())
|
||||
name = strip_titles(name)
|
||||
kept = []
|
||||
for token in name.replace(",", " , ").split():
|
||||
# "a"/"in" are the feminine endings of split titles ("Mag. a", "Dr. in"); never part of a name
|
||||
if token == "," or _is_title(token) or token in SUFFIX_TOKENS:
|
||||
continue
|
||||
kept.append(token)
|
||||
return re.sub(r"\s+", " ", " ".join(kept)).strip(" ,.")
|
||||
|
||||
|
||||
def surname_first(text: str) -> str:
|
||||
"""'Höglinger Tobias, Mag.' / 'AMON Werner, MBA' -> 'Tobias Höglinger' / 'Werner Amon'."""
|
||||
head = text.split(",", 1)[0].strip()
|
||||
parts = head.split()
|
||||
if not parts:
|
||||
return ""
|
||||
upper = [p for p in parts if p.isupper() and len(p) > 1]
|
||||
if upper and len(upper) < len(parts): # Steiermark: surname in capitals, possibly several words
|
||||
surname = " ".join(p.title() if p.isupper() else p for p in upper)
|
||||
given = " ".join(p for p in parts if p not in upper)
|
||||
else:
|
||||
surname, given = parts[0], " ".join(parts[1:])
|
||||
return clean_name(f"{given} {surname}")
|
||||
|
||||
|
||||
def party_short(text: str | None) -> str | None:
|
||||
if not text:
|
||||
return None
|
||||
t = re.sub(r"[()\s]+", " ", text).strip()
|
||||
if t.upper() in PARTY_ALIASES:
|
||||
return PARTY_ALIASES[t.upper()]
|
||||
return t.upper() if len(t) <= 6 else t
|
||||
|
||||
|
||||
def member(name: str, member_id: str | None = None, party: str | None = None, region: str | None = None,
|
||||
since: date | None = None, birth_date: date | None = None, url: str | None = None) -> dict[str, Any]:
|
||||
return {"name": name, "id": member_id or make_slug(name), "party": party_short(party), "district": region,
|
||||
"since": since, "birth_date": birth_date, "url": url}
|
||||
|
||||
|
||||
# -- parsers (pure) -------------------------------------------------------------------------------------------
|
||||
|
||||
def parse_noe_list(html: str, base: str) -> tuple[list[str], list[str]]:
|
||||
"""Profile URLs and further list pages of the NÖ member list."""
|
||||
soup = BeautifulSoup(html, "html.parser")
|
||||
profiles = sorted({urljoin(base, a["href"]) for a in soup.find_all("a", href=re.compile(r"^/personen/[a-z0-9_\-]+$"))})
|
||||
pages = sorted({urljoin(base, a["href"]) for a in soup.find_all("a", href=re.compile(r"/personen\?seite=\d+$"))})
|
||||
return profiles, pages
|
||||
|
||||
|
||||
def parse_noe_profile(html: str, url: str) -> dict[str, Any] | None:
|
||||
soup = BeautifulSoup(html, "html.parser")
|
||||
h1 = soup.find("h1")
|
||||
text = re.sub(r"\s+", " ", soup.get_text(" "))
|
||||
mandate = re.search(r"Abgeordnete[r]? zum NÖ Landtag \(\s*([^)]+?)\s*\) seit (\d{2}\.\d{2}\.\d{4})", text)
|
||||
if not h1 or not mandate:
|
||||
return None
|
||||
born = re.search(r"Geb\.:\s*(\d{2})\.(\d{2})\.(\d{4})", text)
|
||||
district = re.search(r"Wahlkreis:\s*([^:]+?)\s+(?:Klub/Fraktion|Wahlpartei|Sitz)", text)
|
||||
d, m, y = (int(x) for x in mandate.group(2).split("."))
|
||||
return member(clean_name(h1.get_text(" ", strip=True)), url.rstrip("/").rsplit("/", 1)[-1], mandate.group(1),
|
||||
district.group(1).strip() if district else None, date(y, m, d),
|
||||
date(int(born.group(3)), int(born.group(2)), int(born.group(1))) if born else None, url)
|
||||
|
||||
|
||||
def parse_ooe(html: str, base: str) -> list[dict[str, Any]]:
|
||||
soup = BeautifulSoup(html, "html.parser")
|
||||
out = []
|
||||
for li in soup.find_all("li", attrs={"data-title": True, "data-partei": True}):
|
||||
text = re.sub(r"\s+", " ", li.get_text(" "))
|
||||
since = re.search(r"Mitglied des Landtags seit (\d{1,2}\. \w+ \d{4})", text)
|
||||
district = re.search(r"Wahlkreis \d+ - ([^M]+?)\s+Mitglied", text)
|
||||
link = li.find("a", href=True)
|
||||
out.append(member(surname_first(li["data-title"]), None, li["data-partei"],
|
||||
district.group(1).strip() if district else None,
|
||||
german_date(since.group(1)) if since else None, None,
|
||||
urljoin(base, link["href"]) if link else None))
|
||||
return out
|
||||
|
||||
|
||||
def parse_stmk(html: str, base: str) -> list[dict[str, Any]]:
|
||||
soup = BeautifulSoup(html, "html.parser")
|
||||
out = []
|
||||
for row in soup.find_all("tr"):
|
||||
cells = row.find_all("td")
|
||||
link = row.find("a", href=re.compile(r"\.pdf$", re.IGNORECASE))
|
||||
if len(cells) < 2 or not link:
|
||||
continue
|
||||
label = link.get_text(" ", strip=True)
|
||||
out.append(member(surname_first(label), make_slug(label.split(",")[0]), cells[-1].get_text(" ", strip=True),
|
||||
url=base))
|
||||
return out
|
||||
|
||||
|
||||
def parse_tirol(html: str, base: str) -> list[dict[str, Any]]:
|
||||
soup = BeautifulSoup(html, "html.parser")
|
||||
seen, out = set(), []
|
||||
for img in soup.find_all("img", alt=re.compile(r"\(.+\)\s*$")):
|
||||
m = re.match(r"(.+?)\s*\(\s*([^()]+?)\s*\)\s*$", img["alt"])
|
||||
name = clean_name(m.group(1)) if m else ""
|
||||
if name and name not in seen and "Mandat" not in name:
|
||||
seen.add(name)
|
||||
out.append(member(name, party=m.group(2), url=base))
|
||||
return out
|
||||
|
||||
|
||||
def parse_vbg(html: str, base: str) -> list[dict[str, Any]]:
|
||||
soup = BeautifulSoup(html, "html.parser")
|
||||
out = []
|
||||
for box in soup.find_all("div", class_="avlr-standard-ap-mit-bild"):
|
||||
link = box.find("a", href=re.compile(r"/web/landtag/-/"))
|
||||
h3 = box.find("h3")
|
||||
# the party is the last parenthesis ("Mag. (FH)" or "(WU)" in degrees come earlier)
|
||||
parties = re.findall(r"\(([^()]{2,15})\)", box.get_text(" "))
|
||||
if link and h3 and parties:
|
||||
out.append(member(clean_name(h3.get_text(" ", strip=True)), link["href"].rstrip("/").rsplit("/", 1)[-1],
|
||||
parties[-1], url=urljoin(base, link["href"])))
|
||||
return out
|
||||
|
||||
|
||||
def parse_sbg_club(html: str, base: str, party: str) -> list[dict[str, Any]]:
|
||||
soup = BeautifulSoup(html, "html.parser")
|
||||
out = []
|
||||
for a in soup.find_all("a", href=re.compile(r"/politiker-detail-view/")):
|
||||
img = a.find("img", alt=True)
|
||||
if img:
|
||||
out.append(member(clean_name(img["alt"]), a["href"].rstrip("/").rsplit("/", 1)[-1], party,
|
||||
url=urljoin(base, a["href"])))
|
||||
return out
|
||||
|
||||
|
||||
BGLD_MEMBER = re.compile(
|
||||
r"(?:@[\w.\-]+\.at|Bgld\. Landtag)\s+(.{3,90}?)\s+(?:\d\.\s)?(?:Landtagsabgeordnete[r]?|Landtagspräsident(?:in)?|"
|
||||
r"Klubobmann|Klubobfrau|Klubvorsitzende[r]?)\s+(SPÖ|ÖVP|FPÖ|GRÜNE|Grüne|NEOS)\s+\d{4}\s")
|
||||
|
||||
|
||||
def parse_bgld(html: str, base: str) -> list[dict[str, Any]]:
|
||||
"""Club sections list "<name> <function> <party> <postcode> <place> <e-mail>"; only name and party are kept."""
|
||||
text = re.sub(r"[\s]+", " ", BeautifulSoup(html, "html.parser").get_text(" "))
|
||||
out, seen = [], set()
|
||||
for m in BGLD_MEMBER.finditer(text):
|
||||
name = clean_name(m.group(1))
|
||||
if name and name not in seen:
|
||||
seen.add(name)
|
||||
out.append(member(name, party=m.group(2), url=base))
|
||||
return out
|
||||
|
||||
|
||||
def parse_wien(body: bytes, base: str) -> list[dict[str, Any]]:
|
||||
soup = BeautifulSoup(body, "html.parser") # the page declares its (legacy) charset itself
|
||||
out = []
|
||||
for a in soup.find_all("a", href=re.compile(r"Layout=politiker_at")):
|
||||
cell = a.find_parent("td")
|
||||
after = [t for t in (cell.stripped_strings if cell else []) if t not in a.stripped_strings]
|
||||
party = after[-1].split()[-1] if after else None # "SPÖ" or "Klubvorsitzende NEOS"
|
||||
person = parse_qs(urlsplit(a["href"]).query).get("personcd", [None])[0]
|
||||
out.append(member(surname_first(a.get_text(" ", strip=True)), person, party, url=urljoin(base, a["href"])))
|
||||
return out
|
||||
|
||||
|
||||
# -- adapters -------------------------------------------------------------------------------------------------
|
||||
|
||||
def landtag_spec(key: str) -> SourceSpec:
|
||||
region, name, _, url = LANDTAGE[key]
|
||||
return register_spec(SourceSpec(
|
||||
slug=f"landtag-{key}", name=name, kind="official", homepage=url, license="Amtliche Information",
|
||||
access="scrape", trust=90, min_interval=2.0, store_fulltext=False,
|
||||
tos_notes=f"Official member list of the {name} (public role data: name, party, district, mandate dates). "
|
||||
"robots.txt allows the pages used; no contact details or addresses are stored."))
|
||||
|
||||
|
||||
class LandtagMembers(Adapter):
|
||||
key: str = ""
|
||||
schedule = "35 5 * * *"
|
||||
queue = "scrape"
|
||||
creates_persons = True
|
||||
|
||||
def start(self, ctx: RunContext) -> Iterator[Request]:
|
||||
yield Request(url=LANDTAGE[self.key][3], kind="list")
|
||||
|
||||
def members(self, ctx: RunContext, response: Response) -> Iterator[Any]:
|
||||
raise NotImplementedError
|
||||
|
||||
def parse(self, ctx: RunContext, response: Response) -> Iterator[Any]:
|
||||
for item in self.members(ctx, response):
|
||||
if isinstance(item, Request):
|
||||
yield item
|
||||
elif item and item.get("name"):
|
||||
yield from self.records(item)
|
||||
else:
|
||||
ctx.stats["entries_skipped"] += 1
|
||||
|
||||
def records(self, m: dict[str, Any]) -> Iterator[Any]:
|
||||
region, body_name, _, list_url = LANDTAGE[self.key]
|
||||
scheme = f"landtag-{self.key}"
|
||||
ref = PersonRef(ids={scheme: m["id"]}, name=m["name"])
|
||||
yield PersonRecord(ref=ref, full_name=m["name"], birth_date=m["birth_date"], match_by_name=True)
|
||||
body = OrgRef(slug=f"landtag-{self.key}", name=body_name, kind="parliament")
|
||||
party = OrgRef(short_name=m["party"], kind="party", create=False) if m["party"] else None
|
||||
label = f" ({m['party']})" if m["party"] else ""
|
||||
yield Claim(
|
||||
claim_type="role", person=PersonRef(ids={scheme: m["id"]}), org=body, valid_from=m["since"],
|
||||
summary_de=f"{m['name']}: Abgeordnete/r zum {body_name}{label}",
|
||||
summary_en=f"{m['name']}: member of the {body_name}{label}",
|
||||
data={"landtag": self.key, "party": m["party"], "district": m["district"]},
|
||||
evidence=[Evidence(url=m["url"] or list_url, doc_type="register_entry", title=f"{body_name}: Abgeordnete",
|
||||
locator=f"member:{m['id']}")],
|
||||
facet=Role(role="mp", level="state", body=body, party=party, region=region, start=m["since"]),
|
||||
)
|
||||
|
||||
|
||||
def _register(key: str, members_fn) -> None:
|
||||
cls = type(f"Landtag_{key}", (LandtagMembers,), {
|
||||
"key": key, "spec": landtag_spec(key), "name": f"landtag.{key}",
|
||||
"description": f"Current members of the {LANDTAGE[key][1]} (official list)", "members": members_fn,
|
||||
})
|
||||
register(cls)
|
||||
|
||||
|
||||
def _noe(self, ctx, response):
|
||||
if response.request.kind == "list":
|
||||
profiles, pages = parse_noe_list(response.text, response.url)
|
||||
if not response.meta.get("page"):
|
||||
for page in pages:
|
||||
yield Request(url=page, kind="list", meta={"page": True})
|
||||
for url in profiles:
|
||||
yield Request(url=url, kind="profile")
|
||||
else:
|
||||
yield parse_noe_profile(response.text, response.request.url)
|
||||
|
||||
|
||||
def _sbg(self, ctx, response):
|
||||
if response.request.kind == "list":
|
||||
for club in SBG_CLUBS:
|
||||
yield Request(url=f"https://www.salzburg.gv.at/pol/landtag/lt-klubs/{club}", kind="club",
|
||||
meta={"party": SBG_CLUBS[club]})
|
||||
else:
|
||||
yield from parse_sbg_club(response.text, response.url, response.meta["party"])
|
||||
|
||||
|
||||
_register("noe", _noe)
|
||||
_register("ooe", lambda self, ctx, r: iter(parse_ooe(r.text, r.url)))
|
||||
_register("stmk", lambda self, ctx, r: iter(parse_stmk(r.text, r.url)))
|
||||
_register("tirol", lambda self, ctx, r: iter(parse_tirol(r.text, r.url)))
|
||||
_register("vbg", lambda self, ctx, r: iter(parse_vbg(r.text, r.url)))
|
||||
_register("sbg", _sbg)
|
||||
_register("bgld", lambda self, ctx, r: iter(parse_bgld(r.text, r.url)))
|
||||
_register("wien", lambda self, ctx, r: iter(parse_wien(r.body, r.url)))
|
||||
|
||||
|
||||
@register
|
||||
class LandtagStatus(Enricher):
|
||||
"""Closes official Landtag roles of members who are no longer on their list and records which states have a
|
||||
current official list (state key `covered`), which the person_status view uses instead of Wikidata."""
|
||||
|
||||
spec = DERIVED
|
||||
name = "enrich.landtage"
|
||||
description = "Close roles of members who left a Landtag list; mark states covered by an official list"
|
||||
schedule = "15 6 * * *"
|
||||
MIN_SHARE = 0.6 # a run that confirmed fewer members than this share of the seats is treated as broken
|
||||
|
||||
def run(self, ctx: RunContext) -> None:
|
||||
conn = ctx.conn
|
||||
covered = []
|
||||
for key, (_, _, seats, _) in LANDTAGE.items():
|
||||
last = conn.execute(
|
||||
"SELECT started_at FROM run WHERE adapter = %s AND status IN ('ok', 'partial') "
|
||||
"ORDER BY id DESC LIMIT 1", (f"landtag.{key}",)).fetchone()
|
||||
if not last:
|
||||
continue
|
||||
confirmed = conn.execute(
|
||||
"SELECT count(*) AS n FROM claim WHERE created_by = %s AND claim_type = 'role' AND updated_at >= %s",
|
||||
(f"landtag.{key}", last["started_at"])).fetchone()["n"]
|
||||
if confirmed < seats * self.MIN_SHARE:
|
||||
ctx.stats[f"{key}_incomplete"] = confirmed
|
||||
continue
|
||||
covered.append(key)
|
||||
closed = conn.execute(
|
||||
"""WITH gone AS (
|
||||
SELECT r.claim_id FROM role r JOIN claim c ON c.id = r.claim_id
|
||||
WHERE c.created_by = %(adapter)s AND r.end_date IS NULL AND c.updated_at < %(since)s)
|
||||
UPDATE role SET end_date = %(today)s WHERE claim_id IN (SELECT claim_id FROM gone)""",
|
||||
{"adapter": f"landtag.{key}", "since": last["started_at"], "today": date.today()}).rowcount
|
||||
ctx.stats[f"{key}_closed"] = closed
|
||||
ctx.state.set("covered", covered)
|
||||
ctx.stats["covered"] = len(covered)
|
||||
return None
|
||||
@@ -0,0 +1,68 @@
|
||||
"""Landtag member-list parsers, on snippets of each Landtag's real markup (pages fetched November 2026)."""
|
||||
|
||||
from datetime import date
|
||||
|
||||
from politrack.sources import landtage as L
|
||||
|
||||
|
||||
def test_clean_name_drops_functions_and_titles_but_not_names():
|
||||
assert L.clean_name("KO Mag. Markus Abwerzger") == "Markus Abwerzger"
|
||||
assert L.clean_name("Dipl. Päd., DI Carina Laschober-Luif") == "Carina Laschober-Luif"
|
||||
assert L.clean_name("Mag. a Astrid Eisenkopf") == "Astrid Eisenkopf"
|
||||
assert L.clean_name("Ingrid Korosec") == "Ingrid Korosec" # "Ing" is a title, "Ingrid" is not
|
||||
assert L.surname_first("Höglinger Tobias, Mag.") == "Tobias Höglinger"
|
||||
assert L.surname_first("EISEL-EISELSBERG Detlev") == "Detlev Eisel-Eiselsberg"
|
||||
|
||||
|
||||
def test_noe_profile():
|
||||
html = """<html><body><h1>Otto Auer</h1><h2>Lebenslauf</h2><p>Geb.: 04.01.1966 , Hainburg/Donau Beruf: Landwirt
|
||||
Wahlkreis: Bruck an der Leitha Klub/Fraktion: Landtagsklub der Volkspartei Niederösterreich</p>
|
||||
<table><tr><td>Abgeordneter zum NÖ Landtag ( ÖVP )</td><td>seit 23.03.2023</td></tr></table></body></html>"""
|
||||
m = L.parse_noe_profile(html, "https://noe-landtag.gv.at/personen/otto_auer")
|
||||
assert (m["name"], m["id"], m["party"], m["district"]) == ("Otto Auer", "otto_auer", "ÖVP", "Bruck an der Leitha")
|
||||
assert (m["since"], m["birth_date"]) == (date(2023, 3, 23), date(1966, 1, 4))
|
||||
|
||||
|
||||
def test_ooe():
|
||||
html = """<ul><li class="list-item" data-groups="GRÜNE" data-title="Huber-Reiter Brigitte" data-partei="GRÜNE">
|
||||
<h4><a href="/abg/huber">Huber-Reiter Brigitte</a></h4><p>Betriebsratsvorsitzende</p>
|
||||
<p class="mb-2">Bezirk Ried<br> Wahlkreis 2 - Innviertel</p>
|
||||
<p class="mb-0">Mitglied des Landtags seit <span class="text-nowrap">13. November 2025</span></p></li></ul>"""
|
||||
[m] = L.parse_ooe(html, "https://www.land-oberoesterreich.gv.at/12748.htm")
|
||||
assert (m["name"], m["party"], m["district"], m["since"]) == ("Brigitte Huber-Reiter", "GRÜNE", "Innviertel",
|
||||
date(2025, 11, 13))
|
||||
|
||||
|
||||
def test_stmk_tirol_vbg():
|
||||
stmk = """<table><tr><td><a href="/cms/dokumente/1/AMON%20Werner.pdf">AMON Werner, MBA</a></td><td> ÖVP</td></tr>
|
||||
</table>"""
|
||||
assert [(m["name"], m["party"]) for m in L.parse_stmk(stmk, "https://x")] == [("Werner Amon", "ÖVP")]
|
||||
tirol = """<img alt="KO Mag. Markus Abwerzger (FPÖ)"><img alt="Christoph Appler (VP Tirol)">
|
||||
<img alt="Dr. Georg Dornauer (kein Klub)">"""
|
||||
assert [(m["name"], m["party"]) for m in L.parse_tirol(tirol, "https://x")] == [
|
||||
("Markus Abwerzger", "FPÖ"), ("Christoph Appler", "ÖVP"), ("Georg Dornauer", None)]
|
||||
vbg = """<div class="avlr-standard-ap-mit-bild"><a href="https://vorarlberg.at/web/landtag/-/auer-manuela">
|
||||
<figcaption>Bildrechte: mathis.studio</figcaption></a><h3>Mag. (FH) Manuela Auer</h3>
|
||||
<div>Landtagsabgeordnete (SPÖ)</div></div>"""
|
||||
assert [(m["name"], m["id"], m["party"]) for m in L.parse_vbg(vbg, "https://x")] == [
|
||||
("Manuela Auer", "auer-manuela", "SPÖ")]
|
||||
|
||||
|
||||
def test_sbg_bgld_wien():
|
||||
sbg = """<ul><li><a href="/pol/landtag/lt-klubs/politiker-detail-view/pallauf-brigitta">
|
||||
<img alt="Brigitta Pallauf"></a></li></ul>"""
|
||||
assert [(m["name"], m["id"], m["party"]) for m in L.parse_sbg_club(sbg, "https://x", "ÖVP")] == [
|
||||
("Brigitta Pallauf", "pallauf-brigitta", "ÖVP")]
|
||||
bgld = """<p>SPÖ-Abgeordnete zum Bgld. Landtag Gerhard Bachmann Landtagsabgeordneter SPÖ 2423 Deutsch Jahrndorf
|
||||
gerhard.bachmann@spoeklub-bgld.at Mag. a Astrid Eisenkopf Landtagspräsidentin SPÖ 7035 Steinbrunn
|
||||
astrid.eisenkopf@bgld-landtag.at Claudia Schlager 3. Landtagspräsidentin SPÖ 7210 Mattersburg x@y.at</p>"""
|
||||
members = L.parse_bgld(bgld, "https://x")
|
||||
assert [(m["name"], m["party"]) for m in members] == [
|
||||
("Gerhard Bachmann", "SPÖ"), ("Astrid Eisenkopf", "SPÖ"), ("Claudia Schlager", "SPÖ")]
|
||||
assert all(m["district"] is None for m in members) # postal addresses are never kept
|
||||
wien = """<table><tr><td><a href="/advuew/internet/AdvPrSrv.asp?Layout=politiker_at&personcd=2015111618241279">
|
||||
Abrahamczik Nina, Mag<sup>a</sup></a><br>SPÖ</td><td>+43 1 4000 81925</td></tr>
|
||||
<tr><td><a href="/advuew/internet/AdvPrSrv.asp?Layout=politiker_at&personcd=1">Arapovic Selma, Dipl.-Ing in</a>
|
||||
<br>Klubvorsitzende NEOS</td><td>+43 1</td></tr></table>""".encode("utf-8")
|
||||
assert [(m["name"], m["id"], m["party"]) for m in L.parse_wien(wien, "https://www.wien.gv.at/")] == [
|
||||
("Nina Abrahamczik", "2015111618241279", "SPÖ"), ("Selma Arapovic", "1", "NEOS")]
|
||||
+4
-4
@@ -35,6 +35,7 @@ Next.js frontend (de/en), Docker Compose for Unraid.
|
||||
- [x] EP roll-call votes of Austrian MEPs, including amendments and part votes (`europarl.votes`)
|
||||
- [x] Committee memberships of the configured legislative periods (list 250, `parlament.persons`)
|
||||
- [x] Full speech texts and interjections from the protocols (`parlament.speeches`)
|
||||
- [x] Current members of eight Landtage from their official lists (`landtag.*`); status of state MPs from those lists
|
||||
- [x] Event dates from the text (gestern, am Montag, 12. November) instead of publication dates
|
||||
- [x] Election results and preference votes: NR 2024, NR 2019, NR 2017, EU 2024, EU 2019 preference votes (`bmi.elections`)
|
||||
- [x] Former Austrian MEPs of the 2019-2024 term; EC meetings of the 2014-2019 Commission; Bluesky/Mastodon history paging
|
||||
@@ -58,7 +59,7 @@ Next.js frontend (de/en), Docker Compose for Unraid.
|
||||
- [ ] Election results before 2019, NR 2019 state lists, Landtag and municipal elections
|
||||
- [ ] AMA/TAM subsidy recipients (after checking bulk access and terms)
|
||||
- [ ] Cabinet staff and appointments to state-owned companies (from answers to parliamentary questions)
|
||||
- [ ] One adapter per Landtag (OGD where available); municipal councils and mayors
|
||||
- [ ] Landtag votes (only Steiermark has an API) and state election results; municipal councils and mayors
|
||||
- [ ] Lobbying register – if the BMJ ever offers stable URLs or an export
|
||||
|
||||
**Quality**
|
||||
@@ -95,9 +96,8 @@ Not run yet, because PyPI/npm were unreachable on the development network (Octob
|
||||
- Votes in the Nationalrat are faction-level except roll calls (an Austrian parliamentary practice).
|
||||
- Split votes (factions voting differently on parts of a bill) keep the official text but no faction
|
||||
positions, because the summary cannot be attributed reliably.
|
||||
- Activity status: parliament and EP data are authoritative (`status_verified`). Open-ended roles from
|
||||
Wikidata count as current for 15 years after their start, so some state/local politicians may show
|
||||
wrong status until per-state adapters exist.
|
||||
- Activity status: parliament, EP and the official Landtag lists are authoritative (`status_verified`). Kärnten
|
||||
and local offices still rely on Wikidata, whose open-ended roles count as current for 15 years after their start.
|
||||
- Roll-call surnames that are ambiguous without a link in the protocol are skipped, not guessed.
|
||||
- Travel/meeting extraction is rule-based; dates are publication dates of the source.
|
||||
- News coverage is headline-level; mentions are linked by exact name or "office title + surname".
|
||||
|
||||
+1
-1
@@ -54,7 +54,7 @@ Status: ✅ implemented (adapter name in `code`) · 🟡 planned · ⛔ delibera
|
||||
| Meta Ad Library API | political ads of configured pages (`config/meta_pages.yaml`) | Graph API, token after identity confirmation | Meta platform terms | ✅ `ads.meta` (opt-in) |
|
||||
| RIS (data.bka.gv.at) | laws, court decisions | OGD API | CC BY 4.0 | 🟡 decisions are anonymised; only linkable via a case number cited elsewhere |
|
||||
| Lobbying- und Interessenvertretungs-Register (BMJ) | lobbying firms & clients | Lotus Notes web application without stable URLs (deep links return 404) | official work | ⛔ technically not reliably accessible; revisit if an export appears |
|
||||
| State parliaments & governments (9 Länder) | Landtag members, votes | varies; several publish OGD | mostly CC BY 4.0 | 🟡 one adapter per state; until then via Wikidata |
|
||||
| State parliaments (Landtage) | **current members** with party, district and mandate start where published; roles of members who leave a list are closed (`enrich.landtage`) | official member pages of eight Landtage (robots.txt checked per site) | official information | ✅ `landtag.noe`, `landtag.ooe`, `landtag.stmk`, `landtag.tirol`, `landtag.vbg`, `landtag.sbg`, `landtag.bgld`, `landtag.wien`; ⛔ Kärnten (ktn.gv.at disallows all crawlers); 🟡 Landtag votes and state election results |
|
||||
| Municipalities | mayors, councils | fragmented | – | 🟡 via Wikidata |
|
||||
| EU Council participant lists | which minister attended which Council | PDF per meeting | EU re-use policy | 🟡 not implemented yet |
|
||||
| AMA farm subsidies, EU state-aid register (TAM) | subsidy recipients | search interfaces | – | 🟡 not implemented yet (bulk access and terms still to be checked) |
|
||||
|
||||
Reference in new issue
Block a user