Commit Graph
20 Commits
Author SHA1 Message Date
simonandClaude Opus 5.5 7bd0142734 Partial index for claims still awaiting topic tagging
Each enrich.topics batch scanned all claims for untagged ones; after the
history backfill a run took hours and blocked the enrich queue.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 12:23:27 +01:00
simonandClaude Opus 5.5 d03b9643e9 Index chunk.claim_id and chunk.document_id
enrich.embed looks up claims without a chunk on every batch. The existing
unique index is on COALESCE expressions, which that lookup can't use, so
after the history backfill each run took hours and starved the other
enrichers on the same queue.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 11:28:25 +01:00
simonandClaude Opus 5.5 ccf1a4c755 parlament.press: an explicit backfill_days overrides the cursor
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 00:46:57 +01:00
simonandClaude Opus 5.5 405c21a8c5 Backups, freshness monitoring, persistent logs, procurement cleanup
- backup service: nightly pg_dump with retention, recorded as a run
- GET /api/v1/status/sources: 503 when a scheduled source or the backup
  missed its last two slots (for Uptime Kuma)
- worker logs to ${DATA_DIR}/logs, so they survive redeploys
- procurement: reject implausible award dates (1900, contract end dates),
  skip placeholder contractors, credit framework agreements to the BBG

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 00:46:18 +01:00
simonandClaude Opus 5.5 79639e5319 Backfill: former MEPs, older elections, EC 2014-2019, social history paging
- europarl.meps also loads the Austrian MEPs of the 2019-2024 term; roles end
  with the last membership, so former MEPs don't show as current
- bmi.elections: National Council 2017 and EU 2019 preference votes
- eu.commission_meetings: the 2014-2019 Commission
- social.bluesky / social.mastodon: -p backfill_pages=N pages older posts

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 00:34:24 +01:00
simonandClaude Opus 5.5 4bd4e5b2db Add a phone-width audit script for the web app
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 00:00:07 +01:00
simonandClaude Opus 5.5 b7b7f2f648 Add a watchdog that repairs missed schedules and stalled jobs
Every daily and weekly source missed two days in November 2026: Procrastinate
skips a periodic slot that is more than ten minutes overdue, and jobs of
killed workers stayed "doing" forever. Every ten minutes the watchdog fails
jobs of dead workers, closes abandoned run records and defers any scheduled
task whose last slot passed without a run.

Also: EP vote references of joint motions and Council documents, and the
loader now refreshes a vote's reference and stage.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 23:42:19 +01:00
simonandClaude Opus 5.5 a59d475e6b Add official election results and candidates' preference votes (BMI)
National Council 2024 and 2019 and EU 2024: party results for Austria and
the states, and preference votes of candidates who resolve to a known
politician (exact name plus birth year where published). New claim type
election_result, new resolver hint birth_year.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 23:07:51 +01:00
simonandClaude Opus 5.5 b06f17e587 Add EP amendment and part votes, hidden from general lists by default
They come from the same responses as final votes. A sitting has over a
hundred, so they are flagged part_vote and only appear on request: the vote
lists have an option for them, feeds, counts and search chunks leave them out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 13:17:10 +01:00
simonandClaude Opus 5.5 bf19e503f9 Use a raw string for the chunker's regex query
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 09:59:08 +01:00
simonandClaude Opus 5.5 1f945b8bad Stop the chunker looping forever on blank documents
A document whose text is only whitespace produces no chunks, so it was
selected again on every pass and enrich.embed never finished.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 09:56:58 +01:00
simonandClaude Opus 5.5 9c58bb3d8f Add committee memberships; tolerate empty EP declaration responses
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 02:19:25 +01:00
simonandClaude Opus 5.5 b4e26bb296 Let the web container write its fetch cache
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 02:15:06 +01:00
simonandClaude Opus 5.5 256906e9fe Prepare for going public: legal pages, corrections contact, portrait proxy
/impressum and /datenschutz read the operator's details from the environment
so they stay out of this repository, and show a visible notice until set.
Portraits go through an allowlisted /api/img so visitors' IPs stay with us.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 01:22:02 +01:00
simonandClaude Opus 5.5 cd44d8fbba Add EP final roll-call votes of Austrian MEPs
Uses the European Parliament's open data (CC BY 4.0) rather than
HowTheyVote, whose dump is ODbL share-alike and behind a robots.txt block.
Also documents the raw archive ownership step for Unraid.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 01:18:58 +01:00
simonandClaude Opus 5.5 693c32a5e9 Mark posts that disappear from X instead of erasing them; pin mcp below 2
social.x_deletions checks stored posts through the official API. A post X
reports as not found keeps a marked record (author, link, dates, Internet
Archive link if a copy exists) while every copy of its text is removed, as
the Developer Agreement requires. A post that shows up in the timeline again
is restored by the normal upsert.

mcp 2.x removed FastMCP, which crash-looped the MCP server on first deploy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 00:43:36 +01:00
simonandClaude Opus 5.5 0e21730b9f Document coding agents and give them a sandbox image
Agents work on this repository in the agent-envs sandbox: a container
whose only way out is an egress proxy with an allowlist, which also adds
the Gitea bot token. docs/contributing/agents.md covers setting it up
for Politrack (1Password item, .env, commit identity), what it can reach
and its limits; AGENTS.md tells agents how to recognise the sandbox and
behave in it.

backend/tests/Dockerfile.debian gains a `sandbox` target: the agent-envs
agent image plus the backend's dependencies from Debian packages, the
same list as the PyPI-free test image (still the default target). The
sandbox runs `pytest` without PyPI, which the owner's PC can't reach. A
new invariant in AGENTS.md keeps that list in step with pyproject.toml.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 23:59:01 +01:00
simonandClaude Opus 5.5 456f9816f4 Redesign the web app from a written design concept
Design concept and design system in docs/design/: principles (evidence
first, neutral, fair to the people shown, plain language, time as a
dimension, accessible and private), information architecture, core
patterns and a concept for every page; tokens with checked contrast,
component catalogue, content and accessibility rules.

web:
- neutral ink-indigo accent instead of flag red; party colours only as
  small markers next to the name; light/dark theme with a switcher
- stylesheets split into tokens, base, layout, components, print
- new header (search, filter count, language, theme, mobile menu) and
  footer; breadcrumbs, page headers, tabs with counts, filter bars that
  collapse on phones, pagination, notices and empty states
- claim cards with source meter, legal stage, innocence notice inside
  the card and a permalink; claim pages with citation (/angaben/{id})
- profiles with role timeline and key figures; party and organisation
  dashboards; vote list with position chips; calendar agenda on phones;
  tables that stack into cards; loading and error pages; favicon

backend:
- GET /api/v1/claims/{id} and MCP tool get_claim for permalinks
- parties list: evaluate person_status once instead of per party
  (30 s -> 1 s on the sample database, identical result)
- parliament calendar: HTML-decode titles ("84.&nbsp;Sitzung")

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 22:21:44 +01:00
simonandClaude Opus 5.5 45514624f4 Add AGENTS.md for agents picking up work in new sessions
- AGENTS.md: ground rules (legal access, evidence, fairness), invariants
  that are easy to break, commands, and notes on the owner's dev machine;
  CLAUDE.md imports it
- backend/tests/Dockerfile.debian: test image from Debian packages for
  networks where PyPI is unreachable, documented in testing.md
- roadmap: list what v2 has not been verified against yet

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 20:02:33 +01:00
simonandClaude Opus 5.5 583ccba31c Politrack: track Austrian politicians from public sources
Initial import of Politrack, a self-hosted service that tracks Austrian
politicians from federal to municipal level - roles, votes, money flows,
statements, meetings, social media and legal proceedings - with every claim
linked to its sources.

- backend: FastAPI + Postgres/pgvector; adapter framework (fetch -> parse ->
  load -> enrich) with robots.txt/TDM-aware fetching, a raw-response archive
  for reparsing, and a claim/evidence model with corroboration
- sources: Parliament, Wikidata, EU Parliament/Commission, Council of
  Ministers, Rechnungshof party reports and donations, procurement, RTR media
  transparency, political ad libraries, WKStA, news feeds, Wikipedia edits,
  Bluesky, Mastodon, public Telegram channels, YouTube and X (official API)
- web: Next.js frontend in German and English with typed API client
- mcp: MCP server for AI agents and RAG
- docs: architecture, data model, sources, legal notes, Unraid deployment and
  contributor guides; Docker Compose setup for Unraid

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 20:02:01 +01:00