0 Fork 0

Files

T

simonandClaude Opus 5.5 456f9816f4 Redesign the web app from a written design concept

Design concept and design system in docs/design/: principles (evidence
first, neutral, fair to the people shown, plain language, time as a
dimension, accessible and private), information architecture, core
patterns and a concept for every page; tokens with checked contrast,
component catalogue, content and accessibility rules.

web:
- neutral ink-indigo accent instead of flag red; party colours only as
  small markers next to the name; light/dark theme with a switcher
- stylesheets split into tokens, base, layout, components, print
- new header (search, filter count, language, theme, mobile menu) and
  footer; breadcrumbs, page headers, tabs with counts, filter bars that
  collapse on phones, pagination, notices and empty states
- claim cards with source meter, legal stage, innocence notice inside
  the card and a permalink; claim pages with citation (/angaben/{id})
- profiles with role timeline and key figures; party and organisation
  dashboards; vote list with position chips; calendar agenda on phones;
  tables that stack into cards; loading and error pages; favicon

backend:
- GET /api/v1/claims/{id} and MCP tool get_claim for permalinks
- parties list: evaluate person_status once instead of per party
  (30 s -> 1 s on the sample database, identical result)
- parliament calendar: HTML-decode titles ("84.&nbsp;Sitzung")

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

2026-10-28 22:21:44 +01:00

4.9 KiB

Raw Permalink Blame History

Architecture (overview)

This page is the short version. The full design – quality goals, data model, the adapter framework down to the request level, testing and guard-rails – is in architecture/01-design-from-scratch.md; the reasoning behind each decision is in architecture/02-comparison-and-decisions.md.

Deployment

                       ┌──────────────────────────── Unraid host ─────────────────────────────┐
  Browser ──► reverse  │  web (Next.js, de/en) ──► api (FastAPI, typed JSON) ──┐                │
  (SWAG/NPM/Caddy)     │                                                       ▼                │
  AI agents ─────────► │  mcp (MCP server, streamable HTTP) ──────────► PostgreSQL 16           │
  BI / notebooks ────► │  (read-only role politrack_reader) ──────────►  + pgvector, pg_trgm    │
                       │                                                  + procrastinate queue │
                       │  worker-scrape  queues scrape,social (N replicas) ──┤                  │
                       │  worker-bulk    queue bulk (large downloads) ───────┤                  │
                       │  worker-enrich  queue enrich (linking, NLP, corroboration) ─┘          │
                       │  /data/raw  raw-response archive (gzip, content-addressed)             │
                       │  ollama (optional, bge-m3 embeddings)                                  │
                       └────────────────────────────────────────────────────────────────────────┘

One backend image runs every role (API, MCP, workers, CLI). Workers can run on other machines that reach the database (e.g. a scraper box with its own egress IP, a GPU box for embeddings).

Pipeline in one picture

SourceSpec + Adapter.start() ─▶ Fetcher ─▶ raw archive ─▶ Adapter.parse() ─▶ Records ─▶ Loader ─▶ PostgreSQL
            (code)               robots.txt, throttle,       (pure)          typed       resolve,      ▲
                                 TDM, conditional GET                                    identity,     │
                                                                                         evidence      │
                                             Enrichers (linker, events, legal, topics, corroborate) ───┘

Four strict stages – fetch (policy + archive), parse (pure), load (single writer), enrich.
Everything is a claim with evidence; the same fact from two sources is one claim (corroboration by construction).
Sources are code (SourceSpec next to the adapter), synced into the source table.
Re-parse without re-fetch (politrack reparse <adapter>) after parser fixes.
Typed API (Pydantic) → OpenAPI → generated TypeScript types for the web app.

Why this stack

Concern	Choice	Reason
Database	PostgreSQL 16 + pgvector + pg_trgm + unaccent	relational data, German full-text search, fuzzy names and vectors in one easily backed-up system that agents can query
Job queue	Procrastinate (Postgres-backed)	no Redis; cron-style periodic tasks, retries, queueing locks; horizontal workers
Backend	Python 3.12, httpx, protego, psycopg 3, FastAPI	best scraping/NLP ecosystem; one code base for collection and serving
Frontend	Next.js 15 + next-intl	server-side rendering, German at `/`, English at `/en`
Agents / RAG	MCP server, pgvector chunks, `COMMENT ON` everywhere, read-only SQL role	tools or plain SQL; every answer carries citation URLs
Embeddings	Ollama + bge-m3 (optional)	local, multilingual; full-text search works without it

No Scrapy: its strengths (crawling at scale) are not our problem, while its model fights ours (pure parsers, archive-first, per-publisher legal policy) – see ADR-20.

Interfaces for other agents

MCP – http://<host>:8001/mcp: search, find_person, get_person, person_network, list_parties, get_party, find_organization, get_organization, list_claims, get_claim, money_flows, social_posts, wikipedia_edits, calendar, person_votes, former_politicians, list_topics.
REST – http://<host>:8000/api/docs (OpenAPI 3).
SQL – role politrack_reader; start with the view claim_with_evidence; \d+ claim shows the column comments.
RAG – table chunk (text, tsv, embedding vector(1024)), each chunk linked to a claim or document for citations.

4.9 KiB Raw Permalink Blame History Unescape Escape