Politrack: track Austrian politicians from public sources
Initial import of Politrack, a self-hosted service that tracks Austrian politicians from federal to municipal level - roles, votes, money flows, statements, meetings, social media and legal proceedings - with every claim linked to its sources. - backend: FastAPI + Postgres/pgvector; adapter framework (fetch -> parse -> load -> enrich) with robots.txt/TDM-aware fetching, a raw-response archive for reparsing, and a claim/evidence model with corroboration - sources: Parliament, Wikidata, EU Parliament/Commission, Council of Ministers, Rechnungshof party reports and donations, procurement, RTR media transparency, political ad libraries, WKStA, news feeds, Wikipedia edits, Bluesky, Mastodon, public Telegram channels, YouTube and X (official API) - web: Next.js frontend in German and English with typed API client - mcp: MCP server for AI agents and RAG - docs: architecture, data model, sources, legal notes, Unraid deployment and contributor guides; Docker Compose setup for Unraid Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
commit
583ccba31c
187 files changed
+39890
No files matched your search
@@ -0,0 +1,41 @@
|
||||
"""Background jobs on Procrastinate (Postgres-backed queue; no Redis).
|
||||
|
||||
One periodic task is generated per registered adapter/enricher from its `schedule` (cron, UTC) and
|
||||
`queue`. Run workers per queue so heavy jobs don't block light ones:
|
||||
|
||||
procrastinate --app=politrack.worker.app worker --queues=scrape,social
|
||||
procrastinate --app=politrack.worker.app worker --queues=bulk
|
||||
procrastinate --app=politrack.worker.app worker --queues=enrich
|
||||
|
||||
Periodic tasks are deferred by every running worker and de-duplicated by Procrastinate; queueing locks
|
||||
prevent two runs of the same adapter at once. Workers can run on other machines that reach the DB.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
|
||||
from procrastinate import App, PsycopgConnector
|
||||
|
||||
from politrack.config import get_settings
|
||||
from politrack.core import registry
|
||||
from politrack.core.runner import run
|
||||
|
||||
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(name)s: %(message)s")
|
||||
|
||||
app = App(connector=PsycopgConnector(conninfo=get_settings().database_url))
|
||||
|
||||
|
||||
def _make_task(name: str, queue: str, schedule: str | None) -> None:
|
||||
def job(timestamp: int | None = None, **params) -> dict:
|
||||
return run(name, params=params or None)
|
||||
|
||||
job.__name__ = f"run_{name.replace('.', '_')}"
|
||||
task = app.task(name=f"politrack.run.{name}", queue=queue, queueing_lock=name, pass_context=False)(job)
|
||||
if schedule:
|
||||
app.periodic(cron=schedule, periodic_id=name)(task)
|
||||
|
||||
|
||||
registry.load_builtin()
|
||||
for _name, _cls in {**registry.ADAPTERS, **registry.ENRICHERS}.items():
|
||||
_make_task(_name, _cls.queue, _cls.schedule)
|
||||
Reference in new issue
Block a user