simonandClaude Opus 5.5
583ccba31c
Initial import of Politrack, a self-hosted service that tracks Austrian politicians from federal to municipal level - roles, votes, money flows, statements, meetings, social media and legal proceedings - with every claim linked to its sources. - backend: FastAPI + Postgres/pgvector; adapter framework (fetch -> parse -> load -> enrich) with robots.txt/TDM-aware fetching, a raw-response archive for reparsing, and a claim/evidence model with corroboration - sources: Parliament, Wikidata, EU Parliament/Commission, Council of Ministers, Rechnungshof party reports and donations, procurement, RTR media transparency, political ad libraries, WKStA, news feeds, Wikipedia edits, Bluesky, Mastodon, public Telegram channels, YouTube and X (official API) - web: Next.js frontend in German and English with typed API client - mcp: MCP server for AI agents and RAG - docs: architecture, data model, sources, legal notes, Unraid deployment and contributor guides; Docker Compose setup for Unraid Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
42 lines
1.6 KiB
Python
42 lines
1.6 KiB
Python
"""Background jobs on Procrastinate (Postgres-backed queue; no Redis).
|
|
|
|
One periodic task is generated per registered adapter/enricher from its `schedule` (cron, UTC) and
|
|
`queue`. Run workers per queue so heavy jobs don't block light ones:
|
|
|
|
procrastinate --app=politrack.worker.app worker --queues=scrape,social
|
|
procrastinate --app=politrack.worker.app worker --queues=bulk
|
|
procrastinate --app=politrack.worker.app worker --queues=enrich
|
|
|
|
Periodic tasks are deferred by every running worker and de-duplicated by Procrastinate; queueing locks
|
|
prevent two runs of the same adapter at once. Workers can run on other machines that reach the DB.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import logging
|
|
|
|
from procrastinate import App, PsycopgConnector
|
|
|
|
from politrack.config import get_settings
|
|
from politrack.core import registry
|
|
from politrack.core.runner import run
|
|
|
|
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(name)s: %(message)s")
|
|
|
|
app = App(connector=PsycopgConnector(conninfo=get_settings().database_url))
|
|
|
|
|
|
def _make_task(name: str, queue: str, schedule: str | None) -> None:
|
|
def job(timestamp: int | None = None, **params) -> dict:
|
|
return run(name, params=params or None)
|
|
|
|
job.__name__ = f"run_{name.replace('.', '_')}"
|
|
task = app.task(name=f"politrack.run.{name}", queue=queue, queueing_lock=name, pass_context=False)(job)
|
|
if schedule:
|
|
app.periodic(cron=schedule, periodic_id=name)(task)
|
|
|
|
|
|
registry.load_builtin()
|
|
for _name, _cls in {**registry.ADAPTERS, **registry.ENRICHERS}.items():
|
|
_make_task(_name, _cls.queue, _cls.schedule)
|