Mark posts that disappear from X instead of erasing them; pin mcp below 2
social.x_deletions checks stored posts through the official API. A post X reports as not found keeps a marked record (author, link, dates, Internet Archive link if a copy exists) while every copy of its text is removed, as the Developer Agreement requires. A post that shows up in the timeline again is restored by the normal upsert. mcp 2.x removed FastMCP, which crash-looped the MCP server on first deploy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
0e21730b9f
commit
693c32a5e9
18 files changed
+265
-16
No files matched your search
@@ -81,8 +81,9 @@ no public channels and groups are private by design.
|
||||
**Context.** Many Austrian politicians are most active on X; proxies such as Nitter work by violating
|
||||
X's terms. **Decision.** `social.x` uses the official API v2 with the operator's bearer token and does
|
||||
nothing without one. It keeps no raw archive and no full text (`keep_raw=False`, `store_fulltext=False`).
|
||||
The Developer Agreement requires removing posts deleted on X; until a deletion sync exists the adapter is
|
||||
documented as not production-ready.
|
||||
The Developer Agreement requires removing posts deleted on X. For accountability we mark instead of erase:
|
||||
`social.x_deletions` clears every copy of the text and keeps a record that the post existed and disappeared
|
||||
(a new `PostRemoved` record, so the Loader stays the only writer).
|
||||
|
||||
### ADR-23: Organisation-owned accounts
|
||||
**Context.** Party and institution channels (MFG on Telegram, NEOS and the Federal President on Bluesky)
|
||||
|
||||
+1
-1
@@ -11,7 +11,7 @@
|
||||
* **Database right** (§76c ff. UrhG): systematically extracting substantial parts of someone else's curated database (for example MeineAbgeordneten) is not allowed without permission, so we ask for partnerships instead.
|
||||
* **ToS**: we never scrape platforms whose terms prohibit it (X, Meta, most newspapers). We use official APIs or leave the source out. Each source's assessment is recorded in its `SourceSpec.tos_notes` (shown on the method page and in `/api/v1/sources`).
|
||||
* **Raw archive**: responses are archived only where the publisher's terms allow it (`SourceSpec.keep_raw`) and never for TDM-reserved sites. The archive serves re-parsing and auditability, not republication.
|
||||
* **X API**: the Developer Agreement requires removing content of posts that were deleted on X. `social.x` keeps no raw archive or full text, but the deletion sync is still open – don't enable it in production before it exists (see roadmap).
|
||||
* **X API**: the Developer Agreement requires removing content of posts that were deleted on X. `social.x` keeps no raw archive or full text, and `social.x_deletions` checks stored posts four times a day. For accountability, a post that disappears is **marked, not erased**: author, link, dates and a neutral "no longer available" note stay, plus an Internet Archive link if a copy exists, while every copy of X's text is removed. Keeping the text itself would need X's written permission (Twitter cut off Politwoops over this in 2015 and restored access only after an agreement). Check the current terms before enabling `social.x`.
|
||||
* **Messaging platforms**: only public Telegram broadcast channels through Telegram's public web preview. No WhatsApp (no public read access, terms forbid automated collection), no Signal, no private groups.
|
||||
|
||||
## 2. Personal data (GDPR / DSG)
|
||||
|
||||
+1
-1
@@ -31,6 +31,7 @@ Next.js frontend (de/en), Docker Compose for Unraid.
|
||||
- [x] Political ads: Google (bundle), Meta Ad Library (opt-in token)
|
||||
- [x] Social: Bluesky, Mastodon, public Telegram channels, YouTube (key), X via official API (token);
|
||||
party/institution channels; curated `social_accounts.yaml`
|
||||
- [x] X deletion sync (`social.x_deletions`): posts gone from X keep a marked record without their text
|
||||
- [x] Pages: organisations, money explorer, social feed, Wikipedia edits, person network ("follow the money"),
|
||||
say-vs-vote on roll calls
|
||||
|
||||
@@ -43,7 +44,6 @@ Next.js frontend (de/en), Docker Compose for Unraid.
|
||||
## Next
|
||||
|
||||
**Data**
|
||||
- [ ] X deletion sync (Developer Agreement: remove posts deleted on X) – required before enabling `social.x` in production
|
||||
- [ ] Full speech texts and interjections from the protocols; committee memberships (list 250)
|
||||
- [ ] HowTheyVote.eu / EP roll-call votes of Austrian MEPs
|
||||
- [ ] EU Council participant lists, ministries' press releases, Federal President's schedule
|
||||
|
||||
+1
-1
@@ -105,7 +105,7 @@ Accounts come from parliament profiles (verified), Wikidata (unverified until co
|
||||
| Mastodon / Fediverse | public REST API of the account's instance | ✅ `social.mastodon` |
|
||||
| Telegram – public broadcast channels | public web preview `t.me/s/<channel>` (no login, no robots.txt restriction); latest posts | ✅ `social.telegram` |
|
||||
| YouTube | Data API v3 (free quota, `POLITRACK_YOUTUBE_API_KEY`) | ✅ `social.youtube` (opt-in) |
|
||||
| X (Twitter) | **official API v2 only** (paid read access, `POLITRACK_X_BEARER_TOKEN`). No raw archive or full text kept; posts deleted on X must be removed (deletion sync still open, see roadmap) | ✅ `social.x` (opt-in) |
|
||||
| X (Twitter) | **official API v2 only** (paid read access, `POLITRACK_X_BEARER_TOKEN`). No raw archive or full text kept; posts gone from X keep a marked record without their text (`social.x_deletions`) | ✅ `social.x` (opt-in) |
|
||||
| Facebook / Instagram | ToS forbid scraping; Meta Content Library is for researchers only | ⛔ posts · ✅ political ads via the Ad Library API |
|
||||
| TikTok | Research API (academic only) | ⛔ |
|
||||
|
||||
|
||||
Reference in new issue
Block a user