Proposal: Twitter archival bot (Developer needed!)

pilky

This is a proposal for a Twitter/X bot which aims to assist in archiving relevant Twitter content by backing it up to the forum.

By tagging the bot account at the end of a thread, the preceding posts in the thread would get copied and reposted to the forum, just like e.g. the existing Thread Reader App bot.

The motivation for having such a bot is twofold:

Content on Twitter is fragile. Accounts frequently get deleted or banned and all posts by that account get deleted in the process.
It's a way to kickstart the forum and increase it's search engine ranking.

There are a few challenges that need to be addressed:

Avoiding duplicates: If the bot is tagged multiple times in the same post in a thread, or if it is tagged multiple times in different posts in a thread, a new thread on the forum should probably not be created.
Spam: Since anyone can tag the bot account in any post, I suggest that the bot has a way of filtering out irrelevant content. This can be done by: some combination of keywords checks, account whitelists and LLM algorithms.

If you feel up for the task of developing the bot, this forum provides a very nice API:

I suggest making it open-source so that other developers can help improve upon the bot.

Discuss!

brad

@pilky This is an awesome proposal, I think it's a great idea. There are also search issues with Twitter that a forum seems better equipped to handle.

Kruton

@pilky thank you alima

bot-mod

Great idea @pilky. I'll take a look at these API docs.

The knowing could come in handy for other ideas. Imagine. A private peatbot, fine tuned on everything, including retrieval from complete copies of the same documents used to fine tune it for proper citation. You could call it in to a thread to have it comment in context.

zawisza

Maybe some will find this a useful starting point.

After brief but traumatic experience with Twitter API I decided to use ChatGPT to simulate it instead so the code might be not fully correct or optimal as I wasn't able to test it properly. Also, there are many things that need to be added e.g. how to handle quote tweets, images, create tags and titles, or handle errors; that I felt inadequate to address.

ProgestTee

@zawisza This is great, I can def. see this being incorporated.

Read access through the Twitter API costs 100$/month, they removed free access last year.
I guess the bot would have to do some scraping to get the mentions and thread contents, I'm not sure if Twitter limits that somehow.
The bot could still tweet a reply linking to the reposted forum thread.

mazedaze

I have zero knowledge of bot building BUT I have been able to create some Google Aps scripts with help from my new best friend AI CHAT GPT. I also had zero knowledge of Aps scripts before hand.
Chat GPT could be useful.