At geek brekkie yesterday P— mentioned the idea of archiving links that you use with the Internet Archive. This seemed like a great idea to use in deploying my blog. I’ve wanted to add a general link checker to look for broken links. This isn’t quite the same thing but it would be an option for remediating link rot when found. Plus it seemed simple to do. My proof of concept for this also provides an excellent answer for a common question: when have you gone too far for a shell script and should switch to a “real language.” This script has gotten past that line so thought I’d share it.
So now ephemeral was working on my current tweets, but not handling the “historical” ones which twitter deems to be the ones older than the 3,200 most recent. There are APIs that you can use to access those, but they require a paid license. So instead I went into my settings in twitter and requested an archive of my tweets. This takes a number of hours but eventually you get a download link and a while after using that you end up with a huge zip file.
I’ve set up Vicky Lai’s ephemeral to make my tweets, well, ephemeral. But if you follow her README you’ll see the initial config is all manual. I’d rather have that be scripted so I can more easily replicate it in the future. So I wrote a script to create the AWS lambda instance. The prerequisites for this are covered in Vicky’s README but you’ll also need the aws cli, a go install and I find it’s also useful to use named profiles.