YouTube sadly has some horrible bits, but there’s a subset of it that has loads of fascinating math videos. Last week there was a great one which showed a bizarre way to calculate pi using a physics thought experiment.
He left the actual equations that explained why this happens to a later video. Happilly it came out today. What it boils down to is an interaction between the equations for conservation of energy and the conservation of momentum.
Read more
»
One of the nice parts about shell is that you can quickly prototype things. But eventually they become too brittle or limited or complex and you need to switch to a better langauge. Pipelines are powerful, but real data structures are better.
The brevity is nice though. From the ia-save script, these four lines of shell…
1 2 3 4 find posts -name '*.md' -print0 \ | xargs -0 awk '/^\[[^]]*\]: / && $2 ~ /^http/ {print $2}' \ | sed 's/#.
Read more
»
A weird side-effect of putting my home dir in vcsh is that I can write tickets for it. So mild annoyances that can’t be fixed right get a ticket. That way if I have time later I’ll have a better sense of the issue.
You can set up “a git server” with just an ssh account on a server that has git on it. Nowadays though most people run something like Gitlab or Gogs or one of the hosted services like Gitlab, Github or Bitbucket.
Read more
»
One issue with infrastructure sorts of repositories is what to do with sensitive data. Keys, tokens and other secrets shouldn’t be committed to git repos, but they have to go somewhere.
In some cases you can put them in your CI/CD system and import them as variables. But that gets complicated quickly.
One way to address it is to use git crypt. It’s not a standard git extension, but it’s been around for a fair bit of time.
Read more
»
In yesterday’s post the past repo was described.
The first step to getting that to work is to correctly configure
history files in the first place. Some are easy, but some are more
complex.
For zsh and MySQL it’s rather easy. Just put something
like this in your ~/.zshrc:
Read more
»
I’ve written a few articles on using vcsh for tracking your home dir. Unlike previous options vcsh lets me use multiple repositories. My first experiment with this was a past repository.
Lots of Unix tools use the GNU readline library so there are a number of history files to collect. I already was collecting all of them in ~/.history.d. In addition due to problems with NFS mounted home dirs I’d long ago put the hostname in the names of history files as a way to prevent file corruption.
Read more
»
This is why I avoid zfs - at least on Linux. Lots of people say I’m paranoid; that the issue has been decided, but it clearly hasn’t.
I get that it has a lot of benefits. I’m currently working on a FreeBSD based project where zfs will be really beneficial. And I’ve used it before on Solaris.
And the alternatives on Linux (btrfs primarily) still seem too unstable for my liking.
Read more
»
People don’t think of the unix command line as a UI, but it is and it has its own idioms. Nearly all of them are conventions, not hard and fast rules. Because of this sometimes things take on a few meanings.
The first meaning of the dash, "-" is to mark a command line flag like ls -l or mkdir -p. It comes up less often, but another pretty well known meaning is stdout/stdin.
Read more
»
At geek brekkie yesterday P— mentioned the idea of archiving links that you use with the Internet Archive. This seemed like a great idea to use in deploying my blog.
I’ve wanted to add a general link checker to look for broken links. This isn’t quite the same thing but it would be an option for remediating link rot when found. Plus it seemed simple to do.
My proof of concept for this also provides an excellent answer for a common question: when have you gone too far for a shell script and should switch to a “real language.
Read more
»
I use batch to rerun failed cron jobs. I also use it for webhooks. There are three reasons for doing this and why eventually I’ll end up changing even first runs of cronjobs to use batch. They handle load issues, locking issues and return quickly.
The batch command is usually implemented as part of cron and at, but it runs at a certain load, not a certain time. It can be set at different loads when the system is configured, but the idea is that batch run jobs one at a time when the system load is “low”.
Read more
»
Cron jobs sometimes fail and the old way of getting emails from the cron daemon doesn’t really scale. For instance you might have a job that fails from time to time and that’s ok - but fail for too long and it’s a problem. Generally, email as an alerting tool is a Bad Thing and should be avoided.
Since I have prometheus set up for everything, the easiest thing is to use the textfile-collector from node exporter to dump some basic stats.
Read more
»
Yesterday I covered the overview of how this gets deployed. Now for the detail.
The script for testing is more aspirational than functional. It runs a spellcheck on the blog, but doesn’t die if it finds any. I’m supposed to read the output. I never do. I should though. Someday I’ll add consequences to the spellcheck run, but for now at least it tries.
1 2 3 4 5 #!
Read more
»
This website is maintained with hugo which is a static site generator. That means the source is parsed and all of the html, css and javascript are generated and saved as files. It means deployment just requires a plain, basic web server.
But that still does mean it needs to be deployed.
You could deploy just on s3 but for me I already have my own server. So I just deploy it to there.
Read more
»
For some reason a few weeks back I was wondering about using ssh keys to encrypt/decrypt files. Seems like a thing that should be possible, why not? And sure enough, it’s been done.
This won’t be as good as using gpg keys. Specifically without the web of trust it can be hit with MITM attacks, but I think it would be “good enough” for most people in most uses. And in my experience getting people to use gpg is like pulling teeth.
Read more
»
While playing with the Twitter API via a Go lib I saw someone call on people to troll reply to a tweet with a specific piece of text.
This seemed like a really easy thing to search for and block. So I modified the little Go program from starting with ephemeral and put in this as the function run in main().
It searches for the string it’s given and then if the tweet consists solely of that text (compared by turning the string lowercase and then stripping out everything that’s not a letter) it blocks the user.
Read more
»
One nice side effect of using vcsh was developing more complex scripts to help me do things. I didn’t have to worry a script or tool would get lost when a machine inevitibly died.
However before writing a script, sometimes it’s not a bad idea to check and see if someone else already has. Lately many of those that I’ve found have been in Go.
Originally I did these with update but it made update take a long time to run and sometimes with die if a rarely used Go util was broken.
Read more
»
A while back I switched to vcsh. I’ve written a few articles on using it but since then I’ve migrated machines a number of times.
The big issue I’ve found is having to manually install software on each machine. There are things my scripts depend on and things I just expect to have and manually creating them each time is annoying.
The solution obviously is a script. It’s actually used all the time as I might create new dependencies or find new tools I need so I’d want that installed on all machines.
Read more
»
So now ephemeral was working on my current tweets, but not handling the “historical” ones which twitter deems to be the ones older than the 3,200 most recent.
There are APIs that you can use to access those, but they require a paid license. So instead I went into my settings in twitter and requested an archive of my tweets. This takes a number of hours but eventually you get a download link and a while after using that you end up with a huge zip file.
Read more
»
I’ve set up Vicky Lai’s ephemeral to make my tweets, well, ephemeral. But if you follow her README you’ll see the initial config is all manual. I’d rather have that be scripted so I can more easily replicate it in the future.
So I wrote a script to create the AWS lambda instance. The prerequisites for this are covered in Vicky’s README but you’ll also need the aws cli, a go install and I find it’s also useful to use named profiles.
Read more
»
tl;dr my year in vim
Gource is a neat tool for visualising the history of a of a software project. In a way it’s kind of a fun combination of this scene Jurassic Park and version control.
Reading up on it I learned it could also visualise multiple repositories so I decided it would be kind of fun to do just that. I use vcsh to manage my home directory, pass to manage passwords, Hugo for my website and slack for managing my personal servers.
Read more
»