Brain Phrye

code cooking diy fiction personal photos politics reviews tools 

Zsh completion for home grown scripts

I use irssi on a cloud server for my IRC needs. Along with it I’ve written some utility scripts - the most useful is kibo to find all messages sent to myself by searching the irssi logs. Since my home dir is a bit longer lived my scripts keep getting little tweaks and improvements. One I’ve been doing lately is completions. I had hoped yesterday to show a more complicated zsh completion example but so far haven’t gotten it to work. Read more

Zip Slip

While looking into something else I came across the Zip Slip vulnerability. Kind of annoying since it’s nearly a year old and I’ve only heard of it now. Every now and then I keep thinking a neat tool would be something that would run over a project, track every dependency it has and then alert you when there’s a a security issue. However it seems like one of those things people would want but not pay for. Read more

A month of blogging

I still use a feed reader and I follow a blog by a U. Toronto sysadmin. He posts pretty much every day and goes after some pretty meaty topic or another. I’ve been wondering how hard that is and what’s to be gained by it so this January I decided to give it a go. It’s been interesting. I’ve missed a few days - and this weekend diverted into recipes which are kind of technical but not quite. Read more

git filter-branch

I’m working on converting two large svn repos to git. Both have large binary files stored in them so I’m using git lfs to make them more manageable. However the repos are large enough that the normal tools don’t work so I’m using the BFG repo cleaner to do the lfs migration. And this, plus the subgit migration tool leave loads of .gitattributes files laying around. They’re easy to remove, but I need . Read more

Chromebook as dev machine

On Monday I discovered that an old Chromebook I have can now support “LinuxApps”. These are a VM + containers that one can use to run straight Linux programs. They descibe using GUI apps, but I just want compilers, vim, my shell, git and the regular crew of dev tools. And they’re here. Is it the fastest thing ever? No. As I type this I’m having vague flashbacks to connecting up to a shell account via a 9600 baud modem. Read more

Trying tmux

One of the nice things about unix is that a lot of your learning is additive. Learning one tool usually doesn’t replace another. Learning perl didn’t mean I stopped using awk for example. The time spent into learning the various tools on unix is therefore pretty well spent - you’ll get a lot of use from what you’ve learned. However I can think of two circumstance where that didn’t work out. Read more

3blue1brown: The weird way to pi

YouTube sadly has some horrible bits, but there’s a subset of it that has loads of fascinating math videos. Last week there was a great one which showed a bizarre way to calculate pi using a physics thought experiment. He left the actual equations that explained why this happens to a later video. Happilly it came out today. What it boils down to is an interaction between the equations for conservation of energy and the conservation of momentum. Read more

Python vs shell

One of the nice parts about shell is that you can quickly prototype things. But eventually they become too brittle or limited or complex and you need to switch to a better langauge. Pipelines are powerful, but real data structures are better. The brevity is nice though. From the ia-save script, these four lines of shell… 1 2 3 4 find posts -name '*.md' -print0 \ | xargs -0 awk '/^\[[^]]*\]: / && $2 ~ /^http/ {print $2}' \ | sed 's/#. Read more

Ticket home

A weird side-effect of putting my home dir in vcsh is that I can write tickets for it. So mild annoyances that can’t be fixed right get a ticket. That way if I have time later I’ll have a better sense of the issue. You can set up “a git server” with just an ssh account on a server that has git on it. Nowadays though most people run something like Gitlab or Gogs or one of the hosted services like Gitlab, Github or Bitbucket. Read more

Secrets in git repos

One issue with infrastructure sorts of repositories is what to do with sensitive data. Keys, tokens and other secrets shouldn’t be committed to git repos, but they have to go somewhere. In some cases you can put them in your CI/CD system and import them as variables. But that gets complicated quickly. One way to address it is to use git crypt. It’s not a standard git extension, but it’s been around for a fair bit of time. Read more

Saving history with vcsh

I’ve written a few articles on using vcsh for tracking your home dir. Unlike previous options vcsh lets me use multiple repositories. My first experiment with this was a past repository. Lots of Unix tools use the GNU readline library so there are a number of history files to collect. I already was collecting all of them in ~/.history.d. In addition due to problems with NFS mounted home dirs I’d long ago put the hostname in the names of history files as a way to prevent file corruption. Read more

ZFS on Linux

This is why I avoid zfs - at least on Linux. Lots of people say I’m paranoid; that the issue has been decided, but it clearly hasn’t. I get that it has a lot of benefits. I’m currently working on a FreeBSD based project where zfs will be really beneficial. And I’ve used it before on Solaris. And the alternatives on Linux (btrfs primarily) still seem too unstable for my liking. Read more

Dashing git

People don’t think of the unix command line as a UI, but it is and it has its own idioms. Nearly all of them are conventions, not hard and fast rules. Because of this sometimes things take on a few meanings. The first meaning of the dash, "-" is to mark a command line flag like ls -l or mkdir -p. It comes up less often, but another pretty well known meaning is stdout/stdin. Read more

Shell scripts: drawing the line

At geek brekkie yesterday P— mentioned the idea of archiving links that you use with the Internet Archive. This seemed like a great idea to use in deploying my blog. I’ve wanted to add a general link checker to look for broken links. This isn’t quite the same thing but it would be an option for remediating link rot when found. Plus it seemed simple to do. My proof of concept for this also provides an excellent answer for a common question: when have you gone too far for a shell script and should switch to a “real language. Read more

Using batch to avoid problems

I use batch to rerun failed cron jobs. I also use it for webhooks. There are three reasons for doing this and why eventually I’ll end up changing even first runs of cronjobs to use batch. They handle load issues, locking issues and return quickly. The batch command is usually implemented as part of cron and at, but it runs at a certain load, not a certain time. It can be set at different loads when the system is configured, but the idea is that batch run jobs one at a time when the system load is “low”. Read more

Monitoring cron jobs

Cron jobs sometimes fail and the old way of getting emails from the cron daemon doesn’t really scale. For instance you might have a job that fails from time to time and that’s ok - but fail for too long and it’s a problem. Generally, email as an alerting tool is a Bad Thing and should be avoided. Since I have prometheus set up for everything, the easiest thing is to use the textfile-collector from node exporter to dump some basic stats. Read more