A while back, I had extracted some code out of a large file into a separate file and made some mod­i­fi­ca­tions. I wanted to check that the dif­fer­ences were minimal. Let's say that the extracted code had been between lines 123 and 456 of large_old_­file.

diff -u <(sed -n '123,456p;457q' large_old_file) new_file

What's happening here?

• sed -n '123,456p' is printing lines 123–456 of large_old_­file.
• The 457q tells sed to abandon the file at line 457. Otherwise, it will keep reading all the way to the end.
• The <(sed ...) is an example of process sub­sti­tu­tion. The output of the sed invocation becomes the first input of the diff command.

A similar example: Diff …continue.

“Security is 1% technology plus 99% following the procedures correctly” — Tom Limoncelli

Having dealt with GPG last week at work, I remembered that I had intended to write a blog post about how we used GPG, Blackbox, and Paperkey to store secrets in Git at my previous job.

We used Blackbox to manage secrets that were needed during de­vel­op­ment, build, deployment, and runtime. These secrets included AWS cre­den­tials, Docker registry cre­den­tials, our private PyPI cre­den­tials, database cre­den­tials, and cer­tifi­cates. We wanted these secrets to be under version control, but also to be secure.

For example, we had a cre­den­tials.sh that exported en­vi­ron­ment variables, which was managed by Blackbox:

# Save current value 

C-like languages have a ternary operator, cond ? true_re­sult : false_re­sult. Python has true_re­sult if cond else false_re­sult. Bash doesn't have a ternary operator, but there are various workarounds.

I wanted to print succeeded or failed based on the exit code of the previous command in a shell script. In Unix, all programs exit with an integer status code. Successful programs exit with 0; all other values, positive or negative, indicate failure. In Bash, the status code of the previous program is held in $?.

some/command or-other fer example

STATUS="$([ "$?" == 0 ] && echo 'succeeded' || echo 'failed')"
echo "Results: $STATUS"

There are other ways to handle this.

I wanted to diff two files. One of them needed some seds on each line and sorting. I wanted to do that on the fly, without leaving a massaged in­ter­me­di­ate file lying around.

colordiff --unified <(cat orphaned_permalinks.txt
                        | sed 's@http://www.georgevreilly.com/@@'
                        | sed 's/.aspx$/.html/'
                

I had to rename several hundred thousand files today. Thanks to a botched invocation of Im­ageMag­ick, they all looked like unique_pre­fix.png.jpg, whereas we simply wanted unique_pre­fix.jpg.

I found a suitable answer at the Unix Stack­Ex­change. As one of the many variants of parameter sub­sti­tu­tion, Bash supports ${var/Pattern/Re­place­men­t}: “first match of Pattern within var replaced with Re­place­ment.”

for f in *.png.jpg;
do
    mv $f "${f/.png}"
done

The target expression could also have been written as "${f/.png.jpg/.jpg}"

Various Bash guides recommend putting quotes around just about everything. I had a script that contained this line:

sudo apt-get install --yes $(cat "$BUILD/install_on_aws_ubuntu.txt")

While refac­tor­ing, I put another set of quotes around the $(cat ...) out of an abundance of caution:

sudo apt-get install --yes "$(cat "$BUILD/install_on_aws_ubuntu.txt")"

Several other changes later, I couldn't figure out why my script had stopped working.

Here's what happened:

$ cat $HOME/tmp/demo.txt
foo
bar
quux

$ echo $(cat "$HOME/tmp/demo.txt")
foo bar quux

$ echo "$(cat $HOME/tmp/demo.txt)"
foo
bar
quux

The new outer quotes retained the newlines; the original replaced newlines with spaces.

The Bash FAQ has more: specif­i­cal­ly Command Substition and Word Splitting.

Bash has some handy syntax for getting and setting default values. Un­for­tu­nate­ly, it's a collection of punc­tu­a­tion characters, which makes it hard to Google when you can't quite remember the syntax.

Getting a default value using ${var:-fallback}:

# set $LOGDIR to $1 if $1 has a value; otherwise set $LOGDIR to "/var/log"
LOGDIR="${1:-/var/log}"

# use $VERSION unless it's empty or unset; fall back to extracting someprog's version num
build_version=${VERSION:-$(someprog --version | sed 's/[^0-9.]*\([0-9.]*\).*/\1/')}

The colon-dash con­struc­tion is known as the dog's bollocks in typography.

Setting a default value, using ${var:=fallback}:

$ echo $HOME
/Users/georgevreilly
$ echo ${HOME:=/tmp}
/Users/georgevreilly
$ unset HOME
$ echo ${HOME:=/tmp}
/tmp
$ echo $HOME
/tmp
$ cd; pwd
/tmp

Note: := uses the new value in two cases. First, when …continue.

I worked on a Bash script today that sets up various pre­req­ui­sites for our build. We need a recent version of Docker but our Bamboo build agents are running on Ubuntu 14.04, which has a very old version of Docker. The script upgrades Docker when it's first run. The script may be run more than once during the lifetime of the agent, so the second and subsequent calls should not upgrade Docker.

Basically, I wanted

if $DOCKER_VERSION < 1.9; then upgrade_docker; fi

Un­for­tu­nate­ly, it's not that easy in Bash. Here's what I came up with.

install_latest_docker() {
    if docker --version | python -c "min=[1, 9]; import sys; ↩
v=[int(x)