Advanced shell topics

  • Command interpreter
  • Programming language

Note:

  • Does interpretation before execution

Shell options

  • shopt: list shell options
  • shopt -s: set shell options
  • shopt -u: unset shell options
  • set <option>: set shell options

Shell variables

Variables are stored in the “shell environment”, a collection of variables that are available to all programs that are started from a specific shell. The shell environment is inherited by all programs that are started from the shell.

Set a value:

VARIABLE_NAME=<value>

Retrieve a value:

$VARIABLE_NAME

Note:

  • Bash is particular about whitespace

Experiment 15-2

The shell environment

The export command makes a variable available to all programs that are started from the shell.

The env command lists all variables that are available to programs that are started from the shell. It also allows to start a modified shell environment.

Note:

  • Demo export and env

Commands

  • Internal (built-in) commands
  • External commands
  • Aliases
  • Shell functions

Note:

  • External commands fork a new process
  • Explicit path: external
  • Otherwise: Internal/alias/function/PATH

The PATH

Some history on the PATH variable: Command search based on an environment variable started with the PWB/UNIX shell in 1975. Back then, the variable was simply named $p; the shell only allowed single-character names. A few years later, for UNIX V7, Dennis Ritchie, Steve Bourne and John Mashey generalized the PWB design into full-fledged environment variables, and p became PATH. (Source: John Mashey on Mastodon

Note:

  • Shell is as blind as I when I look in the fridge
  • Introduced with the PWB/UNIX shell in 1975 as $p
  • PATH introduced in UNIX V7 by Dennis Ritchie, Steve Bourne and John Mashey

Experiment 15-3

Note:

  • More recent addition: ~/.local/bin

PATH and the PWD

Note:

  • Safety warning about including .

CDPATH

Internal commands

Experiment 15-4

Note:

  • help
  • help echo vs man echo

External commands

Experiment 15-5

Note:

  • type cmd only shows the first match
  • Use type -a to see all matches

Forcing the use of external commands

Use the full path to make sure an external command is used.

Compound commands

Note:

  • One-liners with ;
  • Conditionals with && and ||
  • Return code

Experiment 15-6

Note:

  • Explore && and ||

Time-saving tools

Brace expansion

The curly brace expansion is a feature in the Bash shell that allows users to generate multiple strings with a single command by defining a range of values or a set of options within curly braces {}. When the curly brace expansion is performed, the shell creates a separate string for each possible combination of the options or range of values defined within the braces. This can be used, for example, to quickly create multiple directories or files with similar names, or to generate a list of commands with different arguments. The resulting strings can then be used as arguments to other commands or stored in variables for later use.

Let’s say you want to create a set of directories for different months of the year. Instead of creating them one by one, you can use curly brace expansion to create all the directories at once. Here’s an example:

mkdir {Jan,Feb,Mar,Apr,May,Jun,Jul,Aug,Sep,Oct,Nov,Dec}_2023

When you run this command, the shell will expand the curly braces and create 12 directories, each with a name that combines the month abbreviation and the year 2023. The resulting directories will be:

Jan_2023  Feb_2023  Mar_2023  Apr_2023  May_2023  Jun_2023
Jul_2023  Aug_2023  Sep_2023  Oct_2023  Nov_2023  Dec_2023

The shell can also expand a range of numbers or letters. Let’s say you want to create a set of files with names that follow a specific pattern, such as “file1.txt”, “file2.txt”, and so on. You can use curly brace expansion with a range to quickly generate these file names. Here’s an example:

touch file{1..5}.txt

When you run this command, the shell will expand the curly braces and create 5 files with names file1.txt, file2.txt, file3.txt, file4.txt, and file5.txt.

Elements in the expansion list may even be empty, which allows for a neat trick! Renaming files can be a tedious task when you have to repeat a complicated file name with just a slight alteration. Take this example:

mv Invoice-9388732.pdf Invoice-9388732.pdf.bak

If there are many similarly named files, even tab completion doesn’t make things much easier because you have to be cautious to pick the right alternative. Curly brace expansion to the rescue!

mv Invoice-9388732.pdf{,.bak}

This command expands the argument after mv to two file names, one with a suffix of nothing, i.e. just the file name, and the second with .bak added to its end.

Experiment 15-7

Note:

  • Create directories for every month
  • echo test{0..9}.txt
  • echo test{0..9}-{1..4}.txt
  • echo test{0..9}{a..d}.txt

Renaming made easy

The expansion list can also contain an empty element.

Note:

  • mv Invoice-9388732.pdf{,.bak}

Special pattern characters

aka Globbing

Experiment 15-8

Note:

  • Demo globbing with * and ?

Sets

Sets of characters can be specified with [...].

Experiment 15-9

Note:

  • Demo [...]
  • Demo [^...], [!...]
  • Demo [:alpha:]
  • Demo [:digit:]
  • Demo [:alnum:]
  • Demo [:lower:]

Meta-characters

Characters with special meaning:

  • $
  • *
  • ?
  • ~
  • >
  • |
  • etc.

Iterating over variable values

This works:

mv source/*.txt destination/

This doesn’t work:

FILES="alice.txt bob.txt"
mv source/$FILES destination/

Using grep

The grep command searches files (or STDIN) for a pattern.

The pattern syntax is called regular expressions.

Experiment 15-10

Note:

  • Demo grep

Learning RegEx

To experiment with regular expressions, the website Regex 101 is very helpful.

If you want to explore regular expressions in a more playful way, you can try the following websites: Regex Crossword is, as the name implies, a crossword puzzle based on regular expressions. And if you don’t mind a bit of horror, you can try Slash Escape, a game where you have to escape a dungeon by solving regular expression puzzles.

Note:

  • slash-escape is a horror game - you must escape from a slasher

Finding files

Note:

  • There’s ls -R, but it focuses on the files

Experiment 15-11

Note:

  • Demo find
  • Highlight quoting of globbing characters