The Shell, Pipes & Redirection
What the shell does to a line before any program runs, the three streams every process gets, and how small tools are joined into pipelines.
An interactive Linux lesson: 25 steps, about 35 minutes, on a live simulation in your browser.
You are alice on web01, standing in the directory of a small shop application: three logs, a release note, and a bin directory with two scripts. The shop logged errors this morning. On the way to counting them you will learn what the shell really does with each line you type.
Start with the logs. ls .log prints exactly the three files ending in .log. The obvious reading is that ls knows what means.
What you will learn
Before any program runs
- Who understands the star?
- echo knows nothing about files: The shell rewrites your line before any program starts. A program receives a finished list of words and never sees the star.
- Break it: a space in a file name: The shell splits a line into words at unquoted spaces, and every word is one argument. Quotes mean: this is one word.
- Single quotes, double quotes: Double quotes protect spaces and globs but still expand $variables. Single quotes protect everything.
Variables and $PATH
- Does the script see your variable?: A plain variable belongs to this shell. export puts it in the environment, and every child process starts with a copy of the environment.
- How a command is found: A command name with no slash is searched for in the directories of $PATH, left to right. The first match runs.
- Break it: command not found
Three streams
- Two outputs that look like one: Every process starts with three streams: 0 stdin, 1 stdout, 2 stderr. Both outputs point at the terminal until you say otherwise.
- > replaces, >> appends: The shell opens the redirect target before the program starts. > empties the file first; >> adds to the end.
- 2> and /dev/null: N> redirects stream N. /dev/null is a file that throws away whatever is written to it.
- Both streams in one file: 2>&1 copies where stdout points at that moment. Redirections apply left to right, so the file comes first: > file 2>&1.
- Drill: one file per stream
Pipes
- A pipe: stdout into stdin: A pipe connects one program's stdout to the next program's stdin. Tools that read stdin and write stdout can be chained in any order.
- What travels down a pipe?: A pipe carries stdout only. stderr goes around the pipe to the terminal unless you merge it first: cmd 2>&1 | next.
- tee: save a copy mid-stream: tee is a T-junction: it copies stdin to a file and to stdout, so a pipeline can save an intermediate result and keep going.
- xargs: lines into arguments: A pipe delivers data on stdin. xargs turns that data into arguments for the command it runs.
- $( ): output as part of the line: $(command) runs the command and pastes its stdout into the line, before the outer program starts.
Success, failure, chaining
- Every command leaves a number: Every command ends with an exit status: 0 for success, non-zero for failure. $? holds the status of the last one.
- &&, || and ;: ; means then. && means only if that worked. || means only if that failed.
Writing files, and the sudo trap
- Feeding stdin: < and <<EOF: < makes a file the program's stdin. <<EOF makes the next lines of the script its stdin.
- Break it: sudo echo > file: Redirections are performed by your shell, as you, before the command starts. sudo raises the command, never the redirection.
- The fix: pipe into sudo tee
- Drill: write a file as root
Recap & playground
- Cheat sheet
- Playground