diff --git a/bash4.md b/bash4.md
new file mode 100644
index 0000000..fe213bc
--- /dev/null
+++ b/bash4.md
@@ -0,0 +1,219 @@
+# Bash 4 - a rough overview
+
+:V4:
+
+\
\\
\ \
You +don't have an account yet? Just get one: \" +title="Register" rel="nofollow" +class="register"\>Register\\
\Forgotten your password? Get +a new one: \" +title="Set new password" rel="nofollow" class="resendpwd"\>Set new +password\\
\\ \ \\ + +- **Never use options to `echo`! *Ever*!** Any time you feel tempted to + use `echo -e`, `-n`, or any other special feature of echo, **use + [printf](/commands/builtin/printf) instead!** If portability is a + requirement, you should consider using `printf` *exclusively* and just + ignore that `echo` even exists. If you must use `echo -e` and refuse + to use `printf`, it is usually acceptable to use ''echo \$'...' ''if + targeting only shells that support this special quoting style. + + + +- `ksh93` has a `print` command, which if coding specifically for + `ksh93` should be preferred over `echo`. + [printf](/commands/builtin/printf) still includes most of the + functionality of both, and should usually be the most preferred + option. + +## See also + +- [printf](/commands/builtin/printf) +-You +don't have an account yet? Just get one: \" +title="Register" rel="nofollow" +class="register"\>Register\\
\Forgotten your password? Get +a new one: \" +title="Set new password" rel="nofollow" class="resendpwd"\>Set new +password\\
\\ \ \\ + +- The behavior of function scope is not defined by POSIX, however local + variables are implemented widely by bourne-like shells, and behavior + differs substantially. Even the`dash` shell has local variables. + + + +- In ksh93, using POSIX-style function definitions, `typeset` doesn't + set `local` variables, but rather acts upon variables of the + next-outermost scope (e.g. setting attributes). Using `typeset` within + functions defined using ksh `function name {` syntax, variables follow + roughly + [lexical-scoping](http://community.schemewiki.org/?lexical-scope), + except that functions themselves don't have scope, just like Bash. + This means that even functions defined within a "function's scope" + don't have access to non-local variables except through `namerefs`. + +## See also diff --git a/commands/builtin/mapfile.md b/commands/builtin/mapfile.md new file mode 100644 index 0000000..f9bf9ed --- /dev/null +++ b/commands/builtin/mapfile.md @@ -0,0 +1,220 @@ +# The mapfile builtin command + +## Synopsis + + mapfile [-n COUNT] [-O ORIGIN] [-s COUNT] [-t] [-u FD] [-C CALLBACK] [-c QUANTUM] [ARRAY] + + readarray [-n COUNT] [-O ORIGIN] [-s COUNT] [-t] [-u FD] [-C CALLBACK] [-c QUANTUM] [ARRAY] + +## Description + +This builtin is also accessible using the command name `readarray`. + +`mapfile` is one of the two builtin commands primarily intended for +handling standard input (the other being `read`). `mapfile` reads lines +of standard input and assigns each to the elements of an indexed array. +If no array name is given, the default array name is `MAPFILE`. The +target array must be a "normal" integer indexed array. + +`mapfile` returns success (0) unless an invalid option is given or the +given array `ARRAY` is set readonly. + +| Option | Description | +|---------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `-c QUANTUM` | Specifies the number of lines that have to be read between every call to the callback specified with `-C`. The default QUANTUM is 5000 | +| `-C CALLBACK` | Specifies a callback. The string `CALLBACK` can be any shell code, the index of the array that will be assigned, and the line is appended at evaluation time. | +| `-n COUNT` | Reads at most `COUNT` lines, then terminates. If `COUNT` is 0, then all lines are read (default). | +| `-O ORIGIN` | Starts populating the given array `ARRAY` at the index `ORIGIN` rather than clearing it and starting at index 0. | +| `-s COUNT` | Discards the first `COUNT` lines read. | +| `-t` | Remove any trailing newline from a line read, before it is assigned to an array element. | +| `-u FD` | Read from filedescriptor `FD` rather than standard input. | + +While `mapfile` isn't a common or portable shell feature, it's +functionality will be familiar to many programmers. Almost all +programming languages (aside from shells) with support for compound +datatypes like arrays, and which handle open file objects in the +traditional way, have some analogous shortcut for easily reading all +lines of some input as a standard feature. In Bash, `mapfile` in itself +can't do anything that couldn't already be done using read and a loop, +and if portability is even a slight concern, should never be used. +However, it does *significantly* outperform a read loop, and can make +for shorter and cleaner code - especially convenient for interactive +use. + +## Examples + +Here's a real-world example of interactive use borrowed from Gentoo +workflow. Xorg updates require rebuilding drivers, and the +Gentoo-suggested command is less than ideal, so let's Bashify it. The +first command produces a list of packages, one per line. We can read +those into the array named "args" using `mapfile`, stripping trailing +newlines with the '-t' option. The resulting array is then expanded into +the arguments of the "emerge" command - an interface to Gentoo's package +manager. This type of usage can make for a safe and effective +replacement for xargs(1) in certain situations. Unlike xargs, all +arguments are guaranteed to be passed to a single invocation of the +command with no wordsplitting, pathname expansion, or other monkey +business. + + # eix --only-names -IC x11-drivers | { mapfile -t args; emerge -av1 "${args[@]}" <&1; } + +Note the use of command grouping to keep the emerge command inside the +pipe's subshell and within the scope of "args". Also note the unusual +redirection. This is because the -a flag makes emerge interactive, +asking the user for confirmation before continuing, and checking with +isatty(3) to abort if stdin isn't pointed at a terminal. Since stdin of +the entire command group is still coming from the pipe even though +mapfile has read all available input, we just borrow FD 1 as it just so +happens to be pointing where we want it. More on this over at greycat's +wiki: \$ dash -c 'f() { if command
+ shift 2\>/dev/null; then echo "\$1"; else echo "no args"; fi; }; f'
+
+no args \
While, POSIX requires this behavior, it isn't very
+obvious and some shells don't do it correctly. To work around this, you
+can use something like:
+
+\ \$ mksh -c 'f() { if ! \${1+false} && shift; then echo "\$1";
+else echo "no args"; fi; }; f' no args \
~~The mksh maintainer
+refuses to change either the `shift` or `command` builtins.~~
+[Fixed](https://github.com/MirBSD/mksh/commit/996e05548ab82f7ef2dea61f109cc7b6d13837fa).
+(Thanks!)
+
+- Perhaps almost as bad as the above, busybox sh's `shift` always
+ returns success, even when attempting to shift beyond the final
+ argument. \ \$ bb -c 'f() { if shift; then echo "\$1"; else
+ echo "no args"; fi; }; f'
+
+(no output) \
The above mksh workaround will work in this case
+too.
+
+## See also
diff --git a/commands/builtin/shopt.md b/commands/builtin/shopt.md
new file mode 100644
index 0000000..f049367
--- /dev/null
+++ b/commands/builtin/shopt.md
@@ -0,0 +1,51 @@
+# The shopt builtin command
+
+The `shopt` builtin manages [shell options](/internals/shell_options), a
+set of boolean (`on`/`off`) configuration variables that control the
+behaviour of the shell.
+
+## Synopsis
+
+ shopt [-pqsu] [-o] You +don't have an account yet? Just get one: \" +title="Register" rel="nofollow" +class="register"\>Register\\
\Forgotten your password? Get +a new one: \" +title="Set new password" rel="nofollow" class="resendpwd"\>Set new +password\\
\\ \ \\ + + $ dc << EOF + 2 # put 2 on the stack + d # duplicate i.e. put another 2 on the stack + *p # multiply and print + c p # clear and print + EOF + 4 + dc: stack empty + +`c p` results in an error, as we would expect, as c removes everything +on the stack. *Note: we can use `#` to put comments in the script.* + +If you are lost, you can inspect (i.e. print) the stack using the +command `f`. The stack remains unchanged: + + dc <<< '1 2 d 4+f' + 6 + 2 + 1 + +Note how the first element that will be popped from the stack is printed +first, if you are used to an HP calculator, it's the reverse. + +Don't hesitate to put `f` in the examples of this tutorial, it doesn't +change the result, and it's a good way to see what's going on. + +## Registers + +The GNU `dc` manual says that dc has at least **256 registers** +depending on the range of unsigned char. I'm not sure how you are +supposed to use the NUL byte. Using a register is easy: + + dc < \#!/bin/bash \# Using collapsing functions to turn debug
+messages on/off
+
+\[ "--debug" = "\$1" \] && dbg=echo \|\| dbg=:
+
+\# From now on if you use \$dbg instead of echo, you can select if
+messages will be shown
+
+\$dbg "This message will only be displayed if --debug is specified at
+the command line
diff --git a/howto/conffile.md b/howto/conffile.md
new file mode 100644
index 0000000..934f331
--- /dev/null
+++ b/howto/conffile.md
@@ -0,0 +1,105 @@
+# Config files for your script
+
+![](keywords>bash shell scripting config files include configuration)
+
+## General
+
+For this task, you don't have to write large parser routines (unless you
+want it 100% secure or you want a special file syntax) - you can use the
+Bash source command. The file to be sourced should be formated in
+key="value" format, otherwise bash will try to interpret commands:
+
+ #!/bin/bash
+ echo "Reading config...." >&2
+ source /etc/cool.cfg
+ echo "Config for the username: $cool_username" >&2
+ echo "Config for the target host: $cool_host" >&2
+
+So, where do these variables come from? If everything works fine, they
+are defined in /etc/cool.cfg which is a file that's sourced into the
+current script or shell. Note: this is **not** the same as executing
+this file as a script! The sourced file most likely contains something
+like:
+
+ cool_username="guest"
+ cool_host="foo.example.com"
+
+These are normal statements understood by Bash, nothing special. Of
+course (and, a big disadvantage under normal circumstances) the sourced
+file can contain **everything** that Bash understands, including
+malicious code!
+
+The `source` command also is available under the name `.` (dot). The
+usage of the dot is identical:
+
+ #!/bin/bash
+ echo "Reading config...." >&2
+ . /etc/cool.cfg #note the space between the dot and the leading slash of /etc.cfg
+ echo "Config for the username: $cool_username" >&2
+ echo "Config for the target host: $cool_host" >&2
+
+## Per-user configs
+
+There's also a way to provide a system-wide config file in /etc and a
+custom config in ~/(user's home) to override system-wide defaults. In
+the following example, the if/then construct is used to check for the
+existance of a user-specific config:
+
+ #!/bin/bash
+ echo "Reading system-wide config...." >&2
+ . /etc/cool.cfg
+ if [ -r ~/.coolrc ]; then
+ echo "Reading user config...." >&2
+ . ~/.coolrc
+ fi
+
+## Secure it
+
+As mentioned earlier, the sourced file can contain anything a Bash
+script can. Essentially, it **is** an included Bash script. That creates
+security issues. A malicicios person can "execute" arbitrary code when
+your script is sourcing its config file. You might want to allow only
+constructs in the form `NAME=VALUE` in that file (variable assignment
+syntax) and maybe comments (though technically, comments are
+unimportant). Imagine the following "config file", containing some
+malicious code:
+
+ # cool config file for my even cooler script
+ username=god_only_knows
+ hostname=www.example.com
+ password=secret ; echo rm -rf ~/*
+ parameter=foobar && echo "You've bene pwned!";
+ # hey look, weird code follows...
+ echo "I am the skull virus..."
+ echo rm -fr ~/*
+ mailto=netadmin@example.com
+
+You don't want these `echo`-commands (which could be any other
+commands!) to be executed. One way to be a bit safer is to filter only
+the constructs you want, write the filtered results to a new file and
+source the new file. We also need to be sure something nefarious hasn't
+been added to the end of one of our name=value parameters, perhaps using
+; or && command separators. In those cases, perhaps it is simplest to
+just ignore the line entirely. Egrep (`grep -E`) will help us here, it
+filters by description:
+
+ #!/bin/bash
+ configfile='/etc/cool.cfg'
+ configfile_secured='/tmp/cool.cfg'
+
+ # check if the file contains something we don't want
+ if egrep -q -v '^#|^[^ ]*=[^;]*' "$configfile"; then
+ echo "Config file is unclean, cleaning it..." >&2
+ # filter the original to a new file
+ egrep '^#|^[^ ]*=[^;&]*' "$configfile" > "$configfile_secured"
+ configfile="$configfile_secured"
+ fi
+
+ # now source it, either the original or the filtered variant
+ source "$configfile"
+
+**To make clear what it does:** egrep checks if the file contains
+something we don't want, if yes, egrep filters the file and writes the
+filtered contents to a new file. If done, the original file name is
+changed to the name stored in the variable `configfile`. The file named
+by that variable is sourced, as if it were the original file.
diff --git a/howto/dissectabadoneliner.md b/howto/dissectabadoneliner.md
new file mode 100644
index 0000000..17ac6a8
--- /dev/null
+++ b/howto/dissectabadoneliner.md
@@ -0,0 +1,124 @@
+# Dissect a bad oneliner
+
+``` bash
+$ ls *.zip | while read i; do j=`echo $i | sed 's/.zip//g'`; mkdir $j; cd $j; unzip ../$i; cd ..; done
+```
+
+This is an actual one-liner someone asked about in `#bash`. **There are
+several things wrong with it. Let's break it down!**
+
+``` bash
+$ ls *.zip | while read i; do ...; done
+```
+
+(Please read .) This command
+executes `ls` on the expansion of `*.zip`. Assuming there are filenames
+in the current directory that end in '.zip', ls will give a
+human-readable list of those names. The output of ls is not for parsing.
+But in sh and bash alike, we can loop safely over the glob itself:
+
+``` bash
+$ for i in *.zip; do j=`echo $i | sed 's/.zip//g'`; mkdir $j; cd $j; unzip ../$i; cd ..; done
+```
+
+Let's break it down some more!
+
+``` bash
+j=`echo $i | sed 's/.zip//g'` # where $i is some name ending in '.zip'
+```
+
+The goal here seems to be get the filename without its `.zip` extension.
+In fact, there is a POSIX(r)-compliant command to do this: `basename`
+The implementation here is suboptimal in several ways, but the only
+thing that's genuinely error-prone with this is "`echo $i`". Echoing an
+*unquoted* variable means [wordsplitting](/syntax/expansion/wordsplit)
+will take place, so any whitespace in `$i` will essentially be
+normalized. In `sh` it is necessary to use an external command and a
+subshell to achieve the goal, but we can eliminate the pipe (subshells,
+external commands, and pipes carry extra overhead when they launch, so
+they can really hurt performance in a loop). Just for good measure,
+let's use the more readable, [modern](/syntax/expansion/cmdsubst) `$()`
+construct instead of the old style backticks:
+
+``` bash
+sh $ for i in *.zip; do j=$(basename "$i" ".zip"); mkdir $j; cd $j; unzip ../$i; cd ..; done
+```
+
+In Bash we don't need the subshell or the external basename command. See
+[Substring removal with parameter
+expansion](/syntax/pe#substring_removal):
+
+``` bash
+bash $ for i in *.zip; do j="${i%.zip}"; mkdir $j; cd $j; unzip ../$i; cd ..; done
+```
+
+Let's keep going:
+
+``` bash
+$ mkdir $j; cd $j; ...; cd ..
+```
+
+As a programmer, you **never** know the situation under which your
+program will run. Even if you do, the following best practice will never
+hurt: When a following command depends on the success of a previous
+command(s), check for success! You can do this with the "`&&`"
+conjunction, that way, if the previous command fails, bash will not try
+to execute the following command(s). It's fully POSIX(r). Oh, and
+remember what I said about [wordsplitting](/syntax/expansion/wordsplit)
+in the previous step? Well, if you don't quote `$j`, wordsplitting can
+happen again.
+
+``` bash
+$ mkdir "$j" && cd "$j" && ... && cd ..
+```
+
+That's almost right, but there's one problem -- what happens if `$j`
+contains a slash? Then `cd ..` will not return to the original
+directory. That's wrong! `cd -` causes cd to return to the previous
+working directory, so it's a much better choice:
+
+``` bash
+$ mkdir "$j" && cd "$j" && ... && cd -
+```
+
+(If it occurred to you that I forgot to check for success after cd -,
+good job! You could do this with `{ cd - || break; }`, but I'm going to
+leave that out because it's verbose and I think it's likely that we will
+be able to get back to our original working directory without a
+problem.)
+
+So now we have:
+
+``` bash
+sh $ for i in *.zip; do j=$(basename "$i" ".zip"); mkdir "$j" && cd "$j" && unzip ../$i && cd -; done
+```
+
+``` bash
+bash $ for i in *.zip; do j="${i%.zip}"; mkdir "$j" && cd "$j" && unzip ../$i && cd -; done
+```
+
+Let's throw the `unzip` command back in the mix:
+
+``` bash
+mkdir "$j" && cd "$j" && unzip ../$i && cd -
+```
+
+Well, besides word splitting, there's nothing terribly wrong with this.
+Still, did it occur to you that unzip might already be able to target a
+directory? There isn't a standard for the `unzip` command, but all the
+implementations I've seen can do it with the -d flag. So we can drop the
+cd commands entirely:
+
+``` bash
+$ mkdir "$j" && unzip -d "$j" "$i"
+```
+
+``` bash
+sh $ for i in *.zip; do j=$(basename "$i" ".zip"); mkdir "$j" && unzip -d "$j" "$i"; done
+```
+
+``` bash
+bash $ for i in *.zip; do j="${i%.zip}"; mkdir "$j" && unzip -d "$j" "$i"; done
+```
+
+There! That's as good as it gets.
diff --git a/howto/edit-ed.md b/howto/edit-ed.md
new file mode 100644
index 0000000..a74a6e6
--- /dev/null
+++ b/howto/edit-ed.md
@@ -0,0 +1,386 @@
+# Editing files via scripts with ed
+
+![](keywords>bash shell scripting arguments file editor edit ed sed)
+
+## Why ed?
+
+Like `sed`, `ed` is a line editor. However, if you try to change file
+contents with `sed`, and the file is open elsewhere and read by some
+process, you will find out that GNU `sed` and its `-i` option will not
+allow you to edit the file. There are circumstances where you may need
+that, e.g. editing active and open files, the lack of GNU, or other
+`sed`, with "in-place" option available.
+
+Why `ed`?
+
+- maybe your `sed` doesn't support in-place edit
+- maybe you need to be as portable as possible
+- maybe you need to really edit in-file (and not create a new file like
+ GNU `sed`)
+- last but not least: standard `ed` has very good editing and addressing
+ possibilities, compared to standard `sed`
+
+Don't get me wrong, this is **not** meant as anti-`sed` article! It's
+just meant to show you another way to do the job.
+
+## Commanding ed
+
+Since `ed` is an interactive text editor, it reads and executes commands
+that come from `stdin`. There are several ways to feed our commands to
+ed:
+
+**Pipelines**
+
+ echo '' | ed
+
+To inject the needed newlines, etc. it may be easier to use the builtin
+command, `printf` ("help printf"). Shown here as an example Bash
+function to prefix text to file content:
+
+
+ # insertHead "$text" "$file"
+
+ insertHead() {
+ printf '%s\n' H 1i "$1" . w | ed -s "$2"
+ }
+
+**Here-strings**
+
+ ed <<< ''
+
+**Here-documents**
+
+ ed <
+ EOF
+
+Which one you prefer is your choice. I will use the here-strings, since
+it looks best here IMHO.
+
+There are other ways to provide input to `ed`. For example, process
+substitution. But these should be enough for daily needs.
+
+Since `ed` wants commands separated by newlines, I'll use a special Bash
+quoting method, the C-like strings `$'TEXT'`, as it can interpret a set
+of various escape sequences and special characters. I'll use the `-s`
+option to make it less verbose.
+
+## The basic interface
+
+Check the `ed` manpage for details
+
+Similar to `vi` or `vim`, `ed` has a "command mode" and an "interactive
+mode". For non-interactive use, the command mode is the usual choice.
+
+Commands to `ed` have a simple and regular structure: zero, one, or two
+addresses followed by a single-character command, possibly followed by
+parameters to that command. These addresses specify one or more lines in
+the text buffer. Every command that requires addresses has default
+addresses, so the addresses can often be omitted.
+
+The line addressing is relative to the *current line*. If the edit
+buffer is not empty, the initial value for the *current line* shall be
+the last line in the edit buffer, otherwise zero. Generally, the
+*current line* is the last line affected by a command. All addresses can
+only address single lines, not blocks of lines!
+
+Line addresses or commands using *regular expressions* interpret POSIX
+Basic Regular Expressions (BRE). A null BRE is used to reference the
+most recently used BRE. Since `ed` addressing is only for single lines,
+no RE can ever match a newline.
+
+## Debugging your ed scripts
+
+By default, `ed` is not very talkative and will simply print a "?" when
+an error occurs. Interactively you can use the `h` command to get a
+short message explaining the last error. You can also turn on a mode
+that makes `ed` automatically print this message with the `H` command.
+It is a good idea to always add this command at the beginning of your ed
+scripts:
+
+ bash > ed -s file <<< $'H\n,df'
+ ?
+ script, line 2: Invalid command suffix
+
+While working on your script, you might make errors and destroy your
+file, you might be tempted to try your script doing something like:
+
+ # Works, but there is better
+
+ # copy my original file
+ cp file file.test
+
+ # try my script on the file
+ ed -s file.test <<< $'H\n\nw'
+
+ # see the results
+ cat file.test
+
+There is a much better way though, you can use the ed command `p` to
+print the file, now your testing would look like:
+
+ ed -s file <<< $'H\n\n,p'
+
+the `,` (comma) in front of the `p` command is a shortcut for `1,$`
+which defines an address range for the first to the last line, `,p` thus
+means print the whole file, after it has been modified. When your script
+runs sucessfully, you only have to replace the `,p` by a `w`.
+
+Of course, even if the file is not modified by the `p` command, **it's
+always a good idea to have a backup copy!**
+
+## Editing your files
+
+Most of these things can be done with `sed`. But there are also things
+that can't be done in `sed` or can only be done with very complex code.
+
+### Simple word substitutions
+
+Like `sed`, `ed` also knows the common `s/FROM/TO/` command, and it can
+also take line-addresses. **If no substitution is made on the addressed
+lines, it's considered an error.**
+
+#### Substitutions through the whole file
+
+ ed -s test.txt <<< $',s/Windows(R)-compatible/POSIX-conform/g\nw'
+
+Note: The comma as single address operator is an alias for `1,$`
+("all lines").
+
+#### Substitutions in specific lines
+
+On a line containing `fruits`, do the substitution:
+
+ ed -s test.txt <<< $'/fruits/s/apple/banana/g\nw'
+
+On the 5th line after the line containing `fruits`, do the substitution:
+
+ ed -s test.txt <<< $'/fruits/+5s/apple/banana/g\nw'
+
+### Block operations
+
+#### Delete a block of text
+
+The simple one is a well-known (by position) block of text:
+
+ # delete lines number 2 to 4 (2, 3, 4)
+ ed -s test.txt <<< $'2,5d\nw'
+
+This deletes all lines matching a specific regular expression:
+
+ # delete all lines matching foobar
+ ed -s test.txt <<< $'g/foobar/d\nw'
+
+g/regexp/ applies the command following it to all the lines matching the
+regexp
+
+#### Move a block of text
+
+...using the `m` command: ` m `
+
+This is definitely something that can't be done easily with sed.
+
+ # moving lines 5-9 to the end of the file
+ ed -s test.txt <<< $'5,9m$\nw'
+
+ # moving lines 5-9 to line 3
+ ed -s test.txt <<< $'5,9m3\nw'
+
+#### Copy a block of text
+
+...using the `t` command: ` t `
+
+You use the `t` command just like you use the `m` (move) command.
+
+ # make a copy of lines 5-9 and place it at the end of the file
+ ed -s test.txt <<< $'5,9t$\nw'
+
+ # make a copy of lines 5-9 and place it at line 3
+ ed -s test.txt <<< $'5,9t3\nw'
+
+#### Join all lines
+
+...but leave the final newline intact. This is done by an extra command:
+`j` (join).
+
+ ed -s file <<< $'1,$j\nw'
+
+Compared with two other methods (using `tr` or `sed`), you don't have to
+delete all newlines and manually add one at the end.
+
+### File operations
+
+#### Insert another file
+
+How do you insert another file? As with `sed`, you use the `r` (read)
+command. That inserts another file at the line before the last line (and
+prints the result to stdout - `,p`):
+
+ ed -s FILE1 <<< $'$-1 r FILE2\n,p'
+
+To compare, here's a possible `sed` solution which must use Bash
+arithmetic and the external program `wc`:
+
+ sed "$(($(wc -l < FILE1)-1))r FILE2" FILE1
+
+ # UPDATE here's one which uses GNU sed's "e" parameter for the s-command
+ # it executes the commands found in pattern space. I'll take that as a
+ # security risk, but well, sometimes GNU > security, you know...
+ sed '${h;s/.*/cat FILE2/e;G}' FILE1
+
+Another approach, in two invocations of sed, that avoids the use of
+external commands completely:
+
+ sed $'${s/$/\\n-||-/;r FILE2\n}' FILE1 | sed '0,/-||-/{//!h;N;//D};$G'
+
+## Pitfalls
+
+### ed is not sed
+
+ed and sed might look similar, but the same command(s) might act
+differently:
+
+**\_\_ /foo/d \_\_**
+
+In sed /foo/d will delete all lines matching foo, in ed the commands are
+not repeated on each line so this command will search the next line
+matching foo and delete it. If you want to delete all lines matching
+foo, or do a subsitution on all lines matching foo you have to tell ed
+about it with the g (global) command:
+
+ echo $'1\n1\n3' > file
+
+ #replace all lines matching 1 by "replacement"
+ ed -s file <<< $'g/1/s/1/replacement/\n,p'
+
+ #replace the first line matching 1 by "replacement"
+ #(because it starts searching from the last line)
+ ed -s file <<< $'s/1/replacement/\n,p'
+
+**\_\_ an error stops the script \_\_**
+
+You might think that it's not a problem and that the same thing happens
+with sed and you're right, with the exception that if ed does not find a
+pattern it's an error, while sed just continues with the next line. For
+instance, let's say that you want to change foo to bar on the first line
+of the file and add something after the next line, ed will stop if it
+cannot find foo on the first line, sed will continue.
+
+ #Gnu sed version
+ sed -e '1s/foo/bar/' -e '$a\something' file
+
+ #First ed version, does nothing if foo is not found on the first line:
+ ed -s file <<< $'H\n1s/foo/bar/\na\nsomething\n.\nw'
+
+If you want the same behaviour you can use g/foo/ to trick ed. g/foo/
+will apply the command on all lines matching foo, thus the substitution
+will succeed and ed will not produce an error when foo is not found:
+
+ #Second version will add the line with "something" even if foo is not found
+ ed -s file <<< $'H\n1g/foo/s/foo/bar/\na\nsomething\n.\nw'
+
+In fact, even a substitution that fails after a g/ / command does not
+seem to cause an error, i.e. you can use a trick like g/./s/foo/bar/ to
+attempt the substitution on all non blank lines
+
+### here documents
+
+**\_\_ shell parameters are expanded \_\_**
+
+If you don't quote the delimiter, \$ has a special meaning. This sounds
+obvious but it's easy to forget this fact when you use addresses like
+\$-1 or commands like \$a. Either quote the \$ or the delimiter:
+
+ #fails
+ ed -s file << EOF
+ $a
+ last line
+ .
+ w
+ EOF
+
+ #ok
+ ed -s file << EOF
+ \$a
+ last line
+ .
+ w
+ EOF
+
+ #ok again
+ ed -s file << 'EOF'
+ $a
+ last line
+ .
+ w
+ EOF
+
+**\_\_ "." is not a command \_\_**
+
+The . used to terminate the command "a" must be the only thing on the
+line. take care if you indent the commands:
+
+ #ed doesn't care about the spaces before the commands, but the . must be the only thing on the line:
+ ed -s file << EOF
+ a
+ my content
+ .
+ w
+ EOF
+
+## Simulate other commands
+
+Keep in mind that in all the examples below, the entire file will be
+read into memory.
+
+### A simple grep
+
+ ed -s file <<< 'g/foo/p'
+
+ # equivalent
+ ed -s file <<< 'g/foo/'
+
+The name `grep` is derived from the notaion `g/RE/p` (global =\> regular
+expression =\> print). ref
+
+
+### wc -l
+
+Since the default for the `ed` "print line number" command is the last
+line, a simple `=` (equal sign) will print this line number and thus the
+number of lines of the file:
+
+ ed -s file <<< '='
+
+### cat
+
+Yea, it's a joke...
+
+ ed -s file <<< $',p'
+
+...but a similar thing to `cat` showing line-endings and escapes can be
+done with the `list` command (l):
+
+ ed -s file <<< $',l'
+
+FIXME to be continued
+
+## Links
+
+Reference:
+
+- [Gnu ed](http://www.gnu.org/software/ed/manual/ed_manual.html) - if we
+ had to guess, you're probably using this one.
+- POSIX
+ [ed](http://pubs.opengroup.org/onlinepubs/9699919799/utilities/ed.html#tag_20_38),
+ [ex](http://pubs.opengroup.org/onlinepubs/9699919799/utilities/ex.html#tag_20_40),
+ and
+ [vi](http://pubs.opengroup.org/onlinepubs/9699919799/utilities/vi.html#tag_20_152)
+- - ed cheatsheet on
+ sdf.org
+
+Misc info / tutorials:
+
+- [How can I replace a string with another string in a variable, a
+ stream, a file, or in all the files in a
+ directory?](http://mywiki.wooledge.org/BashFAQ/021) - BashFAQ
diff --git a/howto/getopts_tutorial.md b/howto/getopts_tutorial.md
new file mode 100644
index 0000000..9a466ce
--- /dev/null
+++ b/howto/getopts_tutorial.md
@@ -0,0 +1,358 @@
+# Small getopts tutorial
+
+![](keywords>bash shell scripting arguments positional parameters options getopt getopts)
+
+## Description
+
+**Note that** `getopts` is neither able to parse GNU-style long options
+(`--myoption`) nor XF86-style long options (`-myoption`). So, when you
+want to parse command line arguments in a professional ;-) way,
+`getopts` may or may not work for you. Unlike its older brother `getopt`
+(note the missing *s*!), it's a shell builtin command. The advantages
+are:
+
+- No need to pass the positional parameters through to an external
+ program.
+- Being a builtin, `getopts` can set shell variables to use for parsing
+ (impossible for an *external* process!)
+- There's no need to argue with several `getopt` implementations which
+ had buggy concepts in the past (whitespace, ...)
+- `getopts` is defined in POSIX(r).
+
+------------------------------------------------------------------------
+
+Some other methods to parse positional parameters - using neither
+**getopt** nor **getopts** - are described in: [How to handle positional
+parameters](/scripting/posparams).
+
+### Terminology
+
+It's useful to know what we're talking about here, so let's see...
+Consider the following command line:
+
+ mybackup -x -f /etc/mybackup.conf -r ./foo.txt ./bar.txt
+
+These are all positional parameters, but they can be divided into
+several logical groups:
+
+- `-x` is an **option** (aka **flag** or **switch**). It consists of a
+ dash (`-`) followed by **one** character.
+- `-f` is also an option, but this option has an associated **option
+ argument** (an argument to the option `-f`): `/etc/mybackup.conf`. The
+ option argument is usually the argument following the option itself,
+ but that isn't mandatory. Joining the option and option argument into
+ a single argument `-f/etc/mybackup.conf` is valid.
+- `-r` depends on the configuration. In this example, `-r` doesn't take
+ arguments so it's a standalone option like `-x`.
+- `./foo.txt` and `./bar.txt` are remaining arguments without any
+ associated options. These are often used as **mass-arguments**. For
+ example, the filenames specified for `cp(1)`, or arguments that don't
+ need an option to be recognized because of the intended behavior of
+ the program. POSIX(r) calls them **operands**.
+
+To give you an idea about why `getopts` is useful, The above command
+line is equivalent to:
+
+ mybackup -xrf /etc/mybackup.conf ./foo.txt ./bar.txt
+
+which is complex to parse without the help of `getopts`.
+
+The option flags can be **upper- and lowercase** characters, or
+**digits**. It may recognize other characters, but that's not
+recommended (usability and maybe problems with special characters).
+
+### How it works
+
+In general you need to call `getopts` several times. Each time it will
+use the next positional parameter and a possible argument, if parsable,
+and provide it to you. `getopts` will not change the set of positional
+parameters. If you want to shift them, it must be done manually:
+
+ shift $((OPTIND-1))
+ # now do something with $@
+
+Since `getopts` sets an exit status of *FALSE* when there's nothing left
+to parse, it's easy to use in a while-loop:
+
+ while getopts ...; do
+ ...
+ done
+
+`getopts` will parse options and their possible arguments. It will stop
+parsing on the first non-option argument (a string that doesn't begin
+with a hyphen (`-`) that isn't an argument for any option in front of
+it). It will also stop parsing when it sees the `--` (double-hyphen),
+which means [end of options](/dict/terms/end_of_options).
+
+### Used variables
+
+| variable | description |
+|------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| [OPTIND](/syntax/shellvars#OPTIND) | Holds the index to the next argument to be processed. This is how `getopts` "remembers" its own status between invocations. Also useful to shift the positional parameters after processing with `getopts`. `OPTIND` is initially set to 1, and **needs to be re-set to 1 if you want to parse anything again with getopts** |
+| [OPTARG](/syntax/shellvars#OPTARG) | This variable is set to any argument for an option found by `getopts`. It also contains the option flag of an unknown option. |
+| [OPTERR](/syntax/shellvars#OPTERR) | (Values 0 or 1) Indicates if Bash should display error messages generated by the `getopts` builtin. The value is initialized to **1** on every shell startup - so be sure to always set it to **0** if you don't want to see annoying messages! **`OPTERR` is not specified by POSIX for the `getopts` builtin utility --- only for the C `getopt()` function in `unistd.h` (`opterr`).** `OPTERR` is bash-specific and not supported by shells such as ksh93, mksh, zsh, or dash. |
+
+`getopts` also uses these variables for error reporting (they're set to
+value-combinations which arent possible in normal operation).
+
+### Specify what you want
+
+The base-syntax for `getopts` is:
+
+ getopts OPTSTRING VARNAME [ARGS...]
+
+where:
+
+| `OPTSTRING` | tells `getopts` which options to expect and where to expect arguments (see below) |
+|-------------|------------------------------------------------------------------------------------|
+| `VARNAME` | tells `getopts` which shell-variable to use for option reporting |
+| `ARGS` | tells `getopts` to parse these optional words instead of the positional parameters |
+
+#### The option-string
+
+The option-string tells `getopts` which options to expect and which of
+them must have an argument. The syntax is very simple --- every option
+character is simply named as is, this example-string would tell
+`getopts` to look for `-f`, `-A` and `-x`:
+
+ getopts fAx VARNAME
+
+When you want `getopts` to expect an argument for an option, just place
+a `:` (colon) after the proper option flag. If you want `-A` to expect
+an argument (i.e. to become `-A SOMETHING`) just do:
+
+ getopts fA:x VARNAME
+
+If the **very first character** of the option-string is a `:` (colon),
+which would normally be nonsense because there's no option letter
+preceding it, `getopts` switches to "**silent error reporting mode**".
+In productive scripts, this is usually what you want because it allows
+you to handle errors yourself without being disturbed by annoying
+messages.
+
+#### Custom arguments to parse
+
+The `getopts` utility parses the [positional
+parameters](/scripting/posparams) of the current shell or function by
+default (which means it parses `"$@"`).
+
+You can give your own set of arguments to the utility to parse. Whenever
+additional arguments are given after the `VARNAME` parameter, `getopts`
+doesn't try to parse the positional parameters, but these given words.
+
+This way, you are able to parse any option set you like, here for
+example from an array:
+
+ while getopts :f:h opt "${MY_OWN_SET[@]}"; do
+ ...
+ done
+
+A call to `getopts` **without** these additional arguments is
+**equivalent** to explicitly calling it with `"$@"`:
+
+ getopts ... "$@"
+
+### Error Reporting
+
+Regarding error-reporting, there are two modes `getopts` can run in:
+
+- verbose mode
+- silent mode
+
+For productive scripts I recommend to use the silent mode, since
+everything looks more professional, when you don't see annoying standard
+messages. Also it's easier to handle, since the failure cases are
+indicated in an easier way.
+
+#### Verbose Mode
+
+| invalid option | `VARNAME` is set to `?` (question-mark) and `OPTARG` is unset |
+|-----------------------------|----------------------------------------------------------------------------------------------|
+| required argument not found | `VARNAME` is set to `?` (question-mark), `OPTARG` is unset and an *error message is printed* |
+
+#### Silent Mode
+
+| invalid option | `VARNAME` is set to `?` (question-mark) and `OPTARG` is set to the (invalid) option character |
+|-----------------------------|-----------------------------------------------------------------------------------------------|
+| required argument not found | `VARNAME` is set to `:` (colon) and `OPTARG` contains the option-character in question |
+
+## Using it
+
+### A first example
+
+Enough said - action!
+
+Let's play with a very simple case: only one option (`-a`) expected,
+without any arguments. Also we disable the *verbose error handling* by
+preceding the whole option string with a colon (`:`):
+
+``` bash
+#!/bin/bash
+
+while getopts ":a" opt; do
+ case $opt in
+ a)
+ echo "-a was triggered!" >&2
+ ;;
+ \?)
+ echo "Invalid option: -$OPTARG" >&2
+ ;;
+ esac
+done
+```
+
+I put that into a file named `go_test.sh`, which is the name you'll see
+below in the examples.
+
+Let's do some tests:
+
+#### Calling it without any arguments
+
+ $ ./go_test.sh
+ $
+
+Nothing happened? Right. `getopts` didn't see any valid or invalid
+options (letters preceded by a dash), so it wasn't triggered.
+
+#### Calling it with non-option arguments
+
+ $ ./go_test.sh /etc/passwd
+ $
+
+Again --- nothing happened. The **very same** case: `getopts` didn't see
+any valid or invalid options (letters preceded by a dash), so it wasn't
+triggered.
+
+The arguments given to your script are of course accessible as `$1` -
+`${N}`.
+
+#### Calling it with option-arguments
+
+Now let's trigger `getopts`: Provide options.
+
+First, an **invalid** one:
+
+ $ ./go_test.sh -b
+ Invalid option: -b
+ $
+
+As expected, `getopts` didn't accept this option and acted like told
+above: It placed `?` into `$opt` and the invalid option character (`b`)
+into `$OPTARG`. With our `case` statement, we were able to detect this.
+
+Now, a **valid** one (`-a`):
+
+ $ ./go_test.sh -a
+ -a was triggered!
+ $
+
+You see, the detection works perfectly. The `a` was put into the
+variable `$opt` for our case statement.
+
+Of course it's possible to **mix valid and invalid** options when
+calling:
+
+ $ ./go_test.sh -a -x -b -c
+ -a was triggered!
+ Invalid option: -x
+ Invalid option: -b
+ Invalid option: -c
+ $
+
+Finally, it's of course possible, to give our option **multiple times**:
+
+ $ ./go_test.sh -a -a -a -a
+ -a was triggered!
+ -a was triggered!
+ -a was triggered!
+ -a was triggered!
+ $
+
+The last examples lead us to some points you may consider:
+
+- **invalid options don't stop the processing**: If you want to stop the
+ script, you have to do it yourself (`exit` in the right place)
+- **multiple identical options are possible**: If you want to disallow
+ these, you have to check manually (e.g. by setting a variable or so)
+
+### An option with argument
+
+Let's extend our example from above. Just a little bit:
+
+- `-a` now takes an argument
+- on an error, the parsing exits with `exit 1`
+
+``` bash
+#!/bin/bash
+
+while getopts ":a:" opt; do
+ case $opt in
+ a)
+ echo "-a was triggered, Parameter: $OPTARG" >&2
+ ;;
+ \?)
+ echo "Invalid option: -$OPTARG" >&2
+ exit 1
+ ;;
+ :)
+ echo "Option -$OPTARG requires an argument." >&2
+ exit 1
+ ;;
+ esac
+done
+```
+
+Let's do the very same tests we did in the last example:
+
+#### Calling it without any arguments
+
+ $ ./go_test.sh
+ $
+
+As above, nothing happened. It wasn't triggered.
+
+#### Calling it with non-option arguments
+
+ $ ./go_test.sh /etc/passwd
+ $
+
+The **very same** case: It wasn't triggered.
+
+#### Calling it with option-arguments
+
+**Invalid** option:
+
+ $ ./go_test.sh -b
+ Invalid option: -b
+ $
+
+As expected, as above, `getopts` didn't accept this option and acted
+like programmed.
+
+**Valid** option, but without the mandatory **argument**:
+
+ $ ./go_test.sh -a
+ Option -a requires an argument.
+ $
+
+The option was okay, but there is an argument missing.
+
+Let's provide **the argument**:
+
+ $ ./go_test.sh -a /etc/passwd
+ -a was triggered, Parameter: /etc/passwd
+ $
+
+## See also
+
+- Internal: [posparams](/scripting/posparams)
+- Internal: [case](/syntax/ccmd/case)
+- Internal: [while_loop](/syntax/ccmd/while_loop)
+- POSIX
+ [getopts(1)](http://pubs.opengroup.org/onlinepubs/9699919799/utilities/getopts.html#tag_20_54)
+ and
+ [getopt(3)](http://pubs.opengroup.org/onlinepubs/9699919799/functions/getopt.html)
+- [parse CLI
+ ARGV](https://stackoverflow.com/questions/192249/how-do-i-parse-command-line-arguments-in-bash)
+- [handle command-line arguments (options) to a
+ script](http://mywiki.wooledge.org/BashFAQ/035)
diff --git a/howto/mutex.md b/howto/mutex.md
new file mode 100644
index 0000000..69585fb
--- /dev/null
+++ b/howto/mutex.md
@@ -0,0 +1,202 @@
+# Lock your script (against parallel execution)
+
+![](keywords>bash shell scripting mutex locking run-control)
+
+## Why lock?
+
+Sometimes there's a need to ensure only one copy of a script runs, i.e
+prevent two or more copies running simultaneously. Imagine an important
+cronjob doing something very important, which will fail or corrupt data
+if two copies of the called program were to run at the same time. To
+prevent this, a form of `MUTEX` (**mutual exclusion**) lock is needed.
+
+The basic procedure is simple: The script checks if a specific condition
+(locking) is present at startup, if yes, it's locked - the scipt doesn't
+start.
+
+This article describes locking with common UNIX(r) tools. There are
+other special locking tools available, But they're not standardized, or
+worse yet, you can't be sure they're present when you want to run your
+scripts. **A tool designed for specifically for this purpose does the
+job much better than general purpose code.**
+
+### Other, special locking tools
+
+As told above, a special tool for locking is the preferred solution.
+Race conditions are avoided, as is the need to work around specific
+limits.
+
+- `flock`:
+- `solo`:
+
+## Choose the locking method
+
+The best way to set a global lock condition is the UNIX(r) filesystem.
+Variables aren't enough, as each process has its own private variable
+space, but the filesystem is global to all processes (yes, I know about
+chroots, namespaces, ... special case). You can "set" several things in
+the filesystem that can be used as locking indicator:
+
+- create files
+- update file timestamps
+- create directories
+
+To create a file or set a file timestamp, usually the command touch is
+used. The following problem is implied: A locking mechanism checks for
+the existance of the lockfile, if no lockfile exists, it creates one and
+continues. Those are **two separate steps**! That means it's **not an
+atomic operation**. There's a small amount of time between checking and
+creating, where another instance of the same script could perform
+locking (because when it checked, the lockfile wasn't there)! In that
+case you would have 2 instances of the script running, both thinking
+they are succesfully locked, and can operate without colliding. Setting
+the timestamp is similar: One step to check the timespamp, a second step
+to set the timestamp.
+
+\ **Conclusion:** We need an
+operation that does the check and the locking in one step. \
+
+A simple way to get that is to create a **lock directory** - with the
+mkdir command. It will:
+
+ * create a given directory only if it does not exist, and set a successful exit code
+ * it will set an unsuccesful exit code if an error occours - for example, if the directory specified already exists
+
+With mkdir it seems, we have our two steps in one simple operation. A
+(very!) simple locking code might look like this:
+
+``` bash
+if mkdir /var/lock/mylock; then
+ echo "Locking succeeded" >&2
+else
+ echo "Lock failed - exit" >&2
+ exit 1
+fi
+```
+
+In case `mkdir` reports an error, the script will exit at this point -
+**the MUTEX did its job!**
+
+*If the directory is removed after setting a successful lock, while the
+script is still running, the lock is lost. Doing chmod -w for the parent
+directory containing the lock directory can be done, but it is not
+atomic. Maybe a while loop checking continously for the existence of the
+lock in the background and sending a signal such as USR1, if the
+directory is not found, can be done. The signal would need to be
+trapped. I am sure there there is a better solution than this
+suggestion* --- *[sn18](sunny_delhi18@yahoo.com) 2009/12/19 08:24*
+
+**Note:** While perusing the Internet, I found some people asking if the
+`mkdir` method works "on all filesystems". Well, let's say it should.
+The syscall under `mkdir` is guarenteed to work atomicly in all cases,
+at least on Unices. Two examples of problems are NFS filesystems and
+filesystems on cluster servers. With those two scenarios, dependencies
+exist related to the mount options and implementation. However, I
+successfully use this simple method on an Oracle OCFS2 filesystem in a
+4-node cluster environment. So let's just say "it should work under
+normal conditions".
+
+Another atomic method is setting the `noclobber` shell option
+(`set -C`). That will cause redirection to fail, if the file the
+redirection points to already exists (using diverse `open()` methods).
+Need to write a code example here.
+
+``` bash
+
+if ( set -o noclobber; echo "locked" > "$lockfile") 2> /dev/null; then
+ trap 'rm -f "$lockfile"; exit $?' INT TERM EXIT
+ echo "Locking succeeded" >&2
+ rm -f "$lockfile"
+else
+ echo "Lock failed - exit" >&2
+ exit 1
+fi
+
+```
+
+Another explanation of this basic pattern using `set -C` can be found
+[here](http://pubs.opengroup.org/onlinepubs/9699919799/xrat/V4_xcu_chap02.html#tag_23_02_07).
+
+## An example
+
+This code was taken from a production grade script that controls PISG to
+create statistical pages from my IRC logfiles. There are some
+differences compared to the very simple example above:
+
+- the locking stores the process ID of the locked instance
+- if a lock fails, the script tries to find out if the locked instance
+ still is active (unreliable!)
+- traps are created to automatically remove the lock when the script
+ terminates, or is killed
+
+Details on how the script is killed aren't given, only code relevant to
+the locking process is shown:
+
+``` bash
+#!/bin/bash
+
+# lock dirs/files
+LOCKDIR="/tmp/statsgen-lock"
+PIDFILE="${LOCKDIR}/PID"
+
+# exit codes and text
+ENO_SUCCESS=0; ETXT[0]="ENO_SUCCESS"
+ENO_GENERAL=1; ETXT[1]="ENO_GENERAL"
+ENO_LOCKFAIL=2; ETXT[2]="ENO_LOCKFAIL"
+ENO_RECVSIG=3; ETXT[3]="ENO_RECVSIG"
+
+###
+### start locking attempt
+###
+
+trap 'ECODE=$?; echo "[statsgen] Exit: ${ETXT[ECODE]}($ECODE)" >&2' 0
+echo -n "[statsgen] Locking: " >&2
+
+if mkdir "${LOCKDIR}" &>/dev/null; then
+
+ # lock succeeded, install signal handlers before storing the PID just in case
+ # storing the PID fails
+ trap 'ECODE=$?;
+ echo "[statsgen] Removing lock. Exit: ${ETXT[ECODE]}($ECODE)" >&2
+ rm -rf "${LOCKDIR}"' 0
+ echo "$$" >"${PIDFILE}"
+ # the following handler will exit the script upon receiving these signals
+ # the trap on "0" (EXIT) from above will be triggered by this trap's "exit" command!
+ trap 'echo "[statsgen] Killed by a signal." >&2
+ exit ${ENO_RECVSIG}' 1 2 3 15
+ echo "success, installed signal handlers"
+
+else
+
+ # lock failed, check if the other PID is alive
+ OTHERPID="$(cat "${PIDFILE}")"
+
+ # if cat isn't able to read the file, another instance is probably
+ # about to remove the lock -- exit, we're *still* locked
+ # Thanks to Grzegorz Wierzowiecki for pointing out this race condition on
+ # http://wiki.grzegorz.wierzowiecki.pl/code:mutex-in-bash
+ if [ $? != 0 ]; then
+ echo "lock failed, PID ${OTHERPID} is active" >&2
+ exit ${ENO_LOCKFAIL}
+ fi
+
+ if ! kill -0 $OTHERPID &>/dev/null; then
+ # lock is stale, remove it and restart
+ echo "removing stale lock of nonexistant PID ${OTHERPID}" >&2
+ rm -rf "${LOCKDIR}"
+ echo "[statsgen] restarting myself" >&2
+ exec "$0" "$@"
+ else
+ # lock is valid and OTHERPID is active - exit, we're locked!
+ echo "lock failed, PID ${OTHERPID} is active" >&2
+ exit ${ENO_LOCKFAIL}
+ fi
+
+fi
+```
+
+## Related links
+
+- [BashFAQ/045](http://mywiki.wooledge.org/BashFAQ/045)
+- [Implementation of a shell locking
+ utility](http://wiki.grzegorz.wierzowiecki.pl/code:mutex-in-bash)
diff --git a/howto/pax.md b/howto/pax.md
new file mode 100644
index 0000000..d8a899f
--- /dev/null
+++ b/howto/pax.md
@@ -0,0 +1,347 @@
+# pax - the POSIX archiver
+
+![](keywords>bash shell scripting POSIX archive tar packing zip)
+
+pax can do a lot of fancy stuff, feel free to contribute more awesome
+pax tricks!
+
+## Introduction
+
+The POSIX archiver, `pax`, is an attempt at a standardized archiver with
+the best features of `tar` and `cpio`, able to handle all common archive
+types.
+
+However, this is **not a manpage**, it will **not** list all possible
+options, it will **not** you detailed information about `pax`. It's only
+an introduction.
+
+This article is based on the debianized Berkeley implementation of
+`pax`, but implementation-specific things should be tagged as such.
+Unfortunately, the Debian package doesn't seem to be maintained anymore.
+
+## Overview
+
+### Operation modes
+
+There are four basic operation modes to *list*, *read*, *write* and
+*copy* archives. They're switched with combinations of `-r` and `-w`
+command line options:
+
+| Mode | RW-Options |
+|-------|-----------------|
+| List | *no RW-options* |
+| Read | `-r` |
+| Write | `-w` |
+| Copy | `-r -w` |
+
+#### List
+
+In *list mode*, `pax` writes the list of archive members to standard
+output (a table of contents). If a pattern match is specified on the
+command line, only matching filenames are printed.
+
+#### Read
+
+*Read* an archive. `pax` will read archive data and extract the members
+to the current directory. If a pattern match is specified on the command
+line, only matching filenames are extracted.
+
+When reading an archive, the archive type is determined from the archive
+data.
+
+#### Write
+
+*Write* an archive, which means create a new one or append to an
+existing one. All files and directories specified on the command line
+are inserted into the archive. The archive is written to standard output
+by default.
+
+If no files are specified on the command line, filenames are read from
+`STDIN`.
+
+The write mode is the only mode where you need to specify the archive
+type with `-x `, e.g. `-x ustar`.
+
+#### Copy
+
+*Copy* mode is similar to `cpio` passthrough mode. It provides a way to
+replicate a complete or partial file hierarchy (with all the `pax`
+options, e.g. rewriting groups) to another location.
+
+### Archive data
+
+When you don't specify anything special, `pax` will attempt to read
+archive data from standard input (read/list modes) and write archive
+data to standard output (write mode). This ensures `pax` can be easily
+used as part of a shell pipe construct, e.g. to read a compressed
+archive that's decompressed in the pipe.
+
+The option to specify the pathname of a file to be archived is `-f` This
+file will be used as input or output, depending on the operation
+(read/write/list).
+
+When pax reads an archive, it tries to guess the archive type. However,
+in *write* mode, you must specify which type of archive to append using
+the `-x ` switch. If you omit this switch, a default archive will
+be created (POSIX says it's implementation defined, Berkeley `pax`
+creates `ustar` if no options are specified).
+
+The following archive formats are supported (Berkeley implementation):
+
+| | |
+|:--------|----------------------------|
+| ustar | POSIX TAR format (default) |
+| cpio | POSIX CPIO format |
+| tar | classic BSD TAR format |
+| bcpio | old binary CPIO format |
+| sv4cpio | SVR4 CPIO format |
+| sv4crc | SVR4 CPIO format with CRC |
+
+Berkeley `pax` supports options `-z` and `-j`, similar to GNU `tar`, to
+filter archive files through GZIP/BZIP2.
+
+### Matching archive members
+
+In *read* and *list* modes, you can specify patterns to determine which
+files to list or extract.
+
+- the pattern notation is the one known by a POSIX-shell, i.e. the one
+ known by Bash without `extglob`
+- if the specified pattern matches a complete directory, it affects all
+ files and subdirectories of the specified directory
+- if you specify the `-c` option, `pax` will invert the matches, i.e. it
+ matches all filenames **except** those matching the specified patterns
+- if no patterns are given, `pax` will "match" (list or extract) all
+ files from the archive
+- **To avoid conflicts with shell pathname expansion, it's wise to quote
+ patterns!**
+
+#### Some assorted examples of patterns
+
+ pax -r archive.tar README.txt *.png data/
+
+ # equivalent, extract archive contents directly to a file
+ pax -w -x ustar -f archive.tar README.txt *.png data/
+
+`pax` is in *write* mode, the given filenames are packed into an
+archive:
+
+- `README.txt` is a normal file, it will be packed
+- `*.png` is a pathname glob **for your shell**, the shell will
+ substitute all matching filenames **before** `pax` is executed. The
+ result is a list of filenames that will be packed like the
+ `README.txt` example above
+- `data/` is a directory. **Everything** in this directory will be
+ packed into the archive, i.e. not just an empty directory
+
+When you specify the `-v` option, `pax` will write the pathnames of the
+files inserted into the archive to `STDERR`.
+
+When, and only when, no filename arguments are specified, `pax` attempts
+to read filenames from `STDIN`, separated by newlines. This way you can
+easily combine `find` with `pax`:
+
+ find . -name '*.txt' | pax -wf textfiles.tar -x ustar
+
+### Listing archive contents
+
+The standard output format to list archive members simply is to print
+each filename to a separate line. But the output format can be
+customized to include permissions, timestamps, etc. with the
+`-o listopt=` specification. The syntax of the format
+specification is strongly derived from the `printf(3)` format
+specification.
+
+**Unfortunately** the `pax` utility delivered with Debian doesn't seem
+to support these extended listing formats.
+
+However, `pax` lists archive members in a `ls -l`-like format, when you
+give the `-v` option:
+
+ pax -v **Note:** `-T` is an extension and is not defined by POSIX.
+
+Say you have write-access to a fileserver mounted on your filesystem
+tree. In *copy* mode, you can tell `pax` to copy only files that were
+modified today:
+
+ mkdir /n/mybackups/$(date +%A)/
+ pax -rw -T 0000 data/ /n/mybackups/$(date +%A)/
+
+This is done using the `-T` switch, which normally allows you to specify
+a time window, but in this case, only the start time which means "today
+at midnight".
+
+When you execute this "very simple backup" after your daily work, you
+will have a copy of the modified files.
+
+**Note:** The `%A` format from `date` expands to the name of the
+current day, localized, e.g. "Friday" (en) or "Mittwoch" (de).
+
+The same, but with an archive, can be accomplished by:
+
+ pax -w -T 0000 -f /n/mybackups/$(date +%A)
+
+In this case, the day-name is an archive-file (you don't need a filename
+extension like `.tar` but you can add one, if desired).
+
+### Changing filenames while archiving
+
+`pax` is able to rewrite filenames while archiving or while extracting
+from an archive. This example creates a tar archive containing the
+`holiday_2007/` directory, but the directory name inside the archive
+will be `holiday_pics/`:
+
+ pax -x ustar -w -f holiday_pictures.tar -s '/^holiday_2007/holiday_pics/' holiday_2007/
+
+The option responsible for the string manipulation is the
+`-s `. It takes the string rewrite specification
+as an argument, in the form `/OLD/NEW/[gp]`, which is an `ed(1)`-like
+regular expression (BRE) for `old` and generally can be used like the
+popular sed construct `s/from/to/`. Any non-null character can be used
+as a delimiter, so to mangle pathnames (containing slashes), you could
+use `#/old/path#/new/path#`.
+
+The optional `g` and `p` flags are used to apply substitution
+**(g)**lobally to the line or to **(p)**rint the original and rewritten
+strings to `STDERR`.
+
+Multiple `-s` options can be specified on the command line. They are
+applied to the pathname strings of the files or archive members. This
+happens in the order they are specified.
+
+### Excluding files from an archive
+
+The -s command seen above can be used to exclude a file. The
+substitution must result in a null string: For example, let's say that
+you want to exclude all the CVS directories to create a source code
+archive. We are going to replace the names containing /CVS/ with
+nothing, note the .\* they are needed because we need to match the
+entire pathname.
+
+ pax -w -x ustar -f release.tar -s',.*/CVS/.*,,' myapplication
+
+You can use several -s options, for instance, let's say you also want to
+remove files ending in ~:
+
+ pax -w -x ustar -f release.tar -'s,.*/CVS/.*,,' -'s/.*~//' myapplication
+
+This can also be done while reading an archive, for instance, suppose
+you have an archive containing a "usr" and a "etc" directory but that
+you want to extract only the "usr" directory:
+
+ pax -r -f archive.tar -s',^etc/.*,,' #the etc/ dir is not extracted
+
+### Getting archive filenames from STDIN
+
+Like `cpio`, pax can read filenames from standard input (`stdin`). This
+provides great flexibility - for example, a `find(1)` command may select
+files/directories in ways pax can't do itself. In **write** mode
+(creating an archive) or **copy** mode, when no filenames are given, pax
+expects to read filenames from standard input. For example:
+
+ # Back up config files changed less than 3 days ago
+ find /etc -type f -mtime -3 | pax -x ustar -w -f /backups/etc.tar
+
+ # Copy only the directories, not the files
+ mkdir /target
+ find . -type d -print | pax -r -w -d /target
+
+ # Back up anything that changed since the last backup
+ find . -newer /var/run/mylastbackup -print0 |
+ pax -0 -x ustar -w -d -f /backups/mybackup.tar
+ touch /var/run/mylastbackup
+
+The `-d` option tells pax `not` to recurse into directories it reads
+(`cpio`-style). Without `-d`, pax recurses into all directories
+(`tar`-style).
+
+**Note**: the `-0` option is not standard, but is present in some
+implementations.
+
+## From tar to pax
+
+`pax` can handle the `tar` archive format, if you want to switch to the
+standard tool an alias like:
+
+ alias tar='echo USE PAX, idiot. pax is the standard archiver!; # '
+
+in your `~/.bashrc` can be useful :-D.
+
+Here is a quick table comparing (GNU) `tar` and `pax` to help you to
+make the switch:
+
+| TAR | PAX | Notes |
+|-------------------------------------|------------------------------------------|-----------------------------------------------------------------------|
+| `tar xzvf file.tar.gz` | `pax -rvz -f file.tar.gz` | `-z` is an extension, POSIXly: `gunzip archive.tar.gz` |
+| `tar xjvf file.tar.bz2` | `bunzip2 archive.tar.bz2` | |
+| `tar tzvf file.tar.gz` | `pax -vz -f file.tar.gz` | `-z` is an extension, POSIXly: `gunzip bash shell scripting tutorial redirection redirect file descriptor)
+
+This tutorial is not a complete guide to redirection, it will not cover
+here docs, here strings, name pipes etc... I just hope it'll help you to
+understand what things like `3>&2`, `2>&1` or `1>&3-` do.
+
+# stdin, stdout, stderr
+
+When Bash starts, normally, 3 file descriptors are opened, `0`, `1` and
+`2` also known as standard input (`stdin`), standard output (`stdout`)
+and standard error (`stderr`).
+
+For example, with Bash running in a Linux terminal emulator, you'll see:
+
+ # lsof +f g -ap $BASHPID -d 0,1,2
+ COMMAND PID USER FD TYPE FILE-FLAG DEVICE SIZE/OFF NODE NAME
+ bash 12135 root 0u CHR RW,LG 136,13 0t0 16 /dev/pts/5
+ bash 12135 root 1u CHR RW,LG 136,13 0t0 16 /dev/pts/5
+ bash 12135 root 2u CHR RW,LG 136,13 0t0 16 /dev/pts/5
+
+This `/dev/pts/5` is a pseudo terminal used to emulate a real terminal.
+Bash reads (`stdin`) from this terminal and prints via `stdout` and
+`stderr` to this terminal.
+
+ --- +-----------------------+
+ standard input ( 0 ) ---->| /dev/pts/5 |
+ --- +-----------------------+
+
+ --- +-----------------------+
+ standard output ( 1 ) ---->| /dev/pts/5 |
+ --- +-----------------------+
+
+ --- +-----------------------+
+ standard error ( 2 ) ---->| /dev/pts/5 |
+ --- +-----------------------+
+
+When a command, a compound command, a subshell etc. is executed, it
+inherits these file descriptors. For instance `echo foo` will send the
+text `foo` to the file descriptor `1` inherited from the shell, which is
+connected to `/dev/pts/5`.
+
+# Simple Redirections
+
+## Output Redirection "n\> file"
+
+`>` is probably the simplest redirection.
+
+`echo foo > file`
+
+the `> file` after the command alters the file descriptors belonging to
+the command `echo`. It changes the file descriptor `1` (`> file` is the
+same as `1>file`) so that it points to the file `file`. They will look
+like:
+
+ --- +-----------------------+
+ standard input ( 0 ) ---->| /dev/pts/5 |
+ --- +-----------------------+
+
+ --- +-----------------------+
+ standard output ( 1 ) ---->| file |
+ --- +-----------------------+
+
+ --- +-----------------------+
+ standard error ( 2 ) ---->| /dev/pts/5 |
+ --- +-----------------------+
+
+Now characters written by our command, `echo`, that are sent to the
+standard output, i.e., the file descriptor `1`, end up in the file named
+`file`.
+
+In the same way, command `2> file` will change the standard error and
+will make it point to `file`. Standard error is used by applications to
+print errors.
+
+What will `command 3> file` do? It will open a new file descriptor
+pointing to `file`. The command will then start with:
+
+ --- +-----------------------+
+ standard input ( 0 ) ---->| /dev/pts/5 |
+ --- +-----------------------+
+
+ --- +-----------------------+
+ standard output ( 1 ) ---->| /dev/pts/5 |
+ --- +-----------------------+
+
+ --- +-----------------------+
+ standard error ( 2 ) ---->| /dev/pts/5 |
+ --- +-----------------------+
+
+ --- +-----------------------+
+ new descriptor ( 3 ) ---->| file |
+ --- +-----------------------+
+
+What will the command do with this descriptor? It depends. Often
+nothing. We will see later why we might want other file descriptors.
+
+## Input Redirection "n\< file"
+
+When you run a commandusing `command < file`, it changes the file
+descriptor `0` so that it looks like:
+
+ --- +-----------------------+
+ standard input ( 0 ) <----| file |
+ --- +-----------------------+
+
+ --- +-----------------------+
+ standard output ( 1 ) ---->| /dev/pts/5 |
+ --- +-----------------------+
+
+ --- +-----------------------+
+ standard error ( 2 ) ---->| /dev/pts/5 |
+ --- +-----------------------+
+
+If the command reads from `stdin`, it now will read from `file` and not
+from the console.
+
+As with `>`, `<` can be used to open a new file descriptor for reading,
+`command 3| /dev/pts/5 | ------> ( 0 ) ---->|pipe (read) |
+ --- +--------------+ / --- +--------------+
+ /
+ --- +--------------+ / --- +--------------+
+ ( 1 ) ---->| pipe (write) | / ( 1 ) ---->| /dev/pts |
+ --- +--------------+ --- +--------------+
+
+ --- +--------------+ --- +--------------+
+ ( 2 ) ---->| /dev/pts/5 | ( 2 ) ---->| /dev/pts/ |
+ --- +--------------+ --- +--------------+
+
+This is possible because the redirections are set up by the shell
+**before** the commands are executed, and the commands inherit the file
+descriptors.
+
+# More On File Descriptors
+
+## Duplicating File Descriptor 2\>&1
+
+We have seen how to open (or redirect) file descriptors. Let us see how
+to duplicate them, starting with the classic `2>&1`. What does this
+mean? That something written on the file descriptor `2` will go where
+file descriptor `1` goes. In a shell `command 2>&1` is not a very
+interesting example so we will use `ls /tmp/ doesnotexist 2>&1 | less`
+
+ ls /tmp/ doesnotexist 2>&1 | less
+
+ --- +--------------+ --- +--------------+
+ ( 0 ) ---->| /dev/pts/5 | ------> ( 0 ) ---->|from the pipe |
+ --- +--------------+ / ---> --- +--------------+
+ / /
+ --- +--------------+ / / --- +--------------+
+ ( 1 ) ---->| to the pipe | / / ( 1 ) ---->| /dev/pts |
+ --- +--------------+ / --- +--------------+
+ /
+ --- +--------------+ / --- +--------------+
+ ( 2 ) ---->| to the pipe | / ( 2 ) ---->| /dev/pts/ |
+ --- +--------------+ --- +--------------+
+
+Why is it called *duplicating*? Because after `2>&1`, we have 2 file
+descriptors pointing to the same file. Take care not to call this "File
+Descriptor Aliasing"; if we redirect `stdout` after `2>&1` to a file
+`B`, file descriptor `2` will still be opened on the file `A` where it
+was. This is often misunderstood by people wanting to redirect both
+standard input and standard output to the file. Continue reading for
+more on this.
+
+So if you have two file descriptors `s` and `t` like:
+
+ --- +-----------------------+
+ a descriptor ( s ) ---->| /some/file |
+ --- +-----------------------+
+ --- +-----------------------+
+ a descriptor ( t ) ---->| /another/file |
+ --- +-----------------------+
+
+Using a `t>&s` (where `t` and `s` are numbers) it means:
+
+> Copy whatever file descriptor `s` contains into file descriptor `t`
+
+So you got a copy of this descriptor:
+
+ --- +-----------------------+
+ a descriptor ( s ) ---->| /some/file |
+ --- +-----------------------+
+ --- +-----------------------+
+ a descriptor ( t ) ---->| /some/file |
+ --- +-----------------------+
+
+Internally each of these is represented by a file descriptor opened by
+the operating system's `fopen` calls, and is likely just a pointer to
+the file which has been opened for reading (`stdin` or file descriptor
+`0`) or writing (`stdout` /`stderr`).
+
+Note that the file reading or writing positions are also duplicated. If
+you have already read a line of `s`, then after `t>&s` if you read a
+line from `t`, you will get the second line of the file.
+
+Similarly for output file descriptors, writing a line to file descriptor
+`s` will append a line to a file as will writing a line to file
+descriptor `t`.
+
+\The syntax is somewhat confusing in that you would think
+that the arrow would point in the direction of the copy, but it's
+reversed. So it's `target>&source` effectively.\
+
+So, as a simple example (albeit slightly contrived), is the following:
+
+ exec 3>&1 # Copy 1 into 3
+ exec 1> logfile # Make 1 opened to write to logfile
+ lotsa_stdout # Outputs to fd 1, which writes to logfile
+ exec 1>&3 # Copy 3 back into 1
+ echo Done # Output to original stdout
+
+## Order Of Redirection, i.e., "\> file 2\>&1" vs. "2\>&1 \>file"
+
+While it doesn't matter where the redirections appears on the command
+line, their order does matter. They are set up from left to right.
+
+- `2>&1 >file`
+
+A common error, is to do `command 2>&1 > file` to redirect both `stderr`
+and `stdout` to `file`. Let's see what's going on. First we type the
+command in our terminal, the descriptors look like this:
+
+ --- +-----------------------+
+ standard input ( 0 ) ---->| /dev/pts/5 |
+ --- +-----------------------+
+
+ --- +-----------------------+
+ standard output ( 1 ) ---->| /dev/pts/5 |
+ --- +-----------------------+
+
+ --- +-----------------------+
+ standard error ( 2 ) ---->| /dev/pts/5 |
+ --- +-----------------------+
+
+Then our shell, Bash sees `2>&1` so it duplicates 1, and the file
+descriptor look like this:
+
+ --- +-----------------------+
+ standard input ( 0 ) ---->| /dev/pts/5 |
+ --- +-----------------------+
+
+ --- +-----------------------+
+ standard output ( 1 ) ---->| /dev/pts/5 |
+ --- +-----------------------+
+
+ --- +-----------------------+
+ standard error ( 2 ) ---->| /dev/pts/5 |
+ --- +-----------------------+
+
+That's right, nothing has changed, 2 was already pointing to the same
+place as 1. Now Bash sees `> file` and thus changes `stdout`:
+
+ --- +-----------------------+
+ standard input ( 0 ) ---->| /dev/pts/5 |
+ --- +-----------------------+
+
+ --- +-----------------------+
+ standard output ( 1 ) ---->| file |
+ --- +-----------------------+
+
+ --- +-----------------------+
+ standard error ( 2 ) ---->| /dev/pts/5 |
+ --- +-----------------------+
+
+And that's not what we want.
+
+- `>file 2>&1`
+
+Now let's look at the correct `command >file 2>&1`. We start as in the
+previous example, and Bash sees `> file`:
+
+ --- +-----------------------+
+ standard input ( 0 ) ---->| /dev/pts/5 |
+ --- +-----------------------+
+
+ --- +-----------------------+
+ standard output ( 1 ) ---->| file |
+ --- +-----------------------+
+
+ --- +-----------------------+
+ standard error ( 2 ) ---->| /dev/pts/5 |
+ --- +-----------------------+
+
+Then it sees our duplication `2>&1`:
+
+ --- +-----------------------+
+ standard input ( 0 ) ---->| /dev/pts/5 |
+ --- +-----------------------+
+
+ --- +-----------------------+
+ standard output ( 1 ) ---->| file |
+ --- +-----------------------+
+
+ --- +-----------------------+
+ standard error ( 2 ) ---->| file |
+ --- +-----------------------+
+
+And voila, both `1` and `2` are redirected to file.
+
+## Why sed 's/foo/bar/' file \>file Doesn't Work
+
+This is a common error, we want to modify a file using something that
+reads from a file and writes the result to `stdout`. To do this, we
+redirect stdout to the file we want to modify. The problem here is that,
+as we have seen, the redirections are setup before the command is
+actually executed.
+
+So **BEFORE** sed starts, standard output has already been redirected,
+with the additional side effect that, because we used \>, "file" gets
+truncated. When `sed` starts to read the file, it contains nothing.
+
+## exec
+
+In Bash the `exec` built-in replaces the shell with the specified
+program. So what does this have to do with redirection? `exec` also
+allow us to manipulate the file descriptors. If you don't specify a
+program, the redirection after `exec` modifies the file descriptors of
+the current shell.
+
+For example, all the commands after `exec 2>file` will have file
+descriptors like:
+
+ --- +-----------------------+
+ standard input ( 0 ) ---->| /dev/pts/5 |
+ --- +-----------------------+
+
+ --- +-----------------------+
+ standard output ( 1 ) ---->| /dev/pts/5 |
+ --- +-----------------------+
+
+ --- +-----------------------+
+ standard error ( 2 ) ---->| file |
+ --- +-----------------------+
+
+All the the errors sent to `stderr` by the commands after the
+`exec 2>file` will go to the file, just as if you had the command in a
+script and ran `myscript 2>file`.
+
+`exec` can be used, if, for instance, you want to log the errors the
+commands in your script produce, just add `exec 2>myscript.errors` at
+the beginning of your script.
+
+Let's see another use case. We want to read a file line by line, this is
+easy, we just do:
+
+ while read -r line;do echo "$line";done < file
+
+Now, we want, after printing each line, to do a pause, waiting for the
+user to press a key:
+
+ while read -r line;do echo "$line"; read -p "Press any key" -n 1;done < file
+
+And, surprise this doesn't work. Why? because the shell descriptor of
+the while loop looks like:
+
+ --- +-----------------------+
+ standard input ( 0 ) ---->| file |
+ --- +-----------------------+
+
+ --- +-----------------------+
+ standard output ( 1 ) ---->| /dev/pts/5 |
+ --- +-----------------------+
+
+ --- +-----------------------+
+ standard error ( 2 ) ---->| /dev/pts/5 |
+ --- +-----------------------+
+
+and our read inherits these descriptors, and our command
+(`read -p "Press any key" -n 1`) inherits them, and thus reads from file
+and not from our terminal.
+
+A quick look at `help read` tells us that we can specify a file
+descriptor from which `read` should read. Cool. Now let's use `exec` to
+get another descriptor:
+
+ exec 3| /dev/pts/5 |
+ --- +-----------------------+
+
+ --- +-----------------------+
+ standard output ( 1 ) ---->| /dev/pts/5 |
+ --- +-----------------------+
+
+ --- +-----------------------+
+ standard error ( 2 ) ---->| /dev/pts/5 |
+ --- +-----------------------+
+
+ --- +-----------------------+
+ new descriptor ( 3 ) ---->| file |
+ --- +-----------------------+
+
+and it works.
+
+## Closing The File Descriptors
+
+Closing a file through a file descriptor is easy, just make it a
+duplicate of -. For instance, let's close `stdin <&-` and `stderr 2>&-`:
+
+ bash -c '{ lsof -a -p $$ -d0,1,2 ;} <&- 2>&-'
+ COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
+ bash 10668 pgas 1u CHR 136,2 4 /dev/pts/2
+
+we see that inside the `{}` that only `1` is still here.
+
+Though the OS will probably clean up the mess, it is perhaps a good idea
+to close the file descriptors you open. For instance, if you open a file
+descriptor with `exec 3>file`, all the commands afterwards will inherit
+it. It's probably better to do something like:
+
+ exec 3>file
+ .....
+ #commands that uses 3
+ .....
+ exec 3>&-
+
+ #we don't need 3 any more
+
+I've seen some people using this as a way to discard, say stderr, using
+something like: command 2\>&-. Though it might work, I'm not sure if you
+can expect all applications to behave correctly with a closed stderr.
+
+When in doubt, I use 2\>/dev/null.
+
+# An Example
+
+This example comes from [this post
+(ffe4c2e382034ed9)](http://groups.google.com/group/comp.unix.shell/browse_thread/thread/64206d154894a4ef/ffe4c2e382034ed9#ffe4c2e382034ed9)
+on the comp.unix.shell group:
+
+ {
+ {
+ cmd1 3>&- |
+ cmd2 2>&3 3>&-
+ } 2>&1 >&4 4>&- |
+ cmd3 3>&- 4>&-
+
+ } 3>&2 4>&1
+
+The redirections are processed from left to right, but as the file
+descriptors are inherited we will also have to work from the outer to
+the inner contexts. We will assume that we run this command in a
+terminal. Let's start with the outer `{ } 3>&2 4>&1`.
+
+ --- +-------------+ --- +-------------+
+ ( 0 ) ---->| /dev/pts/5 | ( 3 ) ---->| /dev/pts/5 |
+ --- +-------------+ --- +-------------+
+
+ --- +-------------+ --- +-------------+
+ ( 1 ) ---->| /dev/pts/5 | ( 4 ) ---->| /dev/pts/5 |
+ --- +-------------+ --- +-------------+
+
+ --- +-------------+
+ ( 2 ) ---->| /dev/pts/5 |
+ --- +-------------+
+
+We only made 2 copies of `stderr` and `stdout`. `3>&1 4>&1` would have
+produced the same result here because we ran the command in a terminal
+and thus `1` and `2` go to the terminal. As an exercise, you can start
+with `1` pointing to `file.stdout` and 2 pointing to `file.stderr`, you
+will see why these redirections are very nice.
+
+Let's continue with the right part of the second pipe:
+`| cmd3 3>&- 4>&-`
+
+ --- +-------------+
+ ( 0 ) ---->| 2nd pipe |
+ --- +-------------+
+
+ --- +-------------+
+ ( 1 ) ---->| /dev/pts/5 |
+ --- +-------------+
+
+ --- +-------------+
+ ( 2 ) ---->| /dev/pts/5 |
+ --- +-------------+
+
+It inherits the previous file descriptors, closes 3 and 4 and sets up a
+pipe for reading. Now for the left part of the second pipe
+`{...} 2>&1 >&4 4>&- |`
+
+ --- +-------------+ --- +-------------+
+ ( 0 ) ---->| /dev/pts/5 | ( 3 ) ---->| /dev/pts/5 |
+ --- +-------------+ --- +-------------+
+
+ --- +-------------+
+ ( 1 ) ---->| /dev/pts/5 |
+ --- +-------------+
+
+ --- +-------------+
+ ( 2 ) ---->| 2nd pipe |
+ --- +-------------+
+
+First, The file descriptor `1` is connected to the pipe (`|`), then `2`
+is made a copy of `1` and thus is made an fd to the pipe (`2>&1`), then
+`1` is made a copy of `4` (`>&4`), then `4` is closed. These are the
+file descriptors of the inner `{}`. Lcet's go inside and have a look at
+the right part of the first pipe: `| cmd2 2>&3 3>&-`
+
+ --- +-------------+
+ ( 0 ) ---->| 1st pipe |
+ --- +-------------+
+
+ --- +-------------+
+ ( 1 ) ---->| /dev/pts/5 |
+ --- +-------------+
+
+ --- +-------------+
+ ( 2 ) ---->| /dev/pts/5 |
+ --- +-------------+
+
+It inherits the previous file descriptors, connects 0 to the 1st pipe,
+the file descriptor 2 is made a copy of 3, and 3 is closed. Finally, for
+the left part of the pipe:
+
+ --- +-------------+
+ ( 0 ) ---->| /dev/pts/5 |
+ --- +-------------+
+
+ --- +-------------+
+ ( 1 ) ---->| 1st pipe |
+ --- +-------------+
+
+ --- +-------------+
+ ( 2 ) ---->| 2nd pipe |
+ --- +-------------+
+
+It also inherits the file descriptor of the left part of the 2nd pipe,
+file descriptor `1` is connected to the first pipe, `3` is closed.
+
+The purpose of all this becomes clear if we take only the commands:
+
+ cmd2
+
+ --- +-------------+
+ -->( 0 ) ---->| 1st pipe |
+ / --- +-------------+
+ /
+ / --- +-------------+
+ cmd 1 / ( 1 ) ---->| /dev/pts/5 |
+ / --- +-------------+
+ /
+ --- +-------------+ / --- +-------------+
+ ( 0 ) ---->| /dev/pts/5 | / ( 2 ) ---->| /dev/pts/5 |
+ --- +-------------+ / --- +-------------+
+ /
+ --- +-------------+ / cmd3
+ ( 1 ) ---->| 1st pipe | /
+ --- +-------------+ --- +-------------+
+ ------------>( 0 ) ---->| 2nd pipe |
+ --- +-------------+ / --- +-------------+
+ ( 2 ) ---->| 2nd pipe |/
+ --- +-------------+ --- +-------------+
+ ( 1 ) ---->| /dev/pts/5 |
+ --- +-------------+
+
+ --- +-------------+
+ ( 2 ) ---->| /dev/pts/5 |
+ --- +-------------+
+
+As said previously, as an exercise, you can start with `1` open on a
+file and `2` open on another file to see how the `stdin` from `cmd2` and
+`cmd3` goes to the original `stdin` and how the `stderr` goes to the
+original `stderr`.
+
+# Syntax
+
+I used to have trouble choosing between `0&<3` `3&>1` `3>&1` `->2`
+`-<&0` `&-<0` `0<&-` etc... (I think probably because the syntax is more
+representative of the result, i.e., the redirection, than what is done,
+i.e., opening, closing, or duplicating file descriptors).
+
+If this fits your situation, then maybe the following "rules" will help
+you, a redirection is always like the following:
+
+ lhs op rhs
+
+- `lhs` is always a file description, i.e., a number:
+ - Either we want to open, duplicate, move or we want to close. If the
+ op is `<` then there is an implicit 0, if it's `>` or `>>`, there is
+ an implicit 1.
+
+
+
+- `op` is `<`, `>`, `>>`, `>|`, or `<>`:
+ - `<` if the file decriptor in `lhs` will be read, `>` if it will be
+ written, `>>` if data is to be appended to the file, `>|` to
+ overwrite an existing file or `<>` if it will be both read and
+ written.
+
+
+
+- `rhs` is the thing that the file descriptor will describe:
+ - It can be the name of a file, the place where another descriptor
+ goes (`&1`), or, `&-`, which will close the file descriptor.
+
+You might not like this description, and find it a bit incomplete or
+inexact, but I think it really helps to easily find that, say `&->0` is
+incorrect.
+
+### A note on style
+
+The shell is pretty loose about what it considers a valid redirect.
+While opinions probably differ, this author has some (strong)
+recommendations:
+
+- **Always** keep redirections "tightly grouped" -- that is, **do not**
+ include whitespace anywhere within the redirection syntax except
+ within quotes if required on the RHS (e.g. a filename that contains a
+ space). Since shells fundamentally use whitespace to delimit fields in
+ general, it is visually much clearer for each redirection to be
+ separated by whitespace, but grouped in chunks that contain no
+ unnecessary whitespace.
+
+
+
+- **Do** always put a space between each redirection, and between the
+ argument list and the first redirect.
+
+
+
+- **Always** place redirections together at the very end of a command
+ after all arguments. Never precede a command with a redirect. Never
+ put a redirect in the middle of the arguments.
+
+
+
+- **Never** use the Csh `&>foo` and `>&foo` shorthand redirects. Use the
+ long form `>foo 2>&1`. (see: [obsolete](obsolete))
+
+
+
+ # Good! This is clearly a simple commmand with two arguments and 4 redirections
+ cmd arg1 arg2 /dev/null >&2
+
+ # Good!
+ { cmd1 <<<'my input'; cmd2; } >someFile
+
+ # Bad. Is the "1" a file descriptor or an argument to cmd? (answer: it's the FD). Is the space after the herestring part of the input data? (answer: No).
+ # The redirects are also not delimited in any obvious way.
+ cmd 2>& 1 <<< stuff
+
+ # Hideously Bad. It's difficult to tell where the redirects are and whether they're even valid redirects.
+ # This is in fact one command with one argument, an assignment, and three redirects.
+ foo=barbleh
+
+# Conclusion
+
+I hope this tutorial worked for you.
+
+I lied, I did not explain `1>&3-`, go check the manual ;-)
+
+Thanks to Stéphane Chazelas from whom I stole both the intro and the
+example....
+
+The intro is inspired by this introduction, you'll find a nice exercise
+there too:
+
+- [A Detailed Introduction to I/O and I/O
+ Redirection](http://tldp.org/LDP/abs/html/ioredirintro.html)
+
+The last example comes from this post:
+
+- [comp.unix.shell: piping stdout and stderr to different
+ processes](http://groups.google.com/group/comp.unix.shell/browse_thread/thread/64206d154894a4ef/ffe4c2e382034ed9#ffe4c2e382034ed9)
+
+# See also
diff --git a/howto/start.md b/howto/start.md
new file mode 100644
index 0000000..e209824
--- /dev/null
+++ b/howto/start.md
@@ -0,0 +1 @@
+# HOWTO
diff --git a/howto/testing-your-scripts.md b/howto/testing-your-scripts.md
new file mode 100644
index 0000000..eb505ee
--- /dev/null
+++ b/howto/testing-your-scripts.md
@@ -0,0 +1,94 @@
+The one of the simplest way to check your bash/sh scripts is run it and
+check it output or run it and check the result. This tutorial shows
+how-to use [bashtest](https://github.com/pahaz/bashtest) tool for
+testing your scripts.
+
+### Write simple util
+
+We have a simple **stat.sh** script:
+
+ #!/usr/bin/env bash
+
+ if [ -z "$1" ]
+ then
+ DIR=./
+ else
+ DIR=$1
+ fi
+
+ echo "Evaluate *.py statistics"
+ FILES=$(find $DIR -name '*.py' | wc -l)
+ LINES=$((find $DIR -name '*.py' -print0 | xargs -0 cat) | wc -l)
+ echo "PYTHON FILES: $FILES"
+ echo "PYTHON LINES: $LINES"
+
+This script evaluate the number of python files and the number of python
+code lines in the files. We can use it like **./stat.sh \**
+
+### Create testsuit
+
+Then make test suits for **stat.sh**. We make a directory **testsuit**
+which contain test python files.
+
+**testsuit/main.py**
+
+ import foo
+ print(foo)
+
+**testsuit/foo.py**
+
+ BAR = 1
+ BUZ = BAR + 2
+
+Ok! Our test suit is ready! We have 2 python files which contains 4
+lines of code.
+
+### Write bashtests
+
+Lets write tests. We just write a shell command for testing our work.
+
+Create file **tests.bashtest**:
+
+ $ ./stat.sh testsuit/
+ Evaluate *.py statistics
+ PYTHON FILES: 2
+ PYTHON LINES: 4
+
+This is our test! This is simple. Try to run it.
+
+ # install bashtest if required!
+ $ pip install bashtest
+
+ # run tests
+ $ bashtest *.bashtest
+ 1 items passed all tests:
+ 1 tests in tests.bashtest
+ 1 tests in 1 items.
+ 1 passed and 0 failed.
+ Test passed.
+
+Thats all. We wrote one test. You can write more tests if you want.
+
+ $ ls testsuit/
+ foo.py main.py
+
+ $ ./stat.sh testsuit/
+ Evaluate *.py statistics
+ PYTHON FILES: 2
+ PYTHON LINES: 4
+
+And run tests again:
+
+ $ bashtest *.bashtest
+ 1 items passed all tests:
+ 2 tests in tests.bashtest
+ 2 tests in 1 items.
+ 2 passed and 0 failed.
+ Test passed.
+
+You can find more **.bashtest** examples in the [bashtest github
+repo](https://github.com/pahaz/bashtest). You can also write your
+question or report a bug
+[here](https://github.com/pahaz/bashtest/issues).
+
+Happy testing!
diff --git a/internals/shell_options.md b/internals/shell_options.md
new file mode 100644
index 0000000..215b1aa
--- /dev/null
+++ b/internals/shell_options.md
@@ -0,0 +1,509 @@
+# List of shell options
+
+![](keywords>bash shell scripting options runtime variable behaviour)
+
+This information was taken from a Bash version "`4.1`", every now and
+then new options are added, so likely, this list isn't complete.
+
+The shell-options can be set with the [shopt builtin
+command](/commands/builtin/shopt).
+
+## Shell options
+
+### autocd
+
+| Option: | `autocd` | Since: | 4.0-alpha |
+|:------------|:-----------------|:---------|:----------|
+| Shell mode: | interactive only | Default: | off |
+
+If set, a command name that is the name of a directory is executed as if
+it were the argument to the cd command.
+
+### assoc_expand_once
+
+| Option: | `assoc_expand_once` | Since: | 5.0-alpha |
+|:------------|:--------------------|:---------|:----------|
+| Shell mode: | all | Default: | off |
+
+If set, Bash attempts to expand associative array options only once.
+
+### cdable_vars
+
+| Option: | `cdable_vars` | Since: | unknown |
+|:------------|:--------------|:---------|:--------|
+| Shell mode: | all | Default: | off |
+
+Treat every **non-directory argument** to the `cd`-command as variable
+name containing a directory to `cd` into.
+
+### cdspell
+
+| Option: | `cdspell` | Since: | unknown |
+|:------------|:-----------------|:---------|:--------|
+| Shell mode: | interactive only | Default: | off |
+
+If set, minor errors in the spelling of a directory component in a cd
+command will be corrected. The errors checked for are transposed
+characters, a missing character, and one character too many. If a
+correction is found, the corrected file name is printed, and the command
+proceeds.
+
+### checkhash
+
+| Option: | `checkhash` | Since: | unknown |
+|:------------|:------------|:---------|:--------|
+| Shell mode: | all | Default: | off |
+
+If set, Bash checks that a command found in the hash table exists before
+trying to execute it. If a hashed command no longer exists, a normal
+path search is performed.
+
+### checkjobs
+
+| Option: | `checkjobs` | Since: | 4.0-alpha |
+|:------------|:-----------------|:---------|:----------|
+| Shell mode: | interactive only | Default: | off |
+
+If set, Bash lists the status of any stopped and running jobs before
+exiting an interactive shell. If any jobs are running, this causes the
+exit to be deferred until a second exit is attempted without an
+intervening command. The shell always postpones exiting if any jobs are
+stopped.
+
+### checkwinsize
+
+| Option: | `checkwinsize` | Since: | unknown |
+|:------------|:---------------|:---------|:--------|
+| Shell mode: | all | Default: | on |
+
+If set, Bash checks the window size after each command and, if
+necessary, updates the values of the variables
+[LINES](/syntax/shellvars#LINES) and
+[COLUMNS](/syntax/shellvars#COLUMNS).
+
+### cmdhist
+
+| Option: | `cmdhist` | Since: | unknown |
+|:------------|:----------|:---------|:--------|
+| Shell mode: | all | Default: | off |
+
+If set, Bash attempts to save all lines of a multiple-line command in
+the same history entry. This allows easy re-editing of multi-line
+commands.
+
+### compat31
+
+| Option: | `compat31` | Since: | 3.2 |
+|:------------|:-----------|:---------|:----|
+| Shell mode: | all | Default: | off |
+
+Compatiblity mode for Bash 3.1
+
+### compat32
+
+| Option: | `compat32` | Since: | 4.0 |
+|:------------|:-----------|:---------|:----|
+| Shell mode: | all | Default: | off |
+
+Compatiblity mode for Bash 3.2
+
+### compat40
+
+| Option: | `compat40` | Since: | 4.1-beta |
+|:------------|:-----------|:---------|:---------|
+| Shell mode: | all | Default: | off |
+
+Compatiblity mode for Bash 4.0
+
+### compat41
+
+| Option: | `compat41` | Since: | 4.2-alpha |
+|:------------|:-----------|:---------|:----------|
+| Shell mode: | all | Default: | off |
+
+Compatiblity mode for Bash 4.1
+
+### compat42
+
+| Option: | `compat42` | Since: | 4.3-alpha |
+|:------------|:-----------|:---------|:----------|
+| Shell mode: | all | Default: | off |
+
+Compatiblity mode for Bash 4.2
+
+### compat43
+
+| Option: | `compat43` | Since: | 4.4-alpha |
+|:------------|:-----------|:---------|:----------|
+| Shell mode: | all | Default: | off |
+
+Compatiblity mode for Bash 4.3
+
+### compat44
+
+| Option: | `compat44` | Since: | 5.0-alpha |
+|:------------|:-----------|:---------|:----------|
+| Shell mode: | all | Default: | off |
+
+Compatiblity mode for Bash 4.4
+
+### direxpand
+
+| Option: | `direxpand` | Since: | 4.3-alpha |
+|:------------|:------------|:---------|:-----------------------------------------------------------------------|
+| Shell mode: | all | Default: | off (unless changed on compile-time with `--enable-direxpand-default`) |
+
+If set, bash replaces directory names with the results of word expansion
+when performing filename completion. This changes the contents of the
+readline editing buffer. If not set, bash attempts to preserve what the
+user typed.
+
+### dirspell
+
+| Option: | `dirspell` | Since: | 4.0-alpha |
+|:------------|:-----------|:---------|:----------|
+| Shell mode: | all | Default: | off |
+
+If set, Bash will perform spelling corrections on directory names to
+match a glob.
+
+### dotglob
+
+| Option: | `dotglob` | Since: | unknown |
+|:------------|:----------|:---------|:--------|
+| Shell mode: | all | Default: | off |
+
+If set, Bash includes filenames beginning with a `.` (dot) in the
+results of [pathname expansion](/syntax/expansion/globs).
+
+### execfail
+
+| Option: | `execfail` | Since: | unknown |
+|:------------|:----------------|:---------|:--------|
+| Shell mode: | non-interactive | Default: | off |
+
+If set, a non-interactive shell will not exit if it cannot execute the
+file specified as an argument to the `exec`-builtin command. An
+interactive shell does not exit if `exec` fails.
+
+### expand_aliases
+
+| Option: | `expand_aliases` | Since: | unknown |
+|:------------|:-----------------|:---------|:----------------------------------------|
+| Shell mode: | all | Default: | on (interactive), off (non-interactive) |
+
+If set, aliases are expanded. This option is enabled by default for
+interactive shells.
+
+### extdebug
+
+| Option: | `extdebug` | Since: | 3.0-alpha |
+|:------------|:-----------|:---------|:----------|
+| Shell mode: | all | Default: | off |
+
+If set, behavior intended for use by debuggers is enabled.
+
+### extglob
+
+| Option: | `extglob` | Since: | 2.02-alpha1 |
+|:------------|:----------|:---------|:------------|
+| Shell mode: | all | Default: | off |
+
+If set, the extended [pattern matching](/syntax/pattern) features are
+enabled. See the important note below under [Parser
+configurations](#parser_configurations).
+
+### extquote
+
+| Option: | `extquote` | Since: | 3.0-alpha (?) |
+|:------------|:-----------|:---------|:--------------|
+| Shell mode: | all | Default: | on |
+
+If set, `$'string'` and `$"string"` quoting is performed within
+[parameter expansions](/syntax/pe) enclosed in double quotes. See the
+important note below under [Parser
+configurations](#parser_configurations).
+
+### failglob
+
+| Option: | `failglob` | Since: | 3.0-alpha |
+|:------------|:-----------|:---------|:----------|
+| Shell mode: | all | Default: | off |
+
+If set, patterns which fail to match filenames during pathname expansion
+result in an error message.
+
+### force_fignore
+
+| Option: | `force_fignore` | Since: | 3.0-alpha |
+|:------------|:----------------|:---------|:----------|
+| Shell mode: | interactive | Default: | on |
+
+If set, the suffixes specified by the
+[FIGNORE](/syntax/shellvars#FIGNORE) shell variable cause words to be
+ignored when performing word completion even if the ignored words are
+the only possible completions. This option is enabled by default.
+
+### globasciiranges
+
+| Option: | `globasciiranges` | Since: | 4.3-alpha |
+|:------------|:------------------|:---------|:----------------------------------|
+| Shell mode: | all | Default: | on (configurable at compile time) |
+
+If set, range expressions used in pattern matching behave as if in the
+traditional C locale when performing comparisons. That is, the current
+locale's collating sequence is not taken into account, so b will not
+collate between A and B, and upper-case and lower-case ASCII characters
+will collate together.
+
+### globstar
+
+| Option: | `globstar` | Since: | 4.0-alpha |
+|:------------|:-----------|:---------|:----------|
+| Shell mode: | all | Default: | off |
+
+If set, recursive globbing with `**` is enabled.
+
+### gnu_errfmt
+
+| Option: | `gnu_errfmt` | Since: | 3.0-alpha |
+|:------------|:-------------|:---------|:----------|
+| Shell mode: | all | Default: | off |
+
+If set, shell error messages are written in the "standard GNU error
+message format".
+
+### histappend
+
+| Option: | `histappend` | Since: | unknown |
+|:------------|:----------------|:---------|:--------|
+| Shell mode: | interactive (?) | Default: | off |
+
+If set, the history list is appended to the file named by the value of
+the [HISTFILE](/syntax/shellvars#HISTFILE) variable when the shell
+exits, rather than overwriting the file.
+
+### histreedit
+
+| Option: | `histreedit` | Since: | unknown |
+|:------------|:----------------|:---------|:--------|
+| Shell mode: | interactive (?) | Default: | off |
+
+If set, and readline is being used, a user is given the opportunity to
+re-edit a failed history substitution.
+
+### histverify
+
+| Option: | `histverify` | Since: | unknown |
+|:------------|:----------------|:---------|:--------|
+| Shell mode: | interactive (?) | Default: | off |
+
+Allow to review a history substitution result by loading the resulting
+line into the editing buffer, rather than directly executing it.
+
+### hostcomplete
+
+| Option: | `hostcomplete` | Since: | 2.0-alpha3 |
+|:------------|:----------------|:---------|:-----------|
+| Shell mode: | interactive (?) | Default: | on |
+
+If set, Bash completion also completes hostnames. On by default.
+
+### huponexit
+
+| Option: | `huponexit` | Since: | 2.02-alpha1 |
+|:------------|:------------------|:---------|:------------|
+| Shell mode: | interactive login | Default: | off |
+
+If set, Bash will send the `SIGHUP` signal to all jobs when an
+interactive login shell exits.
+
+### interactive_comments
+
+| Option: | `interactive_comments` | Since: | unknown |
+|:------------|:-----------------------|:---------|:--------|
+| Shell mode: | interactive | Default: | on |
+
+Allow [commenting](/scripting/basics#comments) in interactive shells, on
+by default.
+
+### lastpipe
+
+| Option: | `lastpipe` | Since: | 4.2-alpha |
+|:------------|:-----------|:---------|:----------|
+| Shell mode: | all | Default: | off |
+
+If set, **and job control is not active**, the shell runs the last
+command of a pipeline not executed in the background in the current
+shell environment.
+
+### lithist
+
+| Option: | `lithist` | Since: | unknown |
+|:------------|:------------|:---------|:--------|
+| Shell mode: | interactive | Default: | off |
+
+If set, and the [\#cmdhist](#cmdhist) option is enabled, multi-line
+commands are saved to the history with embedded newlines rather than
+using semicolon separators where possible.
+
+### localvar_inherit
+
+| Option: | `localvar_inherit` | Since: | 5.0-alpha |
+|:------------|:-------------------|:---------|:----------|
+| Shell mode: | all | Default: | off |
+
+If this option is set, a local variable inherits the value of a variable
+with the same name at the nearest preceding scope.
+
+### login_shell
+
+| Option: | `login_shell` | Since: | 2.05a-alpha1 |
+|:------------|:--------------|:---------|:-------------|
+| Shell mode: | all | Default: | n/a |
+
+The option is set when Bash is a login shell. This is a readonly option.
+
+### mailwarn
+
+| Option: | `mailwarn` | Since: | unknown |
+|:------------|:----------------|:---------|:--------|
+| Shell mode: | interactive (?) | Default: | off |
+
+If set, and a file that Bash is checking for mail has been accessed
+since the last time it was checked, the message "The mail in mailfile
+has been read" is displayed.
+
+### no_empty_cmd_completion
+
+| Option: | `mailwarn` | Since: | unknown |
+|:------------|:----------------|:---------|:--------|
+| Shell mode: | interactive (?) | Default: | off |
+
+If set, and readline is being used, Bash will not attempt to search the
+PATH for possible completions when completion is attempted on an empty
+line.
+
+### nocaseglob
+
+| Option: | `nocaseglob` | Since: | 2.02-alpha1 |
+|:------------|:-------------|:---------|:------------|
+| Shell mode: | all | Default: | off |
+
+If set, Bash matches filenames in a case-insensitive fashion when
+performing pathname expansion.
+
+### nocasematch
+
+| Option: | `nocasematch` | Since: | 3.1-alpha1 |
+|:------------|:--------------|:---------|:-----------|
+| Shell mode: | all | Default: | off |
+
+If set, Bash matches patterns in a case-insensitive fashion when
+performing matching while executing `case` or `[[` conditional commands.
+
+### nullglob
+
+| Option: | `nullglob` | Since: | unknown |
+|:------------|:-----------|:---------|:--------|
+| Shell mode: | all | Default: | off |
+
+If set, Bash allows patterns which match no files to expand to a null
+string, rather than themselves.
+
+### progcomp
+
+| Option: | `progcomp` | Since: | 2.04-alpha1 |
+|:------------|:----------------|:---------|:------------|
+| Shell mode: | interactive (?) | Default: | on |
+
+If set, the programmable completion facilities are enabled. This option
+is enabled by default.
+
+### promptvars
+
+| Option: | `promptvars` | Since: | unknown |
+|:------------|:-------------|:---------|:--------|
+| Shell mode: | interactive | Default: | on |
+
+If set, prompt strings undergo parameter expansion, command
+substitution, arithmetic expansion, and quote removal after being
+expanded using the prompt special sequences. This option is enabled by
+default.
+
+### restricted_shell
+
+| Option: | `restricted_shell` | Since: | 2.03-alpha |
+|:------------|:-------------------|:---------|:-----------|
+| Shell mode: | interactive (?) | Default: | off |
+
+The option is set when Bash is a restricted shell. This is a readonly
+option.
+
+### shift_verbose
+
+| Option: | `shift_verbose` | Since: | unknown |
+|:------------|:----------------|:---------|:----------------------|
+| Shell mode: | all | Default: | off, on in POSIX mode |
+
+If set, the shift builtin prints an error message when the shift count
+exceeds the number of positional parameters.
+
+### sourcepath
+
+| Option: | `sourcepath` | Since: | unknown |
+|:------------|:-------------|:---------|:--------|
+| Shell mode: | all | Default: | on |
+
+If set, the source builtin command uses the value of PATH to find the
+directory containing the file supplied as an argument. This option is
+enabled by default.
+
+### syslog_history
+
+| Option: | `syslog_history` | Since: | 5.0-alpha |
+|:------------|:-----------------|:---------|:----------|
+| Shell mode: | unknown | Default: | off |
+
+If set, the shell history is sent to syslog.
+
+This option is undocumented and available only if the shell supports
+syslog.
+
+### xpg_echo
+
+| Option: | `xpg_echo` | Since: | 2.04-beta1 |
+|:------------|:-----------|:---------|:-----------|
+| Shell mode: | all | Default: | off |
+
+If set, the `echo`-builtin command expands backslash-escape sequences by
+default (POSIX, SUS, XPG).
+
+## Parser configurations
+
+Parser configurations change the way the Bash parser recognizes the
+syntax when parsing a line. This, of course, is impossible for a line
+that already was parsed.
+
+There are two options that influence the parsing this way:
+
+- `extglob`
+- `extquote`
+
+Consequence: You **can't** use the new syntax (e.g. the extended
+globbing syntax) and the command to enable it **in the same line**.
+
+ $ shopt -s extglob; echo !(*.txt) # this is the WRONG way!
+ -bash: syntax error near unexpected token `('
+
+You have to configure the parser **before** a line with new syntax is
+parsed:
+
+ $ shopt -s extglob # standalone - CORRECT way!
+ $ echo !(*.txt)
+ ...
+
+## See also
+
+- Internal: [shopt builtin command](/commands/builtin/shopt)
+- Internal: [set builtin command](/commands/builtin/set)
diff --git a/misc/bashphorisms.md b/misc/bashphorisms.md
new file mode 100644
index 0000000..ea4a78a
--- /dev/null
+++ b/misc/bashphorisms.md
@@ -0,0 +1,52 @@
+# The Bashphorisms
+
+Bashphorisms are aphorisms for the IRC channel `#bash` on Freenode. Keep
+in mind that this version is a snapshot, the bashphorisms are changed
+here and there. Also, [another
+snapshot](http://mywiki.wooledge.org/BashFAQ/064).
+
+I think `greycat` was the first one who had the idea, but I'm not sure.
+
+Our bashphorisms can be queried from `greybot` using `!bN`, where `N` is
+the bashphorism number.
+
+And yes, these bashphorisms reflect the daily reality in `#bash`.
+
+| Number | Bashphorism |
+|:-------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| 0 | The questioner will never tell you what they are really doing the first time they ask. |
+| 1 | The questioner's first description of the problem/question will be misleading. |
+| 2 | The questioner will keep changing the question until it drives the helpers in the channel insane. |
+| 3 | Offtopicness will continue until someone asks a bash question that falls under bashphorisms 1 and/or 2, and `greycat` gets pissed off. |
+| 4 | The questioner will not read and apply the answers he is given but will instead continue to practice bashphorism \#1 and bashphorism \#2. |
+| 5 | The ignorant will continually mis-educate the other noobies. |
+| 6 | When given a choice of solutions, the newbie will always choose the wrong one. |
+| 7 | The newbie will always find a reason to say, "It doesn't work." |
+| 8 | If you don't know to whom the bashphorism's referring, it's you. |
+| 9 | All examples given by the questioner will be broken, misleading, wrong, and not representative of the actual question. |
+| 10 | See B1 |
+| 11 | Please apply `(( % 10 ))` to the bashphorism value. |
+| 12 | All logic is deniable; however, some logic will \*plonk\* you if you deny it. |
+| 13 | Everyone ignores greycat when he is right. When he is wrong, it is !b1 |
+| 14 | The newbie doesn't actually know what he's asking. If he did, he wouldn't need to ask. |
+| 15 | The more advanced you are, the more likely you are to be overcomplicating it. |
+| 16 | The more beginner you are, the more likely you are to be overcomplicating it. |
+| 17 | A newbie comes to \#bash to get his script confirmed. He leaves disappointed. |
+| 18 | The newbie will not accept the answer you give, no matter how right it is. |
+| 19 | The newbie is a bloody loon. |
+| 20 | The newbie will always have some excuse for doing it wrong. |
+| 21 | When the newbie's question is ambiguous, the proper interpretation will be whichever one makes the problem the hardest to solve. |
+| 22 | The newcomer will abuse the bot's factoid triggers for their own entertainment until someone gets annoyed enough to ask them to message it privately instead. |
+| 23 | Everyone is a newcomer. |
+| 24 | The newcomer will address greybot as if it were human. |
+| 25 | The newbie won't accept any answer that uses practical or standard tools. |
+| 26 | The newbie will not TELL you about this restriction until you have wasted half an hour. |
+| 27 | The newbie will lie. |
+| 28 | When the full horror of the newbie's true goal is revealed, the newbie will try to restate the goal to trick you into answering. Newbies are stupid. |
+| 29 | It's always git. Or python virtualenv. Or docker. One of those pieces of shit. ALWAYS. |
+| 30 | They won't show you the homework assignment. That would make it too easy. |
+| 31 | Your teacher is a f\*\*king idiot. |
+| 32 | The more horrifyingly wrong a proposed solution is, the more likely it will be used. |
+| 33 | The newbie cannot explain what he is doing, or why. He will show you incomprehensible, nonworking code instead. What? You can't read his mind?! |
+
+Please feel free to correct or extend this page whenever needed.
diff --git a/misc/readthesourceluke.md b/misc/readthesourceluke.md
new file mode 100644
index 0000000..fdac977
--- /dev/null
+++ b/misc/readthesourceluke.md
@@ -0,0 +1,68 @@
+Comments extracted from the bash source and therefore Copyright (C)
+1987-2004 Free Software Foundation, Inc. under the terms of the GNU
+General Public License etc..
+
+from `mailcheck.c`:
+
+``` C
+/* check_mail () is useful for more than just checking mail. Since it has
+ the paranoids dream ability of telling you when someone has read your
+ mail, it can just as easily be used to tell you when someones .profile
+ file has been read, thus letting one know when someone else has logged
+ in. Pretty good, huh? */
+```
+
+From `builtins/read.def`:
+
+``` C
+ /* If there are no variables, save the text of the line read to the
+ variable $REPLY. ksh93 strips leading and trailing IFS whitespace,
+ so that `read x ; echo "$x"' and `read ; echo "$REPLY"' behave the
+ same way, but I believe that the difference in behaviors is useful
+ enough to not do it. Without the bash behavior, there is no way
+ to read a line completely without interpretation or modification
+ unless you mess with $IFS (e.g., setting it to the empty string).
+ If you disagree, change the occurrences of `#if 0' to `#if 1' below. */
+```
+
+from `variables.c`:
+
+``` C
+ /*
+ * 24 October 2001
+ *
+ * I'm tired of the arguing and bug reports. Bash now leaves SSH_CLIENT
+ * and SSH2_CLIENT alone. I'm going to rely on the shell_level check in
+ * isnetconn() to avoid running the startup files more often than wanted.
+ * That will, of course, only work if the user's login shell is bash, so
+ * I've made that behavior conditional on SSH_SOURCE_BASHRC being defined
+ * in config-top.h.
+ */
+```
+
+From `shell.h`:
+
+``` C
+/* Values that can be returned by execute_command (). */
+#define EXECUTION_FAILURE 1
+#define EXECUTION_SUCCESS 0
+
+/* Usage messages by builtins result in a return status of 2. */
+#define EX_BADUSAGE 2
+
+/* Special exit statuses used by the shell, internally and externally. */
+#define EX_RETRYFAIL 124
+#define EX_WEXPCOMSUB 125
+#define EX_BINARY_FILE 126
+#define EX_NOEXEC 126
+#define EX_NOINPUT 126
+#define EX_NOTFOUND 127
+
+#define EX_SHERRBASE 256 /* all special error values are > this. */
+
+#define EX_BADSYNTAX 257 /* shell syntax error */
+#define EX_USAGE 258 /* syntax error in usage */
+#define EX_REDIRFAIL 259 /* redirection failed */
+#define EX_BADASSIGN 260 /* variable assignment error */
+#define EX_EXPFAIL 261 /* word expansion failed */
+```
diff --git a/misc/shell_humor.md b/misc/shell_humor.md
new file mode 100644
index 0000000..a83587d
--- /dev/null
+++ b/misc/shell_humor.md
@@ -0,0 +1,35 @@
+# Shell Humor
+
+Nothing special, just my private collection of some more or less funny
+shell stuff I saw during the years.
+
+Usually Bash and/or Linux (GNU Toolset) specific.
+
+ $ %blow
+ -bash: fg: %blow: no such job
+
+ $ ar m god
+ ar: creating god
+
+ $ touch /pussy
+ touch: cannot touch `/pussy': Permission denied
+
+ $ mount; fsck; fsck; fsck; umount; sleep
+
+ # the lover variant
+ $ unzip; strip; touch; finger; grep; mount; fsck; more; yes; fsck; fsck; umount; sleep
+
+ # it's not directly funny, only because it's not obvious that this is an sed command
+ # for the <<<, it works only in Bash
+ $ sed streetlight <<< reeter
+ lighter
+
+ # see above for comments
+ $ sed statement <<< cat
+ cement
+
+ $ \(-
+ bash: (-: command not found
+
+ $ echo '[q]sa[ln0=aln256%Pln256/snlbx]sb3135071790101768542287578439snlbxq'|dc
+ GET A LIFE!
diff --git a/scripting/bashbehaviour.md b/scripting/bashbehaviour.md
new file mode 100644
index 0000000..1e819ff
--- /dev/null
+++ b/scripting/bashbehaviour.md
@@ -0,0 +1,360 @@
+# Bash's behaviour
+
+![](keywords>bash shell scripting startup files dotfiles modes POSIX)
+
+FIXME incomplete
+
+## Bash startup modes
+
+### Login shell
+
+As a "login shell", Bash reads and sets (executes) the user's profile
+from `/etc/profile` and one of `~/.bash_profile`, `~/.bash_login`, or
+`~/.profile` (in that order, using the first one that's readable!).
+
+When a login shell exits, Bash reads and executes commands from the file
+`~/.bash_logout`, if it exists.
+
+Why an extra login shell mode? There are many actions and variable sets
+that only make sense for the initial user login. That's why all UNIX(r)
+shells have (should have) a "login" mode.
+
+**Methods to start Bash as a login shell:**
+
+- the first character of `argv[0]` is `-` (a hyphen): traditional
+ UNIX(r) shells start from the `login` binary
+- Bash is started with the `-l` option
+- Bash is started with the `--login` option
+
+**Methods to test for login shell mode:**
+
+- the shell option `login_shell` is set
+
+**Related switches:**
+
+- `--noprofile` disables reading of all profile files
+
+### Interactive shell
+
+When Bash starts as an interactive non-login shell, it reads and
+executes commands from `~/.bashrc`. This file should contain, for
+example, aliases, since they need to be defined in every shell as
+they're not inherited from the parent shell.
+
+The feature to have a system-wide `/etc/bash.bashrc` or a similar
+system-wide rc-file is specific to vendors and distributors that ship
+*their own, patched variant of Bash*. The classic way to have a
+system-wide rc file is to `source /etc/bashrc` from every user's
+`~/.bashrc`.
+
+**Methods to test for interactive-shell mode:**
+
+- the special parameter `$-` contains the letter `i` (lowercase I)
+
+**Related switches:**
+
+- `-i` forces the interactive mode
+- `--norc` disables reading of the startup files (e.g.
+ `/etc/bash.bashrc` if supported) and `~/.bashrc`
+- `--rcfile` defines another startup file (instead of `/etc/bash.bashrc`
+ and `~/.bashrc`)
+
+### SH mode
+
+When Bash starts in SH compatiblity mode, it tries to mimic the startup
+behaviour of historical versions of `sh` as closely as possible, while
+conforming to the POSIX(r) standard as well. The profile files read are
+`/etc/profile` and `~/.profile`, if it's a login shell.
+
+If it's not a login shell, the environment variable
+[ENV](/syntax/shellvars#ENV) is evaluated and the resulting filename is
+used as the name of the startup file.
+
+After the startup files are read, Bash enters the [POSIX(r) compatiblity
+mode (for running, not for starting!)](#posix_run_mode).
+
+**Bash starts in `sh` compatiblity mode when:**
+
+- the base filename in `argv[0]` is `sh` (:!: NB: `/bin/sh` may be
+ linked to `/bin/bash`, but that doesn't mean it acts like `/bin/bash`
+ :!:)
+
+### POSIX mode
+
+When Bash is started in POSIX(r) mode, it follows the POSIX(r) standard
+for startup files. In this mode, **interactive shells** expand the
+[ENV](/syntax/shellvars#ENV) variable and commands are read and executed
+from the file whose name is the expanded value.
+No other startup files are read. Hence, a non-interactive shell doesn't
+read any startup files in POSIX(r) mode.
+
+**Bash starts in POSIX(r) mode when:**
+
+- the commandline option `--posix` is specified
+- the environment variable
+ [POSIXLY_CORRECT](/syntax/shellvars#POSIXLY_CORRECT) is set
+
+### Quick startup file reference
+
+- Eventual system-wide rc-files are usually read when `~/.bashrc` would
+ be read (at least Debian GNU/Linux behaves like that)
+- Regardless of the system-wide files in `/etc` which are always read,
+ Bash usually reads the first file found, when multiple choices are
+ given (for user files in `~/`)
+
+| Mode | `/etc/profile` | `~/.bash_profile` | `~/.bash_login` | `~/.profile` | `~/.bashrc` | `${ENV}` |
+|-----------------------|----------------|-------------------|-----------------|--------------|-------------|----------|
+| Login shell | X | X | X | X | \- | \- |
+| Interactive shell | \- | \- | \- | \- | X | \- |
+| SH compatible login | X | \- | \- | X | \- | \- |
+| SH compatible | \- | \- | \- | \- | \- | X |
+| POSIX(r) compatiblity | \- | \- | \- | \- | \- | X |
+
+## Bash run modes
+
+### Normal Bash
+
+### POSIX run mode
+
+In POSIX(r) mode, Bash follows the POSIX(r) standard regarding behaviour
+and parsing (excerpt from a Bash maintainer's document):
+
+ Starting Bash with the `--posix' command-line option or executing `set
+ -o posix' while Bash is running will cause Bash to conform more closely
+ to the POSIX standard by changing the behavior to match that specified
+ by POSIX in areas where the Bash default differs.
+
+ When invoked as `sh', Bash enters POSIX mode after reading the startup
+ files.
+
+ The following lists what's changed when Bash is in `POSIX mode':
+
+ 1. When a command in the hash table no longer exists, Bash will
+ re-search `$PATH' to find the new location. This is also
+ available with `shopt -s checkhash'.
+
+ 2. The message printed by the job control code and builtins when a job
+ exits with a non-zero status is `Done(status)'.
+
+ 3. The message printed by the job control code and builtins when a job
+ is stopped is `Stopped(SIGNAME)', where SIGNAME is, for example,
+ `SIGTSTP'.
+
+ 4. The `bg' builtin uses the required format to describe each job
+ placed in the background, which does not include an indication of
+ whether the job is the current or previous job.
+
+ 5. Reserved words appearing in a context where reserved words are
+ recognized do not undergo alias expansion.
+
+ 6. The POSIX `PS1' and `PS2' expansions of `!' to the history number
+ and `!!' to `!' are enabled, and parameter expansion is performed
+ on the values of `PS1' and `PS2' regardless of the setting of the
+ `promptvars' option.
+
+ 7. The POSIX startup files are executed (`$ENV') rather than the
+ normal Bash files.
+
+ 8. Tilde expansion is only performed on assignments preceding a
+ command name, rather than on all assignment statements on the line.
+
+ 9. The default history file is `~/.sh_history' (this is the default
+ value of `$HISTFILE').
+
+ 10. The output of `kill -l' prints all the signal names on a single
+ line, separated by spaces, without the `SIG' prefix.
+
+ 11. The `kill' builtin does not accept signal names with a `SIG'
+ prefix.
+
+ 12. Non-interactive shells exit if FILENAME in `.' FILENAME is not
+ found.
+
+ 13. Non-interactive shells exit if a syntax error in an arithmetic
+ expansion results in an invalid expression.
+
+ 14. Redirection operators do not perform filename expansion on the word
+ in the redirection unless the shell is interactive.
+
+ 15. Redirection operators do not perform word splitting on the word in
+ the redirection.
+
+ 16. Function names must be valid shell names. That is, they may not
+ contain characters other than letters, digits, and underscores, and
+ may not start with a digit. Declaring a function with an invalid
+ name causes a fatal syntax error in non-interactive shells.
+
+ 17. POSIX special builtins are found before shell functions during
+ command lookup.
+
+ 18. If a POSIX special builtin returns an error status, a
+ non-interactive shell exits. The fatal errors are those listed in
+ the POSIX standard, and include things like passing incorrect
+ options, redirection errors, variable assignment errors for
+ assignments preceding the command name, etc.
+
+ 19. If `CDPATH' is set, the `cd' builtin will not implicitly append
+ the current directory to it. This means that `cd' will fail if no
+ valid directory name can be constructed from any of the entries in
+ `$CDPATH', even if the a directory with the same name as the name
+ given as an argument to `cd' exists in the current directory.
+
+ 20. A non-interactive shell exits with an error status if a variable
+ assignment error occurs when no command name follows the assignment
+ statements. A variable assignment error occurs, for example, when
+ trying to assign a value to a readonly variable.
+
+ 21. A non-interactive shell exits with an error status if the iteration
+ variable in a `for' statement or the selection variable in a
+ `select' statement is a readonly variable.
+
+ 22. Process substitution is not available.
+
+ 23. Assignment statements preceding POSIX special builtins persist in
+ the shell environment after the builtin completes.
+
+ 24. Assignment statements preceding shell function calls persist in the
+ shell environment after the function returns, as if a POSIX
+ special builtin command had been executed.
+
+ 25. The `export' and `readonly' builtin commands display their output
+ in the format required by POSIX.
+
+ 26. The `trap' builtin displays signal names without the leading `SIG'.
+
+ 27. The `trap' builtin doesn't check the first argument for a possible
+ signal specification and revert the signal handling to the original
+ disposition if it is, unless that argument consists solely of
+ digits and is a valid signal number. If users want to reset the
+ handler for a given signal to the original disposition, they
+ should use `-' as the first argument.
+
+ 28. The `.' and `source' builtins do not search the current directory
+ for the filename argument if it is not found by searching `PATH'.
+
+ 29. Subshells spawned to execute command substitutions inherit the
+ value of the `-e' option from the parent shell. When not in POSIX
+ mode, Bash clears the `-e' option in such subshells.
+
+ 30. Alias expansion is always enabled, even in non-interactive shells.
+
+ 31. When the `alias' builtin displays alias definitions, it does not
+ display them with a leading `alias ' unless the `-p' option is
+ supplied.
+
+ 32. When the `set' builtin is invoked without options, it does not
+ display shell function names and definitions.
+
+ 33. When the `set' builtin is invoked without options, it displays
+ variable values without quotes, unless they contain shell
+ metacharacters, even if the result contains nonprinting characters.
+
+ 34. When the `cd' builtin is invoked in LOGICAL mode, and the pathname
+ constructed from `$PWD' and the directory name supplied as an
+ argument does not refer to an existing directory, `cd' will fail
+ instead of falling back to PHYSICAL mode.
+
+ 35. When the `pwd' builtin is supplied the `-P' option, it resets
+ `$PWD' to a pathname containing no symlinks.
+
+ 36. The `pwd' builtin verifies that the value it prints is the same as
+ the current directory, even if it is not asked to check the file
+ system with the `-P' option.
+
+ 37. When listing the history, the `fc' builtin does not include an
+ indication of whether or not a history entry has been modified.
+
+ 38. The default editor used by `fc' is `ed'.
+
+ 39. The `type' and `command' builtins will not report a non-executable
+ file as having been found, though the shell will attempt to
+ execute such a file if it is the only so-named file found in
+ `$PATH'.
+
+ 40. The `vi' editing mode will invoke the `vi' editor directly when
+ the `v' command is run, instead of checking `$FCEDIT' and
+ `$EDITOR'.
+
+ 41. When the `xpg_echo' option is enabled, Bash does not attempt to
+ interpret any arguments to `echo' as options. Each argument is
+ displayed, after escape characters are converted.
+
+
+ There is other POSIX behavior that Bash does not implement by default
+ even when in POSIX mode. Specifically:
+
+ 1. The `fc' builtin checks `$EDITOR' as a program to edit history
+ entries if `FCEDIT' is unset, rather than defaulting directly to
+ `ed'. `fc' uses `ed' if `EDITOR' is unset.
+
+ 2. As noted above, Bash requires the `xpg_echo' option to be enabled
+ for the `echo' builtin to be fully conformant.
+
+
+ Bash can be configured to be POSIX-conformant by default, by specifying
+ the `--enable-strict-posix-default' to `configure' when building.
+
+FIXME help me to find out what breaks in POSIX(r) mode!
+
+**The POSIX(r) mode can be switched on by:**
+
+- Bash starting as `sh` (the basename of `argv[0]` is `sh`)
+- starting Bash with the commandline option `--posix`
+- on startup, the environment variable
+ [POSIXLY_CORRECT](/syntax/shellvars#POSIXLY_CORRECT) is set
+- the command `set -o posix`
+
+**Tests for the POSIX(r) mode:**
+
+- the variable [SHELLOPTS](/syntax/shellvars#SHELLOPTS) contains `posix`
+ in its list
+
+### Restricted shell
+
+In restricted mode, Bash sets up (and runs) a shell environment that's
+far more controlled and limited than the standard shell mode. It acts
+like normal Bash with the following restrictions:
+
+- the `cd` command can't be used to change directories
+- the variables [SHELL](/syntax/shellvars#SHELL),
+ [PATH](/syntax/shellvars#PATH), [ENV](/syntax/shellvars#ENV) and
+ [BASH_ENV](/syntax/shellvars#BASH_ENV) can't be set or unset
+- command names that contain a `/` (slash) can't be called (hence you're
+ limited to `PATH`)
+- filenames containing a `/` (slash) can't be specified as argument to
+ the `source` or `.` builtin command
+- filenames containing a `/` (slash) can't be specified as argument to
+ the `-p` option of the `hash` builtin command
+- function definitions are not inherited from the environment at shell
+ startup
+- the environment variable [SHELLOPTS](/syntax/shellvars#SHELLOPTS) is
+ ignored at startup
+- redirecting output using the `>`, `>|`, `<>`, `>&`, `&>`, and `>>`
+ redirection operators isn't allowed
+- the `exec` builtin command can't replace the shell with another
+ process
+- adding or deleting builtin commands with the `-f` and `-d` options to
+ the enable builtin command is forbidden
+- using the `enable` builtin command to enable disabled shell builtins
+ doesn't work
+- the `-p` option to the `command` builtin command doesn't work
+- turning off restricted mode with `set +r` or `set +o restricted` is
+ (of course) forbidden
+
+The "-r" restrictions are turned on **after** Bash has read its startup
+files.
+
+When the command that is run is a shell script, then the restrictions
+are **turned off** for the (sub-)shell that runs that shell script.
+
+**The restricted shell can be switched on by:**
+
+- calling Bash as `rbash` (the basename of `argv[0]` is `rbash`)
+- calling Bash with the `-r` option
+- calling Bash with the `--restricted` option
+
+**Tests for restricted mode:**
+
+- the special parameter `$-` contains the letter `r` (lowercase R)
+- the shell option `restricted_shell` is set and can be checked by the
+ `shopt` builtin command
diff --git a/scripting/bashchanges.md b/scripting/bashchanges.md
new file mode 100644
index 0000000..9aba355
--- /dev/null
+++ b/scripting/bashchanges.md
@@ -0,0 +1,333 @@
+# Bash changes
+
+This article is an **incomplete overview** of changes to Bash over time.
+Not all changes are listed, just the ones most likely to be useful for
+normal scripting. The overviews are categorized by topic and ordered by
+version.
+
+A useful starting point is the [NEWS
+file](https://github.com/bminor/bash/blob/master/NEWS) in bash sources.
+If you have more detailed information, or historical information about
+Bash versions earlier than V2, feel free to mail me, or use the
+discussion below.
+
+Status: 5.1 (alpha)
+
+## Shell options
+
+Note that the `shopt` builtin command first appeared in Bash 2.0.
+
+For this topic, see also
+
+- [shell_options](/internals/shell_options)
+- [set](/commands/builtin/set)
+
+| Feature or change description | Appeared in Bash version | See also/remarks |
+|:--------------------------------|:-------------------------|:--------------------------------------------------------------------------------|
+| `posix` (for `set -o`) | 1.14.0 | |
+| `hostcomplete` | 2.0-alpha3 | |
+| `expand_aliases` | 2.0 | |
+| `huponexit` | 2.02-alpha1 | |
+| `nocaseglob` | 2.02-alpha1 | |
+| `extglob` | 2.02-alpha1 | together with extended globbing, KSH88 |
+| `restricted_shell` | 2.03-alpha | |
+| `xpg_echo` | 2.04-beta1 | |
+| `progcomp` | 2.04-alpha1 | |
+| `no_empty_command_completion` | 2.04 | |
+| `login_shell` | 2.05a-alpha1 | |
+| `nolog` (for `set -o`) | 2.05a | |
+| `gnu_errfmt` | 3.0-alpha | |
+| `force_fignore` | 3.0-alpha | |
+| `failglob` | 3.0-alpha | |
+| `extquote` | 3.0-alpha | unsure -- verify! |
+| `extdebug` | 3.0-alpha | |
+| `pipefail` (for `set -o`) | 3.0 | |
+| `functrace` (for `set -o`) | 3.0 | |
+| `errtrace` (for `set -o`) | 3.0 | |
+| `nocasematch` | 3.1-alpha1 | |
+| `dirspell` | 4.0-alpha | |
+| `globstar` | 4.0-alpha | |
+| `checkjobs` | 4.0-alpha | |
+| `autocd` | 4.0-alpha | |
+| `set -e` effects more intuitive | 4.0 | not directly specified by POSIX, but in consensus with POSIX WG |
+| `compat40` | 4.1-beta | |
+| `lastpipe` | 4.2-alpha | only works with job control disabled |
+| `compat41` | 4.2-alpha | |
+| `globasciiranges` | 4.3-alpha | enable "character range globbing" to always act as if in `C` locale |
+| `compat42` | 4.3-alpha | |
+| `compat43` | 4.4-alpha | |
+| `compat44` | 5.0-alpha | |
+| `localvar_inherit` | 5.0-alpha | local variables inherit preceeding scope values if they have the same name |
+| `syslog_history` | 5.0-alpha | send history lines to syslog (undocumented, default off) if syslog is supported |
+| `assoc_expand_once` | 5.0-alpha | expand associative array subscripts only one |
+| `globasciiranges` | 5.0-beta | New default: on (default may be configured at compile time) |
+| `localvar_inherit` | 5.0-beta | guard code against inheriting from an incompatible data type |
+| `checkwinsize` | 5.0-beta2 | New default: on |
+| `shift_verbose` | 5.0-beta2 | Default on when in POSIX mode |
+
+### General (all/many builtins)
+
+| Feature or change description | Appeared in Bash version | See also/remarks |
+|:----------------------------------------------------------------------|:-------------------------|:----------------------------------------------------------------|
+| generally return 2 on usage error | 2.0 | |
+| generally accept `--` (end of options) | 2.0 | |
+| (where applicable) implement a `-p` option to produce reusable output | 2.0 | `shopt` and `umask` builtins were fixed to support that in 2.02 |
+
+### printf
+
+For this topic, see also
+
+- [printf](/commands/builtin/printf)
+
+| Feature or change description | Appeared in Bash version | See also/remarks |
+|:------------------------------------------------------------|:-------------------------|:-------------------------------------------------------|
+| new `printf` command | 2.02-alpha1 | |
+| respects `0..` and `0x..` prefixed numbers | 2.04-beta1 | consistency with arithmetic |
+| POSIX(r) length specifiers `j`, `t` and `z` | 2.05a-alpha1 | ISO C99 |
+| POSIX(r) flag `'` | 2.05a-alpha1 | |
+| conversion `a` and `A` | 2.05a-rc1 | if provided by the underlying printf(3) |
+| conversion `F` | 2.05a-rc1 | |
+| conversion `n` | 2.05a-rc1 | |
+| new option `-v` | 3.1-alpha1 | |
+| escape sequences `\"` and `\?` | 3.0-beta1 | |
+| modified option `-v` to assign to individual array elements | 4.1-alpha | |
+| conversion `(...)T` | 4.2-alpha | support stftime(3) date/time format; uses current time |
+| `\uNNNN` and `\UNNNNNNNN` escape sequences | 4.2-alpha | for: `printf`, `echo -e`, `$'...'` |
+
+### Conditional expressions and test command
+
+For this topic, see also
+
+- [conditional_expression](/syntax/ccmd/conditional_expression)
+- [classictest](/commands/classictest)
+
+| Feature or change description | Appeared in Bash version | See also/remarks |
+|:-------------------------------------------------------------------|:-------------------------|:-------------------------------------------------------------------------------------|
+| `test`: `-o`, `==`, `<` and `>` | 2.0 | |
+| `test`: `-N` | 2.02 | |
+| `[[...]]`: new | 2.02-alpha1 | KSH93 |
+| `[[...]]`: regex support (`=~`) | 3.0-alpha | |
+| `[[...]]`: quotable right-hand-side of `=~` forces string matching | 3.2-alpha | for consistency with pattern matching |
+| `[[...]]`: `<` and `>` operators respect locale | 4.1-alpha | for consistency, since 4.1-beta: ensure you have set compatiblity to \>4.0 (default) |
+| `test`/`[`/`[[`: `-v` | 4.2-alpha | check if a variable is set |
+| `test`/`[`/`[[`: `-v` | 4.2-alpha | support array syntax to check for elements |
+| `test`/`[`/`[[`: `-N` accepts nanoseconds | 5.1-alpha | |
+| `test`/`[`/`[[`: `-v` accepts positional parameters | 5.1-alpha | |
+
+### Other builtins and keywords
+
+| Builtin | Feature or change description | Appeared in Bash version | See also/remarks |
+|:----------------------|:-------------------------------------------------------------------------------------------------------------------------------------------|:-------------------------|:----------------------------------------------------------------------------------------------|
+| `bashbug` | new | 1.14.0 | |
+| `select` | new | 1.14.0 | |
+| `disown` | new | 2.0 | |
+| `shopt` | new | 2.0 | [shopt](/commands/builtin/shopt) |
+| `declare` | new options `-a` and `-F` | 2.0 | |
+| `enable` | builtin has basic plugin support (dlopen) | 2.0 | |
+| `exec` | options `-l`, `-c` and `-a` | 2.0 | |
+| `read` | options `-p`, `-e` and `-a` | 2.0 | [read](/commands/builtin/read) |
+| `readonly` | option `-a` | 2.0 | [arrays](/syntax/arrays) |
+| `time` | new keyword | 2.0 | |
+| `shopt` | `-p` (reusable output) | 2.02 | |
+| `umask` | `-p` (reusable output) | 2.02 | |
+| `complete` | new | 2.04-devel | for and together with support for programmable completion |
+| `compgen` | new | 2.04-devel | for and together with support for programmable completion |
+| `read` | options `-t`, `-n`, `-d`, `-s` | 2.04-devel | [read](/commands/builtin/read) |
+| `for ((...;...;...))` | new | 2.04-devel | KSH93 |
+| `set` | print shell functions in a format reusable as input | 2.05-beta1 | |
+| `for` | allow an empty word list | 2.05a-alpha1 | |
+| `read` | new option `-u` | 2.05b-alpha1 | [read](/commands/builtin/read) |
+| `caller` | new | 3.0 | [caller](/commands/builtin/caller) |
+| `coproc` | new | 4.0-alpha | |
+| `declare` | new options `-l` and `-u` | 4.0-alpha | together with case-changing expansion forms |
+| `case` | new action list terminators '';;& and '';& | 4.0-alpha | ksh93: only `;&`. zsh and mksh: `;|`. mksh: all 4, (`;;&` is undocumented Bash compatibility) |
+| `read` | changed `-t` (fractional seconds) | 4.0-alpha | |
+| `mapfile` | new | 4.0-alpha | |
+| `read` | new option `-i` | 4.0-alpha | |
+| `compopt` | new | 4.0-alpha | |
+| `read` | modified option `-t` to test for data | 4.0-beta | |
+| `read` | new option `-N` | 4.1-alpha | |
+| `mapfile` | changed behaviour regarding history spamming | 4.1-alpha | |
+| `declare` | new option `-g` | 4.2-alpha | |
+| `mapfile` | calls the callback with an additional argument: The line (data) | 4.2-alpha | |
+| `cd` | new option `-e` | 4.2-alpha | |
+| `echo` | `\uNNNN` and `\UNNNNNNNN` escape sequences | 4.2-alpha | for: `printf`, `echo -e`, `$'...'` |
+| `exec` | option `-a` to give a `argv[0]` string | 4.2-alpha | |
+| `time` | allowed as a command by itself to display timing values of the shell and its children | 4.2-alpha | POSIX change |
+| `help` | `help` now searches exact topic-strings (i.e. `help read` won't find `readonly` anymore) | 4.3-alpha | |
+| `return` | accept negative values as return value (e.g. `return -1` will show as (8 bit) 255 in the caller) | 4.3-alpha | |
+| `exit` | accept negative values as return value (e.g. `return -1` will show as (8 bit) 255 in the caller) | 4.3-alpha | |
+| `read` | `read` skips `NUL` (ASCII Code 0) in input | 4.3-alpha | |
+| `declare` | new option `-n`/`+n` to support nameref variable type | 4.3-alpha | |
+| `wait` | new option `-n` to wait for the next background job to finish, returning its exit status. | 4.3-alpha | |
+| `read` | `read` checks first variable argument for validity before trying to read inout | 4.3-beta | |
+| `help` | attempts substring matching (as it did through bash-4.2) if exact string matching fails | 4.3-beta2 | |
+| `fc` | interprets option `-0` (zero) as the current command line | 4.3-beta2 | |
+| `cd` | new option `-@` to browse a file's extended attributes (on systems that support `O_XATTR`) | 4.3-rc1 | |
+| `kill` | new option `-L` (upper case ell) to list signals like the normal lowercase option `-l` (compatiblity with some standalone `kill` commands) | 4.4-beta | |
+| `mapfile` | new option `-d` | 4.4-alpha | |
+| `wait` | new option `-f` | 5.0-alpha | |
+| `history` | option `-d` allows negative numbers to index from the end of the history list | 5.0-alpha | |
+| `umask` | allows modes greater than octal 777 | 5.0-alpha | |
+| `times` | honors current locale settings when printing decimal points | 5.0-alpha | |
+| `kill` | New options `-n SIGNUMBER` and `-s SIGNAME` | 5.0-beta2 | [kill](/commands/builtin/kill) |
+| `select` | Support for an empty wordlist following `in` | 5.0-beta2 | |
+| `read` | Option `-e` (use ReadLine to obtain input) now works with arbitrary file descriptors (given by `-u` option) | 5.1-alpha | |
+| `trap` | `-p` option prints signals with SIG_DFL/SIG_IGN on shell start (POSIX mode) | 5.1-alpha | |
+| `unset` | automatically tries to unset a function if the given name is an invalid variable name | 5.1-aplha | |
+| `wait` | option `-n` now accepts a list of jobs | 5.1-alpha | |
+| `wait` | new option `-p NAME` to store PID/JobID (useful when waiting for a list of jobs) | 5.1-alpha | |
+| `local` | new option `-p` to print local variables in the current scope | 5.1-alpha | |
+| `ulimit` | new option `-R` to get/set `RLIMIT_RTTIME` resource | 5.1-alpha | |
+
+## Builtin variables
+
+| Feature or change description | Appeared in Bash version | See also |
+|:--------------------------------------------------|:-------------------------|:---------------------------------------------------------------------------------|
+| `HISTCMD` | 1.14.0 | interactive usage |
+| `PS1`, `PS2`, `PATH`, and `IFS` are unsettable | 2.0 | |
+| `DIRSTACK` array variable | 2.0 | |
+| `PIPESTATUS` array variable | 2.0 | |
+| `BASH_VERSINFO` array variable | 2.0 | |
+| `HOSTNAME` | 2.0 | |
+| `SHELLOPTS` | 2.0 | |
+| `MACHTYPE` | 2.0 | |
+| `GLOBIGNORE` | 2.0 | |
+| `HISTIGNORE` | 2.0 | |
+| respect `LC_ALL` | 2.0 | |
+| respect `LC_MESSAGES` | 2.0 | |
+| respect `LC_CTYPE` | 2.0 | |
+| respect `LC_COLLATE` | 2.0 | |
+| respect `LANG` | 2.0 | |
+| `GROUPS` array variable | 2.01 | |
+| `GROUPS` unsettable/takes (discarded) assignments | 2.04 | |
+| `FUNCNAME` | 2.04 | |
+| respect `LC_NUMERIC` | 2.04 | |
+| `TMOUT` | 2.05b | |
+| `BASH_REMATCH` | 3.0 | together with regex support in `[[...]]` |
+| `BASH_ARGC` | 3.0 | debugger support |
+| `BASH_ARGV` | 3.0 | debugger support |
+| `BASH_SOURCE` | 3.0 | debugger support |
+| `BASH_LINENO` | 3.0 | debugger support |
+| `BASH_SUBSHELL` | 3.0 | debugger support |
+| `BASH_EXECUTION_STRING` | 3.0 | debugger support |
+| `BASH_COMMAND` | 3.0 | debugger support |
+| `HISTTIMEFORMAT` | 3.0 | |
+| `COMP_WORDBREAKS` | 3.0 | |
+| respect `LC_TIME` | 3.1 | |
+| `BASHPID` | 4.0-alpha | Added to mksh R41. |
+| `PROMPT_DIRTRIM` | 4.0 | |
+| `BASH_XTRACEFD` | 4.1-alpha | |
+| `BASHOPTS` | 4.1-alpha | |
+| `FUNCNEST` | 4.2-alpha | |
+| `HISTSIZE` | 4.3-alpha | can be set to negative values for unlimited history length |
+| `HISTFILESIZE` | 4.3-alpha | can be set to negative values for unlimit history file size |
+| `CHILD_MAX` | 4.3-alpha | max. number of exit status of children the shell remembers |
+| `BASH_COMPAT` | 4.3-alpha | set shell compatiblity levels |
+| `EPOCHSECONDS` | 5.0-alpha | expands to the time in seconds since Unix epoch |
+| `EPOCHREALTIME` | 5.0-alpha | expands to the time in seconds since Unix epoch with microsecond granularity |
+| `BASH_ARGV0` | 5.0-alpha | get/set `$0` |
+| `PATH` | 5.0-alpha | Possibility to set a static path for use in a restricted shell (at compile time) |
+| `HISTSIZE` | 5.0-beta | Default can now be set at runtime |
+| `SRANDOM` | 5.1-alpha | New random generator for 32bit numbers (using various methods in the backend) |
+| `ARGV0` | 5.1-alpha | Respected when set in initial shell environment, then initially used to set `$0` |
+| `BASH_REMATCH` | 5.1-alpha | Not readonly anymore |
+| `PROMPT_COMMANDS` | 5.1-alpha | New array variable. List of commands to be executed like `PROMPT_COMMAND` |
+| `SECONDS` | 5.1-alpha | Assignment using arithmetic expressions (is nominally an integer variabnle) |
+| `RANDOM` | 5.1-alpha | Assignment using arithmetic expressions (is nominally an integer variabnle) |
+| `LINENO` | 5.1-alpha | Not an integer variabe |
+
+## Quoting, expansions, substitutions and related
+
+For this topic, see also
+
+- [pe](/syntax/pe).
+
+| Feature or change description | Appeared in Bash version | Remarks |
+|:-----------------------------------------------------------------------------------------------------|:-------------------------|:--------------------------------------------------------------------------------------------------------------|
+| Support for integer-indexed arrays | 2.0 | relevant builtins also got array support |
+| `${PARAMETER//PATTERN/REPLACEMENT}` | 2.0 | |
+| `${PARAMETER:OFFSET:LENGTH}` | 2.0 | |
+| `${!PARAMETER}` (indirection) | 2.0 | |
+| `$"..."` (localized strings) | 2.0 | |
+| `$'...'` (ANSI-C-like strings) | 2.0 | |
+| `\xNNN` in `$'...'` (and `echo -e`) | 2.02-alpha1 | |
+| `$(< FILENAME)` (file content) | 2.02-alpha1 | |
+| globbing (`fnmatch()`) capable of POSIX(r) character classes etc. | 2.02-alpha1 | |
+| extended globbing | 2.02-alpha1 | KSH88 |
+| globbing inside array mass-assignment: `ARRAY=(*.txt)` | 2.03-alpha | |
+| `$'...\'...'` escaped single quote inside ANSI-C-like strings | 2.04-devel | KSH93 |
+| `${!PREFIX*}` (parameter name expansion) | 2.04 | KSH93 |
+| `$'...'` expands `\cx` (Control-x) | 2.05b | |
+| `[:class:]` syntax for pattern matching | 2.05b | KSH93 |
+| `${!ARRAY[@]}` (array index expansion) | 3.0-alpha | KSH93 |
+| `{x..y}` (range brace expansion) | 3.0-alpha | |
+| `$'...'` expands `\xNNN` (Hexdigits) | 3.0 | |
+| `+=` operator for arrays and strings | 3.1-alpha1 | |
+| `${PARAMETER//PATTERN/REPLACEMENT}` behaviour changed | 3.2-alpha | anchoring for global substitution is no longer allowed, changes the way old syntax may work |
+| `${@:0:x}` includes `$0` | 4.0-alpha | |
+| Support for associative arrays | 4.0-alpha | relevant builtins also got associative array support |
+| case modification operators for expansions | 4.0-alpha | |
+| `{0x..0y}` (zeropadding brace expansion) | 4.0-alpha | |
+| numerically indexed arrays can be accessed (on expansion time) from the end using negative indexes | 4.1-alpha | |
+| `\uNNNN` and `\uNNNNNNNN` in `$'...'` | 4.2-alpha | for: `printf`, `echo -e`, `$'...'` |
+| `${PARAMETER:OFFSET:LENGTH}`: Negative `LENGTH` values are used as offset from the end of the string | 4.2-alpha | Substrings only for Bash and ksh93. Works also for argument expansions in zsh. ksh93 can use `${arr[n..-m]}`. |
+| Word expansions like `${foo##bar}` understand indirect variable references | 4.3-beta | |
+| Transformations | 4.4 | |
+| Process substitution now works in POSIX mode | 5.1-alpha | |
+| New transformations: `U`, `u`, `L` | 5.1-alpha | Case-transformation |
+| New transformation: `K` | 5.1-alpha | Display associative arrays as key/value pairs |
+
+## Arithmetic
+
+For this topic, see also
+
+- [arith_expr](/syntax/arith_expr)
+- [arith](/syntax/expansion/arith)
+
+| Feature or change description | Appeared in Bash version | Remarks |
+|:-------------------------------------------|:-------------------------|:------------------------------------------|
+| `((...))` | 2.0-beta2 | KSH93 |
+| ternary operator | 2.0 | |
+| base 64 integer constants | 2.0 | the max. base before is unknown. Anybody? |
+| deprecated `$[...]` in favor of `$((...))` | 2.0 | |
+| exponentiaition operator (`**`) | 2.02-alpha1 | |
+| comma operator `EXPR,EXPR` | 2.04-devel | |
+| pre- and postfix operators | 2.04-devel | |
+
+## Redirection and related
+
+For this topic, see also
+
+- [redirection](/syntax/redirection)
+
+| Feature or change description | Appeared in Bash version | Remarks |
+|:--------------------------------------------------------------------------------------|:-------------------------|:--------|
+| socket redirection (`/dev/tcp/`, `/dev/udp/`) | 2.04-devel | |
+| OS/filesystem-independent support for `/dev/std(in|out|err)` and `/dev/fd/*` | 2.04 | |
+| socket redirection accepts service names | 2.05 | |
+| `[n]<&word-` and `[n]>&word-` FD-duplicate/closing | 2.05b-alpha1 | KSH93 |
+| Here strings: `<<< WORD` | 2.05b-alpha1 | |
+| `|&` (synonym for `2>&1 |`) | 4.0-alpha | |
+| `&>>` (equiv. to `>>FILE 2>&1`) | 4.0-alpha | |
+| `{varname}` style automatic file descriptor allocation | 4.1-alpha | ksh93 |
+| `{varname[idx]}` fd allocation accepts array subscripts and special-meaning variables | 4.3-alpha | ksh93 |
+
+## Misc
+
+| Feature or change description | Appeared in Bash version | See also/remarks |
+|:------------------------------------------------------------------------------------------------|:-------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------|
+| `DEBUG` trap | 2.0 | |
+| `ERR` trap | 2.05a | KSH93 |
+| Support for multibyte characters: Unicode / UTF8 | 2.05b | |
+| `RETURN` trap | 3.0 | ksh93 `EXIT` trap evaluates in caller scope (for `function name {`). Bash `RETURN` in same scope. |
+| `command_not_found_handle` handler function | 4.0-alpha | |
+| official introduction of switchable "compatiblity levels" | 4.0-alpha | `compat31` was introduced in a 3.2 version, mainly because of the incompatibilities that were introduced by the changed `=~` operator |
+| `[[...]]` and `((...))` conditional commands are subject to the `ERR` trap and `set -e` feature | 4.1-alpha | |
+| ACL support for file status checks | 4.1-alpha | |
+| Assignment to negative array indices | 4.3-alpha | ksh93, zsh |
+| `declare`/`typeset -n` | 4.3-alpha | Support for nameref variable type, a variable referencing another one by name |
+| shells started to run process substitutions now run any trap set on `EXIT` | 4.3-beta | |
+| process substitution does not inherit the `v` flag | 5.0-alpha | |
+| `ERR` trap | 5.0-alpha | Reports more reliable line numbers |
+| Variable assignment | 5.0-beta | Assignments preceeding a special builtin that chages variable attributes are not propagated back unless compatiblity mode is 44 or lower |
diff --git a/scripting/basics.md b/scripting/basics.md
new file mode 100644
index 0000000..92e2c5d
--- /dev/null
+++ b/scripting/basics.md
@@ -0,0 +1,351 @@
+# The basics of shell scripting
+
+![](keywords>bash shell scripting basics learning tutorial)
+
+## Script files
+
+A shell script usually resides inside a file. The file can be
+executable, but you can call a Bash script with that filename as a
+parameter:
+
+ bash ./myfile
+
+There is **no need to add a boring filename extension** like `.bash` or
+`.sh`. That is a holdover from UNIX(r), where executables are not tagged
+by the extension, but by **permissions** (filemode). The file name can
+be any combination of legal filename characters. Adding a proper
+filename extension is a convention, nothing else.
+
+ chmod +x ./myfile
+
+If the file is executable, and you want to use it by calling only the
+script name, the shebang must be included in the file.
+
+## The Shebang
+
+The in-file specification of the interpreter of that file, for example:
+
+``` bash
+#!/bin/bash
+echo "Hello world..."
+```
+
+This is interpreted by the kernel [^1] of your system. In general, if a
+file is executable, but not an executable (binary) program, and such a
+line is present, the program specified after `#!` is started with the
+scriptname and all its arguments. These two characters `#` and `!` must
+be **the first two bytes** in the file!
+
+You can follow the process by using `echo` as a fake interpreter:
+
+ #!/bin/echo
+
+We don't need a script body here, as the file will never be interpreted
+and executed by "`echo`". You can see what the Operating System does, it
+calls "`/bin/echo`" with the name of the executable file and following
+arguments.
+
+ $ /home/bash/bin/test testword hello
+ /home/bash/bin/test testword hello
+
+The same way, with `#!/bin/bash` the shell "`/bin/bash`" is called with
+the script filename as an argument. It's the same as executing
+"`/bin/bash /home/bash/bin/test testword hello`"
+
+If the interpreter can be specified with arguments and how long it can
+be is system-specific (see
+[\#!-magic](http://www.in-ulm.de/~mascheck/various/shebang/)). When Bash
+executes a file with a \#!/bin/bash shebang, the shebang itself is
+ignored, since the first character is a hashmark "#", which indicates a
+comment. The shebang is for the operating system, not for the shell.
+Programs that don't ignore such lines, may not work as shebang driven
+interpreters.
+
+\ **Attention:**When the
+specified interpreter is unavailable or not executable (permissions),
+you usually get a "`bad interpreter`" error message., If you get nothing
+and it fails, check the shebang. Older Bash versions will respond with a
+"`no such file or directory`" error for a nonexistant interpreter
+specified by the shebang. \
+
+**Additional note:** When you specify `#!/bin/sh` as shebang and that's
+a link to a Bash, then Bash will run in POSIX(r) mode! See:
+
+- [Bash behaviour](/scripting/bashbehaviour).
+
+A common method is to specify a shebang like
+
+ #!/usr/bin/env bash
+
+...which just moves the location of the potential problem to
+
+- the `env` utility must be located in /usr/bin/
+- the needed `bash` binary must be located in `PATH`
+
+Which one you need, or whether you think which one is good, or bad, is
+up to you. There is no bulletproof portable way to specify an
+interpreter. **It's a common misconception that it solves all problems.
+Period.**
+
+## The standard filedescriptors
+
+Once Initialized, every normal UNIX(r)-program has *at least 3 open
+files*:
+
+- **stdin**: standard input
+- **stdout**: standard output
+- **stderr**: standard error output
+
+Usually, they're all connected to your terminal, stdin as input file
+(keyboard), stdout and stderr as output files (screen). When calling
+such a program, the invoking shell can change these filedescriptor
+connections away from the terminal to any other file (see redirection).
+Why two different output filedescriptors? It's convention to send error
+messages and warnings to stderr and only program output to stdout. This
+enables the user to decide if they want to see nothing, only the data,
+only the errors, or both - and where they want to see them.
+
+When you write a script:
+
+- always read user-input from `stdin`
+- always write diagnostic/error/warning messages to `stderr`
+
+To learn more about the standard filedescriptors, especially about
+redirection and piping, see:
+
+- [An illustrated redirection tutorial](/howto/redirection_tutorial)
+
+## Variable names
+
+It's good practice to use lowercase names for your variables, as shell
+and system-variable names are usually all in UPPERCASE. However, you
+should avoid naming your variables any of the following (incomplete
+list!):
+
+| | | | | | |
+|----------------|-------------|----------------|---------------|------------------|-----------------|
+| `BASH` | `BASH_ARGC` | `BASH_ARGV` | `BASH_LINENO` | `BASH_SOURCE` | `BASH_VERSINFO` |
+| `BASH_VERSION` | `COLUMNS` | `DIRSTACK` | `DISPLAY` | `EDITOR` | `EUID` |
+| `GROUPS` | `HISTFILE` | `HISTFILESIZE` | `HISTSIZE` | `HOME` | `HOSTNAME` |
+| `IFS` | `LANG` | `LANGUAGE` | `LC_ALL` | `LINES` | `LOGNAME` |
+| `LS_COLORS` | `MACHTYPE` | `MAILCHECK` | `OLDPWD` | `OPTERR` | `OPTIND` |
+| `OSTYPE` | `PATH` | `PIPESTATUS` | `PPID` | `PROMPT_COMMAND` | `PS1` |
+| `PS2` | `PS4` | `PS3` | `PWD` | `SHELL` | `SHELLOPTS` |
+| `SHLVL` | `TERM` | `UID` | `USER` | `USERNAME` | `XAUTHORITY` |
+
+This list is incomplete. **The safest way is to use all-lowercase
+variable names.**
+
+## Exit codes
+
+Every program you start terminates with an exit code and reports it to
+the operating system. This exit code can be utilized by Bash. You can
+show it, you can act on it, you can control script flow with it. The
+code is a number between 0 and 255. Values from 126 to 255 are reserved
+for use by the shell directly, or for special purposes, like reporting a
+termination by a signal:
+
+- **126**: the requested command (file) was found, but can't be executed
+- **127**: command (file) not found
+- **128**: according to ABS it's used to report an invalid argument to
+ the exit builtin, but I wasn't able to verify that in the source code
+ of Bash (see code 255)
+- **128 + N**: the shell was terminated by the signal N
+- **255**: wrong argument to the exit builtin (see code 128)
+
+The lower codes 0 to 125 are not reserved and may be used for whatever
+the program likes to report. A value of 0 means **successful**
+termination, a value not 0 means **unsuccessful** termination. This
+behavior (== 0, != 0) is also what Bash reacts to in some flow control
+statements.
+
+An example of using the exit code of the program `grep` to check if a
+specific user is present in /etc/passwd:
+
+``` bash
+if grep ^root /etc/passwd; then
+ echo "The user root was found"
+else
+ echo "The user root was not found"
+fi
+```
+
+A common decision making command is "`test`" or its equivalent "`[`".
+But note that, when calling test with the name "`[`", the square
+brackets are not part of the shell syntax, the left bracket **is** the
+test command!
+
+``` bash
+if [ "$mystring" = "Hello world" ]; then
+ echo "Yeah dude, you entered the right words..."
+else
+ echo "Eeeek - go away..."
+fi
+```
+
+Read more about [the test command](/commands/classictest)
+
+A common exit code check method uses the "`||`" or "`&&`" operators.
+This lets you execute a command based on whether or not the previous
+command completed successfully:
+
+``` bash
+grep ^root: /etc/passwd >/dev/null || echo "root was not found - check the pub at the corner."
+which vi && echo "Your favourite editor is installed."
+```
+
+Please, when your script exits on errors, provide a "FALSE" exit code,
+so others can check the script execution.
+
+## Comments
+
+In a larger, or complex script, it's wise to comment the code. Comments
+can help with debugging or tests. Comments start with the \# character
+(hashmark) and continue to the end of the line:
+
+``` bash
+#!/bin/bash
+# This is a small script to say something.
+echo "Be liberal in what you accept, and conservative in what you send" # say something
+```
+
+The first thing was already explained, it's the so-called shebang, for
+the shell, **only a comment**. The second one is a comment from the
+beginning of the line, the third comment starts after a valid command.
+All three syntactically correct.
+
+### Block commenting
+
+To temporarily disable complete blocks of code you would normally have
+to prefix every line of that block with a \# (hashmark) to make it a
+comment. There's a little trick, using the pseudo command `:` (colon)
+and input redirection. The `:` does nothing, it's a pseudo command, so
+it does not care about standard input. In the following code example,
+you want to test mail and logging, but not dump the database, or execute
+a shutdown:
+
+``` bash
+#!/bin/bash
+# Write info mails, do some tasks and bring down the system in a safe way
+echo "System halt requested" | mail -s "System halt" netadmin@example.com
+logger -t SYSHALT "System halt requested"
+
+##### The following "code block" is effectively ignored
+: <<"SOMEWORD"
+/etc/init.d/mydatabase clean_stop
+mydatabase_dump /var/db/db1 /mnt/fsrv0/backups/db1
+logger -t SYSHALT "System halt: pre-shutdown actions done, now shutting down the system"
+shutdown -h NOW
+SOMEWORD
+##### The ignored codeblock ends here
+```
+
+What happened? The `:` pseudo command was given some input by
+redirection (a here-document) - the pseudo command didn't care about it,
+effectively, the entire block was ignored.
+
+The here-document-tag was quoted here **to avoid substitutions** in the
+"commented" text! Check [redirection with
+here-documents](/syntax/redirection#tag_heredoc) for more
+
+## Variable scope
+
+In Bash, the scope of user variables is generally *global*. That means,
+it does **not** matter whether a variable is set in the "main program"
+or in a "function", the variable is defined everywhere.
+
+Compare the following *equivalent* code snippets:
+
+``` bash
+myvariable=test
+echo $myvariable
+```
+
+``` bash
+myfunction() {
+ myvariable=test
+}
+
+myfunction
+echo $myvariable
+```
+
+In both cases, the variable `myvariable` is set and accessible from
+everywhere in that script, both in functions and in the "main program".
+
+**Attention:** When you set variables in a child process, for
+example a *subshell*, they will be set there, but you will **never**
+have access to them outside of that subshell. One way to create a
+subshell is the pipe. It's all mentioned in a small article about [Bash
+in the processtree](/scripting/processtree)!
+
+### Local variables
+
+Bash provides ways to make a variable's scope *local* to a function:
+
+- Using the `local` keyword, or
+- Using `declare` (which will *detect* when it was called from within a
+ function and make the variable(s) local).
+
+``` bash
+myfunc() {
+local var=VALUE
+
+# alternative, only when used INSIDE a function
+declare var=VALUE
+
+...
+}
+```
+
+The *local* keyword (or declaring a variable using the `declare`
+command) tags a variable to be treated *completely local and separate*
+inside the function where it was declared:
+
+``` bash
+foo=external
+
+printvalue() {
+local foo=internal
+
+echo $foo
+}
+
+
+# this will print "external"
+echo $foo
+
+# this will print "internal"
+printvalue
+
+# this will print - again - "external"
+echo $foo
+```
+
+### Environment variables
+
+The environment space is not directly related to the topic about scope,
+but it's worth mentioning.
+
+Every UNIX(r) process has a so-called *environment*. Other items, in
+addition to variables, are saved there, the so-called *environment
+variables*. When a child process is created (in Bash e.g. by simply
+executing another program, say `ls` to list files), the whole
+environment *including the environment variables* is copied to the new
+process. Reading that from the other side means: **Only variables that
+are part of the environment are available in the child process.**
+
+A variable can be tagged to be part of the environment using the
+`export` command:
+
+``` bash
+# create a new variable and set it:
+# -> This is a normal shell variable, not an environment variable!
+myvariable="Hello world."
+
+# make the variable visible to all child processes:
+# -> Make it an environment variable: "export" it
+export myvariable
+```
+
+[^1]: under specific circumstances, also by the shell itself
diff --git a/scripting/debuggingtips.md b/scripting/debuggingtips.md
new file mode 100644
index 0000000..d590037
--- /dev/null
+++ b/scripting/debuggingtips.md
@@ -0,0 +1,371 @@
+# Debugging a script
+
+![](keywords>bash shell scripting bug debug debugging)
+
+These few lines are not intended as a full-fledged debugging tutorial,
+but as hints and comments about debugging a Bash script.
+
+## Use a unique name for your script
+
+Do **not** name your script `test`, for example! *Why?* `test` is the
+name of a UNIX(r)-command, and most likely built into your shell
+(it's a built-in in Bash) - so you won't be able to run a script with
+the name `test` in a normal way.
+
+**Don't laugh!** This is a classic mistake :-)
+
+## Read the error messages
+
+Many people come into IRC and ask something like *"Why does my script
+fail? I get an error!"*. And when you ask them what the error message
+is, they don't even know. Beautiful.
+
+Reading and interpreting error messages is 50% of your job as debugger!
+Error messages actually **mean** something. At the very least, they can
+give you hints as to where to start debugging. **READ YOUR ERROR
+MESSAGES!**
+
+You may ask yourself why is this mentioned as debugging tip? Well,
+you would be surprised how many shell users ignore the text of error
+messages! When I find some time, I'll paste 2 or 3 IRC log-snips
+here, just to show you that annoying fact.
+
+## Use a good editor
+
+Your choice of editor is a matter of personal preference, but one with
+**Bash syntax highlighting** is highly recommended! Syntax highlighting
+helps you see (you guessed it) syntax errors, such as unclosed quotes
+and braces, typos, etc.
+
+From my personal experience, I can suggest `vim` or `GNU emacs`.
+
+## Write logfiles
+
+For more complex scripts, it's useful to write to a log file, or to the
+system log. Nobody can debug your script without knowing what actually
+happened and what went wrong.
+
+An available syslog interface is `logger` ([online
+manpage](http://unixhelp.ed.ac.uk/CGI/man-cgi?logger+1)).
+
+## Inject debugging code
+
+Insert **echos** everywhere you can, and print to `stderr`:
+
+ echo "DEBUG: current i=$i" >&2
+
+If you read input from **anywhere**, such as a file or [command
+substitution](/syntax/expansion/cmdsubst), print the debug output with
+literal quotes, to see leading and trailing spaces!
+
+ pid=$(< fooservice.pid)
+ echo "DEBUG: read from file: pid=\"$pid\"" >&2
+
+Bash's [printf](/commands/builtin/printf) command has the `%q` format,
+which is handy for verifying whether strings are what they appear to be.
+
+ foo=$(< inputfile)
+ printf "DEBUG: foo is |%q|\n" "$foo" >&2
+ # exposes whitespace (such as CRs, see below) and non-printing characters
+
+## Use shell debug output
+
+There are two useful debug outputs for that task (both are written to
+`stderr`):
+
+- `set -v` mode (`set -o verbose`)
+ - print commands to be executed to `stderr` as if they were read from
+ input (script file or keyboard)
+ - print everything **before** any ([substitution and
+ expansion](/syntax/expansion/intro), ...) is applied
+- `set -x` mode (`set -o xtrace`)
+ - print everything as if it were executed, after [substitution and
+ expansion](/syntax/expansion/intro) is applied
+ - indicate the depth-level of the subshell (by default by prefixing a
+ `+` (plus) sign to the displayed command)
+ - indicate the recognized words after [word
+ splitting](/syntax/expansion/wordsplit) by marking them like `'x y'`
+ - in shell version 4.1, this debug output can be printed to a
+ configurable file descriptor, rather than sdtout by setting the
+ [BASH_XTRACEFD](/syntax/shellvars#BASH_XTRACEFD) variable.
+
+**Hint:** These modes can be entered when calling Bash:
+
+- from commandline: `bash -vx ./myscript`
+- from shebang (OS dependant): `#!/bin/bash -vx`
+
+### Simple example of how to interpret xtrace output
+
+Here's a simple command (a string comparison using the [classic test
+command](/commands/classictest)) executed while in `set -x` mode:
+
+ set -x
+ foo="bar baz"
+ [ $foo = test ]
+
+That fails. Why? Let's see the `xtrace` output:
+
+ + '[' bar baz = test ']'
+
+And now you see that it's ("bar" and "baz") recognized as two separate
+words (which you would have realized if you READ THE ERROR MESSAGES ;)
+). Let's check it...
+
+ # next try
+ [ "$foo" = test ]
+
+`xtrace` now gives
+
+ + '[' 'bar baz' = test ']'
+ ^ ^
+ word markers!
+
+### Making xtrace more useful
+
+(by AnMaster)
+
+`xtrace` output would be more useful if it contained source file and
+line number. Add this assignment [PS4](/syntax/shellvars#PS4) at the
+beginning of your script to enable the inclusion of that information:
+
+ export PS4='+(${BASH_SOURCE}:${LINENO}): ${FUNCNAME[0]:+${FUNCNAME[0]}(): }'
+
+**Be sure to use single quotes here!**
+
+The output would look like this when you trace code *outside a
+function*:
+
+ +(somefile.bash:412): echo 'Hello world'
+
+...and like this when you trace code *inside a function*:
+
+ +(somefile.bash:412): myfunc(): echo 'Hello world'
+
+That helps a lot when the script is long, or when the main script
+sources many other files.
+
+#### Set flag variables with descriptive words
+
+If you test variables that flag the state of options, such as with
+`if [[ -n $option ]];`, consider using descriptive words rather than
+short codes, such as 0, 1, Y, N, because xtrace will show
+`[[ -n word ]]` rather than `[[ -n 1 ]]` when the option is set.
+
+## Debugging commands depending on a set variable
+
+For general debugging purposes you can also define a function and a
+variable to use:
+
+ debugme() {
+ [[ $script_debug = 1 ]] && "$@" || :
+ # be sure to append || : or || true here or use return 0, since the return code
+ # of this function should always be 0 to not influence anything else with an unwanted
+ # "false" return code (for example the script's exit code if this function is used
+ # as the very last command in the script)
+ }
+
+This function does nothing when `script_debug` is unset or empty, but it
+executes the given parameters as commands when `script_debug` is set.
+Use it like this:
+
+ script_debug=1
+ # to turn it off, set script_debug=0
+
+ debugme logger "Sorting the database"
+ database_sort
+ debugme logger "Finished sorting the database, exit code $?"
+
+Of course this can be used to execute something other than echo during
+debugging:
+
+ debugme set -x
+ # ... some code ...
+ debugme set +x
+
+## Dry-run STDIN driven commands
+
+Imagine you have a script that runs FTP commands using the standard FTP
+client:
+
+``` bash
+ftp user@host <Note:** It seems that here-documents (tested on versions
+`1.14.7`, `2.05b`, `3.1.17` and `4.0`) are correctly terminated when
+there is an EOF before the end-of-here-document tag (see
+[redirection](/syntax/redirection)). The reason is unknown, but it seems
+to be deliberate. Bash 4.0 added an extra message for this:
+`` warning: here-document at line delimited by end-of-file (wanted `') ``
+
+### Unexpected end of file while looking for matching ...
+
+ script.sh: line 50: unexpected EOF while looking for matching `"'
+ script.sh: line 100: syntax error: unexpected end of file
+
+This one indicates the double-quote opened in line 50 does not have a
+matching closing quote.
+
+These *unmatched errors* occur with:
+
+- double-quote pairs
+- single-quote pairs (also `$'string'`!)
+- missing a closing `}` with [parameter expansion syntax](/syntax/pe)
+
+### Too many arguments
+
+ bash: test: too many arguments
+
+You most likely forgot to quote a variable expansion somewhere. See the
+example for `xtrace` output from above. External commands may display
+such an error message though in our example, it was the **internal**
+test-command that yielded the error.
+
+### !": event not found
+
+ $ echo "Hello world!"
+ bash: !": event not found
+
+This is not an error per se. It happens in interactive shells, when the
+C-Shell-styled history expansion ("`!searchword`") is enabled. This is
+the default. Disable it like this:
+
+ set +H
+ # or
+ set +o histexpand
+
+### syntax error near unexpected token \`('
+
+When this happens during a script **function definition** or on the
+commandline, e.g.
+
+ $ foo () { echo "Hello world"; }
+ bash: syntax error near unexpected token `('
+
+you most likely have an alias defined with the same name as the function
+(here: `foo`). Alias expansion happens before the real language
+interpretion, thus the alias is expanded and makes your function
+definition invalid.
+
+## The CRLF issue
+
+### What is the CRLF issue?
+
+There's a big difference in the way that UNIX(r) and Microsoft(r) (and
+possibly others) handle the **line endings** of plain text files. The
+difference lies in the use of the CR (Carriage Return) and LF (Line
+Feed) characters.
+
+- MSDOS uses: `\r\n` (ASCII `CR` \#13 `^M`, ASCII LF \#10)
+- UNIX(r) uses: `\n` (ASCII `LF` \#10)
+
+Keep in mind your script is a **plain text file**, and the `CR`
+character means nothing special to UNIX(r) - it is treated like any
+other character. If it's printed to your terminal, a carriage return
+will effectively place the cursor at the beginning of the *current*
+line. This can cause much confusion and many headaches, since lines
+containing CRs are not what they appear to be when printed. In summary,
+CRs are a pain.
+
+### How did a CR end up in my file?
+
+Some possible sources of CRs:
+
+- a DOS/Windows text editor
+- a UNIX(r) text editor that is "too smart" when determining the file
+ content type (and thinks "*it's a DOS text file*")
+- a direct copy and paste from certain webpages (some pastebins are
+ known for this)
+
+### Why do CRs hurt?
+
+CRs can be a nuisance in various ways. They are especially bad when
+present in the shebang/interpreter specified with `#!` in the very first
+line of a script. Consider the following script, written with a
+Windows(r) text editor (`^M` is a symbolic representation of the `CR`
+carriage return character!):
+
+ #!/bin/bash^M
+ ^M
+ echo "Hello world"^M
+ ...
+
+Here's what happens because of the `#!/bin/bash^M` in our shebang:
+
+- the file `/bin/bash^M` doesn't exist (hopefully)
+- So Bash prints an error message which (depending on the terminal, the
+ Bash version, or custom patches!) may or may not expose the problem.
+- the script can't be executed
+
+The error message can vary. If you're lucky, you'll get:
+
+ bash: ./testing.sh: /bin/bash^M: bad interpreter: No such file or directory
+
+which alerts you to the CR. But you may also get the following:
+
+ : bad interpreter: No such file or directory
+
+Why? Because when printed literally, the `^M` makes the cursor go back
+to the beginning of the line. The whole error message is *printed*, but
+you *see* only part of it!
+
+\ It's easy to imagine the `^M` is bad in other places
+too. If you get weird and illogical messages from your script, rule out
+the possibility that`^M` is involved. Find and eliminate it! \
+
+### How can I find and eliminate them?
+
+**To display** CRs (these are only a few examples)
+
+- in VI/VIM: `:set list`
+- with `cat(1)`: `cat -v FILE`
+
+**To eliminate** them (only a few examples)
+
+- blindly with `tr(1)`: `tr -d '\r' FILE.new`
+- controlled with `recode(1)`: `recode MSDOS..latin1 FILE`
+- controlled with `dos2unix(1)`: `dos2unix FILE`
+
+## See also
+
+- [the set builtin command](/commands/builtin/set) (for `-v` and `-x`)
+
+FIXME
+
+- DEBUG trap
diff --git a/scripting/newbie_traps.md b/scripting/newbie_traps.md
new file mode 100644
index 0000000..702bccf
--- /dev/null
+++ b/scripting/newbie_traps.md
@@ -0,0 +1,278 @@
+# Beginner Mistakes
+
+![](keywords>bash shell scripting pitfalls traps beginners)
+
+Here are some typical traps:
+
+## Script execution
+
+### Your perfect Bash script executes with syntax errors
+
+If you write Bash scripts with Bash specific syntax and features, run
+them with Bash, and run them with Bash in native mode.
+
+**Wrong**:
+
+- no shebang
+ - the interpreter used depends on the OS implementation and current
+ shell
+ - **can** be run by calling bash with the script name as an argument,
+ e.g. `bash myscript`
+- `#!/bin/sh` shebang
+ - depends on what `/bin/sh` actually is, for a Bash it means
+ compatiblity mode, **not** native mode
+
+See also:
+
+- [Bash startup mode: SH mode](/scripting/bashbehaviour#sh_mode)
+- [Bash run mode: POSIX mode](/scripting/bashbehaviour#posix_run_mode)
+
+### Your script named "test" doesn't execute
+
+Give it another name. The executable `test` already exists.
+
+In Bash it's a builtin. With other shells, it might be an executable
+file. Either way, it's bad name choice!
+
+Workaround: You can call it using the pathname:
+
+ /home/user/bin/test
+
+## Globbing
+
+### Brace expansion is not globbing
+
+The following command line is not related to globbing (filename
+expansion):
+
+ # YOU EXPECT
+ # -i1.vob -i2.vob -i3.vob ....
+
+ echo -i{*.vob,}
+
+ # YOU GET
+ # -i*.vob -i
+
+**Why?** The brace expansion is simple text substitution. All possible
+text formed by the prefix, the postfix and the braces themselves are
+generated. In the example, these are only two: `-i*.vob` and `-i`. The
+filename expansion happens **after** that, so there is a chance that
+`-i*.vob` is expanded to a filename - if you have files like
+`-ihello.vob`. But it definitely doesn't do what you expected.
+
+Please see:
+
+- [brace](/syntax/expansion/brace)
+
+## Test-command
+
+- `if [ $foo ] ...`
+- `if [-d $dir] ...`
+- ...
+
+Please see:
+
+- [The classic test command -
+ pitfalls](/commands/classictest#pitfalls_summarized)
+
+## Variables
+
+### Setting variables
+
+#### The Dollar-Sign
+
+There is no `$` (dollar-sign) when you reference the **name** of a
+variable! Bash is not PHP!
+
+ # THIS IS WRONG!
+ $myvar="Hello world!"
+
+A variable name preceeded with a dollar-sign always means that the
+variable gets **expanded**. In the example above, it might expand to
+nothing (because it wasn't set), effectively resulting in...
+
+ ="Hello world!"
+
+...which **definitely is wrong**!
+
+When you need the **name** of a variable, you write **only the name**,
+for example
+
+- (as shown above) to set variables: `picture=/usr/share/images/foo.png`
+- to name variables to be used by the `read` builtin command:
+ `read picture`
+- to name variables to be unset: `unset picture`
+
+When you need the **content** of a variable, you prefix its name with
+**a dollar-sign**, like
+
+- echo "The used picture is: \$picture"
+
+#### Whitespace
+
+Putting spaces on either or both sides of the equal-sign (`=`) when
+assigning a value to a variable **will** fail.
+
+ # INCORRECT 1
+ example = Hello
+
+ # INCORRECT 2
+ example= Hello
+
+ # INCORRECT 3
+ example =Hello
+
+The only valid form is **no spaces between the variable name and
+assigned value**:
+
+ # CORRECT 1
+ example=Hello
+
+ # CORRECT 2
+ example=" Hello"
+
+### Expanding (using) variables
+
+A typical beginner's trap is quoting.
+
+As noted above, when you want to **expand** a variable i.e. "get the
+content", the variable name needs to be prefixed with a dollar-sign.
+But, since Bash knows various ways to quote and does word-splitting, the
+result isn't always the same.
+
+Let's define an example variable containing text with spaces:
+
+ example="Hello world"
+
+| Used form | result | number of words |
+|--------------|---------------|-----------------|
+| `$example` | `Hello world` | 2 |
+| `"$example"` | `Hello world` | 1 |
+| `\$example` | `$example` | 1 |
+| `'$example'` | `$example` | 1 |
+
+If you use parameter expansion, you **must** use the **name** (`PATH`)
+of the referenced variables/parameters. i.e. **not** (`$PATH`):
+
+ # WRONG!
+ echo "The first character of PATH is ${$PATH:0:1}"
+
+ # CORRECT
+ echo "The first character of PATH is ${PATH:0:1}"
+
+Note that if you are using variables in [arithmetic
+expressions](/syntax/arith_expr), then the bare **name** is allowed:
+
+ ((a=$a+7)) # Add 7 to a
+ ((a = a + 7)) # Add 7 to a. Identical to the previous command.
+ ((a += 7)) # Add 7 to a. Identical to the previous command.
+
+ a=$((a+7)) # POSIX-compatible version of previous code.
+
+Please see:
+
+- [words](/syntax/words)
+- [quoting](/syntax/quoting)
+- [wordsplit](/syntax/expansion/wordsplit)
+- [pe](/syntax/pe)
+
+### Exporting
+
+Exporting a variable means giving **newly created** (child-)processes a
+copy of that variable. It does **not** copy a variable created in a
+child process back to the parent process. The following example does
+**not** work, since the variable `hello` is set in a child process (the
+process you execute to start that script `./script.sh`):
+
+ $ cat script.sh
+ export hello=world
+
+ $ ./script.sh
+ $ echo $hello
+ $
+
+Exporting is one-way. The direction is from parent process to child
+process, not the reverse. The above example **will** work, when you
+don't execute the script, but include ("source") it:
+
+ $ source ./script.sh
+ $ echo $hello
+ world
+ $
+
+In this case, the export command is of no use.
+
+Please see:
+
+- [processtree](/scripting/processtree)
+
+## Exit codes
+
+### Reacting to exit codes
+
+If you just want to react to an exit code, regardless of its specific
+value, you **don't need** to use `$?` in a test command like this:
+
+``` bash
+grep ^root: /etc/passwd >/dev/null 2>&1
+
+if [ $? -ne 0 ]; then
+ echo "root was not found - check the pub at the corner"
+fi
+```
+
+This can be simplified to:
+
+``` bash
+if ! grep ^root: /etc/passwd >/dev/null 2>&1; then
+ echo "root was not found - check the pub at the corner"
+fi
+```
+
+Or, simpler yet:
+
+``` bash
+grep ^root: /etc/passwd >/dev/null 2>&1 || echo "root was not found - check the pub at the corner"
+```
+
+If you need the specific value of `$?`, there's no other choice. But if
+you need only a "true/false" exit indication, there's no need for `$?`.
+
+See also:
+
+- [Exit codes](/scripting/basics#exit_codes)
+
+### Output vs. Return Value
+
+It's important to remember the different ways to run a child command,
+and whether you want the output, the return value, or neither.
+
+When you want to run a command (or a pipeline) and save (or print) the
+**output**, whether as a string or an array, you use Bash's `$(command)`
+syntax:
+
+ $(ls -l /tmp)
+ newvariable=$(printf "foo")
+
+When you want to use the **return value** of a command, just use the
+command, or add ( ) to run a command or pipeline in a subshell:
+
+ if grep someuser /etc/passwd ; then
+ # do something
+ fi
+
+ if ( w | grep someuser | grep sqlplus ) ; then
+ # someuser is logged in and running sqlplus
+ fi
+
+Make sure you're using the form you intended:
+
+ # WRONG!
+ if $(grep ERROR /var/log/messages) ; then
+ # send alerts
+ fi
+
+Please see:
+
+- [intro](/syntax/ccmd/intro)
+- [cmdsubst](/syntax/expansion/cmdsubst)
diff --git a/scripting/nonportable.md b/scripting/nonportable.md
new file mode 100644
index 0000000..9c14fb9
--- /dev/null
+++ b/scripting/nonportable.md
@@ -0,0 +1,265 @@
+# Portability talk
+
+![](keywords>bash shell scripting portability POSIX portable)
+
+The script programming language of BASH is based on the Bourne Shell
+syntax, with some extensions and derivations.
+
+If scripts need to be portable, some of the BASH-specific syntax
+elements should be avoided. Others should be avoided for all scripts,
+e.g. if there is a corresponding POSIX(r)-compatible syntax (see
+[obsolete](/scripting/obsolete)).
+
+Some syntax elements have a BASH-specific, and a portable[^1]) pendant.
+In these cases the portable syntax should be preferred.
+
+
+
+
+construct
+portable equivalent
+Description
+Portability
+
+
+
+
+source\ FILE
+. FILE
+include a script file
+Bourne shell (bash, ksh, POSIX(r), zsh,
+...)
+
+
+declare
\ keyword
+typeset
keyword
+define local variables (or variables
+with special attributes)
+ksh, zsh, ..., not
+POSIX!
+
+
+command\ <<<\ WORD
+command <<MARKER WORD MARKER
+a here-string, a special form of the
+here-document, avoid it in portable scripts!
+POSIX(r)
+
+
+export VAR=VALUE
+VAR=VALUE export VAR
+Though POSIX(r) allows it, some shells
+don't want the assignment and the exporting in one command
+POSIX(r), zsh, ksh, ...
+
+
+(( MATH ))
+: $(( MATH ))
+POSIX(r) does't define an arithmetic
+compund command, many shells don't know it. Using the pseudo-command
+:
and the arithmetic expansion $(( ))
is a
+kind of workaround here. Attention: Not all shell
+support assignment like $(( a = 1 + 1 ))
! Also see below
+for a probably more portable solution.
+all POSIX(r) compatible shells
+
+
+[[\ EXPRESSION\ ]]
+[ EXPRESSION ]
+or
+test EXPRESSION
+The Bashish test keyword is reserved by
+POSIX(r), but not defined. Use the old fashioned way with the
+test
command. See the
+classic test command
+POSIX(r) and others
+
+
+COMMAND\ <\ <(\ ...INPUTCOMMANDS...\ )
+INPUTCOMMANDS\ >\ TEMPFILE COMMAND\ <\ TEMPFILE
+Process substitution (here used with
+redirection); use the old fashioned way (tempfiles)
+POSIX(r) and others
+
+
+((echo X);(echo Y))
+( (echo X); (echo Y) )
+Nested subshells (separate the inner
+()
from the outer ()
by spaces, to not confuse
+the shell regarding arithmetic control operators)
+POSIX(r) and others
+
+
+
+
+## Portability rationale
+
+Here is some assorted portability information. Take it as a small guide
+to make your scripts a bit more portable. It's not complete (it never
+will be!) and it's not very detailed (e.g. you won't find information
+about how which shell technically forks off which subshell). It's just
+an assorted small set of portability guidelines. *-Thebonsai*
+
+FIXME UNIX shell gurus out there, please be patient with a newbie like
+me and give comments and hints instead of flames.
+
+### Environment (exported) variables
+
+When a new value is assigned to an **existing environment variable**,
+there are two possibilities:
+
+The *new value* is seen by subsequent programs
+
+- without any special action (e.g. Bash)
+- only after an explicit export with `export VARIABLE` (e.g. Sun's
+ `/bin/sh`)
+
+Since an extra `export` doesn't hurt, the safest and most portable way
+is to always (re-)export a changed variable if you want it to be seen by
+subsequent processes.
+
+### Arithmetics
+
+Bash has a special compound command to do arithmetic without expansion.
+However, POSIX has no such command. In the table at the top, there's the
+`: $((MATH))` construct mentioned as possible alternative. Regarding the
+exit code, a 100% equivalent construct would be:
+
+ # Bash (or others) compound command
+ if ((MATH)); then
+ ...
+
+ # portable equivalent command
+ if [ "$((MATH))" -ne 0 ]; then
+ ...
+
+Quotes around the arithmetic expansion `$((MATH))` should not be
+necessary as per POSIX, but Bash and AT&T-KSH perform word-splitting on
+aritrhmetic expansions, so the most portable is *with quotes*.
+
+### echo command
+
+The overall problem with `echo` is, that there are 2 (maybe more)
+mainstream flavours around. The only case where you may safely use an
+`echo` on all systems is: Echoing non-variable arguments that don't
+start with a `-` (dash) and don't contain a `\` (backslash).
+
+Why? (list of known behaviours)
+
+- may or may not automatically interpret backslash escpape codes in the
+ strings
+- may or may not automatically interpret switches (like `-n`)
+- may or may not ignore "end of options" tag (`--`)
+- `echo -n` and `echo -e` are neither portable nor standard (**even
+ within the same shell**, depending on the version or environment
+ variables or the build options, especially KSH93 and Bash)
+
+For these, and possibly other, reasons, POSIX (SUS) standardized the
+existance of [the `printf` command](/commands/builtin/printf).
+
+### Parameter expansions
+
+- `${var:x:x}` is KSH93/Bash specific
+- `${var/../..}` and `${var//../..}` are KSH93/Bash specific
+- `var=$*` and `var=$@` are not handled the same in all shells if the
+ first char of IFS is not " " (space). `var="$*"` should work (except
+ the Bourne shell always joins the expansions with space)
+
+### Special variables
+
+#### PWD
+
+[PWD](/syntax/shellvars#PWD) is POSIX but not Bourne. Most shells are
+*not POSIX* in that they don't ignore the value of the `PWD` environment
+variable. Workaround to fix the value of `PWD` at the start of your
+script:
+
+ pwd -P > dev/null
+
+#### RANDOM
+
+[RANDOM](/syntax/shellvars#RANDOM) is Bash/KSH/ZSH specific variable
+that will give you a random number up to 32767 (2^15-1). Among many
+other available external options, you can use awk to generate a random
+number. There are multiple implementations of awk and which version your
+system uses will depend. Most modern systems will call 'gawk' (i.e. GNU
+awk) or 'nawk'. 'oawk' (i.e. Original/Old awk) does not have the rand()
+or srand() functions, so is best avoided.
+
+ # 'gawk' can produce random numbers using srand(). In this example, 10 integers between 1 and 500:
+ randpm=$(gawk -v min=1 -v max=500 -v nNum=10 'BEGIN { srand(systime() + PROCINFO["pid"]); for (i = 0; i < nNum; ++i) {print int(min + rand() * (max - min)} }')
+
+ # 'nawk' and 'mawk' does the same, but needs a seed to be provided for its rand() function. In this example we use $(date)
+ randpm=$(mawk -v min=1 -v max=500 -v nNum=10 -v seed="$(date +%Y%M%d%H%M%S)" 'BEGIN { srand(seed); for (i = 0; i < nNum; ++i) {print int(min + rand() * (max - min)} }')
+
+*Yes, I'm not an `awk` expert, so please correct it, rather than
+complaining about possible stupid code!*
+
+ # Well, seeing how this //is// BASH-hackers.org I kinda missed the bash way of doing the above ;-)
+ # print a number between 0 and 500 :-)
+ printf $(( 500 * RANDOM / 32767 ))
+
+ # Or print 30 random numbers between 0 and 10 ;)
+ X=0; while (( X++ < 30 )); do echo $(( 10 * RANDOM / 32767 )); done
+
+#### SECONDS
+
+[SECONDS](/syntax/shellvars#SECONDS) is KSH/ZSH/Bash specific. Avoid it.
+Find another method.
+
+### Check for a command in PATH
+
+The [PATH](/syntax/shellvars#PATH) variable is a colon-delimited list of
+directory names, so it's basically possible to run a loop and check
+every `PATH` component for the command you're looking for and for
+executability.
+
+However, this method doesn't look nice. There are other ways of doing
+this, using commands that are *not directly* related to this task.
+
+#### hash
+
+The `hash` command is used to make the shell store the full pathname of
+a command in a lookup-table (to avoid re-scanning the `PATH` on every
+command execution attempt). Since it has to do a `PATH` search, it can
+be used for this check.
+
+For example, to check if the command `ls` is available in a location
+accessible by `PATH`:
+
+ if hash ls >/dev/null 2>&1; then
+ echo "ls is available"
+ fi
+
+Somewhat of a mass-check:
+
+ for name in ls grep sed awk; do
+ if ! hash "$name" >/dev/null 2>&1; then
+ echo "FAIL: Missing command '$name'"
+ exit 1
+ fi
+ done
+
+Here (bash 3), `hash` also respects builtin commands. I don't know if
+this works everywhere, but it seems logical.
+
+#### command
+
+The `command` command is used to explicitly call an external command,
+rather than a builtin with the same name. For exactly this reason, it
+has to do a `PATH` search, and can be used for this check.
+
+For example, to check if the command `sed` is available in a location
+accessible by `PATH`:
+
+ if command -v sed >/dev/null 2>&1; then
+ echo "sed is available"
+ fi
+
+[^1]: "portable" doesn't necessarily mean it's POSIX, it can also mean
+ it's "widely used and accepted", and thus maybe more portable than
+ POSIX(r
diff --git a/scripting/obsolete.md b/scripting/obsolete.md
new file mode 100644
index 0000000..e69de29
diff --git a/scripting/posparams.md b/scripting/posparams.md
new file mode 100644
index 0000000..a860a24
--- /dev/null
+++ b/scripting/posparams.md
@@ -0,0 +1,396 @@
+# Handling positional parameters
+
+![](keywords>bash shell scripting arguments positional parameters options)
+
+## Intro
+
+The day will come when you want to give arguments to your scripts. These
+arguments are known as **positional parameters**. Some relevant special
+parameters are described below:
+
+| Parameter(s) | Description |
+|------------------|---------------------------------------------------------------------------------------------------------------------------------|
+| `$0` | the first positional parameter, equivalent to `argv[0]` in C, see [the first argument](/scripting/posparams#the_first_argument) |
+| `$FUNCNAME` | the function name (**attention**: inside a function, `$0` is still the `$0` of the shell, **not** the function name) |
+| `$1 ... $9` | the argument list elements from 1 to 9 |
+| `${10} ... ${N}` | the argument list elements beyond 9 (note the [parameter expansion](/syntax/pe) syntax!) |
+| `$*` | all positional parameters except `$0`, see [mass usage](/scripting/posparams#mass_usage) |
+| `$@` | all positional parameters except `$0`, see [mass usage](/scripting/posparams#mass_usage) |
+| `$#` | the number of arguments, not counting `$0` |
+
+These positional parameters reflect exactly what was given to the script
+when it was called.
+
+Option-switch parsing (e.g. `-h` for displaying help) is not performed
+at this point.
+
+See also [the dictionary entry for "parameter"](/dict/terms/parameter).
+
+## The first argument
+
+The very first argument you can access is referenced as `$0`. It is
+usually set to the script's name exactly as called, and it's set on
+shell initialization:
+
+Testscript - it just echos `$0`:
+
+ #!/bin/bash
+ echo "$0"
+
+You see, `$0` is always set to the name the script is called with (`>`
+is the prompt...):
+
+ > ./testscript
+ ./testscript
+
+ > /usr/bin/testscript
+ /usr/bin/testscript
+
+However, this isn't true for login shells:
+
+ > echo "$0"
+ -bash
+
+In other terms, `$0` is not a positional parameter, it's a special
+parameter independent from the positional parameter list. It can be set
+to anything. In the **ideal** case it's the pathname of the script, but
+since this gets set on invocation, the invoking program can easily
+influence it (the `login` program does that for login shells, by
+prefixing a dash, for example).
+
+Inside a function, `$0` still behaves as described above. To get the
+function name, use `$FUNCNAME`.
+
+## Shifting
+
+The builtin command `shift` is used to change the positional parameter
+values:
+
+- `$1` will be discarded
+- `$2` will become `$1`
+- `$3` will become `$2`
+- ...
+- in general: `$N` will become `$N-1`
+
+The command can take a number as argument: Number of positions to shift.
+e.g. `shift 4` shifts `$5` to `$1`.
+
+## Using them
+
+Enough theory, you want to access your script-arguments. Well, here we
+go.
+
+### One by one
+
+One way is to access specific parameters:
+
+ #!/bin/bash
+ echo "Total number of arguments: $#"
+ echo "Argument 1: $1"
+ echo "Argument 2: $2"
+ echo "Argument 3: $3"
+ echo "Argument 4: $4"
+ echo "Argument 5: $5"
+
+While useful in another situation, this way is lacks flexibility. The
+maximum number of arguments is a fixedvalue - which is a bad idea if you
+write a script that takes many filenames as arguments.
+
+=\> forget that one
+
+### Loops
+
+There are several ways to loop through the positional parameters.
+
+------------------------------------------------------------------------
+
+You can code a [C-style for-loop](/syntax/ccmd/c_for) using `$#` as the
+end value. On every iteration, the `shift`-command is used to shift the
+argument list:
+
+ numargs=$#
+ for ((i=1 ; i <= numargs ; i++))
+ do
+ echo "$1"
+ shift
+ done
+
+Not very stylish, but usable. The `numargs` variable is used to store
+the initial value of `$#` because the shift command will change it as
+the script runs.
+
+------------------------------------------------------------------------
+
+Another way to iterate one argument at a time is the `for` loop without
+a given wordlist. The loop uses the positional parameters as a wordlist:
+
+ for arg
+ do
+ echo "$arg"
+ done
+
+Advantage: The positional parameters will be preserved
+
+------------------------------------------------------------------------
+
+The next method is similar to the first example (the `for` loop), but it
+doesn't test for reaching `$#`. It shifts and checks if `$1` still
+expands to something, using the [test command](/commands/classictest):
+
+ while [ "$1" ]
+ do
+ echo "$1"
+ shift
+ done
+
+Looks nice, but has the disadvantage of stopping when `$1` is empty
+(null-string). Let's modify it to run as long as `$1` is defined (but
+may be null), using [parameter expansion for an alternate
+value](/syntax/pe#use_an_alternate_value):
+
+ while [ "${1+defined}" ]; do
+ echo "$1"
+ shift
+ done
+
+### Getopts
+
+There is a [small tutorial dedicated to
+`getopts`](/howto/getopts_tutorial) (*under construction*).
+
+## Mass usage
+
+### All Positional Parameters
+
+Sometimes it's necessary to just "relay" or "pass" given arguments to
+another program. It's very inefficient to do that in one of these loops,
+as you will destroy integrity, most likely (spaces!).
+
+The shell developers created `$*` and `$@` for this purpose.
+
+As overview:
+
+| Syntax | Effective result |
+|:-------|:----------------------------|
+| `$*` | `$1 $2 $3 ... ${N}` |
+| `$@` | `$1 $2 $3 ... ${N}` |
+| `"$*"` | `"$1c$2c$3c...c${N}"` |
+| `"$@"` | `"$1" "$2" "$3" ... "${N}"` |
+
+Without being quoted (double quotes), both have the same effect: All
+positional parameters from `$1` to the last one used are expanded
+without any special handling.
+
+When the `$*` special parameter is double quoted, it expands to the
+equivalent of: `"$1c$2c$3c$4c........$N"`, where 'c' is the first
+character of `IFS`.
+
+But when the `$@` special parameter is used inside double quotes, it
+expands to the equivanent of...
+
+`"$1" "$2" "$3" "$4" ..... "$N"`
+
+...which **reflects all positional parameters as they were set
+initially** and passed to the script or function. If you want to re-use
+your positional parameters to **call another program** (for example in a
+wrapper-script), then this is the choice for you, use double quoted
+`"$@"`.
+
+Well, let's just say: **You almost always want a quoted `"$@"`!**
+
+### Range Of Positional Parameters
+
+Another way to mass expand the positional parameters is similar to what
+is possible for a range of characters using [substring
+expansion](/syntax/pe#substring_expansion) on normal parameters and the
+mass expansion range of [arrays](/syntax/arrays).
+
+`${@:START:COUNT}`
+
+`${*:START:COUNT}`
+
+`"${@:START:COUNT}"`
+
+`"${*:START:COUNT}"`
+
+The rules for using `@` or `*` and quoting are the same as above. This
+will expand `COUNT` number of positional parameters beginning at
+`START`. `COUNT` can be omitted (`${@:START}`), in which case, all
+positional parameters beginning at `START` are expanded.
+
+If `START` is negative, the positional parameters are numbered in
+reverse starting with the last one.
+
+`COUNT` may not be negative, i.e. the element count may not be
+decremented.
+
+**Example:** START at the last positional parameter:
+
+ echo "${@: -1}"
+
+**Attention**: As of Bash 4, a `START` of `0` includes the
+special parameter `$0`, i.e. the shell name or whatever \$0 is set to,
+when the positional parameters are in use. A `START` of `1` begins at
+`$1`. In Bash 3 and older, both `0` and `1` began at `$1`.
+
+## Setting Positional Parameters
+
+Setting positional parameters with command line arguments, is not the
+only way to set them. The [builtin command, set](/commands/builtin/set)
+may be used to "artificially" change the positional parameters from
+inside the script or function:
+
+ set "This is" my new "set of" positional parameters
+
+ # RESULTS IN
+ # $1: This is
+ # $2: my
+ # $3: new
+ # $4: set of
+ # $5: positional
+ # $6: parameters
+
+It's wise to signal "end of options" when setting positional parameters
+this way. If not, the dashes might be interpreted as an option switch by
+`set` itself:
+
+ # both ways work, but behave differently. See the article about the set command!
+ set -- ...
+ set - ...
+
+Alternately this will also preserve any verbose (-v) or tracing (-x)
+flags, which may otherwise be reset by `set`
+
+ set -$- ...
+
+FIXME continue
+
+## Production examples
+
+### Using a while loop
+
+To make your program accept options as standard command syntax:
+
+`COMMAND [options] ` \# Like 'cat -A file.txt'
+
+See simple option parsing code below. It's not that flexible. It doesn't
+auto-interpret combined options (-fu USER) but it works and is a good
+rudimentary way to parse your arguments.
+
+ #!/bin/sh
+ # Keeping options in alphabetical order makes it easy to add more.
+
+ while :
+ do
+ case "$1" in
+ -f | --file)
+ file="$2" # You may want to check validity of $2
+ shift 2
+ ;;
+ -h | --help)
+ display_help # Call your function
+ # no shifting needed here, we're done.
+ exit 0
+ ;;
+ -u | --user)
+ username="$2" # You may want to check validity of $2
+ shift 2
+ ;;
+ -v | --verbose)
+ # It's better to assign a string, than a number like "verbose=1"
+ # because if you're debugging the script with "bash -x" code like this:
+ #
+ # if [ "$verbose" ] ...
+ #
+ # You will see:
+ #
+ # if [ "verbose" ] ...
+ #
+ # Instead of cryptic
+ #
+ # if [ "1" ] ...
+ #
+ verbose="verbose"
+ shift
+ ;;
+ --) # End of all options
+ shift
+ break;
+ -*)
+ echo "Error: Unknown option: $1" >&2
+ exit 1
+ ;;
+ *) # No more options
+ break
+ ;;
+ esac
+ done
+
+ # End of file
+
+### Filter unwanted options with a wrapper script
+
+This simple wrapper enables filtering unwanted options (here: `-a` and
+`--all` for `ls`) out of the command line. It reads the positional
+parameters and builds a filtered array consisting of them, then calls
+`ls` with the new option set. It also respects the `--` as "end of
+options" for `ls` and doesn't change anything after it:
+
+ #!/bin/bash
+
+ # simple ls(1) wrapper that doesn't allow the -a option
+
+ options=() # the buffer array for the parameters
+ eoo=0 # end of options reached
+
+ while [[ $1 ]]
+ do
+ if ! ((eoo)); then
+ case "$1" in
+ -a)
+ shift
+ ;;
+ --all)
+ shift
+ ;;
+ -[^-]*a*|-a?*)
+ options+=("${1//a}")
+ shift
+ ;;
+ --)
+ eoo=1
+ options+=("$1")
+ shift
+ ;;
+ *)
+ options+=("$1")
+ shift
+ ;;
+ esac
+ else
+ options+=("$1")
+
+ # Another (worse) way of doing the same thing:
+ # options=("${options[@]}" "$1")
+ shift
+ fi
+ done
+
+ /bin/ls "${options[@]}"
+
+### Using getopts
+
+There is a [small tutorial dedicated to
+`getopts`](/howto/getopts_tutorial) (*under construction*).
+
+## See also
+
+- Internal: [getopts_tutorial](/howto/getopts_tutorial)
+- Internal: [while_loop](/syntax/ccmd/while_loop)
+- Internal: [c_for](/syntax/ccmd/c_for)
+- Internal: [arrays](/syntax/arrays) (for equivalent syntax for
+ mass-expansion)
+- Internal: [Substring expansion on a
+ parameter](/syntax/pe#substring_expansion) (for equivalent syntax for
+ mass-expansion)
+- Dictionary, internal: [parameter](/dict/terms/parameter)
diff --git a/scripting/processtree.md b/scripting/processtree.md
new file mode 100644
index 0000000..6fe0c3d
--- /dev/null
+++ b/scripting/processtree.md
@@ -0,0 +1,175 @@
+# Bash and the process tree
+
+![](keywords>bash shell scripting processes pipes variables environment)
+
+## The process tree
+
+The processes in UNIX(r) are - unlike other systems - **organized as a
+tree**. Every process has a parent process that started, or is
+responsible, for it. Every process has its own **context memory** (Not
+the memory where the process stores its data, rather, the memory where
+data is stored that doesn't directly belong to the process, but is
+needed to run the process) i.e. **The environment**.
+
+Every process has its **own** environment space.
+
+The environment stores, among other things, data that's useful to us,
+the **environment variables**. These are strings in common `NAME=VALUE`
+form, but they are not related to shell variables. A variable named
+`LANG`, for example, is used by every program that looks it up in its
+environment to determinate the current locale.
+
+**Attention:** A variable that is set, like with `MYVAR=Hello`,
+is **not** automatically part of the environment. You need to put it
+into the environment with the bash builtin command `export`:
+
+ export MYVAR
+
+Common system variables like [PATH](/syntax/shellvars#PATH) or
+[HOME](/syntax/shellvars#HOME) are usually part of the environment (as
+set by login scripts or programs).
+
+## Executing programs
+
+All the diagrams of the process tree use names like "`xterm`" or
+"`bash`", but that's just to make it easier to understand what's going
+on, it doesn't mean those processes are actually executed.
+
+Let's take a short look at what happens when you "execute a program"
+from the Bash prompt, a program like "ls":
+
+ $ ls
+
+Bash will now perform **two steps**:
+
+- It will make a copy of itself
+- The copy will replace itself with the "ls" program
+
+The copy of Bash will inherit the environment from the "main Bash"
+process: All environment variables will also be copied to the new
+process. This step is called **forking**.
+
+For a short moment, you have a process tree that might look like this...
+
+ xterm ----- bash ----- bash(copy)
+
+...and after the "second Bash" (the copy) replaces itself with the `ls`
+program (the copy execs it), it might look like
+
+ xterm ----- bash ----- ls
+
+If everything was okay, the two steps resulted in one program being run.
+The copy of the environment from the first step (forking) becomes the
+environment for the final running program (in this case, `ls`).
+
+**What is so important about it?** In our example, what the
+program `ls` does inside its own environment, it can't affect the
+environment of its parent process (in this case, `bash`). The
+environment was copied when ls was executed. Nothing is "copied back" to
+the parent environment when `ls` terminates.
+
+## Bash playing with pipes
+
+Pipes are a very powerful tool. You can connect the output of one
+process to the input of another process. We won't delve into piping at
+this point, we just want to see how it looks in the process tree. Again,
+we execute some commands, this time, we'll run `ls` and `grep`:
+
+ $ ls | grep myfile
+
+It results in a tree like this:
+
+ +-- ls
+ xterm ----- bash --|
+ +-- grep
+
+Note once again, `ls` can't influence the `grep` environment, `grep`
+can't influence the `ls` environment, and neither `grep` nor `ls` can
+influence the `bash` environment.
+
+**How is that related to shell programming?!?**
+
+Well, imagine some Bash code that reads data from a pipe. For example,
+the internal command `read`, which reads data from *stdin* and puts it
+into a variable. We run it in a loop here to count input lines:
+
+ counter=0
+
+ cat /etc/passwd | while read; do ((counter++)); done
+ echo "Lines: $counter"
+
+What? It's 0? Yes! The number of lines might not be 0, but the variable
+`$counter` still is 0. Why? Remember the diagram from above? Rewriting
+it a bit, we have:
+
+ +-- cat /etc/passwd
+ xterm ----- bash --|
+ +-- bash (while read; do ((counter++)); done)
+
+See the relationship? The forked Bash process will count the lines like
+a charm. It will also set the variable `counter` as directed. But if
+everything ends, this extra process will be terminated - **your
+"counter" variable is gone.** You see a 0 because in the main shell it
+was 0, and wasn't changed by the child process!
+
+**So, how do we count the lines?** Easy: **Avoid the subshell.**
+The details don't matter, the important thing is the shell that sets the
+counter must be the "main shell". For example:
+
+ counter=0
+
+ while read; do ((counter++)); done output.txt
+
+### Command substitution
+
+With [command substitution](/syntax/expansion/cmdsubst) you re-use the
+output of another command as text in your command line, for example to
+set a variable. The other command is run in a subshell:
+
+ number_of_users=$(cat /etc/passwd | wc -l)
+
+Note that, in this example, a second subshell was created by using a
+pipe in the command substitution:
+
+ +-- cat /etc/passwd
+ xterm ----- bash ----- bash (cmd. subst.) --|
+ +-- wc -l
diff --git a/scripting/style.md b/scripting/style.md
new file mode 100644
index 0000000..dbae0e6
--- /dev/null
+++ b/scripting/style.md
@@ -0,0 +1,404 @@
+# Scripting with style
+
+FIXME continue
+
+These are some coding guidelines that helped me to read and understand
+my own code over the years. They also will help to produce code that
+will be a bit more robust than "if something breaks, I know how to fix
+it".
+
+This is not a bible, of course. But I have seen so much ugly and
+terrible code (not only in shell) during all the years, that I'm 100%
+convinced there needs to be *some* code layout and style. No matter
+which one you use, use it throughout your code (at least don't change it
+within the same shellscript file); don't change your code layout with
+your mood.
+
+Some good code layout helps you to read your own code after a while. And
+of course it helps others to read the code.
+
+## Indentation guidelines
+
+Indentation is nothing that technically influences a script, it's only
+for us humans.
+
+I'm used to seeing/using indentation of *two space characters* (though
+many may prefer 4 spaces, see below in the discussion section):
+
+- it's easy and fast to type
+- it's not a hard-tab that's displayed differently in different
+ environments
+- it's wide enough to give a visual break and small enough to not waste
+ too much space on the line
+
+Speaking of hard-tabs: Avoid them if possible. They only make trouble. I
+can imagine one case where they're useful: Indenting
+[here-documents](/syntax/redirection#here_documents).
+
+### Breaking up lines
+
+Whenever you need to break lines of long code, you should follow one of
+these two rules:
+
+**Indention using command width:**
+
+ activate some_very_long_option \
+ some_other_option
+
+**Indention using two spaces:**
+
+ activate some_very_long_option \
+ some_other_option
+
+Personally, with some exceptions, I prefer the first form because it
+supports the visual impression of "these belong together".
+
+### Breaking compound commands
+
+[Compound commands](/syntax/ccmd/intro) form the structures that make a
+shell script different from a stupid enumeration of commands. Usually
+they contain a kind of "head" and a "body" that contains command lists.
+This type of compound command is relatively easy to indent.
+
+I'm used to (not all points apply to all compound commands, just pick
+the basic idea):
+
+- put the introducing keyword and the initial command list or parameters
+ on one line ("head")
+- put the "body-introducing" keyword on the same line
+- the command list of the "body" on separate lines, indented by two
+ spaces
+- put the closing keyword on a separated line, indented like the initial
+ introducing keyword
+
+What?! Well, here again:
+
+##### Symbolic
+
+ HEAD_KEYWORD parameters; BODY_BEGIN
+ BODY_COMMANDS
+ BODY_END
+
+##### if/then/elif/else
+
+This construct is a bit special, because it has keywords (`elif`,
+`else`) "in the middle". The visually appealing way is to indent them
+like this:
+
+ if ...; then
+ ...
+ elif ...; then
+ ...
+ else
+ ...
+ fi
+
+##### for
+
+ for f in /etc/*; do
+ ...
+ done
+
+##### while/until
+
+ while [[ $answer != [YyNn] ]]; do
+ ...
+ done
+
+##### The case construct
+
+The `case` construct might need a bit more discussion here, since its
+structure is a bit more complex.
+
+In general, every new "layer" gets a new indentation level:
+
+ case $input in
+ hello)
+ echo "You said hello"
+ ;;
+ bye)
+ echo "You said bye"
+ if foo; then
+ bar
+ fi
+ ;;
+ *)
+ echo "You said something weird..."
+ ;;
+ esac
+
+Some notes:
+
+- if not 100% needed, the optional left parenthesis on the pattern is
+ not used
+- the patterns (`hello)`) and the corresponding action terminator (`;;`)
+ are indented at the same level
+- the action command lists are indented one more level (and continue to
+ have their own indentation, if needed)
+- though optional, the very last action terminator is given
+
+## Syntax and coding guidelines
+
+### Cryptic constructs
+
+Cryptic constructs, we all know them, we all love them. If they are not
+100% needed, avoid them, since nobody except you may be able to decipher
+them.
+
+It's - just like in C - the middle ground between smart, efficient and
+readable.
+
+If you need to use a cryptic construct, include a comment that explains
+what your "monster" does.
+
+### Variable names
+
+Since all reserved variables are `UPPERCASE`, the safest way is to only
+use `lowercase` variable names. This is true for reading user input,
+loop counting variables, etc., ... (in the example: `file`)
+
+- prefer `lowercase` variables
+- if you use `UPPERCASE` names, **do not use reserved variable names**
+ (see
+ [SUS](http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap08.html#tag_08)
+ for an incomplete list)
+- if you use `UPPERCASE` names, prepend the name with a unique prefix
+ (`MY_` in the example below)
+
+
+
+ #!/bin/bash
+
+ # the prefix 'MY_'
+ MY_LOG_DIRECTORY=/var/adm/
+
+ for file in "$MY_LOG_DIRECTORY"/*; do
+ echo "Found Logfile: $file"
+ done
+
+### Variable initialization
+
+As in C, it's always a good idea to initialize your variables, though,
+the shell will initialize fresh variables itself (better: Unset
+variables will generally behave like variables containing a null
+string).
+
+It's no problem to pass an **environment variable** to the script. If
+you blindly assume that all variables you use for the first time are
+**empty**, anybody can **inject** content into a variable by passing it
+via the environment.
+
+The solution is simple and effective: **Initialize them**
+
+ my_input=""
+ my_array=()
+ my_number=0
+
+If you do that for every variable you use, then you also have some
+in-code documentation for them.
+
+### Parameter expansion
+
+Unless you are really sure what you're doing, **quote every parameter
+expansion**.
+
+There are some cases where this isn't needed from a technical point of
+view, e.g.
+
+- inside `[[ ... ]]` (other than the RHS of the `==`, `!=`, and `=~`
+ operators)
+- the parameter (`WORD`) in `case $WORD in ....`
+- variable assignment: `VAR=$WORD`
+
+But quoting these is never a mistake. If you quote every parameter
+expansion, you'll be safe.
+
+If you need to parse a parameter as a list of words, you can't quote, of
+course, e.g.
+
+ list="one two three"
+
+ # you MUST NOT quote $list here
+ for word in $list; do
+ ...
+ done
+
+### Function names
+
+Function names should be all `lowercase` and meaningful. The function
+names should be human readable. A function named `f1` may be easy and
+quick to write down, but for debugging and especially for other people,
+it reveals nothing. Good names help document your code without using
+extra comments.
+
+**do not use command names for your functions**. e.g. naming a script or
+function `test`, will collide with the UNIX `test` command.
+
+Unless absolutely necessary, only use alphanumeric characters and the
+underscore for function names. `/bin/ls` is a valid function name in
+Bash, but is not a good idea.
+
+### Command substitution
+
+As noted in [the article about command
+substitution](/syntax/expansion/cmdsubst), you should use the `$( ... )`
+form.
+
+If portability is a concern, use the backquoted form `` ` ... ` ``.
+
+In any case, if other expansions and word splitting are not wanted, you
+should quote the command substitution!
+
+### Eval
+
+Well, like Greg says: **"If eval is the answer, surely you are asking
+the wrong question."**
+
+Avoid it, unless absolutely neccesary:
+
+- `eval` can be your neckshot
+- there are most likely other ways to achieve what you want
+- if possible, re-think the way your script works, if it seems you can't
+ avoid `eval` with your current method
+- if you really, really, have to use it, then take care, and be sure
+ about what you're doing
+
+## Basic structure
+
+The basic structure of a script simply reads:
+
+ #!SHEBANG
+
+ CONFIGURATION_VARIABLES
+
+ FUNCTION_DEFINITIONS
+
+ MAIN_CODE
+
+### The shebang
+
+If possible (I know it's not always possible!), use [a
+shebang](/dict/terms/shebang).
+
+Be careful with `/bin/sh`: The argument that "on Linux `/bin/sh` is
+Bash" **is a lie** (and technically irrelevant)
+
+The shebang serves two purposes for me:
+
+- it specifies the interpreter to be used when the script file is called
+ directly: If you code for Bash, specify `bash`!
+- it documents the desired interpreter (so: use `bash` when you write a
+ Bash-script, use `sh` when you write a general Bourne/POSIX script,
+ ...)
+
+### Configuration variables
+
+I call variables that are meant to be changed by the user "configuration
+variables" here.
+
+Make them easy to find (directly at the top of the script), give them
+meaningful names and maybe a short comment. As noted above, use
+`UPPERCASE` for them only when you're sure about what you're doing.
+`lowercase` will be the safest.
+
+### Function definitions
+
+Unless there are reasons not to, all function definitions should be
+declared before the main script code runs. This gives a far better
+overview and ensures that all function names are known before they are
+used.
+
+Since a function isn't parsed before it is executed, you usually don't
+have to ensure they're in a specific order.
+
+The portable form of the function definition should be used, without the
+`function` keyword (here using the [grouping compound
+command](/syntax/ccmd/grouping_plain)):
+
+ getargs() {
+ ...
+ }
+
+Speaking about the command grouping in function definitions using
+`{ ...; }`: If you don't have a good reason to use another compound
+command directly, you should always use this one.
+
+## Behaviour and robustness
+
+### Fail early
+
+**Fail early**, this sounds bad, but usually is good. Failing early
+means to error out as early as possible when checks indicate an error or
+unmet condition. Failing early means to error out **before** your script
+begins its work in a potentially broken state.
+
+### Availability of commands
+
+If you use external commands that may not be present on the path, or not
+installed, check for their availability, then tell the user they're
+missing.
+
+Example:
+
+ my_needed_commands="sed awk lsof who"
+
+ missing_counter=0
+ for needed_command in $my_needed_commands; do
+ if ! hash "$needed_command" >/dev/null 2>&1; then
+ printf "Command not found in PATH: %s\n" "$needed_command" >&2
+ ((missing_counter++))
+ fi
+ done
+
+ if ((missing_counter > 0)); then
+ printf "Minimum %d commands are missing in PATH, aborting\n" "$missing_counter" >&2
+ exit 1
+ fi
+
+### Exit meaningfully
+
+The [exit code](/dict/terms/exit_status) is your only way to directly
+communicate with the calling process without any special provisions.
+
+If your script exits, provide a meaningful exit code. That minimally
+means:
+
+- `exit 0` (zero) if everything is okay
+- `exit 1` - in general non-zero - if there was an error
+
+This, **and only this**, will enable the calling component to check the
+operation status of your script.
+
+You know: **"One of the main causes of the fall of the Roman Empire was
+that, lacking zero, they had no way to indicate successful termination
+of their C programs."** *-- Robert Firth*
+
+## Misc
+
+### Output and appearance
+
+- if the script is interactive, if it works for you and if you think
+ this is a nice feature, you can try to [save the terminal content and
+ restore it](/snipplets/screen_saverestore) after execution
+- output clean and understandable screen messages
+- if applicable, you can use colors or specific prefixes to tag error
+ and warning messages
+ - make it easy for the user to identify those messages
+- write normal output to `STDOUT`. write error, warning and diagnostic
+ messages to `STDERR`
+ - enables message filtering
+ - keeps the script from mixing output data with diagnostic, or error
+ messages
+ - if the script gives syntax help (`-?` or `-h` or `--help`
+ arguments), it should go to `STDOUT`
+- if applicable, write error/diagnostic messages to a logfile
+ - avoids screen clutter
+ - messages are available for diagnostic use
+
+### Input
+
+- never blindly assume anything. If you want the user to input a number,
+ **check for numeric input, leading zeros**, etc. If you have specific
+ format or content needs, **always validate the input!**
+
+### Tooling
diff --git a/scripting/terminalcodes.md b/scripting/terminalcodes.md
new file mode 100644
index 0000000..db6a27b
--- /dev/null
+++ b/scripting/terminalcodes.md
@@ -0,0 +1,403 @@
+# Terminal codes (ANSI/VT100) introduction
+
+![](keywords>bash shell scripting colors cursor control vt100 ansi)
+
+Terminal (control) codes are used to issue specific commands to your
+terminal. This can be related to switching colors or positioning the
+cursor, i.e. anything that can't be done by the application itself.
+
+## How it technically works
+
+A terminal control code is a special sequence of characters that is
+printed (like any other text). If the terminal understands the code, it
+won't display the character-sequence, but will perform some action. You
+can print the codes with a simple `echo` command.
+
+**Note:** I see codes referenced as "Bash colors" sometimes
+(several "Bash tutorials" etc...): That's a completely incorrect
+definition.
+
+## The tput command
+
+Because there's a large number of different terminal control languages,
+usually a system has an intermediate communication layer. The real codes
+are looked up in a database **for the currently detected terminal type**
+and you give standardized requests to an API or (from the shell) to a
+command.
+
+One of these commands is `tput`. Tput accepts a set of acronyms called
+*capability names* and any parameters, if appropriate, then looks up the
+correct escape sequences for the detected terminal in the `terminfo`
+database and prints the correct codes (the terminal hopefully
+understands).
+
+## The codes
+
+In this list I'll focus on ANSI/VT100 control codes for the most common
+actions - take it as quick reference. The documentation of your terminal
+or the `terminfo` database is always the preferred source when something
+is unclear! Also the `tput` acronyms are usually the ones dedicated for
+ANSI escapes!
+
+I listed only the most relevant codes, of course, any ANSI terminal
+understands many more! But let's keep the discussion centered on common
+shell scripting ;-)
+
+If I couldn't find a matching ANSI escape, you'll see a :?: as the code.
+Feel free to mail me or fix it.
+
+The ANSI codes always start with the ESC character. (ASCII 0x1B or octal
+033) This isn't part of the list, but **you should avoid using the ANSI
+codes directly - use the `tput` command!**
+
+All codes that can be used with `tput` can be found in terminfo(5). (on
+OpenBSD at least) See [OpenBSD's
+terminfo(5)](http://www.openbsd.org/cgi-bin/man.cgi?query=terminfo&apropos=0&sektion=5&manpath=OpenBSD+Current&arch=i386&format=html)
+under the Capabilities section. The *cap-name* is the code to use
+with tput. A description of each code is also provided.
+
+### General useful ASCII codes
+
+The **Ctrl-Key** representation is simply associating the non-printable
+characters from ASCII code 1 with the printable (letter) characters from
+ASCII code 65 ("A"). ASCII code 1 would be `^A` (Ctrl-A), while ASCII
+code 7 (BEL) would be `^G` (Ctrl-G). This is a common representation
+(and input method) and historically comes from one of the VT series of
+terminals.
+
+| Name | decimal | octal | hex | C-escape | Ctrl-Key | Description |
+|-------|---------|-------|------|----------|----------|--------------------------------|
+| `BEL` | 7 | 007 | 0x07 | `\a` | `^G` | Terminal bell |
+| `BS` | 8 | 010 | 0x08 | `\b` | `^H` | Backspace |
+| `HT` | 9 | 011 | 0x09 | `\t` | `^I` | Horizontal TAB |
+| `LF` | 10 | 012 | 0x0A | `\n` | `^J` | Linefeed (newline) |
+| `VT` | 11 | 013 | 0x0B | `\v` | `^K` | Vertical TAB |
+| `FF` | 12 | 014 | 0x0C | `\f` | `^L` | Formfeed (also: New page `NP`) |
+| `CR` | 13 | 015 | 0x0D | `\r` | `^M` | Carriage return |
+| `ESC` | 27 | 033 | 0x1B | `` | `^[` | Escape character |
+| `DEL` | 127 | 177 | 0x7F | `` | `` | Delete character |
+
+### Cursor handling
+
+
+
+
+ANSI
+terminfo equivalent
+Description
+
+
+
+
+[ <X> ; <Y> H
+[ <X> ; <Y> f
+cup <X> <Y>
+Home-positioning to X
and Y
+coordinates
+:!: it seems that ANSI uses 1-1 as home while tput
uses
+0-0
+
+
+[ H
+home
+Move cursor to home position (0-0)
+
+
+7
+sc
+Save current cursor position
+
+
+8
+rc
+Restore saved cursor position
+
+
+:?: most likely a normal code like \b
+cub1
+move left one space (backspace)
+
+
+VT100 [ ? 25 l
+civis
+make cursor invisible
+
+
+VT100 [ ? 25 h
+cvvis
+make cursor visible
+
+
+
+
+### Erasing text
+
+
+
+
+ANSI
+terminfo equivalent
+Description
+
+
+
+
+[ K
+[ 0 K
+el
+Clear line from current
+cursor position to end of line
+
+
+[ 1 K
+el1
+Clear line from beginning
+to current cursor position
+
+
+[ 2 K
+el2
:?:
+Clear whole line (cursor
+position unchanged)
+
+
+
+
+### General text attributes
+
+| ANSI | terminfo equivalent | Description |
+|---------|-----------------------------|----------------------------------------------|
+| `[ 0 m` | `sgr0` | Reset all attributes |
+| `[ 1 m` | `bold` | Set "bright" attribute |
+| `[ 2 m` | `dim` | Set "dim" attribute |
+| `[ 3 m` | `smso` | Set "standout" attribute |
+| `[ 4 m` | set `smul` unset `rmul` :?: | Set "underscore" (underlined text) attribute |
+| `[ 5 m` | `blink` | Set "blink" attribute |
+| `[ 7 m` | `rev` | Set "reverse" attribute |
+| `[ 8 m` | `invis` | Set "hidden" attribute |
+
+### Foreground coloring
+
+| ANSI | terminfo equivalent | Description |
+|:----------|:--------------------|:----------------------------------------------|
+| `[ 3 0 m` | `setaf 0` | Set **foreground** to color \#0 - **black** |
+| `[ 3 1 m` | `setaf 1` | Set **foreground** to color \#1 - **red** |
+| `[ 3 2 m` | `setaf 2` | Set **foreground** to color \#2 - **green** |
+| `[ 3 3 m` | `setaf 3` | Set **foreground** to color \#3 - **yellow** |
+| `[ 3 4 m` | `setaf 4` | Set **foreground** to color \#4 - **blue** |
+| `[ 3 5 m` | `setaf 5` | Set **foreground** to color \#5 - **magenta** |
+| `[ 3 6 m` | `setaf 6` | Set **foreground** to color \#6 - **cyan** |
+| `[ 3 7 m` | `setaf 7` | Set **foreground** to color \#7 - **white** |
+| `[ 3 9 m` | `setaf 9` | Set **default** color as foreground color |
+
+### Background coloring
+
+| ANSI | terminfo equivalent | Description |
+|:----------|:--------------------|:----------------------------------------------|
+| `[ 4 0 m` | `setab 0` | Set **background** to color \#0 - **black** |
+| `[ 4 1 m` | `setab 1` | Set **background** to color \#1 - **red** |
+| `[ 4 2 m` | `setab 2` | Set **background** to color \#2 - **green** |
+| `[ 4 3 m` | `setab 3` | Set **background** to color \#3 - **yellow** |
+| `[ 4 4 m` | `setab 4` | Set **background** to color \#4 - **blue** |
+| `[ 4 5 m` | `setab 5` | Set **background** to color \#5 - **magenta** |
+| `[ 4 6 m` | `setab 6` | Set **background** to color \#6 - **cyan** |
+| `[ 4 7 m` | `setab 7` | Set **background** to color \#7 - **white** |
+| `[ 4 9 m` | `setab 9` | Set **default** color as background color |
+
+### Misc codes
+
+#### Save/restore screen
+
+Used capabilities: `smcup`, `rmcup`
+
+You've undoubtedly already encountered programs that restore the
+terminal contents after they do their work (like `vim`). This can be
+done by the following commands:
+
+ # save, clear screen
+ tput smcup
+ clear
+
+ # example "application" follows...
+ read -n1 -p "Press any key to continue..."
+ # example "application" ends here
+
+ # restore
+ tput rmcup
+
+These features require that certain capabilities exist in your
+termcap/terminfo. While `xterm` and most of its clones (`rxvt`, `urxvt`,
+etc) will support the instructions, your operating system may not
+include references to them in its default xterm profile. (FreeBSD, in
+particular, falls into this category.) If \`tput smcup\` appears to do
+nothing for you, and you don't want to modify your system
+termcap/terminfo data, and you KNOW that you are using a compatible
+xterm application, the following may work for you:
+
+ echo -e '\033[?47h' # save screen
+ echo -e '\033[?47l' # restore screen
+
+Certain software uses these codes (via their termcap capabilities) as
+well. You may have seen the screen save/restore in `less`, `vim`, `top`,
+`screen` and others. Some of these applications may also provide
+configuration options to \*disable\* this behaviour. For example, `less`
+has a `-X` option for this, which can also be set in an environment
+variable:
+
+ export LESS=X
+ less /path/to/file
+
+Similarly, `vim` can be configured not to "restore" the screen by adding
+the following to your `~/.vimrc`:
+
+ set t_ti= t_te=
+
+#### Additional colors
+
+Some terminal emulators support additional colors. The most common
+extension used by xterm-compatible terminals supports 256 colors. These
+can be generated by `tput` with `seta{f,b} [0-255]` when the `TERM`
+value has a `-256color` suffix. [Some
+terminals](https://gist.github.com/XVilka/8346728#now-supporting-truecolour)
+also support full 24-bit colors, and any X11 color code can be written
+directly into a special escape sequence. ([More
+infos](https://gist.github.com/XVilka/8346728)) Only a few programs make
+use of anything beyond 256 colors, and tput doesn't know about them.
+Colors beyond 16 usually only apply to modern terminal emulators running
+in graphical environments.
+
+The Virtual Terminal implemented in the Linux kernel supports only 16
+colors, and the usual default terminfo entry for `TERM=linux` defines
+only 8. There is sometimes an alternate "linux-16color" that you can
+switch to, to get the other 8 colors.
+
+## Bash examples
+
+### Hardcoded colors
+
+ printf '%b\n' 'It is \033[31mnot\033[39m intelligent to use \033[32mhardcoded ANSI\033[39m codes!'
+
+### Colors using tput
+
+Directly inside the echo:
+
+ echo "TPUT is a $(tput setaf 2)nice$(tput setaf 9) and $(tput setaf 5)user friendly$(tput setaf 9) terminal capability database."
+
+With preset variables:
+
+ COL_NORM="$(tput setaf 9)"
+ COL_RED="$(tput setaf 1)"
+ COL_GREEN="$(tput setaf 2)"
+ echo "It's ${COL_RED}red${COL_NORM} and ${COL_GREEN}green${COL_NORM} - have you seen?"
+
+### Misc
+
+HOME function
+
+ home() {
+ # yes, actually not much shorter ;-)
+ tput home
+ }
+
+### Silly but nice effect
+
+ #!/bin/bash
+
+ DATA[0]=" _/ _/ _/ _/ "
+ DATA[1]=" _/_/_/_/_/ _/_/_/ _/_/_/ _/_/_/ _/_/_/ "
+ DATA[2]=" _/ _/ _/ _/ _/ _/ _/_/ _/ _/"
+ DATA[3]="_/_/_/_/_/ _/ _/ _/ _/ _/_/ _/ _/ "
+ DATA[4]=" _/ _/ _/_/_/ _/_/_/ _/_/_/ _/ _/ "
+
+ # virtual coordinate system is X*Y ${#DATA} * 5
+
+ REAL_OFFSET_X=0
+ REAL_OFFSET_Y=0
+
+ draw_char() {
+ V_COORD_X=$1
+ V_COORD_Y=$2
+
+ tput cup $((REAL_OFFSET_Y + V_COORD_Y)) $((REAL_OFFSET_X + V_COORD_X))
+
+ printf %c ${DATA[V_COORD_Y]:V_COORD_X:1}
+ }
+
+
+ trap 'exit 1' INT TERM
+ trap 'tput setaf 9; tput cvvis; clear' EXIT
+
+ tput civis
+ clear
+
+ while :; do
+
+ for ((c=1; c <= 7; c++)); do
+ tput setaf $c
+ for ((x=0; x<${#DATA[0]}; x++)); do
+ for ((y=0; y<=4; y++)); do
+ draw_char $x $y
+ done
+ done
+ done
+
+ done
+
+### Mandelbrot set
+
+This is a slightly modified version of Charles Cooke's colorful
+Mandelbrot plot scripts ([original w/
+screenshot](http://earth.gkhs.net/ccooke/shell.html)) -- ungolfed,
+optimized a bit, and without hard-coded terminal escapes. The `colorBox`
+function is [memoized](http://en.wikipedia.org/wiki/Memoization) to
+collect `tput` output only when required and output a new escape only
+when a color change is needed. This limits the number of `tput` calls to
+at most 16, and reduces raw output by more than half. The `doBash`
+function uses integer arithmetic, but is still ksh93-compatible (run as
+e.g. `bash ./mandelbrot` to use it). The ksh93-only floating-point
+`doKsh` is almost 10x faster than `doBash` (thus the ksh shebang by
+default), but uses only features that don't make the Bash parser crash.
+
+ #!/usr/bin/env ksh
+
+ # Charles Cooke's 16-color Mandelbrot
+ # http://earth.gkhs.net/ccooke/shell.html
+ # Combined Bash/ksh93 flavors by Dan Douglas (ormaaj)
+
+ function doBash {
+ typeset P Q X Y a b c i v x y
+ for ((P=10**8,Q=P/100,X=320*Q/cols,Y=210*Q/lines,y=-105*Q,v=-220*Q,x=v;y<105*Q;x=v,y+=Y)); do
+ for ((;xbash shell scripting review tutorial list recommendation)
+
+Here's a list of some Bash tutorials.
+
+The primary purpose of that list is to lead beginners to *good
+tutorials* and not to the wrong ones. However, the secondary purpose is
+to provide information to increase quality of the linked tutorials.
+
+My experience shows that nobody is interested when you "just send a mail
+to the author", even if he links a big "contact me" in his article(s).
+This is another try of influencing the Bash world.
+
+## List
+
+This is a test for the data plugin. For now, please use the next
+section.
+
+---- datatable tutorial ---- cols : %pageid%, recindex headers : Short
+name, Recommendation Index sort : recindex filter : type=tutorial
+
+------------------------------------------------------------------------
+
+## Recommendations
+
+**Note** that these recommendations are my personal opinion. Please
+**contact** me
+
+- if you have reviews or new sites
+- if you're not okay with a recommendation
+- if you're the author of a mentioned site (remove link, copyright,
+ discussion, ...)
+- etc...
+
+The recommendation-indicator "REC" used below is a number between 1 and
+10 visualized as a bar:
+
+- \