Under construction
This is an archive of the old wiki.bash-hackers.org with the goal of preserving and improving the content by the community in a modern way and format.
The most recent version of each page that had content was automatically converted to Markdown and can be found here. Pandoc did its best, but there is still some work to do as not all pages are formatted correctly. So for everyone who is interested in helping out, feel free to open PRs. Any help is welcome.
This wiki is intended to hold documentation of any kind about GNU Bash. The main motivation was to provide human-readable documentation and information so users aren't forced to read every bit of the Bash manpage - which can be difficult to understand. However, the docs here are not meant as a newbie tutorial.
This wiki and any programs found in this wiki are free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
This wiki and its programs are distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program. If not, see http://www.gnu.org/licenses/.
Stranger! Feel free to comment or edit the contents on GitHub. Use GitHub Issues to submit bugs and GitHub Discussions for enhancements, requests and general feedback/discussion.
","tags":["bash","shell","linux","scripting"]},{"location":"#scripting-and-general-information","title":"Scripting and general information","text":"There is a section that holds small code snippets.
See also some Bash source code excerpts.
","tags":["bash","shell","linux","scripting"]},{"location":"#how-to","title":"How to","text":"Doing specific tasks: concepts, methods, ideas:
kojoro){ ...; } command grouping ( ... ) command grouping in a subshell Conditionals [[ ... ]] conditional expression if ...; then ...; fi conditional branching case ... esac pattern-based branching Loops for word in ...; do ...; done classic for-loop for ((x=1; x<=10; x++)); do ...; done C-style for-loop while ...; do ...; done while loop until ...; do ...; done until loop Misc (( ... )) arithmetic evaluation select word in ...; do ...; done user selections","tags":["bash","shell","linux","scripting"]},{"location":"#expansions-and-substitutions","title":"Expansions and substitutions","text":"Introduction to expansions and substitutions {A,B,C} {A..C} Brace expansion ~/ ~root/ Tilde expansion $FOO ${BAR%.mp3} Parameter expansion `command` $(command) Command substitution <(command.md) >(command) Process substitution $((1 + 2 + 3)) $[4 + 5 + 6] Arithmetic expansion Hello <---> Word! Word splitting /data/*-av/*.mp? Pathname expansion","tags":["bash","shell","linux","scripting"]},{"location":"#builtin-commands","title":"Builtin Commands","text":"This is a selection of builtin commands and command-like keywords, loosely arranged by their common uses. These are provided directly by the shell, rather than invoked as standalone external commands.
","tags":["bash","shell","linux","scripting"]},{"location":"#declaration-commands","title":"Declaration commands","text":"Note
Commands that set and query attributes/types, and manipulate simple datastructures.
Command Description Alt Type declare Display or set shell variables or functions along with attributes.typeset builtin export Display or set shell variables, also giving them the export attribute. typeset -x special builtin eval Evaluate arguments as shell code. special builtin local Declare variables as having function local scope. builtin readonly Mark variables or functions as read-only. typeset -r special builtin unset Unset variables and functions. special builtin shift Shift positional parameters special builtin","tags":["bash","shell","linux","scripting"]},{"location":"#io","title":"IO","text":"Note
Commands for reading/parsing input, or producing/formatting output of standard streams.
Command Description Alt Type coproc Co-processes: Run a command in the background with pipes for reading / writing its standard streams. keyword echo Create output from arguments. builtin mapfile Read lines of input into an array.readarray builtin printf \"advanced echo\" builtin read Read input into variables or arrays, or split strings into fields using delimiters. builtin","tags":["bash","shell","linux","scripting"]},{"location":"#configuration-and-debugging","title":"Configuration and Debugging","text":"Note
Commands that modify shell behavior, change special options, assist in debugging.
Command Description Alt Type caller Identify/print execution frames. builtin set Set the positional parameters and/or set options that affect shell behaviour. special builtin shopt set/get some bash-specific shell options. builtin","tags":["bash","shell","linux","scripting"]},{"location":"#control-flow-and-data-processing","title":"Control flow and data processing","text":"Note
Commands that operate on data and/or affect control flow.
Command Description Alt Typecolon \"true\" null command. true special builtin . (dot) Source external files. source special builtin false Fail at doing nothing. builtin continue / break continue with or break out of loops. special builtin let Arithmetic evaluation simple command. builtin return Return from a function with a specified exit status. special builtin [] The classic test simple command. test builtin","tags":["bash","shell","linux","scripting"]},{"location":"#process-and-job-control","title":"Process and Job control","text":"Note
Commands related to jobs, signals, process groups, subshells.
Command Description Alt Type exec Replace the current shell process or set redirections. special builtin exit Exit the shell. special builtin trap Set signal handlers or output the current handlers. special builtin kill Send a signal to specified process(es.md) builtintimes Display process times. special builtin wait Wait for background jobs and asynchronous lists. builtin","tags":["bash","shell","linux","scripting"]},{"location":"#dictionary","title":"Dictionary","text":"A list of expressions, words, and their meanings can be found under the Dict tab.
","tags":["bash","shell","linux","scripting"]},{"location":"#links","title":"Links","text":"","tags":["bash","shell","linux","scripting"]},{"location":"#official-bash-links","title":"Official Bash links","text":"git clone git.sv.gnu.org/srv/git/bash.gitVisit us in ircs://irc.libera.chat:6697, channel #bash ;-)
If you have critiques or suggestions, please feel free to send a mail using the contact form on the right. Note that there is a simple discussion option below every article.
Please also see the imprint if you have problems with the site and its contents (legality, ...)!
It also would be nice to drop a line when
Simply: Reader's feedback.
","tags":["bash","shell","linux","scripting"]},{"location":"bash4/","title":"Bash 4 - a rough overview","text":"Attention
Since Bash 4 has been around for quite some time now (4.3 will come soon), I consider it to be \"standard\". This page is not maintained anymore and is left here to keep your links working. See the bashchanges page for new stuff introduced.
Besides many bugfixes since Bash 3.2, Bash 4 will bring some interesting new features for shell users and scripters. See also bashchanges for a small general overview with more details.
Not all of the changes and news are included here, just the biggest or most interesting ones. The changes to completion, and the readline component are not covered. Though, if you're familiar with these parts of Bash (and Bash 4), feel free to write a chapter here.
The complete list of fixes and changes is in the CHANGES or NEWS file of your Bash 4 distribution.
The current available stable version is 4.4.18 release (February 03, 2018):
Bash 4 introduces the concepts of coprocesses, a well known feature of other shells. The basic concept is simple: It will start any command in the background and set up an array that is populated with accessible files that represent the filedescriptors of the started process.
In other words: It lets you start a process in background and communicate with its input and output data streams.
See The coproc keyword
"},{"location":"bash4/#the-new-mapfile-builtin","title":"The new \"mapfile\" builtin","text":"The mapfile builtin is able to map the lines of a file directly into an array. This avoids having to fill an array yourself using a loop. It enables you to define the range of lines to read, and optionally call a callback, for example to display a progress bar.
See: mapfile
"},{"location":"bash4/#changes-to-the-case-keyword","title":"Changes to the \"case\" keyword","text":"The case construct understands two new action list terminators:
The ;& terminator causes execution to continue with the next action list (rather than terminate the case construct).
The ;;& terminator causes the case construct to test the next given pattern instead of terminating the whole execution.
See case
"},{"location":"bash4/#changes-to-the-declare-builtin","title":"Changes to the \"declare\" builtin","text":"The -p option now prints all attributes and values of declared variables (or functions, when used with -f). The output is fully re-usable as input.
The new option -l declares a variable in a way that the content is converted to lowercase on assignment. For uppercase, the same applies to -u. The option -c causes the content to be capitalized before assignment.
declare -A declares associative arrays (see below).
The read builtin command has some interesting new features.
The -t option to specify a timeout value has been slightly tuned. It now accepts fractional values and the special value 0 (zero). When -t 0 is specified, read immediately returns with an exit status indicating if there's data waiting or not. However, when a timeout is given, and the read builtin times out, any partial data recieved up to the timeout is stored in the given variable, rather than lost. When a timeout is hit, read exits with a code greater than 128.
A new option, -i, was introduced to be able to preload the input buffer with some text (when Readline is used, with -e). The user is able to change the text, or press return to accept it.
See read
"},{"location":"bash4/#changes-to-the-help-builtin","title":"Changes to the \"help\" builtin","text":"The builtin itself didn't change much, but the data displayed is more structured now. The help texts are in a better format, much easier to read.
There are two new options: -d displays the summary of a help text, -m displays a manpage-like format.
Besides the use of the 512 bytes blocksize everywhere in POSIX mode, ulimit supports two new limits: -b for max socket buffer size and -T for max number of threads.
The brace expansion was tuned to provide expansion results with leading zeros when requesting a row of numbers.
See brace
"},{"location":"bash4/#parameter-expansion","title":"Parameter Expansion","text":"Methods to modify the case on expansion time have been added.
On expansion time you can modify the syntax by adding operators to the parameter name.
See Case modification on parameter expansion
"},{"location":"bash4/#substring-expansion","title":"Substring expansion","text":"When using substring expansion on the positional parameters, a starting index of 0 now causes \\$0 to be prepended to the list (if the positional parameters are used). Before, this expansion started with \\$1:
# this should display $0 on Bash v4, $1 on Bash v3\necho ${@:0:1}\nThere's a new shell option globstar. When enabled, Bash will perform recursive globbing on ** \u2013 this means it matches all directories and files from the current position in the filesystem, rather than only the current level.
The new shell option dirspell enables spelling corrections on directory names during globbing.
See globs
"},{"location":"bash4/#associative-arrays","title":"Associative Arrays","text":"Besides the classic method of integer indexed arrays, Bash 4 supports associative arrays.
An associative array is an array indexed by an arbitrary string, something like
declare -A ASSOC\n\nASSOC[First]=\"first element\"\nASSOC[Hello]=\"second element\"\nASSOC[Peter Pan]=\"A weird guy\"\nSee arrays
"},{"location":"bash4/#redirection","title":"Redirection","text":"There is a new &>> redirection operator, which appends the standard output and standard error to the named file. This is the same as the good old >>FILE 2>&1 notation.
The parser now understands |& as a synonym for 2>&1 |, which redirects the standard error for a command through a pipe.
See redirection
"},{"location":"bash4/#interesting-new-shell-variables","title":"Interesting new shell variables","text":"Variable Description BASHPID contains the PID of the current shell (this is different than what$$ does!) PROMPT_DIRTRIM specifies the max. level of unshortened pathname elements in the prompt FUNCNEST control the maximum number of shell function recursions See shellvars
"},{"location":"bash4/#interesting-new-shell-options","title":"Interesting new Shell Options","text":"The mentioned shell options are off by default unless otherwise mentioned.
Option Descriptioncheckjobs check for and report any running jobs at shell exit compat* set compatiblity modes for older shell versions (influences regular expression matching in [[ ... ]] dirspell enables spelling corrections on directory names during globbing globstar enables recursive globbing with ** lastpipe (4.2) to execute the last command in a pipeline in the current environment See shell_options
"},{"location":"bash4/#misc","title":"Misc","text":"command_not_found_handle, supplying the command words as the function arguments. This can be used to display userfriendly messages or perform different command searchesset -e (errexit) mode was changed, it now acts more intuitive (and is better documented in the manpage).xtrace (set -x/set +x) feature is configurable since Bash 4.1 (previously, it was fixed to stderr): a variable named BASH_XTRACEFD can be set to the filedescriptor that should get the outputconfig-top.h)test <EXPRESSION>
[ <EXPRESSION> ]
This command allows you to do various tests and sets its exit code to 0 (true) or 1 (false) whenever such a test succeeds or not. Using this exit code, it's possible to let Bash react on the result of such a test, here by using the command in an if-statement:
#!/bin/bash\n\n# test if /etc/passwd exists\nif test -e /etc/passwd; then\n echo \"Alright man...\" >&2\nelse\n echo \"Yuck! Where is it??\" >&2\n exit 1\nfi\nThe syntax of the test command is relatively easy. Usually it's the command name test followed by a test type (here -e for \"file exists\") followed by test-type-specific values (here the filename to check, /etc/passwd).
There's a second standardized command that does exactly the same: the command [ \u2013 the difference just is that it's called [ and the last argument to the command must be a ]: It forms [ <EXPRESSION> ].
Let's rewrite the above example to use it:
#!/bin/bash\n\n# test if /etc/passwd exists\nif [ -e /etc/passwd ]; then\n echo \"Alright man...\" >&2\nelse\n echo \"Yuck! Where is it??\" >&2\n exit 1\nfi\nOne might think now that these [ and ] belong to the syntax of Bash's if-clause: No they don't! It's a simple, ordinary command, still!
Another thing you have to remember is that if the test command wants one parameter for a test, you have to give it one parameter. Let's check for some of your music files:
#!/bin/bash\n\nmymusic=\"/data/music/Van Halen/Van Halen - Right Now.mp3\"\n\nif [ -e \"$mymusic\" ]; then\n echo \"Let's rock\" >&2\nelse\n echo \"No music today, sorry...\" >&2\n exit 1\nfi\nAs you definitely noted, the filename contains spaces. Since we call a normal ordinary command (test or [) the shell will word-split the expansion of the variable mymusic: You need to quote it when you don't want the test-command to complain about too many arguments for this test-type! If you didn't understand it, please read the article about words....
Please also note that the file-tests want one filename to test. Don't give a glob (filename-wildcards) as it can expand to many filenames => too many arguments!
Another common mistake is to provide too few arguments:
[ \"$mystring\"!=\"test\" ]\nThis provides exactly one test-argument to the command. With one parameter, it defaults to the -n test: It tests if a provided string is empty (FALSE) or not (TRUE) - due to the lack of spaces to separate the arguments the shown command always ends TRUE!
Well, I addressed several basic rules, now let's see what the test-command can do for you. The Bash test-types can be split into several sections: file tests, string tests, arithmetic tests, misc tests. Below, the tests marked with are non-standard tests (i.e. not in SUS/POSIX/etc..).
"},{"location":"commands/classictest/#file-tests","title":"File tests","text":"This section probably holds the most tests, I'll list them in some logical order. Since Bash 4.1, all tests related to permissions respect ACLs, if the underlying filesystem/OS supports them.
Operator syntax Description -a True if exists. (not recommended, may collide with-a for AND, see below) -e True if exists. -f True, if exists and is a regular file. -d True, if exists and is a directory. -c True, if exists and is a character special file. -b True, if exists and is a block special file. -p True, if exists and is a named pipe (FIFO). -S True, if exists and is a socket file. -L True, if exists and is a symbolic link. -h True, if exists and is a symbolic link. -g True, if exists and has sgid bit set. -u True, if exists and has suid bit set. -r True, if exists and is readable. -w True, if exists and is writable. -x True, if exists and is executable. -s True, if exists and has size bigger than 0 (not empty). -t True, if file descriptor is open and refers to a terminal. -nt True, if is newer than (mtime). -ot True, if is older than (mtime). -ef True, if and refer to the same device and inode numbers."},{"location":"commands/classictest/#string-tests","title":"String tests","text":"Operator syntax Description -z True, if is empty. -n True, if is not empty (this is the default operation). = True, if the strings are equal. != True, if the strings are not equal. < True if sorts before lexicographically (pure ASCII, not current locale!). Remember to escape! Use < > True if sorts after lexicographically (pure ASCII, not current locale!). Remember to escape! Use >"},{"location":"commands/classictest/#arithmetic-tests","title":"Arithmetic tests","text":"Operator syntax Description -eq True, if the integers are equal. -ne True, if the integers are NOT equal. -le True, if the first integer is less than or equal second one. -ge True, if the first integer is greater than or equal second one. -lt True, if the first integer is less than second one. -gt True, if the first integer is greater than second one."},{"location":"commands/classictest/#misc-syntax","title":"Misc syntax","text":"Operator syntax Description -a True, if and are true (AND). Note that -a also may be used as a file test (see above) -o True, if either or is true (OR). ! True, if is false (NOT). ( ) Group a test (for precedence). Attention: In normal shell-usage, the ( and ) must be escaped; use ( and )! -o True, if the shell option is set. -v True if the variable has been set. Use var[n] for array elements. -R True if the variable has been set and is a nameref variable (since 4.3-alpha)"},{"location":"commands/classictest/#number-of-arguments-rules","title":"Number of Arguments Rules","text":"The test builtin, especially hidden under its [ name, may seem simple but is in fact causing a lot of trouble sometimes. One of the difficulty is that the behaviour of test not only depends on its arguments but also on the number of its arguments.
Here are the rules taken from the manual:
Note
This is for the command test, for [ the number of arguments is calculated without the final ], for example [ ] follows the \"zero arguments\" rule.
! (exclamation mark), the expression is true if, and only if, the second argument is null!, the value is the negation of the two-argument test using the second and third arguments( and the third argument is exactly ), the result is the one-argument test of the second argument. Otherwise, the expression is false. The -a and -o operators are considered binary operators in this case (Attention: This means the operator -a is not a file operator in this case!)!, the result is the negation of the three-argument expression composed of the remaining arguments. Otherwise, the expression is parsed and evaluated according to precedence using the rules listed aboveThese rules may seem complex, but it's not so bad in practice. Knowing them might help you to explain some of the \"unexplicable\" behaviours you might encounter:
var=\"\"\nif [ -n $var ]; then echo \"var is not empty\"; fi\nThis code prints \"var is not empty\", even though -n something is supposed to be true if $var is not empty - why?
Here, as $var is not quoted, word splitting occurs and $var results in actually nothing (Bash removes it from the command's argument list!). So the test is in fact [ -n ] and falls into the \"one argument\" rule, the only argument is \"-n\" which is not null and so the test returns true. The solution, as usual, is to quote the parameter expansion: [ -n \"$var\" ] so that the test has always 2 arguments, even if the second one is the null string.
These rules also explain why, for instance, -a and -o can have several meanings.
"},{"location":"commands/classictest/#and-and-or","title":"AND and OR","text":""},{"location":"commands/classictest/#the-prefered-way","title":"The Prefered Way","text":"The way often recommended to logically connect several tests with AND and OR is to use several single test commands and to combine them with the shell && and || list control operators.
See this:
if [ -n \"$var\"] && [ -e \"$var\"]; then\n echo \"\\$var is not null and a file named $var exists!\"\nfi\nThe return status of AND and OR lists is the exit status of the last command executed in the list
command1 && command2, command2 is executed if, and only if, command1 returns an exit status of zero (true)command1 \u2502\u2502 command2, command2 is executed if, and only if, command1 returns a non-zero exit status (false)The logical operators AND and OR for the test-command itself are -a and -o, thus:
if [ -n \"$var\" -a -e \"$var\" ] ; then\n echo \"\\$var is not null and a file named $var exists\"\nfi\nThey are not && or ||:
$ if [ -n \"/tmp\" && -d \"/tmp\"]; then echo true; fi # DOES NOT WORK\nbash: [: missing `]'\nYou might find the error message confusing, [ does not find the required final ], because as seen above && is used to write a list of commands. The if statement actually sees two commands:
[ -n \"/tmp\"-d \"/tmp\" ]...which must fail.
"},{"location":"commands/classictest/#why-you-should-avoid-using-a-and-o","title":"Why you should avoid using -a and -o","text":""},{"location":"commands/classictest/#if-portability-is-a-concern","title":"If portability is a concern","text":"POSIX\u00ae/SUSv3 does not specify the behaviour of test in cases where there are more than 4 arguments. If you write a script that might not be executed by Bash, the behaviour might be different! 1
Let's say, we want to check the following two things (AND):
Let's see:
if [ -z \"false\" -a -z \"$(echo I am executed >&2)\" ] ; then ... \n=> The arguments are all expanded before test runs, thus the echo-command is executed.
if [ -z \"false\" ] && [ -z \"$(echo I am not executed >&2)\" ]; then... \n=> Due to the nature of the && list operator, the second test-command runs only if the first test-command returns true, our echo-command is not executed.
Note
In my opinion, -a and -o are also less readable [pgas]
Take care if you convert your scripts from using -a and -o to use the list way (&& and ||):
-a has precedence over -o&& and || have equal precedenceThat means, you can get different results, depending on the manner of use:
$ if [ \"true\" ] || [ -e /does/not/exist ] && [ -e /does/not/exist ]; then echo true; else echo false; fi\nfalse\n\n$ if [ \"true\" -o -e /does/not/exist -a -e /does/not/exist ]; then echo true; else echo false;fi\ntrue\nAs a result you have to think about it a little or add precedence control (parenthesis).
For && and || parenthesis means (shell-ly) grouping the commands, and since ( ... ) introduces a subshell we will use { ... } instead:
$ if [ \"true\" ] || { [ -e /does/not/exist ] && [ -e /does/not/exist ] ;} ; then echo true; else echo false; fi\ntrue\nFor the test command, the precedence parenthesis are, as well, ( ), but you need to escape or quote them, so that the shell doesn't try to interpret them:
$ if [ ( \"true\" -o -e /does/not/exist ) -a -e /does/not/exist ]; then echo true; else echo false; fi\nfalse\n\n# equivalent, but less readable IMHO:\n$ if [ '(' \"true\" -o -e /does/not/exist ')' -a -e /does/not/exist ]; then echo true; else echo false; fi\nfalse\nAs for AND and OR, there are 2 ways to negate a test with the shell keyword ! or passing ! as an argument to test.
Here ! negates the exit status of the command test which is 0 (true), and the else part is executed:
if ! [ -d '/tmp' ]; then echo \"/tmp doesn't exists\"; else echo \"/tmp exists\"; fi\nHere the test command itself exits with status 1 (false) and the else is also executed:
if [ ! -d '/tmp' ]; then echo \"/tmp doesn't exists\"; else echo \"/tmp exists\"; fi\nUnlike for AND and OR, both methods for NOT have an identical behaviour, at least for doing one single test.
"},{"location":"commands/classictest/#pitfalls-summarized","title":"Pitfalls summarized","text":"In this section you will get all the mentioned (and maybe more) possible pitfalls and problems in a summary.
"},{"location":"commands/classictest/#general","title":"General","text":"Here's the copy of a mail on bug-bash list. A user asking a question about using the test command in Bash, he's talking about a problem, which you may have already had yourself:
From: (PROTECTED)\nSubject: -d option not working. . .?\nDate: Tue, 11 Sep 2007 21:51:59 -0400\nTo: bug-bash@gnu.org\n\nHi All,\n\nI've got a script that I'm trying to set up, but it keeps telling me \nthat \"[-d command not found\". Can someone please explain what is \nwrong with this?:\n\n\n\n\n#!/bin/sh\n\nfor i in $*\ndo\n{\n if [-d $i]\n then\n echo \"$i is a directory! Yay!\"\n else\n echo \"$i is not a directory!\"\n fi\n}\ndone\n\nRegards\nSee the problem regarding the used test-command (the other potential problems are not of interest here)?
[-d $i]\nHe simply didn't know that test or [ is a normal, simple command. Well, here's the answer he got. I quote it here, because it's a well written text that addresses most of the common issues with the \"classic\" test command:
From: Bob Proulx (EMAIL PROTECTED)\nSubject: Re: -d option not working. . .?\nDate: Wed, 12 Sep 2007 10:32:35 -0600\nTo: bug-bash@gnu.org\n\n> (QUOTED TEXT WAS REMOVED)\n\nThe shell is first and foremost a way to launch other commands. The\nsyntax is simply \"if\" followed by a command-list, (e.g. if /some/foo;\nor even if cmd1; cmd2; cmd3; then). Plus the '( ... )' syntax is\nalready taken by the use of starting a subshell.\n\nAs I recall in the original shell language the file test operator was\nnot built-in. It was provided by the standalone '/bin/test' command.\nThe result was effectively this:\n\n if /bin/test -d somedir\n\nAlthough the full path /bin/test was never used. I showed it that way\nhere for emphasis that following the 'if' statement is a command list.\nNormally it would simply have been:\n\n if test -d somedir\n\nOf course that is fine and for the best portability that style is\nstill the recommended way today to use the test command. But many\npeople find that it looks different from other programming languages.\nTo make the test operator (note I mention the test operator and not\nthe shell language, this is a localized change not affecting the\nlanguage as a whole) look more like other programming languages the\n'test' program was coded to ignore the last argument if it was a ']'.\nThen a copy of the test program could be used as the '[' program.\n\n ...modify /bin/test to ignore ']' as last argument...\n cp /bin/test /bin/[\n\nThis allows:\n\n if [ -d somedir ]\n\nDoesn't that look more normal? People liked it and it caught on. It\nwas so popular that both 'test' and '[' are now shell built-ins. They\ndon't launch an external '/bin/test' program anymore. But they *used*\nto launch external programs. Therefore argument parsing is the same\nas if they still did launch an external program. This affects\nargument parsing.\n\n it test -f *.txt\n test: too many arguments\n\nOops. I have twenty .txt files and so test got one -f followed by the\nfirst file followed by the remaining files. (e.g. test -f 1.txt 2.txt\n3.txt 4.txt)\n\n if test -d $file\n test: argument expected\n\nOops. I meant to set file.\n\n file=/path/some/file\n if test -d $file\n\nIf variables such as that are not set then they wlll be expanded by\nthe shell before passing them to the (possibly external) command and\ndisappear entirely. This is why test arguments should always be quoted.\n\n if test -d \"$file\"\n if [ -d \"$file\" ]\n\nActually today test is defined that if only one argument is given as\nin this case \"test FOO\" then then test returns true if the argument is\nnon-zero in text length. Because \"-d\" is non-zero length \"test -d\" is\ntrue. The number of arguments affects how test parses the args. This\navoids a case where depending upon the data may look like a test\noperator.\n\n DATA=\"something\"\n if test \"$DATA\" # true, $DATA is non-zero length\n\n DATA=\"\"\n if test \"$DATA\" # false, $DATA is zero length\n\nBut the problem case is how should test handle an argument that looks\nlike an operator? This used to generate errors but now because it is\nonly one argument is defined to be the same as test -n $DATA.\n\n DATA=\"-d\"\n if test \"$DATA\" # true, $DATA is non-zero length\n if test -d # true, same as previous case.\n\nBecause test and [ are possibly external commands all of the parts of\nthem are chosen to avoid shell metacharacters. The Fortran operator\nnaming was well known at the time (e.g. .gt., .eq., etc.) and was\npressed into service for the shell test operator too. Comming from\nFortran using -gt, -eq, etc. looked very normal.\n\nIncorrect use generating unlikely to be intended results:\n\n if test 5 > 2 # true, \"5\" is non-zero length, creates file named \"2\"\n\nIntended use:\n\n if test 5 -gt 2 # true (and no shell meta characters needing quoting)\n\nThen much later, sometime in the mid 1980's, the Korn sh decided to\nimprove upon this situation. A new test operator was introduced.\nThis one was always a shell built-in and therefore could act upon the\nshell arguments directly. This is '[[' which is a shell keyword.\n(Keyword, metacharacters, builtins, all are different.) Because the\nshell processes [[ internally all arguments are known and do not need\nto be quoted.\n\n if [[ -d $file ]] # okay\n if [[ 5 > 2 ]] # okay\n\nI am sure that I am remembering a detail wrong but hopefully this is\nuseful as a gentle introduction and interesting anyway.\n\nBob\nI hope this text protects you a bit from stepping from one pitfall into the next.
I find it very interesting and informative, that's why I quoted it here. Many thanks, Bob, also for the permission to copy the text here!
"},{"location":"commands/classictest/#code-examples","title":"Code examples","text":""},{"location":"commands/classictest/#snipplets","title":"Snipplets","text":"Some code snipplets follow, different ways of shell reaction is used.
test \"$MYVAR\"[ \"$MYVAR\" ]test ! -d /home/user/foo && mkdir /home/user/foo[ ! -d /home/user/foo ] && mkdir /home/user/fooif [ ! -d /home/user/foo ]; then mkdir /home/user/foo; fitest $# -ge 1 -a \"$1\" = \"Hello\" || exit 1[ $# -ge 1 ] && [ \"$1\" = \"Hello\" ] || exit 1 (see lists description)Using a for-loop to iterate through all entries of a directory, if an entry is a directory ([ -d \"$fn\" ]), print its name:
for fn in *; do\n [ -d \"$fn\" ] && echo \"$fn\"\ndone\nOf course, one can wonder what is the use of including the parenthesis in the specification without defining the behaviour with more than 4 arguments or how usefull are the examples with 7 or 9 arguments attached to the specification. \u21a9
caller [FRAMENUMBER]\nThe caller builtin command is used to print execution frames of subroutine calls. Without giving a framenumber, the topmost execution frame information is printed (\"who called me\") wile linenumber and filename.
When an execution frame number is given (0 - topmost), the linenumber, the subroutine (function) and the filename is printed. When an invalid execution frame number is given, it exists FALSE. This way it can be used in a loop (see the examples section below).
The code below defines a function die that is used to exit the program. It prints a list of execution frames, starting with the topmost frame (0). The topmost frame is the \"caller of the die function\", in this case function \"f1\".
This way, you can print a \"stack trace\" for debugging or logging purposes.
The code is made very simple, just to show the basic purposes.
#!/bin/bash\n\ndie() {\n local frame=0\n while caller $frame; do\n ((++frame));\n done\n echo \"$*\"\n exit 1\n}\n\nf1() { die \"*** an error occured ***\"; }\nf2() { f1; }\nf3() { f2; }\n\nf3\nOutput
12 f1 ./callertest.sh\n13 f2 ./callertest.sh\n14 f3 ./callertest.sh\n16 main ./callertest.sh\n*** an error occured ***\ncaller produces no output unless used within a script that's run from a real file. It isn't particularly useful for interactive use, but can be used to create a decent die function to track down errors in moderately complex scripts. { bash /dev/stdin; } <<<$'f(){ g; }\\ng(){ h; }\\nh(){ while caller $((n++)); do :; done; }\\nf'caller is an \"expr\" (whatever that means). Only an integer is actually allowed, with no special interpretation of an \"expression\" as far as we can tell.caller is not specified by POSIX(R)caller builtin command appeared in Bash version 3.0cd [-L|-P] [DIRECTORY]\n\ncd -\nThe cd builtin command is used to change the current working directory
cd DIRECTORY)cd -) as saved in the OLDPWD shell variableDIRECTORY argument)The cd builtin command searches the directories listed in CDPATH for a matching directory.
The default behaviour is to follow symbolic links unless the -P option is given or the shell is configured to do so (see the -P option of the set builtin command).
-L Follow symbolic links (default) -P Do not follow symbolic links -@ Browse a file's extended attributed, if supported"},{"location":"commands/builtin/cd/#exit-status","title":"Exit status","text":"cd\ncd -\n-P option of the set builtin commanddeclare [-aAfFgilnrtux] [-p] [NAME[=VALUE] ...]\n\n# obsolete typeset synonym\ntypeset [-aAfFgilnrtux] [-p] [NAME[=VALUE] ...]\ndeclare is used to display or set variables along with variable attributes. When used to display variables/functions and their value, the output is re-usable as input for the shell.
If no NAME is given, it displays the values of all variables or functions when restricted by the -f option.
If NAME is followed by =VALUE, declare also sets the value for a variable.
When used in a function, declare makes NAMEs local variables, unless used with the -g option.
Don't use it's synonym typeset when coding for Bash, since it's tagged as obsolete.
Below, [-+]X indicates an attribute, use -X to set the attribute, +X to remove it.
[-+]a make NAMEs indexed arrays (removing with +a is valid syntax, but leads to an error message) [-+]A make NAMEs associative arrays [-+]c Undocumented convert NAMEs to \"capcase\" on assignment (makes the first letter upper-case and the rest lower). Requires Bash built with -DCASEMOD_CAPCASE -f restrict action or display to function names and definitions (removing with +f is valid syntax, but leads to an error message) -F restrict display to function names only (plus line number and source file when debugging) -g create global variables when used in a shell function; otherwise ignored (by default, declare declares local scope variables when used in shell functions) [-+]i make NAMEs have the \"integer\" attribute [-+]l convert NAMEs to lower case on assignment (makes sure the variable contains only lower case letters) [-+]n make NAME a reference to the variable named by its value. Introduced in Bash 4.3-alpha. ''${!NAME}'' reveals the reference variable name, VALUE. Use unset -n NAME to unset the variable. (unset -v NAME unsets the VALUE variable.) Use [[ -R NAME ]] to test if NAME has been set to a VALUE, another variable's name. -p display the attributes and value of each NAME [-+]r make NAMEs readonly (removing with +r is valid syntax, but not possible) [-+]t make NAMEs have the \"trace\" attribute (effective only for functions) [-+]u convert NAMEs to upper case on assignment (makes sure the variable contains only upper case letters) [-+]x make NAMEs exported"},{"location":"commands/builtin/declare/#return-status","title":"Return status","text":"Status Reason 0 no error != 0 invalid option != 0 invalid variable name given != 0 attempt to define a function using -f != 0 assignment to a readonly variable != 0 removing the readonly-attribute from a readonly variable != 0 assignment to an array variable without the compound assignment syntax (array=(...)) != 0 attempt to use +a to \"destroy\" an array != 0 attemt to display a non-existent function with -f"},{"location":"commands/builtin/declare/#notes","title":"Notes","text":"Unix shells offer very few datatypes. Bash and some other shells extend this by allowing \"attributes\" to be set on variable names. The only attributes specified by POSIX are export and readonly, which are set by their own dedicated builtins. Datatypes in bash have a few other interesting capabilities such as the ability to modify data on assignment.
declare -f can be used to display all defined functions...
$ declare -f\nfoo ()\n{\n echo \"FOO is BAR\"\n}\nworld ()\n{\n echo \"Hello World!\"\n}\n...or just a specific defined function.
$ declare -f foo\nfoo ()\n{\n echo \"FOO is BAR\"\n}\nBash 4.3 adds a new way to indirectly reference variables. typeset -n or declare -n can be used to make a variable indirectly refer to another. In Bash, the lvalue of the assignment given to typeset -n or declare -n will refer to the variable whose name is expanded on the RHS.
typeset -n is used in the example below. See notes below.
# Sum a set of arrays and assign the result indirectly, also printing each intermediary result (without portability workarounds)\n# sum name arrname [ arrname ... ]\nfunction sum {\n typeset -n _result=$1 _arr\n typeset IFS=+\n _result=0\n for _arr in \"${@:2}\"; do # Demonstrate the special property of \"for\" on a nameref.\n (( _result += ${_arr[*]} ))\n printf '%s = %d\\n' \"${!_result}\" \"$_result\" # Demonstrate the special property of ${!ref} on a nameref.\n done\n}\n\na=(1 2 3) b=(6 5 4) c=(2 4 6)\nsum total a b c\nprintf 'Final value of \"total\" is: %d\\n' \"$total\"\ntypeset -n is currently implemented in ksh93, mksh, and Bash 4.3. Bash and mksh's implementations are quite similar, but much different from ksh93's. See Portability considerations for details. ksh93 namerefs are much more powerful than Bash's.
declare is not specified by POSIX\u00aedeclare is unique to Bash and totally non-portable with the possible exception of Zsh in Bash compatibility mode. Bash marks the synonym typeset as obsolete, which in Bash behaves identically to declare. All other Korn-like shells use typeset, so it probably isn't going away any time soon. Unfortunately, being a non-standard builtin, typeset differs significantly between shells. ksh93 also considers typeset a special builtin, while Bash does not - even in POSIX mode. If you use typeset, you should attempt to only use it in portable ways.export in the next version of POSIX.echo [-neE] [arg ...]\necho outputs it's args to stdout, separated by spaces, followed by a newline. The return status is always 0. If the shopt option xpg_echo is set, Bash dynamically determines whether echo should expand escape characters (listed below) by default based on the current platform. echo doesn't interpret -- as the end of options, and will simply print this string if given.
-n The trailing newline is suppressed. -e Interpretation of the following backslash-escaped characters (below) is enabled. -E Disables the interpretation of these escape characters, even on systems where they are interpreted by default."},{"location":"commands/builtin/echo/#escape-sequences","title":"Escape sequences","text":"Escape Description \\a alert (bell) \\b backspace \\c suppress further output \\e \\E an escape character \\f form feed \\n new line \\r carriage return \\t horizontal tab \\v vertical tab \\\\ backslash \\0nnn the eight-bit character whose value is the octal value nnn (zero to three octal digits) \\xHH the eight-bit character whose value is the hexadecimal value HH (one or two hex digits) \\uHHHH the Unicode (ISO/IEC 10646) character whose value is the hexadecimal value HHHH (one to four hex digits) \\UHHHHHHHH the Unicode (ISO/IEC 10646) character whose value is the hexadecimal value HHHHHHHH (one to eight hex digits)"},{"location":"commands/builtin/echo/#examples","title":"Examples","text":""},{"location":"commands/builtin/echo/#portability-considerations","title":"Portability considerations","text":"echo is a portability train wreck. No major shell follows POSIX completely, and any shell that attempts to do so should be considered horribly broken. SUSv4 specifies that echo shall not include any options. Further, it specifies that the behavior of -n as a first argument shall be determined by the implementation, unless XSI is followed, in which case -n is always treated as a string, and backslash escapes are interpreted by default. dash has the misfeature of following this and interpreting escapes by default, but includes a -n feature for suppressing newlines nevertheless.
In practice, if you're able to assume a korn-like shell including bash, mksh, or zsh, echo when used in simple cases is generally reliable. For example, in the very common situation in which echo is supplied with a single argument and whose output is to have a newline appended, using echo is considered common practice.
Never use options to echo! Ever! Any time you feel tempted to use echo -e, -n, or any other special feature of echo, use printf instead! If portability is a requirement, you should consider using printf exclusively and just ignore that echo even exists. If you must use echo -e and refuse to use printf, it is usually acceptable to use \\'\\'echo \\$\\'...\\' \\'\\'if targeting only shells that support this special quoting style.
ksh93 has a print command, which if coding specifically for ksh93 should be preferred over echo. printf still includes most of the functionality of both, and should usually be the most preferred option.
eval: eval [arg ...]\neval takes its arguments, concatenates them separated by spaces, and executes the resulting string as Bash code in the current execution environment. eval in Bash works in essentially the same way as most other languages that have an eval function. Perhaps the easiest way to think about eval is that it works in the same way as running \\'\\'bash -c \\\"bash code...\\\" \\'\\'from a script, except in the case of eval, the given code is executed in the current shell environment rather than a child process.
In this example, the literal text within the here-document is executed as Bash code exactly as though it were to appear within the script in place of the eval command below it.
#!/usr/bin/env bash\n{ myCode=$(</dev/stdin); } <<\\EOF\n... arbitrary bash code here ...\nEOF\n\neval \"$myCode\"\nFrequently, eval is used to cause side-effects by performing a pass of expansion on the code before executing the resulting string. This allows for things that otherwise wouldn't be possible with ordinary Bash syntax. This also, of course, makes eval the most powerful command in all of shell programming (and in most other languages for that matter).
This code defines a set of identical functions using the supplied names. eval is the only way to achieve this effect.
main() {\n local fun='() { echo \"$FUNCNAME\"; }' x\n\n for x in {f..n}; do\n eval \"${x}${fun}\"\n done\n\n \"$@\"\n}\n\nmain \"$@\"\nThe printf %q format string performs shell escaping on its arguments. This makes printf %q the \"anti-eval\" - with each pass of a string through printf requiring another eval to peel off the escaping again.
while (( ++n <= 5 )) || ! evalBall=\"eval $evalBall\"; do\n printf -v evalBall 'eval %q' \"printf $n;${evalBall-printf '0\\n'}\"\ndone\n$evalBall\nThe above example is mostly fun and games but illustrates the printf %q property.
Since all current POSIX-compatible shells lack support for first-class functions, it can be tempting and sometimes useful to simulate some of their effect using eval to evaluate a string containing code.
This example shows partial application using eval.
function partial {\n eval shift 2 \\; function \"$1\" \\{ \"$2\" \"$(printf '%q ' \"${@:3}\")\" '\"$@\"; }'\n}\n\nfunction repeat {\n [[ $1 == +([0-9]) ]] || return\n typeset n\n while ((n++ < $1)); do\n \"${@:2}\"\n done\n}\n\npartial print3 repeat 3 printf '%s ' # Create a new function named print3\nprint3 hi # Print \"hi\" 3 times\necho\nThis is very easy to do incorrectly and not usually considered idiomatic of Bash if used extensively. However abstracting eval behind functions that validate their input and/or make clear which input must be controlled carefully by the caller is a good way to use it.
"},{"location":"commands/builtin/eval/#portability-considerations","title":"Portability considerations","text":"Unfortunately, because eval is a special builtin, it only gets its own environment in Bash, and only when Bash is not in POSIX mode. In all other shells plus Bash in POSIX mode, the environment of eval will leak out into the surrounding environment. It is possible to work around this limitation by prefixing special builtins with the command regular builtin, but current versions of ~~ksh93~~ and zsh don't do this properly (fixed in ksh 93v- 2012-10-24 alpha). Earlier versions of zsh work (with setopt POSIX_BUILTINS -- looks like a regression). This works correctly in Bash POSIX mode, Dash, and mksh.
eval is another one of the few Bash builtins with keyword-like conditional parsing of arguments that are in the form of compound assignments.
$ ( eval a=( a b\\ c d ); printf '<%s> ' \"${a[@]}\"; echo ) # Only works in Bash. $ ( x=a; eval \"$x\"=( a b\\ c d ); printf '<%s> ' \"${a[@]}\"; echo ) # Argument is no longer in the form of a valid assignment, therefore ordinary parsing rules apply. -bash: syntax error near unexpected token `(' $ ( x=a; eval \"$x\"'=( a b\\ c d )'; printf '<%s> ' \"${a[@]}\"; echo ) # Proper quoting then gives us the expected results.
We don't know why Bash does this. Since parentheses are metacharacters, they must ordinary be quoted or escaped when used as arguments. The first example above is the same error as the second in all non-Bash shells, even those with compound assignment.
In the case of eval it isn't recommended to use this behavior, because unlike e.g. declare, the initial expansion is still subject to all expansions including word-splitting and pathname expansion.
$ ( set -x; touch 'x+=(\\[[123]\\]=*)' 'x+=([3]=yo)'; eval x+=(*); echo \"${x[@]}\" )\n+ touch 'x+=(\\[[123]\\]=*)' 'x+=([3]=yo)'\n+ eval 'x+=(\\[[123]\\]=*)' 'x+=([3]=yo)'\n++ x+=(\\[[123]\\]=*)\n++ x+=([3]=yo)\n+ echo '[[123]]=*' yo\n[[123]]=* yo\nOther commands known to be affected by compound assignment arguments include: let, declare, typeset, local, export, and readonly. More oddities below show both similarities and differences to commands like declare. The rules for eval appear identical to those of let.
printf %q work-alike for POSIX.exec [-a NAME] [-cl] [COMMAND] [ARG...] [REDIRECTION...]\nThe exec builtin command is used to
If only redirections are given, the redirections affect the current shell without executing any program.
"},{"location":"commands/builtin/exec/#options","title":"Options","text":"Option Description-a NAME Passes NAME as zeroth argument for the program to be executed -c Execute the program with an empty (cleared) environment -l Prepends a dash (-) to the zeroth argument of the program to be executed, similar to what the login program does"},{"location":"commands/builtin/exec/#exit-status","title":"Exit status","text":"exec returns failureexec returns failuremyprog=/bin/ls\necho \"This is the wrapper script, it will exec $myprog\"\n\n# do some vodoo here, probably change the arguments etc.\n# well, stuff a wrapper is there for\n\nexec \"$myprog\" \"$@\"\n# open it\nexec 3< input.txt\n\n# for example: read one line from the file(-descriptor)\nread -u 3 LINE\n# or\nread LINE <&3\n\n# finally, close it\nexec 3<&-\nTo redirect the whole stdout and stderr of the shell or shellscript to a file, you can use the exec builtin command:
exec >/var/adm/my.log 2>&1\n\n# script continues here...\n*POSIX(r) specifies error code ranges:\n * if ''exec'' can't find the program to execute, the error code shall be 126\n * on a redirection error, the error code shall be between 1 and 125\n* the ''-a NAME'' option appeared in Bash 4.2-alpha\n* POSIX(r) does **not** specify any options for ''exec'' (like ''-c'', ''-l'', ''-a NAME'').\nexit [N]\nThe exit command terminates the current shell (or script).
If N is given, the return code to the parent process is set to N. If not, the returned status the the status of the most recently executed command (i.e. $?).
A trap on EXIT is executed before the shell exits, except the executed exit command is part of an already running trap.
There are no options.
"},{"location":"commands/builtin/exit/#exit-status","title":"Exit status","text":"Naturally, you can't ask for the exit status from within the shell that executed the exit command, because the shell exits.
exit 3\nN is specified, but its value is not between 0 and 255 inclusively, the exit status is undefined.export [-fn] [NAME[=VALUE] ...]\nexport -p\nThe export builtin command is used to mark variables or functions referenced by NAME for automatic export to the environment. If NAME is a shell variable, a value VALUE can be assigned before exporting it.
-f refer to shell functions -n remove the export property from any referenced NAME -p print all exported variables, with -f, print all exported functions - all in a format re-usable as input An argument of -- disables further option processing.
NAME is invalid"},{"location":"commands/builtin/export/#examples","title":"Examples","text":"Set the display to use when launching a GUI application (useful during SSH sessions):
export DISPLAY=\":0\"\nSet your default text editor (e.g. SublimeText):
export EDITOR=subl\n-p option is specifiedkill [-s SIGNAL | -n SIGNALNUMBER | -SIGNAL] PID|JOB\n\nkill -l|-L [SIGNAL...]\nThe kill command is used to send signals to processes specified by their PID or their JOB-specification.
The signal(s) to be specified can have the following formats:
SIG<name>)SIG-prefix (<name>)Without any specified signal, the command sends the SIGTERM-signal.
The kill command is a Bash builtin command instead of relying on the external kill command of the operating system to
-s SIGNAL specifies the signal to send -n SIGNALNUMBER specifies the signal to send -SIGNAL specifies the signal to send -l [SIGNAL...] Lists supported/known signal numbers and their symbolic name. If SIGNAL is given, only list this signal, translated (if a number is given the symbolic name is printed, and vice versa) -L [SIGNAL...] Same as -l [SIGNAL] (compatiblity option)"},{"location":"commands/builtin/kill/#return-status","title":"Return status","text":"Status Reason 0 no error/success !=0 invalid option !=0 invalid signal specification !=0 error returned by the system function (e.g. insufficient permissions to send to a specific process)"},{"location":"commands/builtin/kill/#examples","title":"Examples","text":""},{"location":"commands/builtin/kill/#list-supported-signals","title":"List supported signals","text":"kill -l\nkill -9 12345\n\nkill -KILL 12345\n\nkill -SIGKILL 12345\nlet arg [arg ...]\nThe let builtin command evaluates each supplied word from left to right as an arithmetic expression and returns an exit code according to the truth value of the rightmost expression.
arg evaluated to not 0 (arithmetic \"true\")arg evaluated to 0 (arithmetic \"false\")For this return code mapping, please see this section. They work in the same way as ((.
let is very similar to (( - the only difference being let is a builtin (simple command), and (( is a compound command. The arguments to let are therefore subject to all the same expansions and substitutions as any other simple command - requiring proper quoting and escaping - whereas the contents of (( aren't subject to word-splitting or pathname expansion (almost never desirable for arithmetic). For this reason, the arithmetic compound command should generally be preferred over let.
$ let 'b = a' \"(a += 3) + $((a = 1)), b++\"\n$ echo \"$a - $b - $?\"\n4 - 2 - 0\nIs equivalent to the arithmetic evaluation compound command:
$ (( b = a, (a += 3) + $((a = 1)), b++ ))\n$ echo \"$a - $b - $?\"\n4 - 2 - 0\nRemember that inside arithmetic evaluation contexts, all other expansions are processed as usual (from left-to-right), and the resulting text is evaluated as an arithmetic expression. Arithmetic already has a way to control precedence using parentheses, so it's very rare to need to nest arithmetic expansions within one another. It's used above only to illustrate how this precedence works.
Unlike ((, being a simple command let has its own environment. In Bash, built-ins that can set variables process any arithmetic under their own environment, which makes the variable effectively \"local\" to the builtin unless the variable is also set or modified by the builtin. This differs in other shells, such as ksh93, where environment assignments to regular builtins are always local even if the variable is modified by the builtin.
~ $ ( y=1+1 let x=y; declare -p x y )\ndeclare -- x=\"2\"\nbash: declare: y: not found\n\n ~ $ ( y=1+1 let x=y++; declare -p x y )\ndeclare -- x=\"2\"\ndeclare -- y=\"3\"\nThis can be useful in certain situations where a temporary variable is needed.
"},{"location":"commands/builtin/let/#portability-considerations","title":"Portability considerations","text":"let command is not specified by POSIX\u00ae. The portable alternative is: [ \"$(( <EXPRESSION> ))\" -ne 0 ]. To make portable scripts simpler and cleaner, let can be defined as: # POSIX let() { IFS=, command eval test '$(($*))' -ne 0 } Aside from differences in supported arithmetic features, this should be identical to the Bash/Ksh let.let has some legacy purpose. Both let and 1 were ksh88 features and almost identical in terms of portability as everything that inherited one also tended to get the other. Don't choose let over (( expecting it to work in more places.let, the above should always be preferred. Both arithmetic expansions and the [ test operator are specified by POSIX\u00ae and satisfy almost all of expr's use-cases. Unlike let, expr cannot assign directly to bash variables but instead returns a result on stdout. expr takes each operator it recognizes as a separate word and then concatenates them into a single expression that's evaluated according to it's own rules (which differ from shell arithmetic). let parses each word it recieves on its own and evaluates it as an expression without generating any output other than a return code.let is one of the Bash commands with special parsing for arguments formatted like compound array assignments. See: eval for details. There are no known practical uses for this. Parentheses are treated as grouping operators and compound assignment is not possible by arithmetic expressions....\u00a0\u21a9
local [option] name[=value] ...\nlocal is identical to declare in every way, and takes all the same options, with 3 exceptions:
declare and local within a function have the same effect on variable scope, including the -g option.local with no options prints variable names and values in the same format as declare with no options, except the variables are filtered to print only locals that were set in the same scope from which local was called. Variables in parent scopes are not printed.local is not specified by POSIX. Most bourne-like shells don't have a builtin called local, but some such as dash and the busybox shell do.
The behavior of function scope is not defined by POSIX, however local variables are implemented widely by bourne-like shells, and behavior differs substantially. Even thedash shell has local variables.
In ksh93, using POSIX-style function definitions, typeset doesn't set local variables, but rather acts upon variables of the next-outermost scope (e.g. setting attributes). Using typeset within functions defined using ksh function name { syntax, variables follow roughly lexical-scoping, except that functions themselves don't have scope, just like Bash. This means that even functions defined within a \"function's scope\" don't have access to non-local variables except through namerefs.
mapfile [-n COUNT] [-O ORIGIN] [-s COUNT] [-t] [-u FD] [-C CALLBACK] [-c QUANTUM] [ARRAY]\n\nreadarray [-n COUNT] [-O ORIGIN] [-s COUNT] [-t] [-u FD] [-C CALLBACK] [-c QUANTUM] [ARRAY]\nThis builtin is also accessible using the command name readarray.
mapfile is one of the two builtin commands primarily intended for handling standard input (the other being read). mapfile reads lines of standard input and assigns each to the elements of an indexed array. If no array name is given, the default array name is MAPFILE. The target array must be a \"normal\" integer indexed array.
mapfile returns success (0) unless an invalid option is given or the given array ARRAY is set readonly.
-c QUANTUM Specifies the number of lines that have to be read between every call to the callback specified with -C. The default QUANTUM is 5000 -C CALLBACK Specifies a callback. The string CALLBACK can be any shell code, the index of the array that will be assigned, and the line is appended at evaluation time. -n COUNT Reads at most COUNT lines, then terminates. If COUNT is 0, then all lines are read (default). -O ORIGIN Starts populating the given array ARRAY at the index ORIGIN rather than clearing it and starting at index 0. -s COUNT Discards the first COUNT lines read. -t Remove any trailing newline from a line read, before it is assigned to an array element. -u FD Read from filedescriptor FD rather than standard input. While mapfile isn't a common or portable shell feature, it's functionality will be familiar to many programmers. Almost all programming languages (aside from shells) with support for compound datatypes like arrays, and which handle open file objects in the traditional way, have some analogous shortcut for easily reading all lines of some input as a standard feature. In Bash, mapfile in itself can't do anything that couldn't already be done using read and a loop, and if portability is even a slight concern, should never be used. However, it does significantly outperform a read loop, and can make for shorter and cleaner code - especially convenient for interactive use.
Here's a real-world example of interactive use borrowed from Gentoo workflow. Xorg updates require rebuilding drivers, and the Gentoo-suggested command is less than ideal, so let's Bashify it. The first command produces a list of packages, one per line. We can read those into the array named \"args\" using mapfile, stripping trailing newlines with the '-t' option. The resulting array is then expanded into the arguments of the emerge command - an interface to Gentoo's package manager. This type of usage can make for a safe and effective replacement for xargs(1) in certain situations. Unlike xargs, all arguments are guaranteed to be passed to a single invocation of the command with no wordsplitting, pathname expansion, or other monkey business.
# eix --only-names -IC x11-drivers | { mapfile -t args; emerge -av1 \"${args[@]}\" <&1; }\nNote the use of command grouping to keep the emerge command inside the pipe's subshell and within the scope of \"args\". Also note the unusual redirection. This is because the -a flag makes emerge interactive, asking the user for confirmation before continuing, and checking with isatty(3) to abort if stdin isn't pointed at a terminal. Since stdin of the entire command group is still coming from the pipe even though mapfile has read all available input, we just borrow FD 1 as it just so happens to be pointing where we want it. More on this over at greycat's wiki: http://mywiki.wooledge.org/BashFAQ/024
"},{"location":"commands/builtin/mapfile/#the-callback","title":"The callback","text":"This is one of the more unusual features of a Bash builtin. As far as I'm able to tell, the exact behavior is as follows: If defined, as each line is read, the code contained within the string argument to the -C flag is evaluated and executed before the assignment of each array element. There are no restrictions to this string, which can be any arbitrary code, however, two additional \"words\" are automatically appended to the end before evaluation: the index, and corresponding line of data to be assigned to the next array element. Since all this happens before assignment, the callback feature cannot be used to modify the element to be assigned, though it can read and modify any array elements already assigned.
A very simple example might be to use it as a kind of progress bar. This will print a dot for each line read. Note the escaped comment to hide the appended words from printf.
$ printf '%s\\n' {1..5} | mapfile -c 1 -C 'printf . \\#' )\n.....\nReally, the intended usage is for the callback to just contain the name of a function, with the extra words passed to it as arguments. If you're going to use callbacks at all, this is probably the best way because it allows for easy access to the arguments with no ugly \"code in a string\".
$ foo() { echo \"|$1|\"; }; mapfile -n 11 -c 2 -C 'foo' <file\n|2|\n|4|\netc..\nFor the sake of completeness, here are some more complicated examples inspired by a question asked in #bash - how to prepend something to every line of some input, and then output even and odd lines to separate files. This is far from the best possible answer, but hopefully illustrates the callback behavior:
$ { printf 'input%s\\n' {1..10} | mapfile -c 1 -C '>&$(( (${#x[@]} % 2) + 3 )) printf -- \"%.sprefix %s\"' x; } 3>outfile0 4>outfile1\n$ cat outfile{0,1}\nprefix input1\nprefix input3\nprefix input5\nprefix input7\nprefix input9\nprefix input2\nprefix input4\nprefix input6\nprefix input8\nprefix input10\nSince redirects are syntactically allowed anywhere in a command, we put it before the printf to stay out of the way of additional arguments. Rather than opening \"outfile<n>\" for appending on each call by calculating the filename, open an FD for each first and calculate which FD to send output to by measuring the size of x mod 2. The zero-width format specification is used to absorb the index number argument.
Another variation might be to add each of these lines to the elements of separate arrays. I'll leave dissecting this one as an exercise for the reader. This is quite the hack but illustrates some interesting properties of printf -v and mapfile -C (which you should probably never use in real code).
$ y=( 'odd[j]' 'even[j++]' ); printf 'input%s\\n' {1..10} | { mapfile -tc 1 -C 'printf -v \"${y[${#x[@]} % 2]}\" -- \"%.sprefix %s\"' x; printf '%s\\n' \"${odd[@]}\" '' \"${even[@]}\"; }\nprefix input1\nprefix input3\nprefix input5\nprefix input7\nprefix input9\n\nprefix input2\nprefix input4\nprefix input6\nprefix input8\nprefix input10\nThis example based on yet another #bash question illustrates mapfile in combination with read. The sample input is the heredoc to main. The goal is to build a \"struct\" based upon records in the input file made up of the numbers following the colon on each line. Every 3rd line is a key followed by 2 corresponding fields. The showRecord function takes a key and returns the record.
#!/usr/bin/env bash\n\nshowRecord() {\n printf 'key[%d] = %d, %d\\n' \"$1\" \"${vals[@]:keys[$1]*2:2}\"\n}\n\nparseRecords() {\n trap 'unset -f _f' RETURN\n _f() {\n local x\n IFS=: read -r _ x\n ((keys[x]=n++))\n }\n local n\n\n _f\n mapfile -tc2 -C _f \"$1\"\n eval \"$1\"'=(\"${'\"$1\"'[@]##*:}\")' # Return the array with some modification\n}\n\nmain() {\n local -a keys vals\n parseRecords vals\n showRecord \"$1\"\n}\n\nmain \"$1\" <<-\"EOF\"\nfabric.domain:123\nroutex:1\nroutey:2\nfabric.domain:321\nroutex:6\nroutey:4\nEOF\nFor example, running scriptname 321 would output key[321] = 6, 4. Every 2 lines read by mapfile, the function _f is called, which reads one additional line. Since the first line in the file is a key, and _f is responsible for the keys, it gets called first so that mapfile starts by reading the second line of input, calling _f with each subsequent 2 iterations. The RETURN trap is unimportant.
mapfile filling the readline history buffer with calls to the CALLBACK. This was fixed in 4.1 beta.mapfile -n reads an extra line beyond the last line assigned to the array, through Bash. Fixed in 4.2.35.mapfile callbacks could cause a crash if the variable being assigned is manipulated in certain ways. https://lists.gnu.org/archive/html/bug-bash/2013-01/msg00039.html. Fixed in 4.3.FIXME
This is a very big topic that needs experience - please fill in missing information, extend the descriptions, and correct the details if you can!
Bash-Builtin
This is about the Bash-builtin command printf - however, the description should be nearly identical for an external command that follows POSIX\u00ae.
GNU Awk expects a comma after the format string and between each of the arguments of a printf command. For examples, see: code snippet.
Unlike other documentations, I don't want to redirect you to the manual page for the printf() C function family. However, if you're more experienced, that should be the most detailed description for the format strings and modifiers.
Due to conflicting historical implementations of the echo command, POSIX\u00ae recommends that printf is preferred over echo.
The printf command provides a method to print preformatted text similar to the printf() system interface (C function). It's meant as successor for echo and has far more features and possibilities.
Beside other reasons, POSIX\u00ae has a very good argument to recommend it: Both historical main flavours of the echo command are mutual exclusive, they collide. A \"new\" command had to be invented to solve the issue.
printf <FORMAT> <ARGUMENTS...>\nThe text format is given in <FORMAT>, while all arguments the formatstring may point to are given after that, here, indicated by <ARGUMENTS...>.
Thus, a typical printf-call looks like:
printf \"Surname: %s\\nName: %s\\n\" \"$SURNAME\" \"$FIRSTNAME\"\nwhere \"Surname: %s\\nName: %s\\n\" is the format specification, and the two variables are passed as arguments, the %s in the formatstring points to (for every format specifier you give, printf awaits one argument!).
-v VAR If given, the output is assigned to the variable VAR instead of printed to stdout (comparable to sprintf() in some way) The -v Option can't assign directly to array indexes in Bash versions older than Bash 4.1.
Danger
In versions newer than 4.1, one must be careful when performing expansions into the first non-option argument of printf as this opens up the possibility of an easy code injection vulnerability.
$ var='-vx[$(echo hi >&2)]'; printf \"$var\" hi; declare -p x\nhi\ndeclare -a x='([0]=\"hi\")'\n...where the echo can of course be replaced with any arbitrary command. If you must, either specify a hard-coded format string or use -- to signal the end of options. The exact same issue also applies to read, and a similar one to mapfile, though performing expansions into their arguments is less common.
"},{"location":"commands/builtin/printf/#arguments","title":"Arguments","text":"Of course in shell-meaning the arguments are just strings, however, the common C-notations plus some additions for number-constants are recognized to give a number-argument to printf:
N A normal decimal number 0N An octal number 0xN A hexadecimal number 0XN A hexadecimal number \"X (a literal double-quote infront of a character): interpreted as number (underlying codeset) don't forget escaping 'X (a literal single-quote infront of a character): interpreted as number (underlying codeset) don't forget escaping If more arguments than format specifiers are present, then the format string is re-used until the last argument is interpreted. If fewer format specifiers than arguments are present, then number-formats are set to zero, while string-formats are set to null (empty).
Take care to avoid word splitting, as accidentally passing the wrong number of arguments can produce wildly different and unexpected results. See this article.
Attention
When a numerical format expects a number, the internal printf-command will use the common Bash arithmetic rules regarding the base. A command like the following example will throw an error, since 08 is not a valid octal number (00 to 07!):
printf '%d\\n' 08\nThe format string interpretion is derived from the C printf() function family. Only format specifiers that end in one of the letters diouxXfeEgGaAcs are recognized.
To print a literal % (percent-sign), use %% in the format string.
Again: Every format specifier expects an associated argument provided!
These specifiers have different names, depending who you ask. But they all mean the same: A placeholder for data with a specified format:
%b Print the associated argument while interpreting backslash escapes in there %q Print the associated argument shell-quoted, reusable as input %d Print the associated argument as signed decimal number %i Same as %d %o Print the associated argument as unsigned octal number %u Print the associated argument as unsigned decimal number %x Print the associated argument as unsigned hexadecimal number with lower-case hex-digits (a-f) %X Same as %x, but with upper-case hex-digits (A-F) %f Interpret and print the associated argument as floating point number %e Interpret the associated argument as double, and print it in <N>\u00b1e<N> format %E Same as %e, but with an upper-case E in the printed format %g Interprets the associated argument as double, but prints it like %f or %e %G Same as %g, but print it like %E %c Interprets the associated argument as char: only the first character of a given argument is printed %s Interprets the associated argument literally as string %n Assigns the number of characters printed so far to the variable named in the corresponding argument. Can't specify an array index. If the given name is already an array, the value is assigned to the zeroth element. %a Interprets the associated argument as double, and prints it in the form of a C99 hexadecimal floating-point literal. %A Same as %a, but print it like %E %(FORMAT)T output the date-time string resulting from using FORMAT as a format string for strftime(3). The associated argument is the number of seconds since Epoch, or -1 (current time) or -2 (shell startup time). If no corresponding argument is supplies, the current time is used as default %% No conversion is done. Produces a % (percent sign) Some of the mentioned format specifiers can modify their behaviour by getting a format modifier:
"},{"location":"commands/builtin/printf/#modifiers","title":"Modifiers","text":"To be more flexible in the output of numbers and strings, the printf command allows format modifiers. These are specified between the introductory % and the character that specifies the format:
printf \"%50s\\n\" \"This field is 50 characters wide...\"\n<N> Any number: Specifies a minimum field width, if the text to print is shorter, it's padded with spaces, if the text is longer, the field is expanded . The dot: Together with a field width, the field is not expanded when the text is longer, the text is truncated instead. \"%.s\" is an undocumented equivalent for \"%.0s\", which will force a field width of zero, effectively hiding the field from output * The asterisk: the width is given as argument before the string or number. Usage (the \"*\" corresponds to the \"20\"): printf \"%*s\\n\" 20 \"test string\" # \"Alternative format\" for numbers: see table below - Left-bound text printing in the field (standard is right-bound) 0 Pads numbers with zeros, not spaces <space> Pad a positive number with a space, where a minus (-) is for negative numbers + Prints all numbers signed (+ for positive, - for negative) ' For decimal conversions, the thousands grouping separator is applied to the integer portion of the output according to the current LC_NUMERIC The \"alternative format\" modifier #:
%#o The octal number is printed with a leading zero, unless it's zero itself %#x, %#X The hex number is printed with a leading \"0x\"/\"0X\", unless it's zero %#g, %#G The float number is printed with trailing zeros until the number of digits for the current precision is reached (usually trailing zeros are not printed) all number formats except %d, %o, %x, %X Always print a decimal point in the output, even if no digits follow it"},{"location":"commands/builtin/printf/#precision","title":"Precision","text":"The precision for a floating- or double-number can be specified by using .<DIGITS>, where <DIGITS> is the number of digits for precision. If <DIGITS> is an asterisk (*), the precision is read from the argument that precedes the number to print, like (prints 4,3000000000):
printf \"%.*f\\n\" 10 4,3\nThe format .*N to specify the N'th argument for precision does not work in Bash.
For strings, the precision specifies the maximum number of characters to print (i.e., the maximum field width). For integers, it specifies the number of digits to print (zero-padding!).
"},{"location":"commands/builtin/printf/#escape-codes","title":"Escape codes","text":"These are interpreted if used anywhere in the format string, or in an argument corresponding to a %b format.
\\\\ Prints the character \\ (backslash) \\a Prints the alert character (ASCII code 7 decimal) \\b Prints a backspace \\f Prints a form-feed \\n Prints a newline \\r Prints a carriage-return \\t Prints a horizontal tabulator \\v Prints a vertical tabulator \\\" Prints a ' \\? Prints a ? <NNN> Interprets <NNN> as octal number and prints the corresponding character from the character set \\0<NNN> same as <NNN> \\x<NNN> Interprets <NNN> as hexadecimal number and prints the corresponding character from the character set (3 digits) \\u<NNNN> same as \\x<NNN>, but 4 digits \\U<NNNNNNNN> same as \\x<NNN>, but 8 digits The following additional escape and extra rules apply only to arguments associated with a %b format:
\\c Terminate output similarly to the \\c escape used by echo -e. printf produces no additional output after coming across a \\c escape in a %b argument. \\', \\\", and \\? are not removed.\\0 may contain up to four digits. (POSIX specifies up to three).These are also respects in which %b differs from the escapes used by \\$\\'...\\' style quoting.
printf \"%d\\n\" 0x41printf \"%d\\n\" -0x41printf \"%+d\\n\" 0x41printf \"%o\\n\" 65printf \"%05o\\n\" 65 (5 characters width, padded with zeros)printf \"%d\\n\"Aprintf \"%d\\n\" \\'Aprintf \"%d\\n\" \"'A\"GREETERprintf -v GREETER \"Hello %s\" \"$LOGNAME\"tput to get the current line widthprintf \"%*s\\n\" $(tput cols) \"Hello world!\"This small loop prints all numbers from 0 to 127 in
for ((x=0; x <= 127; x++)); do\n printf '%3d | %04o | 0x%02x\\n' \"$x\" \"$x\" \"$x\"\ndone\nThis code here will take a common MAC address and rewrite it into a well-known format (regarding leading zeros or upper/lowercase of the hex digits, ...):
the_mac=\"0:13:ce:7:7a:ad\"\n\n# lowercase hex digits\nthe_mac=\"$(printf \"%02x:%02x:%02x:%02x:%02x:%02x\" 0x${the_mac//:/ 0x})\"\n\n# or the uppercase-digits variant\nthe_mac=\"$(printf \"%02X:%02X:%02X:%02X:%02X:%02X\" 0x${the_mac//:/ 0x})\"\nThis code was found in Solaris manpage for echo(1).
Solaris version of /usr/bin/echo is equivalent to:
printf \"%b\\n\" \"$*\"\nSolaris /usr/ucb/echo is equivalent to:
if [ \"X$1\" = \"X-n\" ]\nthen\n shift\n printf \"%s\" \"$*\"\nelse\n printf \"%s\\n\" \"$*\"\nfi\nWorking off the replacement echo, here is a terse implementation of prargs:
printf '\"%b\"\\n' \"$0\" \"$@\" | nl -v0 -s\": \"\nA small trick: Combining printf and parameter expansion to draw a line
length=40\nprintf -v line '%*s' \"$length\"\necho ${line// /-}\nor:
length=40\neval printf -v line '%.0s-' {1..$length}\nThe %(...)T format string is a direct interface to strftime(3).
$ printf 'This is week %(%U/%Y)T.\\n' -1\nThis is week 52/2010.\nPlease read the manpage of strftime(3) to get more information about the supported formats.
Awk also derives its printf() function from C, and therefore has similar format specifiers. However, in all versions of awk the space character is used as a string concatenation operator, so it cannot be used as an argument separator. Arguments to awk printf must be separated by commas. Some versions of awk do not require printf arguments to be surrounded by parentheses, but you should use them anyway to provide portability.
In the following example, the two strings are concatenated by the intervening space so that no argument remains to fill the format.
$ echo \"Foo\" | awk '{ printf \"%s\\n\" $1 }'\nawk: (FILENAME=- FNR=1) fatal: not enough arguments to satisfy format string\n `%s\nFoo'\n ^ ran out for this one\nSimply replacing the space with a comma and adding parentheses yields correct awk syntax.
$ echo \"Foo\" | awk '{ printf( \"%s\\n\", $1 ) }'\nFoo\nWith appropriate metacharacter escaping the bash printf can be called from inside awk (as from perl and other languages that support shell callout) as long as you don't care about program efficiency or readability.
echo \"Foo\" | awk '{ system( \"printf \\\"%s\\\\n \\\" \\\"\" $1 \"\\\"\" ) }'\nFoo\nThe a, A, e, E, f, F, g, and G conversions are supported by Bash, but not required by POSIX.
There is no wide-character support (wprintf). For instance, if you use %c, you're actually asking for the first byte of the argument. Likewise, the maximum field width modifier (dot) in combination with %s goes by bytes, not characters. This limits some of printf's functionality to working with ascii only. ksh93's printf supports the L modifier with %s and %c (but so far not %S or %C) in order to treat precision as character width, not byte count. zsh appears to adjust itself dynamically based upon LANG and LC_CTYPE. If LC_CTYPE=C, zsh will throw \"character not in range\" errors, and otherwise supports wide characters automatically if a variable-width encoding is set for the current locale.
Bash recognizes and skips over any characters present in the length modifiers specified by POSIX during format string parsing.
builtins/printf.def
#define LENMODS \"hjlLtz\"\n...\n/* skip possible format modifiers */\nmodstart = fmt;\nwhile (*fmt && strchr (LENMODS, *fmt))\nfmt++;\n/usr/bin/printf or equivalent. The mksh maintainer recommends using print. The development version (post- R40f) adds a new parameter expansion in the form of ${name@Q} which fills the role of printf %q -- expanding in a shell-escaped format. printf -v functionality can be closely matched by var=$(printf ...) without a big performance hit.# Illustrates Bash-like behavior. Redefining printf is usually unnecessary / not recommended.\nfunction printf {\n case $1 in\n -v)\n shift\n nameref x=$1\n shift\n x=$(command printf \"$@\")\n ;;\n *)\n command printf \"$@\"\n esac\n}\nbuiltin cut\nprint $$\nprintf -v 'foo[2]' '%d\\n' \"$(cut -d ' ' -f 1 /proc/self/stat)\"\ntypeset -p foo\n# 22461\n# typeset -a foo=([2]=22461)\nprint may be useful for ksh compatibility and to overcome some of echo's portability pitfalls. Bash, ksh93, and zsh's print have an -f option which takes a printf format string and applies it to the remaining arguments. Bash lists the synopsis as: print: print [-Rnprs] [-u unit] [-f format] [arguments]. However, only -Rrnfu are actually functional. Internally, -p is a noop (it doesn't tie in with Bash coprocs at all), and -s only sets a flag but has no effect. -Cev are unimplemented.printf -v way is slightly different to the way using command-substitution. Command substitution removes trailing newlines before substituting the text, printf -v preserves all output.printf examplesread something about read here!
"},{"location":"commands/builtin/read/#synopsis","title":"Synopsis","text":"read [-ers] [-u <FD>] [-t <TIMEOUT>] [-p <PROMPT>] [-a <ARRAY>] [-n <NCHARS>] [-N <NCHARS>] [-d <DELIM>] [-i <TEXT>] [<NAME...>]\nThe read builtin reads one line of data (text, user input, ...) from standard input or a supplied filedescriptor number into one or more variables named by <NAME...>.
Since Bash 4.3-alpha, read skips any NUL (ASCII code 0) characters in input.
If <NAME...> is given, the line is word-split using IFS variable, and every word is assigned to one <NAME>. The remaining words are all assigned to the last <NAME> if more words than variable names are present.
Info
If no <NAME> is given, the whole line read (without performing word-splitting!) is assigned to the shell variable REPLY. Then, REPLY really contains the line as it was read, without stripping pre- and postfix spaces and other things!
while read -r; do\n printf '\"%s\"\\n' \"$REPLY\"\ndone <<<\" a line with prefix and postfix space \"\nIf a timeout is given, or if the shell variable TMOUT is set, it is counted from initially waiting for input until the completion of input (i.e. until the complete line is read). That means the timeout can occur during input, too.
"},{"location":"commands/builtin/read/#options","title":"Options","text":"Option Description-a <ARRAY> read the data word-wise into the specified array <ARRAY> instead of normal variables -d <DELIM> recognize <DELIM> as data-end, rather than <newline> -e on interactive shells: use Bash's readline interface to read the data. Since version 5.1-alpha, this can also be used on specified file descriptors using -u -i <STRING> preloads the input buffer with text from <STRING>, only works when Readline (-e) is used -n <NCHARS> reads <NCHARS> characters of input, then quits -N <NCHARS> reads <NCHARS> characters of input, ignoring any delimiter, then quits -p <PROMPT> the prompt string <PROMPT> is output (without a trailing automatic newline) before the read is performed -r raw input - disables interpretion of backslash escapes and line-continuation in the read data -s secure input - don't echo input if on a terminal (passwords!) -t <TIMEOUT> wait for data <TIMEOUT> seconds, then quit (exit code 1). Fractional seconds (\"5.33\") are allowed since Bash 4. A value of 0 immediately returns and indicates if data is waiting in the exit code. Timeout is indicated by an exit code greater than 128. If timeout arrives before data is read completely (before end-of-line), the partial data is saved. -u <FD> use the filedescriptor number <FD> rather than stdin (0) When both, -a <ARRAY> and a variable name <NAME> is given, then the array is set, but not the variable.
Of course it's valid to set individual array elements without using -a:
read MYARRAY[5]\nWarning
Reading into array elements using the syntax above may cause pathname expansion to occur.
Example: You are in a directory with a file named x1, and you want to read into an array x, index 1 with
read x[1]\nthen pathname expansion will expand to the filename x1 and break your processing!
Even worse, if nullglob is set, your array/index will disappear.
To avoid this, either disable pathname expansion or quote the array name and index:
read 'x[1]'\n-t) !=0 invalid filedescriptor supplied to -u !=0 end-of-file reached"},{"location":"commands/builtin/read/#read-without-r","title":"read without -r","text":"Essentially all you need to know about -r is to ALWAYS use it. The exact behavior you get without -r is completely useless even for weird purposes. It basically allows the escaping of input which matches something in IFS, and also escapes line continuations. It's explained pretty well in the POSIX read spec.
2012-05-23 13:48:31 geirha it should only remove the backslashes, not change \\n and \\t and such into newlines and tabs\n2012-05-23 13:49:00 ormaaj so that's what read without -r does?\n2012-05-23 13:49:16 geirha no, -r doesn't remove the backslashes\n2012-05-23 13:49:34 ormaaj I thought read <<<'str' was equivalent to read -r <<<$'str'\n2012-05-23 13:49:38 geirha # read x y <<< 'foo\\ bar baz'; echo \"<$x><$y>\"\n2012-05-23 13:49:40 shbot geirha: <foo bar><baz>\n2012-05-23 13:50:32 geirha no, read without -r is mostly pointless. Damn bourne\n2012-05-23 13:51:08 ormaaj So it's mostly (entirely) used to escape spaces\n2012-05-23 13:51:24 ormaaj and insert newlines\n2012-05-23 13:51:47 geirha ormaaj: you mostly get the same effect as using \\ at the prompt\n2012-05-23 13:52:04 geirha echo \\\" outputs a \" , read x <<< '\\\"' reads a \"\n2012-05-23 13:52:32 ormaaj oh weird\n2012-05-23 13:52:46 * ormaaj struggles to think of a point to that...\n2012-05-23 13:53:01 geirha ormaaj: ask Bourne :P\n2012-05-23 13:53:20 geirha (not Jason)\n2012-05-23 13:53:56 ormaaj hm thanks anyway :)\nA rudimentary replacement for the cat command: read lines of input from a file and print them on the terminal.
opossum() {\n while read -r; do\n printf \"%s\\n\" \"$REPLY\"\n done <\"$1\"\n}\nNote: Here, read -r and the default REPLY is used, because we want to have the real literal line, without any mangeling. printf is used, because (depending on settings), echo may interpret some baskslash-escapes or switches (like -n).
Remember the MSDOS pause command? Here's something similar:
pause() {\n local dummy\n read -s -r -p \"Press any key to continue...\" -n 1 dummy\n}\nNotes:
-s to suppress terminal echo (printing)-r to not interpret special characters (like waiting for a second character if somebody presses the backslash)Read can be used to split a string:
var=\"one two three\"\nread -r col1 col2 col3 <<< \"$var\"\nprintf \"col1: %s col2: %s col3 %s\\n\" \"$col1\" \"$col2\" \"$col3\"\nTake care that you cannot use a pipe:
echo \"$var\" | read col1 col2 col3 # does not work!\nprintf \"col1: %s col2: %s col3 %s\\n\" \"$col1\" \"$col2\" \"$col3\"\nWhy? because the commands of the pipe run in subshells that cannot modify the parent shell. As a result, the variables col1, col2 and col3 of the parent shell are not modified (see article: processtree).
If the variable has more fields than there are variables, the last variable get the remaining of the line:
read col1 col2 col3 <<< \"one two three four\"\nprintf \"%s\\n\" \"$col3\" #prints three four\nBy default reads separates the line in fields using spaces or tabs. You can modify this using the special variable IFS, the Internal Field Separator.
IFS=\":\" read -r col1 col2 <<< \"hello:world\"\nprintf \"col1: %s col2: %s\\n\" \"$col1\" \"$col2\"\nHere we use the var=value command syntax to set the environment of read temporarily. We could have set IFS normally, but then we would have to take care to save its value and restore it afterward (OLD=$IFS IFS=\":\"; read ....;IFS=$OLD).
The default IFS is special in that 2 fields can be separated by one or more space or tab. When you set IFS to something besides whitespace (space or tab), the fields are separated by exactly one character:
IFS=\":\" read -r col1 col2 col3 <<< \"hello::world\"\nprintf \"col1: %s col2: %s col3 %s\\n\" \"$col1\" \"$col2\" \"$col3\"\nSee how the :: in the middle infact defines an additional empty field.
The fields are separated by exactly one character, but the character can be different between each field:
IFS=\":|@\" read -r col1 col2 col3 col4 <<< \"hello:world|in@bash\"\nprintf \"col1: %s col2: %s col3 %s col4 %s\\n\" \"$col1\" \"$col2\" \"$col3\" \"$col4\"\nasksure() {\necho -n \"Are you sure (Y/N)? \"\nwhile read -r -n 1 -s answer; do\n if [[ $answer = [YyNn] ]]; then\n [[ $answer = [Yy] ]] && retval=0\n [[ $answer = [Nn] ]] && retval=1\n break\n fi\ndone\n\necho # just a final linefeed, optics...\n\nreturn $retval\n}\n\n### using it\nif asksure; then\n echo \"Okay, performing rm -rf / then, master....\"\nelse\n echo \"Pfff...\"\nfi\nNote: The -i option was introduced with Bash 4
read -e -p \"Enter the path to the file: \" -i \"/usr/local/etc/\" FILEPATH\nThe user will be prompted, he can just accept the default, or edit it.
"},{"location":"commands/builtin/read/#multichar-ifs-parsing-a-simple-datetime-string","title":"Multichar-IFS: Parsing a simple date/time string","text":"Here, IFS contains both, a colon and a space. The fields of the date/time string are recognized correctly.
datetime=\"2008:07:04 00:34:45\"\nIFS=\": \" read -r year month day hour minute second <<< \"$datetime\"\n-r option (raw read); -r is not only POSIX, you can find it in earlier Bourne source codeREPLY is not POSIX\u00ae, you need to set IFS to the empty string to get the whole line for shells that don't know REPLY. while IFS= read -r line; do ... done < text.txtfixed in 4.2-rc1\u00a0\u21a9
readonly [-p] [-a] [-A] [-f] [NAME[=VALUE] ...]\nThe readonly builtin command is used to mark variables or functions as read-only, which means unchangeable. This implies that it can't be unset anymore. A readonly variable may not be redefined in child scopes. A readonly global may not be redefined as a function local variable. Simple command environment assignments may not reference readonly variables.
-a refer to normal arrays -A refer to associative arrays -f refer to functions -p print all read-only variables or functions, -a, -A and -f can be used to filter. The output is reusable as input An argument of -- disables further option processing.
NAME is invalid"},{"location":"commands/builtin/readonly/#examples","title":"Examples","text":""},{"location":"commands/builtin/readonly/#portability-considerations","title":"Portability considerations","text":"-p option is specifiedreturn [N]\nThe return command returns from a shell function.
If N is given, the return code to the caller is set to N. If not, the returned status the the status of the most recently executed command (i.e. $?).
There are no options.
"},{"location":"commands/builtin/return/#exit-status","title":"Exit status","text":"If everything is okay, the return command doesn't come back. If it comes back, there was a problem in doing the return.
return was called while not being in a shell function or sourced file"},{"location":"commands/builtin/return/#examples","title":"Examples","text":""},{"location":"commands/builtin/return/#portability-considerations","title":"Portability considerations","text":""},{"location":"commands/builtin/return/#see-also","title":"See also","text":"FIXME
incomplete text, examples, maybe extended description
"},{"location":"commands/builtin/set/#synopsis","title":"Synopsis","text":"set [--abefhkmnptuvxBCHP] <-o OPTIONNAME> [-][--] <POSPARAMS>\nset is primarily made to
<POSPARAMS>Without any options, set displays all shell- and environment-variables (only is POSIX-mode) in a re-usable format NAME=VALUE.
All attributes below can be switched on using -X and switched off using +X. This is done because of the historical meaning of the - to set flags (true for most commands on UNIX\u00ae).
-a allexport Automatically mark new and altered variables to be exported to subsequent environments. -b notify Don't wait for the next prompt to print when showing the reports for a terminated background job (only with job control) -e errexit When set, the shell exits when a simple command in a command list exits non-zero (FALSE). This is not done in situations, where the exit code is already checked (if, while, until, ||, &&) -f noglob Disable pathname expansion (globbing) -h hashall Remembers the location of commands when they're called (hashing). Enabled by default. -k keyword Allows to place environment-assignments everywhere in the commandline, not only infront of the called command. -m monitor Monitor mode. With job control, a short descriptive line is printed when a backgroud job ends. Default is \"on\" for interactive shells (with job control). -n noexec Read and parse but do not execute commands - useful for checking scripts for syntax errors. Ignored by interactive shells. -o Set/unset attributes with long option names, e.g. set -o noglob. The long option names are in the second column of this table. If no option name is given, all options are printed with their current status. -p privileged Turn on privileged mode. -t onecmd Exit after reading and executing one command. -u nounset Treat unset variables as an error when performing parameter expansion. Non-interactive shells exit on this error. -v verbose Print shell input lines as they are read - useful for debugging. -x xtrace Print commands just before execution - with all expansions and substitutions done, and words marked - useful for debugging. -B braceexpand The shell performs brace expansion This is on by default. -C noclobber Don't overwrite files on redirection operations. You can override that by specifying the >| redirection operator when needed. See redirection -E errtrace ERR-traps are inherited by by shell functions, command substitutions, and commands executed in a subshell environment. -H histexpand Enable !-style history expansion. Defaults to on for interactive shells. -P physical Don't follow symlinks when changing directories - use the physical filesystem structure. -T functrace DEBUG- and RETURN-traps are inherited by subsequent environments, like -E for ERR trap. - \"End of options\" - all following arguments are assigned to the positional parameters, even when they begin with a dash. -x and -v options are turned off. Positional parameters are unchanged (unlike using --!) when no further arguments are given. -- If no arguments follow, the positional parameters are unset. With arguments, the positional parameters are set, even if the strings begin with a - (dash) like an option. Long options usable with -o without a short equivalent emacs Use an emacs-style command line editing interface. This is enabled by default when the shell is interactive, unless the shell is started with --noediting option. history If set, command historization is done (enabled by default on interactive shells) ignoreeof The effect is as if the shell command IGNOREEOF=10 had been executed. See shell variables. nolog (currently ignored) pipefail If set, the exit code from a pipeline is different from the normal (\"last command in pipeline\") behaviour: TRUE when no command failed, FALSE when something failed (code of the rightmost command that failed) posix When set, Bash runs in POSIX mode. vi Enables a vi-style command line editing interface."},{"location":"commands/builtin/set/#examples","title":"Examples","text":"Tag a part of a shell script to output debugging information (-x):
#!/bin/bash\n...\nset -x # on\n...\nset +x # off\n...\nset and its basic behaviour and options are specified by POSIX\u00ae. However, options that influence Bash-specific things are not portable, naturally.
shift [n]\nThe shift builtin command is used to \"shift\" the positional parameters by the given number n or by 1, if no number is given.
This means, the number and the position of the positional parameters are changed. The very first positional parameter is discarded, the second becomes the first one, etc.
Imagine the following set of positional parameters ($1 to $4):
When you use shift 1, they will be changed to:
The special parameter $# will reflect the final number of positional parameters.
If the number given is 0, no changes are made to the positional parameters.
"},{"location":"commands/builtin/shift/#options","title":"Options","text":"There are no options.
"},{"location":"commands/builtin/shift/#return-status","title":"Return status","text":"Status Reason 0 no error 1 non-numeric argument 1 given number (or the default 1) is bigger than the number of actually present positional parameters 1 given number is negative"},{"location":"commands/builtin/shift/#examples","title":"Examples","text":""},{"location":"commands/builtin/shift/#portability-considerations","title":"Portability considerations","text":"shift builtin command is specified by POSIX\u00ae.Many shells will throw a fatal error when attempting to shift more than the number of positional parameters. POSIX does not require that behavior. Bash (even in POSIX mode) and Zsh return 1 when there are no args, and no error output is produced unless the shift_verbose shopt option is enabled. Ksh93, pdksh, posh, mksh, and dash, all throw useless fatal shell errors.
$ dash -c 'f() { if shift; then echo \"$1\"; else echo \"no args\"; fi; }; f'\n dash: 1: shift: can't shift that many\ncommand builtin to suppress fatal errors caused by special builtins. $ dash -c 'f() { if command shift 2>/dev/null; then echo \"$1\"; else echo \"no args\"; fi; }; f'\nno args\n$ mksh -c 'f() { if ! ${1+false} && shift; then echo \"$1\"; else echo \"no args\"; fi; }; f'\nno args\nshift or command builtins. Fixed. (Thanks!) Perhaps almost as bad as the above, busybox sh's shift always returns success, even when attempting to shift beyond the final argument.
$ bb -c 'f() { if shift; then echo \"$1\"; else echo \"no args\"; fi; }; f'\n(no output)\nThe shopt builtin manages shell options, a set of boolean (on/off) configuration variables that control the behaviour of the shell.
shopt [-pqsu] [-o] <OPTNAME...>\nNote: Some of these options and other shell options can also be set with the set builtin.
"},{"location":"commands/builtin/shopt/#options","title":"Options","text":"Option Description-o Restrict the values of <OPTNAME...> to only those also known by the set builtin -p Print all shell options and their current value. Default. -q Quiet mode. Set exit code if named option is set. For multiple options: TRUE if all options are set, FALSE otherwise -s Enable (set) the shell options named by <OPTNAME...> or list all enabled options if no names are given -u Disabe (unset) the shell options named by <OPTNAME...> or list all disabled options if no names are given As noted above, if only -s or -u are given without any option names, only the currently enabled (-s) or disabled (-u) options are printed.
When listing options, the exit code is TRUE (0), if all options are enabled, FALSE otherwise.
When setting/unsetting an option, the exit code is TRUE unless the named option doesn't exitst.
Enable the nullglob option:
shopt -s nullglob\nThe shopt command is not portable accross different shells.
trap [-lp] [[ARGUMENT] SIGNAL]\nThe trap command is used to \"trap\" signals and other events. In this context, \"trapping\" means to install handler code.
The shell code ARGUMENT is to be read and executed whenever the shell receives a signal or another event SIGNAL. The given SIGNAL specification can be
SIGTERMTERMtrap -l), e.g. 15EXITWithout any options or operands, trap prints a list of installed traps in a reusable format (equivalent to the -p option).
Special ARGUMENTs
ARGUMENT is absent or - (dash), the signal/event handler is reset to its original valueARGUMENT is the null string, the signal/event is ignoredSpecial events
Name Code DescriptionEXIT 0 executed on shell exit DEBUG executed before every simple command RETURN executed when a shell function or a sourced code finishes executing ERR executed each time a command's failure would cause the shell to exit when the -e option (errexit) is enabled"},{"location":"commands/builtin/trap/#options","title":"Options","text":"Option Description -l print a list of signal names and their corresponding numbers -p display the trap commands associated with each signal specification in a reusable format"},{"location":"commands/builtin/trap/#return-status","title":"Return status","text":"Status Reason 0 no error/success !=0 invalid option !=0 invalid signal specification"},{"location":"commands/builtin/trap/#examples","title":"Examples","text":""},{"location":"commands/builtin/trap/#list-installed-traps","title":"List installed traps","text":"trap\ntrap '' INT\ntrap is specified by POSIX(R) without the -l and -p optionsEXIT (0) is valid as an event-e (errexit) optionunset [-f|v] [-n] [NAME ...]\nThe unset builtin command is used to unset values and attributes of shell variables and functions. Without any option, unset tries to unset a variable first, then a function.
-f treats each NAME as a function name -v treats each NAME as a variable name -n treats each NAME as a name reference and unsets the variable itself rather than the variable it references"},{"location":"commands/builtin/unset/#exit-status","title":"Exit status","text":"Status Reason 0 no error !=0 invalid option !=0 invalid combination of options (-v and -f) !=0 a given NAME is read-only"},{"location":"commands/builtin/unset/#examples","title":"Examples","text":"unset -v EDITOR\n\nunset -f myfunc1 myfunc2\nIn bash, unset has some interesting properties due to its unique dynamic scope. If a local variable is both declared and unset (by calling unset on the local) from within the same function scope, then the variable appears unset to that scope and all child scopes until either returning from the function, or another local variable of the same name is declared underneath where the original variable was unset. In other words, the variable looks unset to everything until returning from the function in which the variable was set (and unset), at which point variables of the same name from higher scopes are uncovered and accessible once again.
If however unset is called from a child scope relative to where a local variable has been set, then the variable of the same name in the next-outermost scope becomes visible to its scope and all children - as if the variable that was unset was never set to begin with. This property allows looking upwards through the stack as variable names are unset, so long as unset and the local it unsets aren't together in the same scope level.
Here's a demonstration of this behavior.
#!/usr/bin/env bash\n\nFUNCNEST=10\n\n# Direct recursion depth.\n# Search up the stack for the first non-FUNCNAME[1] and count how deep we are.\ncallDepth() {\n # Strip \"main\" off the end of FUNCNAME[@] if current function is named \"main\" and\n # Bash added an extra \"main\" for non-interactive scripts.\n if [[ main == !(!(\"${FUNCNAME[1]}\")|!(\"${FUNCNAME[-1]}\")) && $- != *i* ]]; then\n local -a 'fnames=(\"${FUNCNAME[@]:1:${#FUNCNAME[@]}-2}\")'\n else\n local -a 'fnames=(\"${FUNCNAME[@]:1}\")'\n fi\n\n if (( ! ${#fnames[@]} )); then\n printf 0\n return\n fi\n\n local n\n while [[ $fnames == ${fnames[++n]} ]]; do\n :\n done\n\n printf -- $n\n}\n\n# This function is the magic stack walker.\nunset2() {\n unset -v -- \"$@\"\n}\n\nf() {\n local a\n if (( (a=$(callDepth)) <= 4 )); then\n (( a == 1 )) && unset a\n (( a == 2 )) && declare -g a='global scope yo'\n f\n else\n trap 'declare -p a' DEBUG\n unset2 a # declare -- a=\"5\"\n unset a a # declare -- a=\"4\"\n unset a # declare -- a=\"2\"\n unset a # ./unset-tests: line 44: declare: a: not found\n : # declare -- a=\"global scope yo\"\n fi\n}\n\na='global scope'\nf\n\n# vim: set fenc=utf-8 ff=unix ts=4 sts=4 sw=4 ft=sh nowrap et:\noutput:
declare -- a=\"5\"\ndeclare -- a=\"4\"\ndeclare -- a=\"2\"\n./unset-tests: line 44: declare: a: not found\ndeclare -- a=\"global scope yo\"\nSome things to observe:
unset2 is only really needed once. We remain 5 levels deep in f's for the remaining unset calls, which peel away the outer layers of a's.Like several other Bash builtins that take parameter names, unset expands its arguments.
~ $ ( a=({a..d}); unset 'a[2]'; declare -p a )\ndeclare -a a='([0]=\"a\" [1]=\"b\" [3]=\"d\")'\nAs usual in such cases, it's important to quote the args to avoid accidental results such as globbing.
~ $ ( a=({a..d}) b=a c=d d=1; set -x; unset \"${b}[\"{2..3}-c\\]; declare -p a )\n+ unset 'a[2-1]' 'a[3-1]'\n+ declare -p a\ndeclare -a a='([0]=\"a\" [3]=\"d\")'\nOf course hard to follow indirection is still possible whenever arithmetic is involved, also as shown above, even without extra expansions.
In Bash, the unset builtin only evaluates array subscripts if the array itself is set.
~ $ ( unset -v 'a[$(echo a was set >&2)0]' )\n ~ $ ( a=(); unset -v 'a[$(echo a was set >&2)0]' )\na was set\nQuoting POSIX:
If neither -f nor -v is specified, name refers to a variable; if a variable by that name does not exist, it is unspecified whether a function by that name, if any, shall be unset.\nTherefore, it is recommended to explicitly specify -f or -v when using unset. Also, I prefer it as a matter of style.
unset utilitywait [-f] [-n] [-p VARNAME] [ID...]\nThe wait builtin command is used to wait for job completion and return exit status.
ID is a job specification, it waits for all processes in the pipeline of this jobID is given-n option)-f option)ID may be an operating system process identifier or a shell job specification.
-n Waits for \"the next\" child to exit (as opposed to \"all children\" without this option). Accepts a list of IDs (jobs) -f Waits for the termination of the given ID (instead of waiting for a status change only) -p VARNAME When waiting for a list (-n) or all jobs, writes the job ID to the job that was actually terminated into the variable VARNAME"},{"location":"commands/builtin/wait/#return-status","title":"Return status","text":"The return status is the return status of the job waited for, or
Status Reason 0 waited for all jobs in shell's job list 1 the givenID is not a valid job or process ID"},{"location":"commands/builtin/wait/#examples","title":"Examples","text":""},{"location":"commands/builtin/wait/#portability-considerations","title":"Portability considerations","text":""},{"location":"commands/builtin/wait/#see-also","title":"See also","text":""},{"location":"dict/directory/","title":"Directory","text":"In terms of UNIX\u00ae, a directory is a special file which contains a list of hardlinks to other files. These other files also can be directories of course, so it's possible to create a \\\"hierarchy of directories\\\" - the UNIX\u00ae-typical filesystem structure.
The structure begins at the special directory / (root directory) and all other directory entries are subdirectories of it.
The options of UNIX\u00ae utilities usually are introduced with a dash (-) character.
This is problematic when a non-option argument has to be specified that begins with a dash. A common example for this are filenames.
Many utilities use the convention to specify two consecutive dashes (--) to signal \\\"end of options at this point\\\". Beyond this tag, no options are processed anymore, even if an argument begins with a dash.
Example: You want to list (ls) the file with the name -hello. With common option processing, this could end up in the ls-options -h, -e, -l and -o and probably in an error message about invalid options. You use this to avoid the wrong option processing:
ls -- -hello\nPOSIX\u00ae specifies that every utility should follow this rule (see ch. 12.2 Utility Syntax Guidelines), except
echo (historical reasons)test (obvious parsing reasons)The exit status is a numeric value that is returned by a program to the calling program or shell. In C programs, this is represented by the return value of the main() function or the value you give to exit(3). The only part of the number that matters are the least significant 8 bits, which means there are only values from 0 to 255.
In the shell, every operation generates an exit status (return status), even if no program is called. An example for such an operation is a redirection.
The parameter to the
exit (exit the shell/script)return (return from a function)builtin commands serve the purpose of giving the exit status to the calling component.
This - and only this - makes it possible to determinate the success or failure of an operation. For scripting, always set exit codes.
"},{"location":"dict/exit_status/#values","title":"Values","text":"The code is a number between 0 and 255, where the part from 126 to 255 is reserved to be used by the Bash shell directly or for special purposes, like reporting a termination by a signal:
Code Description 0 success 1-255 failure (in general) 126 the requested command (file) can't be executed (but was found) 127 command (file) not found 128 according to ABS it's used to report an invalid argument to the exit builtin, but I wasn't able to verify that in the source code of Bash (see code 255) 128+N the shell was terminated by the signal N (also used like this by various other programs) 255 wrong argument to the exit builtin (see code 128)The lower codes 0 to 125 are not reserved and may be used for whatever the program likes to report. A value of 0 means successful termination, a value not 0 means unsuccessful termination. This behavior (== 0, != 0) is also what Bash reacts on in some code flow control statements like if or while.
Tables of shell behavior involving non-portable side-effects or common bugs with exit statuses. Note heirloom doesn't support pipeline negation (! pipeline).
:; : `false` `echo $? >&2`1 1 1 1 0 0 0 0 1
false; eval; echo $?0 0 0 0 0 1 0 1 0
x=`false` eval echo $?1 1 1 1 0 0 0 0 1
eval echo \\$? <&0`false`1 1 1 1 0 0 0 0 1
while :; do ! break; done; echo $?1 1 1 1 0 0 1 1 -
false; :\necho $?discussion\n1\n1\n1\n0\n1\n1\n1\n1\n\n\n(exit 2); for x in \"`exit 3`\"; do echo $?; done\n3\n3\n3\n3\n2\n2\n0\n0\n3"},{"location":"dict/exit_status/#functions","title":"functions","text":"Measuring side-effects during the function call, during return, and\ntransparency of the return builtin.
\n\n\n\ntest\nbash\nbash(POSIX)\nzsh(emulate ksh)\nksh93\nmksh\nposh\ndash\nbusybox\nheirloom\n\n\n\n\nf() { echo $?; }; :; f `false`\n1\n1\n1\n1\n0\n0\n0\n0\n1\n\n\nf() { return; }; false; f; echo $?\n1\n1\n1\n0\n1\n1\n1\n1\n1\n\n\nf() { return $?; }; false; f; echo $?\n1\n1\n1\n1\n1\n1\n1\n1\n1\n\n\nf() { ! return; }; f; echo $?\n0\n0\n1\n0\n0\n0\n1\n1\n-\n\n\nf() { ! return; }; false; f; echo $?\n1\n1\n0\n0\n1\n1\n0\n0\n-\n\n\nf() { return; }; x=`false` f; echo $?\n1\n1\n1\n1\n0\n0\n0\n0\n0\n\n\nf() { return; }; f <&0`false`; echo $?\n1\n1\n1\n1\n0\n0\n0\n0\n1\n\n\nf() { x=`false` return; }; f; echo $?\n1\n1\n1\n0\n0\n0\n0\n0\n1\n\n\nf() { return <&0`false`; }; f; echo $?\n1\n1\n1\n0\n0\n0\n0\n0\n1\n\n\nf() { x=`false` return <&0`false`; }; f; echo $?\n1\n1\n1\n1\n0\n0\n0\n0\n1"},{"location":"dict/exit_status/#caseesac","title":"case..esac","text":"Statuses measured within the command and after, with matching and\nnon-matching patterns.
\n\n\n\ntest\nbash\nbash(POSIX)\nzsh(emulate ksh)\nksh93\nmksh\nposh\ndash\nbusybox\nheirloom\n\n\n\n\n(exit 2); case x in x) echo $?;; esac\n2\n2\n0\n2\n2\n2\n0\n0\n2\n\n\n(exit 2); case `exit 3`x in x) echo $?;; esac\n3\n3\n0\n3\n2\n2\n0\n0\n3\n\n\n(exit 2); case x in `exit 4`x) echo $?;; esac\n4\n4\n4\n4\n2\n2\n0\n0\n4\n\n\n(exit 2); case `exit 3`x in `exit 4`x) echo $?;; esac\n4\n4\n4\n4\n2\n2\n0\n0\n4\n\n\n(exit 2); case x in x);; esac; echo $?\n0\n0\n0\n0\n0\n0\n0\n0\n2\n\n\n(exit 2); case x in \"\");; esac; echo $?\n0\n0\n0\n0\n0\n0\n0\n0\n2\n\n\n(exit 2); case `exit 3`x in x);; esac; echo $?\n0\n0\n0\n3\n0\n0\n0\n0\n3\n\n\n(exit 2); case `exit 3`x in \"\");; esac; echo $?\n0\n0\n0\n3\n0\n0\n0\n0\n3\n\n\n(exit 2); case x in `exit 4`x);; esac; echo $?\n0\n0\n0\n4\n0\n0\n0\n0\n4\n\n\n(exit 2); case x in `exit 4`);; esac; echo $?\n0\n0\n4\n4\n0\n0\n0\n0\n4\n\n\n(exit 2); case `exit 3`x in `exit 4`);; esac; echo $?\n0\n0\n4\n4\n0\n0\n0\n0\n4\n\n\n(exit 2); case `exit 3`x in `exit 4`x);; esac; echo $?\n0\n0\n0\n4\n0\n0\n0\n0\n4"},{"location":"dict/file/","title":"File","text":"A file is a pool of data in the
filesystem. On userlevel, it's referenced using a name, a hardlink to the file.If a file is not referenced anymore (number of hardlinks to it drops to 1) then the space allocated for that file is re-used, unless it's still used by some process.
The file-data splits into actual payload (file contents) and some metadata like filesize, filemode or timestamps. The metadata is stored in the
inode.Strictly spoken, a hardlink (also called \\\"filename\\\") points to the
"},{"location":"dict/file/#see-also","title":"See also","text":"inodewhich organizes a file, not to the file itself.
This timestamp indicates when a file was last accessed (read). cating a file or executing a shellscript will set it, for example.
This timestamp is set, whenever the metadata of a file (stored in the responsible inode) is set. The metadata includes for example:
and some other things. ctime will also be updated when a file is written to (when mtime is updated.
The mtime is set, whenever a file's contents are changed, for example by editing a file.
"},{"location":"dict/globbing/","title":"Globbing","text":"Globbing is the procedure of
Unlike MSDOS, where the called program had to interpret the patterns, the globbing on UNIX\u00ae is done by the shell, the matched filenames are given as parameters to a called command:
$ cat *.txt\nreally executes
$ cat 1.txt 3.txt foobar.txt XXX.txt\nThe term \\\"glob\\\" originates back in the UNIX\u00ae days where an executable glob (from \\\"global\\\") existed which was used to expand pattern-matching characters. Later, this functionality was built into the shell. There's still a library function called glob() (POSIX\u00ae), which serves the same purpose.
Also the article for:
A hardlink associates a filename with a file. That name is an entry in a directory listing. Of course a file can have more hardlinks to it (usually the number of hardlinks to a file is limited), but all hardlinks to a file must reside on the same filesystem as the file itself!
What you usually call a file is just a name for that file, and thus, a hardlink.
The difference between a symbolic link and a hard link is that there is no easy way to differentiate between a \\'real\\' file and a hard link, let's take a look at the example:
* create an empty file
$ touch a\n* create a hard link \\'b\\' and sym link \\'c\\' to empty file
$ ln a b\n$ ln -s a c\nas you can see file(1) can't differentiate between a real file \\'a\\' and a hard link \\'b\\', but it can tell \\'c\\' is a sym link
$ file *\na: empty\nb: empty\nc: symbolic link to `a'\nls -i prints out the inode numbers of files, if two files have the same inode number AND are on the same file system it means they are hardlinked.
$ ls -i *\n5262 a 5262 b 5263 c\nhard links don't consume additional space on the filesystem, the space is freed when the last hard link pointing to it is deleted.
"},{"location":"dict/hardlink/#see-also","title":"See also","text":"The interpreter directive, usually called shebang, is the character sequence starting with #! (hash, exclamation-point) at the beginning of the very first line of an executable text file on unixoid operating systems.
The program loader of the operating system may use this line to load an interpreter for this file when executed. This makes it a self-executable script.
A shebang will typically look like
#!/bin/bash\nSince the line starting with # is a comment for the shell (and some other scripting languages), it's ignored.
Regarding the shebang, there are various, differences between operating systems, including:
#! and before the pathname of the interpreterPOSIX\u00ae doesn't specify the shebang, though in general it's commonly supported by operating systems.
"},{"location":"dict/interpreter_directive/#see-also","title":"See also","text":"Also the article for: variable, positional parameter, special parameter
In Bash, a parameter is simply an entity that stores values and can be referenced. Depending on the type, the parameters can be set directly, only indirectly, or only automatically by the shell.
Bash knows 3 types of parameters:
A shell variable is a parameter denoted by a variable name:
A value can be assigned to a variable, using the variable's name and an equal-sign:
NAME=VALUE\nOnce a variable is set, it exists and can only be unset by the unset builtin command.
The nullstring is a valid value:
NAME=\nNAME=\"\"\nA positional parameter is denoted by a number other than 0 (zero).
Positional parameters reflect the shell's arguments that are not given to the shell itself (in practise, the script arguments, also the function arguments). You can't directly assign to the positional parameters, however, the set builtin command can be used to indirectly set them.
The first to ninth positional parameter is referenced by $1 to $9. All following positional parameters (tenth and above) must be referenced by the number given in curly braces, i.e., ${10} or ${432}.
Unlike popular belief, $0 is not a positional parameter.
See also the scripting article about handling positional parameters.
"},{"location":"dict/parameter/#special-parameters","title":"special parameters","text":"There are a bunch of special parameters, which are set by the shell. Direct assignment to them is not possible. These parameter names are formed of one character.
Please see shellvars.
"},{"location":"dict/parameter/#see-also","title":"See also","text":"POSIX\u00ae is a family of standards defined by the IEEE to give a minimum API and interface standardization across the variants of UNIX\u00ae operating systems.
One part of it is the standardization of minimum functionality and behaviour of the system shell and some utilities (commands).
On UNIX\u00ae, the shell is the main interaction tool between the user-level and the system. That doesn't necessarily mean the user always sits infront of a shell, but it's integral part of the system, not only an \\\"optional commandline interpreter\\\".
The main job of a shell is to execute commands as a user requests them. This behaviour alone doesn't help much. A shell knits some intelligence and flow control around the possibility to execute commands - it's a complete commandline-oriented user-interface (UI).
FIXME
tbd.
"},{"location":"dict/shell/#see-also","title":"See also","text":""},{"location":"dict/shell/#see-also-external","title":"See also (external)","text":"Unlike a regular file (a bunch of accessible data organized on a filesystem), it's a special filename that points to a ressource or similar:
Since a directory also is only a file, you can count it as special file, too.
"},{"location":"dict/special_file/#see-also","title":"See also","text":"A symlink (symbolic link) is a \\\"normal\\\" file, which contains a pointer to another filename. Since it really only points to another filename it can
dc(1) is a non standard, but commonly found, reverse-polish Desk Calculator. According to Ken Thompson, \"dc is the oldest language on Unix; it was written on the PDP-7 and ported to the PDP-11 before Unix [itself] was ported\".
Historically the standard bc(1) has been implemented as a front-end to dc.
","tags":["bash","shell","scripting","arithmetic","calculate"]},{"location":"howto/calculate-dc/#simple-calculation","title":"Simple calculation","text":"In brief, the reverse polish notation means the numbers are put on the stack first, then an operation is applied to them. Instead of writing 1+1, you write 1 1+.
By default dc, unlike bc, doesn't print anything, the result is pushed on the stack. You have to use the \"p\" command to print the element at the top of the stack. Thus a simple operation looks like:
$ dc <<< '1 1+pq'\n2\nI used a \"here string\" present in bash 3.x, ksh93 and zsh. if your shell doesn't support this, you can use echo '1 1+p' | dc or if you have GNU dc, you can use dc -e '1 1 +p'.
Of course, you can also just run dc and enter the commands.
The classic operations are:
+-/*%^vGNU dc adds a couple more.
To input a negative number you need to use the _ (underscore) character:
$ dc <<< '1_1-p'\n2\nYou can use the digits 0 to 9 and the letters A to F as numbers, and a dot (.) as a decimal point. The A to F must be capital letters in order not to be confused with the commands specified with lower case characters. A number with a letter is considered hexadecimal:
dc <<< 'Ap'\n10\nThe output is converted to base 10 by default
","tags":["bash","shell","scripting","arithmetic","calculate"]},{"location":"howto/calculate-dc/#scale-and-base","title":"Scale And Base","text":"dc is a calulator with abitrary precision, by default this precision is 0. thus dc <<< \"5 4/p\" prints \"1\".
We can increase the precision using the k command. It pops the value at the top of the stack and uses it as the precision argument:
dc <<< '2k5 4/p' # prints 1.25\ndc <<< '4k5 4/p' # prints 1.2500\ndc <<< '100k 2vp'\n1.4142135623730950488016887242096980785696718753769480731766797379907\\\n324784621070388503875343276415727\ndc supports large precision arguments.
You can change the base used to output (print) the numbers with o and the base used to input (type) the numbers with i:
dc << EOF\n20 p# prints 20, output is in base 10\n16o # the output is now in base 2 16\n20p # prints 14, in hex\n16i # the output is now in hex\np # prints 14 this doesn't modify the number in the stack\n10p # prints 10 the output is done in base 16\nEOF\nNote: when the input value is modified, the base is modified for all commands, including i:
dc << EOF\n16i 16o # base is 16 for input and output\n10p # prints 10\n10i # ! set the base to 10 i.e. to 16 decimal\n17p # prints 17\nEOF\nThis code prints 17 while we might think that 10i reverts the base back to 10 and thus the number should be converted to hex and printed as 11. The problem is 10 was typed while the input base 16, thus the base was set to 10 hexadecimal, i.e. 16 decimal.
dc << EOF\n16o16o10p #prints 10\nAi # set the base to A in hex i.e. 10\n17p # prints 11 in base 16\nEOF\nThere are two basic commands to manipulate the stack:
d duplicates the top of the stackc clears the stack
$ dc << EOF 2 # put 2 on the stack d # duplicate i.e. put another 2 on the stack *p # multiply and print c p # clear and print EOF 4 dc: stack empty
c p results in an error, as we would expect, as c removes everything on the stack. Note: we can use # to put comments in the script.
If you are lost, you can inspect (i.e. print) the stack using the command f. The stack remains unchanged:
dc <<< '1 2 d 4+f'\n6\n2\n1\nNote how the first element that will be popped from the stack is printed first, if you are used to an HP calculator, it's the reverse.
Don't hesitate to put f in the examples of this tutorial, it doesn't change the result, and it's a good way to see what's going on.
The GNU dc manual says that dc has at least 256 registers depending on the range of unsigned char. I\\'m not sure how you are supposed to use the NUL byte. Using a register is easy:
dc <<EOF\n12 # put 12 on the stack\nsa # remove it from the stack (s), and put it in register 'a'\n10 # put 10 on the stack\nla # read (l) the value of register 'a' and push it on the stack\n+p # add the 2 values and print\nEOF\nThe above snippet uses newlines to embed comments, but it doesn't really matter, you can use echo '12sa10la+p'| dc, with the same results.
The register can contain more than just a value, each register is a stack on its own.
dc <<EOF\n12sa #store 12 in 'a'\n6Sa # with a capital S the 6 is removed\n # from the main stack and pushed on the 'a' stack\nlap # prints 6, the value at the top of the 'a' stack\nlap # still prints 6\nLap # prints 6 also but with a capital L, it pushes the value in 'a'\n # to the main stack and pulls it from the 'a' stack\nlap # prints 12, which is now at the top of the stack\nEOF\ndc lets you push arbitrary strings on the stack when the strings are enclosed in []. You can print it with p: dc <<< '[Hello World!]p' and you can evalute it with x: dc <<< '[1 2+]xp'.
This is not that interesting until combined with registers. First, let's say we want to calculate the square of a number (don't forget to include f if you get lost!):
dc << EOF\n3 # push our number on the stack\nd # duplicate it i.e. push 3 on the stack again\nd**p # duplicate again and calculate the product and print\nEOF\nNow we have several cubes to calculate, we could use dd** several times, or use a macro.
dc << EOF\n[dd**] # push a string\nsa # save it in register a\n3 # push 3 on the stack\nlax # push the string \"dd**\" on the stack and execute it\np # print the result\n4laxp # same operation for 4, in one line\nEOF\ndc can execute a macro stored in a register using the lR x combo, but it can also execute macros conditionally. >a will execute the macro stored in the register a, if the top of the stack is greater than the second element of the stack. Note: the top of the stack contains the last entry. When written, it appears as the reverse of what we are used to reading:
dc << EOF\n[[Hello World]p] sR # store in 'R' a macro that prints Hello World\n2 1 >R # do nothing 1 is at the top 2 is the second element\n1 2 >R # prints Hello World\nEOF\nSome dc have >R <R =R, GNU dc had some more, check your manual. Note that the test \"consumes\" its operands: the 2 first elements are popped off the stack (you can verify that dc <<< \"[f]sR 2 1 >R 1 2 >R f\" doesn't print anything)
Have you noticed how we can include a macro (string) in a macro? and as dc relies on a stack we can, in fact, use the macro recursively (have your favorite control-c key combo ready ;)) :
dc << EOF\n[ [Hello World] p # our macro starts by printing Hello World\n lRx ] # and then executes the macro in R\nsR # we store it in the register R\nlRx # and finally executes it.\nEOF\nWe have recursivity, we have test, we have loops:
dc << EOF\n[ li # put our index i on the stack\n p # print it, to see what's going on\n 1 - # we decrement the index by one\n si # store decremented index (i=i-1)\n 0 li >L # if i > 0 then execute L\n] sL # store our macro with the name L\n\n10 si # let's give to our index the value 10\nlLx # and start our loop\nEOF\nOf course code written this way is far too easy to read! Make sure to remove all those extra spaces newlines and comments:
dc <<< '[lip1-si0li>L]sL10silLx'\ndc <<< '[p1-d0<L]sL10lLx' # use the stack instead of a register\nI'll let you figure out the second example, it's not hard, it uses the stack instead of a register for the index.
","tags":["bash","shell","scripting","arithmetic","calculate"]},{"location":"howto/calculate-dc/#next","title":"Next","text":"Check your dc manual, i haven't decribed everything, like arrays (only documented with \"; : are used by bc(1) for array operations\" on solaris, probably because echo \\'1 0:a 0Sa 2 0:a La 0;ap\\' | dc results in //Segmentation Fault (core dump) //, the latest solaris uses GNU dc)
You can find more info and dc programs here:
And more example, as well as a dc implementation in python here:
The manual for the 1971 dc from Bell Labs:
A collapsing function is a function whose behavior changes depending upon the circumstances under which it's run. Function collapsing is useful when you find yourself repeatedly checking a variable whose value never changes.
","tags":["bash","shell","scripting","example","function","collapse"]},{"location":"howto/collapsing_functions/#how-do-i-make-a-function-collapse","title":"How do I make a function collapse?","text":"Function collapsing requires some static feature in the environment. A common example is a script that gives the user the option of having \"verbose\" output.
#!/bin/bash\n\n[[ $1 = -v || $1 = --verbose ]] && verbose=1\n\nchatter() {\n if [[ $verbose ]]; then\n chatter() {\n echo \"$@\"\n }\n chatter \"$@\"\n else\n chatter() {\n :\n }\n fi\n}\n\necho \"Waiting for 10 seconds.\"\nfor i in {1..10}; do\n chatter \"$i\"\n sleep 1\ndone\nThe first time you run chatter(), the function redefines itself based on the value of verbose. Thereafter, chatter doesn't check $verbose, it simply is. Further calls to the function reflect its collapsed nature. If verbose is unset, chatter will echo nothing, with no extra effort from the developer.
FIXME
Add more examples!
# Somewhat more portable find -executable\n# FIXME/UNTESTED (I don't have access to all of the different versions of find.)\n# Usage: find PATH ARGS -- use find like normal, except use -executable instead of\n# various versions of -perm /+ blah blah and hacks\nfind() {\n hash find || { echo 'find not found!'; exit 1; }\n # We can be pretty sure \"$0\" should be executable.\n if [[ $(command find \"$0\" -executable 2> /dev/null) ]]; then\n unset -f find # We can just use the command find\n elif [[ $(command find \"$0\" -perm /u+x 2> /dev/null) ]]; then\n find() {\n typeset arg args\n for arg do\n [[ $arg = -executable ]] && args+=(-perm /u+x) || args+=(\"$arg\")\n done\n command find \"${args[@]}\"\n }\n elif [[ $(command find \"$0\" -perm +u+x 2> /dev/null) ]]; then\n find() {\n typeset arg args\n for arg do\n [[ $arg = -executable ]] && args+=(-perm +u+x) || args+=(\"$arg\")\n done\n command find \"${args[@]}\"\n }\n else # Last resort\n find() {\n typeset arg args\n for arg do\n [[ $arg = -executable ]] && args+=(-exec test -x {} \\; -print) || args+=(\"$arg\")\n done\n command find \"${args[@]}\"\n }\n fi\n find \"$@\"\n}\n#!/bin/bash\n# Using collapsing functions to turn debug messages on/off\n\n[ \"--debug\" = \"$1\" ] && dbg=echo || dbg=:\n\n\n# From now on if you use $dbg instead of echo, you can select if messages will be shown\n\n$dbg \"This message will only be displayed if --debug is specified at the command line\nFor this task, you don't have to write large parser routines (unless you want it 100% secure or you want a special file syntax) - you can use the Bash source command. The file to be sourced should be formated in key=\"value\" format, otherwise bash will try to interpret commands:
#!/bin/bash\necho \"Reading config....\" >&2\nsource /etc/cool.cfg\necho \"Config for the username: $cool_username\" >&2\necho \"Config for the target host: $cool_host\" >&2\nSo, where do these variables come from? If everything works fine, they are defined in /etc/cool.cfg which is a file that's sourced into the current script or shell. Note: this is not the same as executing this file as a script! The sourced file most likely contains something like:
cool_username=\"guest\"\ncool_host=\"foo.example.com\"\nThese are normal statements understood by Bash, nothing special. Of course (and, a big disadvantage under normal circumstances) the sourced file can contain everything that Bash understands, including malicious code!
The source command also is available under the name . (dot). The usage of the dot is identical:
#!/bin/bash\necho \"Reading config....\" >&2\n. /etc/cool.cfg #note the space between the dot and the leading slash of /etc.cfg\necho \"Config for the username: $cool_username\" >&2\necho \"Config for the target host: $cool_host\" >&2\nThere's also a way to provide a system-wide config file in /etc and a custom config in ~/(user's home) to override system-wide defaults. In the following example, the if/then construct is used to check for the existance of a user-specific config:
#!/bin/bash\necho \"Reading system-wide config....\" >&2\n. /etc/cool.cfg\nif [ -r ~/.coolrc ]; then\n echo \"Reading user config....\" >&2\n . ~/.coolrc\nfi\nAs mentioned earlier, the sourced file can contain anything a Bash script can. Essentially, it is an included Bash script. That creates security issues. A malicicios person can \"execute\" arbitrary code when your script is sourcing its config file. You might want to allow only constructs in the form NAME=VALUE in that file (variable assignment syntax) and maybe comments (though technically, comments are unimportant). Imagine the following \"config file\", containing some malicious code:
# cool config file for my even cooler script\nusername=god_only_knows\nhostname=www.example.com\npassword=secret ; echo rm -rf ~/*\nparameter=foobar && echo \"You've bene pwned!\";\n# hey look, weird code follows...\necho \"I am the skull virus...\"\necho rm -fr ~/*\nmailto=netadmin@example.com\nYou don't want these echo-commands (which could be any other commands!) to be executed. One way to be a bit safer is to filter only the constructs you want, write the filtered results to a new file and source the new file. We also need to be sure something nefarious hasn't been added to the end of one of our name=value parameters, perhaps using ; or && command separators. In those cases, perhaps it is simplest to just ignore the line entirely. Egrep (grep -E) will help us here, it filters by description:
#!/bin/bash\nconfigfile='/etc/cool.cfg'\nconfigfile_secured='/tmp/cool.cfg'\n\n# check if the file contains something we don't want\nif egrep -q -v '^#|^[^ ]*=[^;]*' \"$configfile\"; then\n echo \"Config file is unclean, cleaning it...\" >&2\n # filter the original to a new file\n egrep '^#|^[^ ]*=[^;&]*' \"$configfile\" > \"$configfile_secured\"\n configfile=\"$configfile_secured\"\nfi\n\n# now source it, either the original or the filtered variant\nsource \"$configfile\"\nTo make clear what it does: egrep checks if the file contains something we don't want, if yes, egrep filters the file and writes the filtered contents to a new file. If done, the original file name is changed to the name stored in the variable configfile. The file named by that variable is sourced, as if it were the original file.
This filter allows only NAME=VALUE and comments in the file, but it doesn't prevent all methods of code execution. I will address that later.
$ ls *.zip | while read i; do j=`echo $i | sed 's/.zip//g'`; mkdir $j; cd $j; unzip ../$i; cd ..; done\nThis is an actual one-liner someone asked about in #bash. There are several things wrong with it. Let's break it down!
$ ls *.zip | while read i; do ...; done\n(Please read http://mywiki.wooledge.org/ParsingLs.) This command executes ls on the expansion of *.zip. Assuming there are filenames in the current directory that end in '.zip', ls will give a human-readable list of those names. The output of ls is not for parsing. But in sh and bash alike, we can loop safely over the glob itself:
$ for i in *.zip; do j=`echo $i | sed 's/.zip//g'`; mkdir $j; cd $j; unzip ../$i; cd ..; done\nLet's break it down some more!
j=`echo $i | sed 's/.zip//g'` # where $i is some name ending in '.zip'\nThe goal here seems to be get the filename without its .zip extension. In fact, there is a POSIX\u00ae-compliant command to do this: basename The implementation here is suboptimal in several ways, but the only thing that's genuinely error-prone with this is \"echo $i\". Echoing an unquoted variable means wordsplitting will take place, so any whitespace in $i will essentially be normalized. In sh it is necessary to use an external command and a subshell to achieve the goal, but we can eliminate the pipe (subshells, external commands, and pipes carry extra overhead when they launch, so they can really hurt performance in a loop). Just for good measure, let's use the more readable, modern $() construct instead of the old style backticks:
sh $ for i in *.zip; do j=$(basename \"$i\" \".zip\"); mkdir $j; cd $j; unzip ../$i; cd ..; done\nIn Bash we don't need the subshell or the external basename command. See Substring removal with parameter expansion:
bash $ for i in *.zip; do j=\"${i%.zip}\"; mkdir $j; cd $j; unzip ../$i; cd ..; done\nLet's keep going:
$ mkdir $j; cd $j; ...; cd ..\nAs a programmer, you never know the situation under which your program will run. Even if you do, the following best practice will never hurt: When a following command depends on the success of a previous command(s), check for success! You can do this with the \"&&\" conjunction, that way, if the previous command fails, bash will not try to execute the following command(s). It's fully POSIX\u00ae. Oh, and remember what I said about wordsplitting in the previous step? Well, if you don't quote $j, wordsplitting can happen again.
$ mkdir \"$j\" && cd \"$j\" && ... && cd ..\nThat's almost right, but there's one problem -- what happens if $j contains a slash? Then cd .. will not return to the original directory. That's wrong! cd - causes cd to return to the previous working directory, so it's a much better choice:
$ mkdir \"$j\" && cd \"$j\" && ... && cd -\n(If it occurred to you that I forgot to check for success after cd -, good job! You could do this with { cd - || break; }, but I'm going to leave that out because it's verbose and I think it's likely that we will be able to get back to our original working directory without a problem.)
So now we have:
sh $ for i in *.zip; do j=$(basename \"$i\" \".zip\"); mkdir \"$j\" && cd \"$j\" && unzip ../$i && cd -; done\nbash $ for i in *.zip; do j=\"${i%.zip}\"; mkdir \"$j\" && cd \"$j\" && unzip ../$i && cd -; done\nLet's throw the unzip command back in the mix:
mkdir \"$j\" && cd \"$j\" && unzip ../$i && cd -\nWell, besides word splitting, there's nothing terribly wrong with this. Still, did it occur to you that unzip might already be able to target a directory? There isn't a standard for the unzip command, but all the implementations I've seen can do it with the -d flag. So we can drop the cd commands entirely:
$ mkdir \"$j\" && unzip -d \"$j\" \"$i\"\nsh $ for i in *.zip; do j=$(basename \"$i\" \".zip\"); mkdir \"$j\" && unzip -d \"$j\" \"$i\"; done\nbash $ for i in *.zip; do j=\"${i%.zip}\"; mkdir \"$j\" && unzip -d \"$j\" \"$i\"; done\nThere! That's as good as it gets.
"},{"location":"howto/edit-ed/","title":"Editing files via scripts with ed","text":"","tags":["bash","shell","scripting","arguments","file","editor","edit","ed","sed"]},{"location":"howto/edit-ed/#why-ed","title":"Why ed?","text":"Like sed, ed is a line editor. However, if you try to change file contents with sed, and the file is open elsewhere and read by some process, you will find out that GNU sed and its -i option will not allow you to edit the file. There are circumstances where you may need that, e.g. editing active and open files, the lack of GNU, or other sed, with \"in-place\" option available.
Why ed?
sed doesn't support in-place editsed)ed has very good editing and addressing possibilities, compared to standard sedDon't get me wrong, this is not meant as anti-sed article! It's just meant to show you another way to do the job.
Since ed is an interactive text editor, it reads and executes commands that come from stdin. There are several ways to feed our commands to ed:
Pipelines
echo '<ED-COMMANDS>' | ed <FILE>\nTo inject the needed newlines, etc. it may be easier to use the builtin command, printf (\"help printf\"). Shown here as an example Bash function to prefix text to file content:
# insertHead \"$text\" \"$file\"\n\ninsertHead() {\n printf '%s\\n' H 1i \"$1\" . w | ed -s \"$2\"\n}\nHere-strings
ed <FILE> <<< '<ED-COMMANDS>'\nHere-documents
ed <FILE> <<EOF\n<ED-COMMANDS>\nEOF\nWhich one you prefer is your choice. I will use the here-strings, since it looks best here IMHO.
There are other ways to provide input to ed. For example, process substitution. But these should be enough for daily needs.
Since ed wants commands separated by newlines, I'll use a special Bash quoting method, the C-like strings $'TEXT', as it can interpret a set of various escape sequences and special characters. I'll use the -s option to make it less verbose.
Check the ed manpage for details
Similar to vi or vim, ed has a \"command mode\" and an \"interactive mode\". For non-interactive use, the command mode is the usual choice.
Commands to ed have a simple and regular structure: zero, one, or two addresses followed by a single-character command, possibly followed by parameters to that command. These addresses specify one or more lines in the text buffer. Every command that requires addresses has default addresses, so the addresses can often be omitted.
The line addressing is relative to the current line. If the edit buffer is not empty, the initial value for the current line shall be the last line in the edit buffer, otherwise zero. Generally, the current line is the last line affected by a command. All addresses can only address single lines, not blocks of lines!
Line addresses or commands using regular expressions interpret POSIX Basic Regular Expressions (BRE). A null BRE is used to reference the most recently used BRE. Since ed addressing is only for single lines, no RE can ever match a newline.
By default, ed is not very talkative and will simply print a \"?\" when an error occurs. Interactively you can use the h command to get a short message explaining the last error. You can also turn on a mode that makes ed automatically print this message with the H command. It is a good idea to always add this command at the beginning of your ed scripts:
bash > ed -s file <<< $'H\\n,df'\n?\nscript, line 2: Invalid command suffix\nWhile working on your script, you might make errors and destroy your file, you might be tempted to try your script doing something like:
# Works, but there is better\n\n# copy my original file\ncp file file.test\n\n# try my script on the file\ned -s file.test <<< $'H\\n<ed commands>\\nw'\n\n# see the results\ncat file.test\nThere is a much better way though, you can use the ed command p to print the file, now your testing would look like:
ed -s file <<< $'H\\n<ed commands>\\n,p'\nthe , (comma) in front of the p command is a shortcut for 1,$ which defines an address range for the first to the last line, ,p thus means print the whole file, after it has been modified. When your script runs sucessfully, you only have to replace the ,p by a w.
Of course, even if the file is not modified by the p command, it's always a good idea to have a backup copy!
Most of these things can be done with sed. But there are also things that can't be done in sed or can only be done with very complex code.
Like sed, ed also knows the common s/FROM/TO/ command, and it can also take line-addresses. If no substitution is made on the addressed lines, it's considered an error.
ed -s test.txt <<< $',s/Windows(R)-compatible/POSIX-conform/g\\nw'\nNote: The comma as single address operator is an alias for 1,$ (\"all lines\").
On a line containing fruits, do the substitution:
ed -s test.txt <<< $'/fruits/s/apple/banana/g\\nw'\nOn the 5th line after the line containing fruits, do the substitution:
ed -s test.txt <<< $'/fruits/+5s/apple/banana/g\\nw'\nThe simple one is a well-known (by position) block of text:
# delete lines number 2 to 4 (2, 3, 4)\ned -s test.txt <<< $'2,5d\\nw'\nThis deletes all lines matching a specific regular expression:
# delete all lines matching foobar\ned -s test.txt <<< $'g/foobar/d\\nw'\ng/regexp/ applies the command following it to all the lines matching the regexp
","tags":["bash","shell","scripting","arguments","file","editor","edit","ed","sed"]},{"location":"howto/edit-ed/#move-a-block-of-text","title":"Move a block of text","text":"...using the m command: <ADDRESS> m <TARGET-ADDRESS>
This is definitely something that can't be done easily with sed.
# moving lines 5-9 to the end of the file\ned -s test.txt <<< $'5,9m$\\nw'\n\n# moving lines 5-9 to line 3\ned -s test.txt <<< $'5,9m3\\nw'\n...using the t command: <ADDRESS> t <TARGET-ADDRESS>
You use the t command just like you use the m (move) command.
# make a copy of lines 5-9 and place it at the end of the file\ned -s test.txt <<< $'5,9t$\\nw'\n\n# make a copy of lines 5-9 and place it at line 3\ned -s test.txt <<< $'5,9t3\\nw'\n...but leave the final newline intact. This is done by an extra command: j (join).
ed -s file <<< $'1,$j\\nw'\nCompared with two other methods (using tr or sed), you don't have to delete all newlines and manually add one at the end.
How do you insert another file? As with sed, you use the r (read) command. That inserts another file at the line before the last line (and prints the result to stdout - ,p):
ed -s FILE1 <<< $'$-1 r FILE2\\n,p'\nTo compare, here's a possible sed solution which must use Bash arithmetic and the external program wc:
sed \"$(($(wc -l < FILE1)-1))r FILE2\" FILE1\n\n# UPDATE here's one which uses GNU sed's \"e\" parameter for the s-command\n# it executes the commands found in pattern space. I'll take that as a\n# security risk, but well, sometimes GNU > security, you know...\nsed '${h;s/.*/cat FILE2/e;G}' FILE1\nAnother approach, in two invocations of sed, that avoids the use of external commands completely:
sed $'${s/$/\\\\n-||-/;r FILE2\\n}' FILE1 | sed '0,/-||-/{//!h;N;//D};$G'\ned and sed might look similar, but the same command(s) might act differently:
__ /foo/d __
In sed /foo/d will delete all lines matching foo, in ed the commands are not repeated on each line so this command will search the next line matching foo and delete it. If you want to delete all lines matching foo, or do a subsitution on all lines matching foo you have to tell ed about it with the g (global) command:
echo $'1\\n1\\n3' > file\n\n#replace all lines matching 1 by \"replacement\"\ned -s file <<< $'g/1/s/1/replacement/\\n,p'\n\n#replace the first line matching 1 by \"replacement\"\n#(because it starts searching from the last line)\ned -s file <<< $'s/1/replacement/\\n,p'\n__ an error stops the script __
You might think that it's not a problem and that the same thing happens with sed and you're right, with the exception that if ed does not find a pattern it's an error, while sed just continues with the next line. For instance, let's say that you want to change foo to bar on the first line of the file and add something after the next line, ed will stop if it cannot find foo on the first line, sed will continue.
#Gnu sed version\nsed -e '1s/foo/bar/' -e '$a\\something' file\n\n#First ed version, does nothing if foo is not found on the first line:\ned -s file <<< $'H\\n1s/foo/bar/\\na\\nsomething\\n.\\nw'\nIf you want the same behaviour you can use g/foo/ to trick ed. g/foo/ will apply the command on all lines matching foo, thus the substitution will succeed and ed will not produce an error when foo is not found:
#Second version will add the line with \"something\" even if foo is not found\ned -s file <<< $'H\\n1g/foo/s/foo/bar/\\na\\nsomething\\n.\\nw'\nIn fact, even a substitution that fails after a g/ / command does not seem to cause an error, i.e. you can use a trick like g/./s/foo/bar/ to attempt the substitution on all non blank lines
","tags":["bash","shell","scripting","arguments","file","editor","edit","ed","sed"]},{"location":"howto/edit-ed/#here-documents","title":"here documents","text":"__ shell parameters are expanded __
If you don't quote the delimiter, $ has a special meaning. This sounds obvious but it's easy to forget this fact when you use addresses like $-1 or commands like $a. Either quote the $ or the delimiter:
#fails\ned -s file << EOF\n$a\nlast line\n.\nw\nEOF\n\n#ok\ned -s file << EOF\n$a\nlast line\n.\nw\nEOF\n\n#ok again\ned -s file << 'EOF'\n$a\nlast line\n.\nw\nEOF\n__ \".\" is not a command __
The . used to terminate the command \"a\" must be the only thing on the line. take care if you indent the commands:
#ed doesn't care about the spaces before the commands, but the . must be the only thing on the line:\ned -s file << EOF\n a\nmy content\n.\n w\nEOF\nKeep in mind that in all the examples below, the entire file will be read into memory.
","tags":["bash","shell","scripting","arguments","file","editor","edit","ed","sed"]},{"location":"howto/edit-ed/#a-simple-grep","title":"A simple grep","text":"ed -s file <<< 'g/foo/p'\n\n# equivalent\ned -s file <<< 'g/foo/'\nThe name grep is derived from the notaion g/RE/p (global => regular expression => print). ref http://www.catb.org/~esr/jargon/html/G/grep.html
Since the default for the ed \"print line number\" command is the last line, a simple = (equal sign) will print this line number and thus the number of lines of the file:
ed -s file <<< '='\nYea, it's a joke...
ed -s file <<< $',p'\n...but a similar thing to cat showing line-endings and escapes can be done with the list command (l):
ed -s file <<< $',l'\nFIXME
to be continued
","tags":["bash","shell","scripting","arguments","file","editor","edit","ed","sed"]},{"location":"howto/edit-ed/#links","title":"Links","text":"Reference:
Misc info / tutorials:
Note that getopts is neither able to parse GNU-style long options (--myoption) nor XF86-style long options (-myoption). So, when you want to parse command line arguments in a professional ;-) way, getopts may or may not work for you. Unlike its older brother getopt (note the missing s!), it's a shell builtin command. The advantages are:
getopts can set shell variables to use for parsing (impossible for an external process!)getopt implementations which had buggy concepts in the past (whitespace, ...)getopts is defined in POSIX\u00ae.Some other methods to parse positional parameters - using neither getopt nor getopts - are described in: How to handle positional parameters.
","tags":["bash","shell","scripting","arguments","positional","parameters","options","getopt","getopts"]},{"location":"howto/getopts_tutorial/#terminology","title":"Terminology","text":"It's useful to know what we're talking about here, so let's see... Consider the following command line:
mybackup -x -f /etc/mybackup.conf -r ./foo.txt ./bar.txt\nThese are all positional parameters, but they can be divided into several logical groups:
-x is an option (aka flag or switch). It consists of a dash (-) followed by one character.-f is also an option, but this option has an associated option argument (an argument to the option -f): /etc/mybackup.conf. The option argument is usually the argument following the option itself, but that isn't mandatory. Joining the option and option argument into a single argument -f/etc/mybackup.conf is valid.-r depends on the configuration. In this example, -r doesn't take arguments so it's a standalone option like -x../foo.txt and ./bar.txt are remaining arguments without any associated options. These are often used as mass-arguments. For example, the filenames specified for cp(1), or arguments that don't need an option to be recognized because of the intended behavior of the program. POSIX\u00ae calls them operands.To give you an idea about why getopts is useful, The above command line is equivalent to:
mybackup -xrf /etc/mybackup.conf ./foo.txt ./bar.txt\nwhich is complex to parse without the help of getopts.
The option flags can be upper- and lowercase characters, or digits. It may recognize other characters, but that's not recommended (usability and maybe problems with special characters).
","tags":["bash","shell","scripting","arguments","positional","parameters","options","getopt","getopts"]},{"location":"howto/getopts_tutorial/#how-it-works","title":"How it works","text":"In general you need to call getopts several times. Each time it will use the next positional parameter and a possible argument, if parsable, and provide it to you. getopts will not change the set of positional parameters. If you want to shift them, it must be done manually:
shift $((OPTIND-1))\n# now do something with $@\nSince getopts sets an exit status of FALSE when there's nothing left to parse, it's easy to use in a while-loop:
while getopts ...; do\n ...\ndone\ngetopts will parse options and their possible arguments. It will stop parsing on the first non-option argument (a string that doesn't begin with a hyphen (-) that isn't an argument for any option in front of it). It will also stop parsing when it sees the -- (double-hyphen), which means end of options.
getopts \"remembers\" its own status between invocations. Also useful to shift the positional parameters after processing with getopts. OPTIND is initially set to 1, and needs to be re-set to 1 if you want to parse anything again with getopts OPTARG This variable is set to any argument for an option found by getopts. It also contains the option flag of an unknown option. OPTERR (Values 0 or 1) Indicates if Bash should display error messages generated by the getopts builtin. The value is initialized to 1 on every shell startup - so be sure to always set it to 0 if you don't want to see annoying messages! OPTERR is not specified by POSIX for the getopts builtin utility --- only for the C getopt() function in unistd.h (opterr). OPTERR is bash-specific and not supported by shells such as ksh93, mksh, zsh, or dash. getopts also uses these variables for error reporting (they're set to value-combinations which arent possible in normal operation).
The base-syntax for getopts is:
getopts OPTSTRING VARNAME [ARGS...]\nwhere:
Option DescriptionOPTSTRING tells getopts which options to expect and where to expect arguments (see below) VARNAME tells getopts which shell-variable to use for option reporting ARGS tells getopts to parse these optional words instead of the positional parameters","tags":["bash","shell","scripting","arguments","positional","parameters","options","getopt","getopts"]},{"location":"howto/getopts_tutorial/#the-option-string","title":"The option-string","text":"The option-string tells getopts which options to expect and which of them must have an argument. The syntax is very simple --- every option character is simply named as is, this example-string would tell getopts to look for -f, -A and -x:
getopts fAx VARNAME\nWhen you want getopts to expect an argument for an option, just place a : (colon) after the proper option flag. If you want -A to expect an argument (i.e. to become -A SOMETHING) just do:
getopts fA:x VARNAME\nIf the very first character of the option-string is a : (colon), which would normally be nonsense because there's no option letter preceding it, getopts switches to \"silent error reporting mode\". In productive scripts, this is usually what you want because it allows you to handle errors yourself without being disturbed by annoying messages.
The getopts utility parses the positional parameters of the current shell or function by default (which means it parses \"$@\").
You can give your own set of arguments to the utility to parse. Whenever additional arguments are given after the VARNAME parameter, getopts doesn't try to parse the positional parameters, but these given words.
This way, you are able to parse any option set you like, here for example from an array:
while getopts :f:h opt \"${MY_OWN_SET[@]}\"; do\n ...\ndone\nA call to getopts without these additional arguments is equivalent to explicitly calling it with \"$@\":
getopts ... \"$@\"\nRegarding error-reporting, there are two modes getopts can run in:
For productive scripts I recommend to use the silent mode, since everything looks more professional, when you don't see annoying standard messages. Also it's easier to handle, since the failure cases are indicated in an easier way.
","tags":["bash","shell","scripting","arguments","positional","parameters","options","getopt","getopts"]},{"location":"howto/getopts_tutorial/#verbose-mode","title":"Verbose Mode","text":"failure message invalid optionVARNAME is set to ? (question-mark) and OPTARG is unset required argument not found VARNAME is set to ? (question-mark), OPTARG is unset and an error message is printed","tags":["bash","shell","scripting","arguments","positional","parameters","options","getopt","getopts"]},{"location":"howto/getopts_tutorial/#silent-mode","title":"Silent Mode","text":"failure message invalid option VARNAME is set to ? (question-mark) and OPTARG is set to the (invalid) option character required argument not found VARNAME is set to : (colon) and OPTARG contains the option-character in question","tags":["bash","shell","scripting","arguments","positional","parameters","options","getopt","getopts"]},{"location":"howto/getopts_tutorial/#using-it","title":"Using it","text":"","tags":["bash","shell","scripting","arguments","positional","parameters","options","getopt","getopts"]},{"location":"howto/getopts_tutorial/#a-first-example","title":"A first example","text":"Enough said - action!
Let's play with a very simple case: only one option (-a) expected, without any arguments. Also we disable the verbose error handling by preceding the whole option string with a colon (:):
#!/bin/bash\n\nwhile getopts \":a\" opt; do\n case $opt in\n a)\n echo \"-a was triggered!\" >&2\n ;;\n \\?)\n echo \"Invalid option: -$OPTARG\" >&2\n ;;\n esac\ndone\nI put that into a file named go_test.sh, which is the name you'll see below in the examples.
Let's do some tests:
","tags":["bash","shell","scripting","arguments","positional","parameters","options","getopt","getopts"]},{"location":"howto/getopts_tutorial/#calling-it-without-any-arguments","title":"Calling it without any arguments","text":"$ ./go_test.sh\n$\nNothing happened? Right. getopts didn't see any valid or invalid options (letters preceded by a dash), so it wasn't triggered.
$ ./go_test.sh /etc/passwd\n$\nAgain --- nothing happened. The very same case: getopts didn't see any valid or invalid options (letters preceded by a dash), so it wasn't triggered.
The arguments given to your script are of course accessible as $1 - ${N}.
Now let's trigger getopts: Provide options.
First, an invalid one:
$ ./go_test.sh -b\nInvalid option: -b\n$\nAs expected, getopts didn't accept this option and acted like told above: It placed ? into $opt and the invalid option character (b) into $OPTARG. With our case statement, we were able to detect this.
Now, a valid one (-a):
$ ./go_test.sh -a\n-a was triggered!\n$\nYou see, the detection works perfectly. The a was put into the variable $opt for our case statement.
Of course it's possible to mix valid and invalid options when calling:
$ ./go_test.sh -a -x -b -c\n-a was triggered!\nInvalid option: -x\nInvalid option: -b\nInvalid option: -c\n$\nFinally, it's of course possible, to give our option multiple times:
$ ./go_test.sh -a -a -a -a\n-a was triggered!\n-a was triggered!\n-a was triggered!\n-a was triggered!\n$\nThe last examples lead us to some points you may consider:
exit in the right place)Let's extend our example from above. Just a little bit:
-a now takes an argumentexit 1#!/bin/bash\n\nwhile getopts \":a:\" opt; do\n case $opt in\n a)\n echo \"-a was triggered, Parameter: $OPTARG\" >&2\n ;;\n \\?)\n echo \"Invalid option: -$OPTARG\" >&2\n exit 1\n ;;\n :)\n echo \"Option -$OPTARG requires an argument.\" >&2\n exit 1\n ;;\n esac\ndone\nLet's do the very same tests we did in the last example:
","tags":["bash","shell","scripting","arguments","positional","parameters","options","getopt","getopts"]},{"location":"howto/getopts_tutorial/#calling-it-without-any-arguments_1","title":"Calling it without any arguments","text":"$ ./go_test.sh\n$\nAs above, nothing happened. It wasn't triggered.
","tags":["bash","shell","scripting","arguments","positional","parameters","options","getopt","getopts"]},{"location":"howto/getopts_tutorial/#calling-it-with-non-option-arguments_1","title":"Calling it with non-option arguments","text":"$ ./go_test.sh /etc/passwd\n$\nThe very same case: It wasn't triggered.
","tags":["bash","shell","scripting","arguments","positional","parameters","options","getopt","getopts"]},{"location":"howto/getopts_tutorial/#calling-it-with-option-arguments_1","title":"Calling it with option-arguments","text":"Invalid option:
$ ./go_test.sh -b\nInvalid option: -b\n$\nAs expected, as above, getopts didn't accept this option and acted like programmed.
Valid option, but without the mandatory argument:
$ ./go_test.sh -a\nOption -a requires an argument.\n$\nThe option was okay, but there is an argument missing.
Let's provide the argument:
$ ./go_test.sh -a /etc/passwd\n-a was triggered, Parameter: /etc/passwd\n$\nSometimes there's a need to ensure only one copy of a script runs, i.e prevent two or more copies running simultaneously. Imagine an important cronjob doing something very important, which will fail or corrupt data if two copies of the called program were to run at the same time. To prevent this, a form of MUTEX (mutual exclusion) lock is needed.
The basic procedure is simple: The script checks if a specific condition (locking) is present at startup, if yes, it's locked - the scipt doesn't start.
This article describes locking with common UNIX\u00ae tools. There are other special locking tools available, But they're not standardized, or worse yet, you can't be sure they're present when you want to run your scripts. A tool designed for specifically for this purpose does the job much better than general purpose code.
","tags":["bash","shell","scripting","mutex","locking","run-control"]},{"location":"howto/mutex/#other-special-locking-tools","title":"Other, special locking tools","text":"As told above, a special tool for locking is the preferred solution. Race conditions are avoided, as is the need to work around specific limits.
flock: http://www.kernel.org/pub/software/utils/script/flock/solo: http://timkay.com/solo/The best way to set a global lock condition is the UNIX\u00ae filesystem. Variables aren't enough, as each process has its own private variable space, but the filesystem is global to all processes (yes, I know about chroots, namespaces, ... special case). You can \"set\" several things in the filesystem that can be used as locking indicator:
To create a file or set a file timestamp, usually the command touch is used. The following problem is implied: A locking mechanism checks for the existance of the lockfile, if no lockfile exists, it creates one and continues. Those are two separate steps! That means it's not an atomic operation. There's a small amount of time between checking and creating, where another instance of the same script could perform locking (because when it checked, the lockfile wasn't there)! In that case you would have 2 instances of the script running, both thinking they are succesfully locked, and can operate without colliding. Setting the timestamp is similar: One step to check the timespamp, a second step to set the timestamp.
Conclusion: We need an operation that does the check and the locking in one step.
A simple way to get that is to create a lock directory - with the mkdir command. It will:
* create a given directory only if it does not exist, and set a successful exit code\n * it will set an unsuccesful exit code if an error occours - for example, if the directory specified already exists\nWith mkdir it seems, we have our two steps in one simple operation. A (very!) simple locking code might look like this:
if mkdir /var/lock/mylock; then\n echo \"Locking succeeded\" >&2\nelse\n echo \"Lock failed - exit\" >&2\n exit 1\nfi\nIn case mkdir reports an error, the script will exit at this point - the MUTEX did its job!
If the directory is removed after setting a successful lock, while the script is still running, the lock is lost. Doing chmod -w for the parent directory containing the lock directory can be done, but it is not atomic. Maybe a while loop checking continously for the existence of the lock in the background and sending a signal such as USR1, if the directory is not found, can be done. The signal would need to be trapped. I am sure there there is a better solution than this suggestion --- sn18 2009/12/19 08:24*
Note: While perusing the Internet, I found some people asking if the mkdir method works \"on all filesystems\". Well, let's say it should. The syscall under mkdir is guarenteed to work atomicly in all cases, at least on Unices. Two examples of problems are NFS filesystems and filesystems on cluster servers. With those two scenarios, dependencies exist related to the mount options and implementation. However, I successfully use this simple method on an Oracle OCFS2 filesystem in a 4-node cluster environment. So let's just say \"it should work under normal conditions\".
Another atomic method is setting the noclobber shell option (set -C). That will cause redirection to fail, if the file the redirection points to already exists (using diverse open() methods). Need to write a code example here.
if ( set -o noclobber; echo \"locked\" > \"$lockfile\") 2> /dev/null; then\n trap 'rm -f \"$lockfile\"; exit $?' INT TERM EXIT\n echo \"Locking succeeded\" >&2\n rm -f \"$lockfile\"\nelse\n echo \"Lock failed - exit\" >&2\n exit 1\nfi\nAnother explanation of this basic pattern using set -C can be found here.
This code was taken from a production grade script that controls PISG to create statistical pages from my IRC logfiles. There are some differences compared to the very simple example above:
Details on how the script is killed aren't given, only code relevant to the locking process is shown:
#!/bin/bash\n\n# lock dirs/files\nLOCKDIR=\"/tmp/statsgen-lock\"\nPIDFILE=\"${LOCKDIR}/PID\"\n\n# exit codes and text\nENO_SUCCESS=0; ETXT[0]=\"ENO_SUCCESS\"\nENO_GENERAL=1; ETXT[1]=\"ENO_GENERAL\"\nENO_LOCKFAIL=2; ETXT[2]=\"ENO_LOCKFAIL\"\nENO_RECVSIG=3; ETXT[3]=\"ENO_RECVSIG\"\n\n###\n### start locking attempt\n###\n\ntrap 'ECODE=$?; echo \"[statsgen] Exit: ${ETXT[ECODE]}($ECODE)\" >&2' 0\necho -n \"[statsgen] Locking: \" >&2\n\nif mkdir \"${LOCKDIR}\" &>/dev/null; then\n\n # lock succeeded, install signal handlers before storing the PID just in case\n # storing the PID fails\n trap 'ECODE=$?;\n echo \"[statsgen] Removing lock. Exit: ${ETXT[ECODE]}($ECODE)\" >&2\n rm -rf \"${LOCKDIR}\"' 0\n echo \"$$\" >\"${PIDFILE}\"\n # the following handler will exit the script upon receiving these signals\n # the trap on \"0\" (EXIT) from above will be triggered by this trap's \"exit\" command!\n trap 'echo \"[statsgen] Killed by a signal.\" >&2\n exit ${ENO_RECVSIG}' 1 2 3 15\n echo \"success, installed signal handlers\"\n\nelse\n\n # lock failed, check if the other PID is alive\n OTHERPID=\"$(cat \"${PIDFILE}\")\"\n\n # if cat isn't able to read the file, another instance is probably\n # about to remove the lock -- exit, we're *still* locked\n # Thanks to Grzegorz Wierzowiecki for pointing out this race condition on\n # http://wiki.grzegorz.wierzowiecki.pl/code:mutex-in-bash\n if [ $? != 0 ]; then\n echo \"lock failed, PID ${OTHERPID} is active\" >&2\n exit ${ENO_LOCKFAIL}\n fi\n\n if ! kill -0 $OTHERPID &>/dev/null; then\n # lock is stale, remove it and restart\n echo \"removing stale lock of nonexistant PID ${OTHERPID}\" >&2\n rm -rf \"${LOCKDIR}\"\n echo \"[statsgen] restarting myself\" >&2\n exec \"$0\" \"$@\"\n else\n # lock is valid and OTHERPID is active - exit, we're locked!\n echo \"lock failed, PID ${OTHERPID} is active\" >&2\n exit ${ENO_LOCKFAIL}\n fi\n\nfi\npax can do a lot of fancy stuff, feel free to contribute more awesome pax tricks!
","tags":["bash","shell","scripting","POSIX","archive","tar","packing","zip"]},{"location":"howto/pax/#introduction","title":"Introduction","text":"The POSIX archiver, pax, is an attempt at a standardized archiver with the best features of tar and cpio, able to handle all common archive types.
However, this is not a manpage, it will not list all possible options, it will not you detailed information about pax. It's only an introduction.
This article is based on the debianized Berkeley implementation of pax, but implementation-specific things should be tagged as such. Unfortunately, the Debian package doesn't seem to be maintained anymore.
There are four basic operation modes to list, read, write and copy archives. They're switched with combinations of -r and -w command line options:
-r Write -w Copy -r -w","tags":["bash","shell","scripting","POSIX","archive","tar","packing","zip"]},{"location":"howto/pax/#list","title":"List","text":"In list mode, pax writes the list of archive members to standard output (a table of contents). If a pattern match is specified on the command line, only matching filenames are printed.
Read an archive. pax will read archive data and extract the members to the current directory. If a pattern match is specified on the command line, only matching filenames are extracted.
When reading an archive, the archive type is determined from the archive data.
","tags":["bash","shell","scripting","POSIX","archive","tar","packing","zip"]},{"location":"howto/pax/#write","title":"Write","text":"Write an archive, which means create a new one or append to an existing one. All files and directories specified on the command line are inserted into the archive. The archive is written to standard output by default.
If no files are specified on the command line, filenames are read from STDIN.
The write mode is the only mode where you need to specify the archive type with -x <TYPE>, e.g. -x ustar.
Copy mode is similar to cpio passthrough mode. It provides a way to replicate a complete or partial file hierarchy (with all the pax options, e.g. rewriting groups) to another location.
When you don't specify anything special, pax will attempt to read archive data from standard input (read/list modes) and write archive data to standard output (write mode). This ensures pax can be easily used as part of a shell pipe construct, e.g. to read a compressed archive that's decompressed in the pipe.
The option to specify the pathname of a file to be archived is -f This file will be used as input or output, depending on the operation (read/write/list).
When pax reads an archive, it tries to guess the archive type. However, in write mode, you must specify which type of archive to append using the -x <TYPE> switch. If you omit this switch, a default archive will be created (POSIX says it's implementation defined, Berkeley pax creates ustar if no options are specified).
The following archive formats are supported (Berkeley implementation):
ustar POSIX TAR format (default) cpio POSIX CPIO format tar classic BSD TAR format bcpio old binary CPIO format sv4cpio SVR4 CPIO format sv4crc SVR4 CPIO format with CRCBerkeley pax supports options -z and -j, similar to GNU tar, to filter archive files through GZIP/BZIP2.
In read and list modes, you can specify patterns to determine which files to list or extract.
extglob-c option, pax will invert the matches, i.e. it matches all filenames except those matching the specified patternspax will \"match\" (list or extract) all files from the archivepax -r <myarchive.tar 'data/sales/*.txt' 'data/products/*.png'\n\npax -r <myarchive.tar 'data/sales/year_200[135].txt'\n# should be equivalent to\npax -r <myarchive.tar 'data/sales/year_2001.txt' 'data/sales/year_2003.txt' 'data/sales/year_2005.txt'\nThis is a brief description of using pax as a normal archiver system, like you would use tar.
This task is done with basic syntax
# archive contents to stdout\npax -w >archive.tar README.txt *.png data/\n\n# equivalent, extract archive contents directly to a file\npax -w -x ustar -f archive.tar README.txt *.png data/\npax is in write mode, the given filenames are packed into an archive:
README.txt is a normal file, it will be packed*.png is a pathname glob for your shell, the shell will substitute all matching filenames before pax is executed. The result is a list of filenames that will be packed like the README.txt example abovedata/ is a directory. Everything in this directory will be packed into the archive, i.e. not just an empty directoryWhen you specify the -v option, pax will write the pathnames of the files inserted into the archive to STDERR.
When, and only when, no filename arguments are specified, pax attempts to read filenames from STDIN, separated by newlines. This way you can easily combine find with pax:
find . -name '*.txt' | pax -wf textfiles.tar -x ustar\nThe standard output format to list archive members simply is to print each filename to a separate line. But the output format can be customized to include permissions, timestamps, etc. with the -o listopt=<FORMAT> specification. The syntax of the format specification is strongly derived from the printf(3) format specification.
Unfortunately the pax utility delivered with Debian doesn't seem to support these extended listing formats.
However, pax lists archive members in a ls -l-like format, when you give the -v option:
pax -v <myarchive.tar\n# or, of course\npax -vf myarchive.tar\nYou can extract all files, or files (not) matching specific patterns from an archive using constructs like:
# \"normal\" extraction\npax -rf myarchive.tar '*.txt'\n\n# with inverted pattern\npax -rf myarchive.tar -c '*.txt'\nTo copy directory contents to another directory, similar to a cp -a command, use:
mkdir destdir\npax -rw dir destdir #creates a copy of dir in destdir/, i.e. destdir/dir\nTo copy directory contents to another directory on a remote system, use:
pax -w localdir | ssh user@host \"cd distantdest && pax -r -v\"\npax -w localdir | gzip | ssh user@host \"cd distantdir && gunzip | pax -r -v\" #compress the sent data\nThese commands create a copy of localdir in distandir (distantdir/dir) on the remote machine.
","tags":["bash","shell","scripting","POSIX","archive","tar","packing","zip"]},{"location":"howto/pax/#advanced-usage","title":"Advanced usage","text":"","tags":["bash","shell","scripting","POSIX","archive","tar","packing","zip"]},{"location":"howto/pax/#backup-your-daily-work","title":"Backup your daily work","text":"Note: -T is an extension and is not defined by POSIX.
Say you have write-access to a fileserver mounted on your filesystem tree. In copy mode, you can tell pax to copy only files that were modified today:
mkdir /n/mybackups/$(date +%A)/\npax -rw -T 0000 data/ /n/mybackups/$(date +%A)/\nThis is done using the -T switch, which normally allows you to specify a time window, but in this case, only the start time which means \"today at midnight\".
When you execute this \"very simple backup\" after your daily work, you will have a copy of the modified files.
Note: The %A format from date expands to the name of the current day, localized, e.g. \"Friday\" (en) or \"Mittwoch\" (de).
The same, but with an archive, can be accomplished by:
pax -w -T 0000 -f /n/mybackups/$(date +%A)\nIn this case, the day-name is an archive-file (you don't need a filename extension like .tar but you can add one, if desired).
pax is able to rewrite filenames while archiving or while extracting from an archive. This example creates a tar archive containing the holiday_2007/ directory, but the directory name inside the archive will be holiday_pics/:
pax -x ustar -w -f holiday_pictures.tar -s '/^holiday_2007/holiday_pics/' holiday_2007/\nThe option responsible for the string manipulation is the -s <REWRITE-SPECIFICATION>. It takes the string rewrite specification as an argument, in the form /OLD/NEW/[gp], which is an ed(1)-like regular expression (BRE) for old and generally can be used like the popular sed construct s/from/to/. Any non-null character can be used as a delimiter, so to mangle pathnames (containing slashes), you could use #/old/path#/new/path#.
The optional g and p flags are used to apply substitution (g)lobally to the line or to (p)rint the original and rewritten strings to STDERR.
Multiple -s options can be specified on the command line. They are applied to the pathname strings of the files or archive members. This happens in the order they are specified.
The -s command seen above can be used to exclude a file. The substitution must result in a null string: For example, let's say that you want to exclude all the CVS directories to create a source code archive. We are going to replace the names containing /CVS/ with nothing, note the .* they are needed because we need to match the entire pathname.
pax -w -x ustar -f release.tar -s',.*/CVS/.*,,' myapplication\nYou can use several -s options, for instance, let's say you also want to remove files ending in ~:
pax -w -x ustar -f release.tar -'s,.*/CVS/.*,,' -'s/.*~//' myapplication\nThis can also be done while reading an archive, for instance, suppose you have an archive containing a \"usr\" and a \"etc\" directory but that you want to extract only the \"usr\" directory:
pax -r -f archive.tar -s',^etc/.*,,' #the etc/ dir is not extracted\nLike cpio, pax can read filenames from standard input (stdin). This provides great flexibility - for example, a find(1) command may select files/directories in ways pax can't do itself. In write mode (creating an archive) or copy mode, when no filenames are given, pax expects to read filenames from standard input. For example:
# Back up config files changed less than 3 days ago\nfind /etc -type f -mtime -3 | pax -x ustar -w -f /backups/etc.tar\n\n# Copy only the directories, not the files\nmkdir /target\nfind . -type d -print | pax -r -w -d /target\n\n# Back up anything that changed since the last backup\nfind . -newer /var/run/mylastbackup -print0 |\n pax -0 -x ustar -w -d -f /backups/mybackup.tar\ntouch /var/run/mylastbackup\nThe -d option tells pax not to recurse into directories it reads (cpio-style). Without -d, pax recurses into all directories (tar-style).
Note: the -0 option is not standard, but is present in some implementations.
pax can handle the tar archive format, if you want to switch to the standard tool an alias like:
alias tar='echo USE PAX, idiot. pax is the standard archiver!; # '\nin your ~/.bashrc can be useful :-D.
Here is a quick table comparing (GNU) tar and pax to help you to make the switch:
tar xzvf file.tar.gz pax -rvz -f file.tar.gz -z is an extension, POSIXly: gunzip <file.tar.gz | pax -rv tar czvf archive.tar.gz path ... pax -wvz -f archive.tar.gz path ... -z is an extension, POSIXly: pax -wv path | gzip > archive.tar.gz tar xjvf file.tar.bz2 bunzip2 <file.tar.bz2 | pax -rv tar cjvf archive.tar.bz2 path ... pax -wv path | bzip2 > archive.tar.bz2 tar tzvf file.tar.gz pax -vz -f file.tar.gz -z is an extension, POSIXly: gunzip <file.tar.gz | pax -v pax might not create ustar (tar) archives by default but its own pax format, add -x ustar if you want to ensure pax creates tar archives!
This tutorial is not a complete guide to redirection, it will not cover here docs, here strings, name pipes etc... I just hope it'll help you to understand what things like 3>&2, 2>&1 or 1>&3- do.
When Bash starts, normally, 3 file descriptors are opened, 0, 1 and 2 also known as standard input (stdin), standard output (stdout) and standard error (stderr).
For example, with Bash running in a Linux terminal emulator, you'll see:
# lsof +f g -ap $BASHPID -d 0,1,2\nCOMMAND PID USER FD TYPE FILE-FLAG DEVICE SIZE/OFF NODE NAME\nbash 12135 root 0u CHR RW,LG 136,13 0t0 16 /dev/pts/5\nbash 12135 root 1u CHR RW,LG 136,13 0t0 16 /dev/pts/5\nbash 12135 root 2u CHR RW,LG 136,13 0t0 16 /dev/pts/5\nThis /dev/pts/5 is a pseudo terminal used to emulate a real terminal. Bash reads (stdin) from this terminal and prints via stdout and stderr to this terminal.
--- +-----------------------+\nstandard input ( 0 ) ---->| /dev/pts/5 |\n --- +-----------------------+\n\n --- +-----------------------+\nstandard output ( 1 ) ---->| /dev/pts/5 |\n --- +-----------------------+\n\n --- +-----------------------+\nstandard error ( 2 ) ---->| /dev/pts/5 |\n --- +-----------------------+\nWhen a command, a compound command, a subshell etc. is executed, it inherits these file descriptors. For instance echo foo will send the text foo to the file descriptor 1 inherited from the shell, which is connected to /dev/pts/5.
> is probably the simplest redirection.
echo foo > file
the > file after the command alters the file descriptors belonging to the command echo. It changes the file descriptor 1 (> file is the same as 1>file) so that it points to the file file. They will look like:
--- +-----------------------+\nstandard input ( 0 ) ---->| /dev/pts/5 |\n --- +-----------------------+\n\n --- +-----------------------+\nstandard output ( 1 ) ---->| file |\n --- +-----------------------+\n\n --- +-----------------------+\nstandard error ( 2 ) ---->| /dev/pts/5 |\n --- +-----------------------+\nNow characters written by our command, echo, that are sent to the standard output, i.e., the file descriptor 1, end up in the file named file.
In the same way, command 2> file will change the standard error and will make it point to file. Standard error is used by applications to print errors.
What will command 3> file do? It will open a new file descriptor pointing to file. The command will then start with:
--- +-----------------------+\nstandard input ( 0 ) ---->| /dev/pts/5 |\n --- +-----------------------+\n\n --- +-----------------------+\nstandard output ( 1 ) ---->| /dev/pts/5 |\n --- +-----------------------+\n\n --- +-----------------------+\nstandard error ( 2 ) ---->| /dev/pts/5 |\n --- +-----------------------+\n\n --- +-----------------------+\nnew descriptor ( 3 ) ---->| file |\n --- +-----------------------+\nWhat will the command do with this descriptor? It depends. Often nothing. We will see later why we might want other file descriptors.
","tags":["bash","shell","scripting","tutorial","redirection","redirect","file","descriptor"]},{"location":"howto/redirection_tutorial/#input-redirection-n-file","title":"Input Redirection \"n< file\"","text":"When you run a commandusing command < file, it changes the file descriptor 0 so that it looks like:
--- +-----------------------+\nstandard input ( 0 ) <----| file |\n --- +-----------------------+\n\n --- +-----------------------+\nstandard output ( 1 ) ---->| /dev/pts/5 |\n --- +-----------------------+\n\n --- +-----------------------+\nstandard error ( 2 ) ---->| /dev/pts/5 |\n --- +-----------------------+\nIf the command reads from stdin, it now will read from file and not from the console.
As with >, < can be used to open a new file descriptor for reading, command 3<file. Later we will see how this can be useful.
What does this | do? Among other things, it connects the standard output of the command on the left to the standard input of the command on the right. That is, it creates a special file, a pipe, which is opened as a write destinaton for the left command, and as a read source for the right command.
echo foo | cat\n\n --- +--------------+ --- +--------------+\n( 0 ) ---->| /dev/pts/5 | ------> ( 0 ) ---->|pipe (read) |\n --- +--------------+ / --- +--------------+\n /\n --- +--------------+ / --- +--------------+\n( 1 ) ---->| pipe (write) | / ( 1 ) ---->| /dev/pts |\n --- +--------------+ --- +--------------+\n\n --- +--------------+ --- +--------------+\n( 2 ) ---->| /dev/pts/5 | ( 2 ) ---->| /dev/pts/ |\n --- +--------------+ --- +--------------+\nThis is possible because the redirections are set up by the shell before the commands are executed, and the commands inherit the file descriptors.
","tags":["bash","shell","scripting","tutorial","redirection","redirect","file","descriptor"]},{"location":"howto/redirection_tutorial/#more-on-file-descriptors","title":"More On File Descriptors","text":"","tags":["bash","shell","scripting","tutorial","redirection","redirect","file","descriptor"]},{"location":"howto/redirection_tutorial/#duplicating-file-descriptor-21","title":"Duplicating File Descriptor 2>&1","text":"We have seen how to open (or redirect) file descriptors. Let us see how to duplicate them, starting with the classic 2>&1. What does this mean? That something written on the file descriptor 2 will go where file descriptor 1 goes. In a shell command 2>&1 is not a very interesting example so we will use ls /tmp/ doesnotexist 2>&1 | less
ls /tmp/ doesnotexist 2>&1 | less\n\n --- +--------------+ --- +--------------+\n( 0 ) ---->| /dev/pts/5 | ------> ( 0 ) ---->|from the pipe |\n --- +--------------+ / ---> --- +--------------+\n / /\n --- +--------------+ / / --- +--------------+\n( 1 ) ---->| to the pipe | / / ( 1 ) ---->| /dev/pts |\n --- +--------------+ / --- +--------------+\n /\n --- +--------------+ / --- +--------------+\n( 2 ) ---->| to the pipe | / ( 2 ) ---->| /dev/pts/ |\n --- +--------------+ --- +--------------+\nWhy is it called duplicating? Because after 2>&1, we have 2 file descriptors pointing to the same file. Take care not to call this \"File Descriptor Aliasing\"; if we redirect stdout after 2>&1 to a file B, file descriptor 2 will still be opened on the file A where it was. This is often misunderstood by people wanting to redirect both standard input and standard output to the file. Continue reading for more on this.
So if you have two file descriptors s and t like:
--- +-----------------------+\n a descriptor ( s ) ---->| /some/file |\n --- +-----------------------+\n --- +-----------------------+\n a descriptor ( t ) ---->| /another/file |\n --- +-----------------------+\nUsing a t>&s (where t and s are numbers) it means:
Copy whatever file descriptor s contains into file descriptor t
So you got a copy of this descriptor:
--- +-----------------------+\n a descriptor ( s ) ---->| /some/file |\n --- +-----------------------+\n --- +-----------------------+\n a descriptor ( t ) ---->| /some/file |\n --- +-----------------------+\nInternally each of these is represented by a file descriptor opened by the operating system's fopen calls, and is likely just a pointer to the file which has been opened for reading (stdin or file descriptor 0) or writing (stdout /stderr).
Note that the file reading or writing positions are also duplicated. If you have already read a line of s, then after t>&s if you read a line from t, you will get the second line of the file.
Similarly for output file descriptors, writing a line to file descriptor s will append a line to a file as will writing a line to file descriptor t.
Tip
The syntax is somewhat confusing in that you would think that the arrow would point in the direction of the copy, but it's reversed. So it's target>&source effectively.
So, as a simple example (albeit slightly contrived), is the following:
exec 3>&1 # Copy 1 into 3\nexec 1> logfile # Make 1 opened to write to logfile\nlotsa_stdout # Outputs to fd 1, which writes to logfile\nexec 1>&3 # Copy 3 back into 1\necho Done # Output to original stdout\n> file 2>&1\" vs. \"2>&1 >file\"","text":"While it doesn't matter where the redirections appears on the command line, their order does matter. They are set up from left to right.
2>&1 >fileA common error, is to do command 2>&1 > file to redirect both stderr and stdout to file. Let's see what's going on. First we type the command in our terminal, the descriptors look like this:
--- +-----------------------+\nstandard input ( 0 ) ---->| /dev/pts/5 |\n --- +-----------------------+\n\n --- +-----------------------+\nstandard output ( 1 ) ---->| /dev/pts/5 |\n --- +-----------------------+\n\n --- +-----------------------+\nstandard error ( 2 ) ---->| /dev/pts/5 |\n --- +-----------------------+\nThen our shell, Bash sees 2>&1 so it duplicates 1, and the file descriptor look like this:
--- +-----------------------+\nstandard input ( 0 ) ---->| /dev/pts/5 |\n --- +-----------------------+\n\n --- +-----------------------+\nstandard output ( 1 ) ---->| /dev/pts/5 |\n --- +-----------------------+\n\n --- +-----------------------+\nstandard error ( 2 ) ---->| /dev/pts/5 |\n --- +-----------------------+\nThat's right, nothing has changed, 2 was already pointing to the same place as 1. Now Bash sees > file and thus changes stdout:
--- +-----------------------+\nstandard input ( 0 ) ---->| /dev/pts/5 |\n --- +-----------------------+\n\n --- +-----------------------+\nstandard output ( 1 ) ---->| file |\n --- +-----------------------+\n\n --- +-----------------------+\nstandard error ( 2 ) ---->| /dev/pts/5 |\n --- +-----------------------+\nAnd that's not what we want.
>file 2>&1Now let's look at the correct command >file 2>&1. We start as in the previous example, and Bash sees > file:
--- +-----------------------+\nstandard input ( 0 ) ---->| /dev/pts/5 |\n --- +-----------------------+\n\n --- +-----------------------+\nstandard output ( 1 ) ---->| file |\n --- +-----------------------+\n\n --- +-----------------------+\nstandard error ( 2 ) ---->| /dev/pts/5 |\n --- +-----------------------+\nThen it sees our duplication 2>&1:
--- +-----------------------+\nstandard input ( 0 ) ---->| /dev/pts/5 |\n --- +-----------------------+\n\n --- +-----------------------+\nstandard output ( 1 ) ---->| file |\n --- +-----------------------+\n\n --- +-----------------------+\nstandard error ( 2 ) ---->| file |\n --- +-----------------------+\nAnd voila, both 1 and 2 are redirected to file.
This is a common error, we want to modify a file using something that reads from a file and writes the result to stdout. To do this, we redirect stdout to the file we want to modify. The problem here is that, as we have seen, the redirections are setup before the command is actually executed.
So BEFORE sed starts, standard output has already been redirected, with the additional side effect that, because we used >, \"file\" gets truncated. When sed starts to read the file, it contains nothing.
In Bash the exec built-in replaces the shell with the specified program. So what does this have to do with redirection? exec also allow us to manipulate the file descriptors. If you don't specify a program, the redirection after exec modifies the file descriptors of the current shell.
For example, all the commands after exec 2>file will have file descriptors like:
--- +-----------------------+\nstandard input ( 0 ) ---->| /dev/pts/5 |\n --- +-----------------------+\n\n --- +-----------------------+\nstandard output ( 1 ) ---->| /dev/pts/5 |\n --- +-----------------------+\n\n --- +-----------------------+\nstandard error ( 2 ) ---->| file |\n --- +-----------------------+\nAll the the errors sent to stderr by the commands after the exec 2>file will go to the file, just as if you had the command in a script and ran myscript 2>file.
exec can be used, if, for instance, you want to log the errors the commands in your script produce, just add exec 2>myscript.errors at the beginning of your script.
Let's see another use case. We want to read a file line by line, this is easy, we just do:
while read -r line;do echo \"$line\";done < file\nNow, we want, after printing each line, to do a pause, waiting for the user to press a key:
while read -r line;do echo \"$line\"; read -p \"Press any key\" -n 1;done < file\nAnd, surprise this doesn't work. Why? because the shell descriptor of the while loop looks like:
--- +-----------------------+\nstandard input ( 0 ) ---->| file |\n --- +-----------------------+\n\n --- +-----------------------+\nstandard output ( 1 ) ---->| /dev/pts/5 |\n --- +-----------------------+\n\n --- +-----------------------+\nstandard error ( 2 ) ---->| /dev/pts/5 |\n --- +-----------------------+\nand our read inherits these descriptors, and our command (read -p \"Press any key\" -n 1) inherits them, and thus reads from file and not from our terminal.
A quick look at help read tells us that we can specify a file descriptor from which read should read. Cool. Now let's use exec to get another descriptor:
exec 3<file\n while read -u 3 line;do echo \"$line\"; read -p \"Press any key\" -n 1;done\nNow the file descriptors look like:
--- +-----------------------+\nstandard input ( 0 ) ---->| /dev/pts/5 |\n --- +-----------------------+\n\n --- +-----------------------+\nstandard output ( 1 ) ---->| /dev/pts/5 |\n --- +-----------------------+\n\n --- +-----------------------+\nstandard error ( 2 ) ---->| /dev/pts/5 |\n --- +-----------------------+\n\n --- +-----------------------+\nnew descriptor ( 3 ) ---->| file |\n --- +-----------------------+\nand it works.
","tags":["bash","shell","scripting","tutorial","redirection","redirect","file","descriptor"]},{"location":"howto/redirection_tutorial/#closing-the-file-descriptors","title":"Closing The File Descriptors","text":"Closing a file through a file descriptor is easy, just make it a duplicate of -. For instance, let's close stdin <&- and stderr 2>&-:
bash -c '{ lsof -a -p $$ -d0,1,2 ;} <&- 2>&-'\n COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME\n bash 10668 pgas 1u CHR 136,2 4 /dev/pts/2\nwe see that inside the {} that only 1 is still here.
Though the OS will probably clean up the mess, it is perhaps a good idea to close the file descriptors you open. For instance, if you open a file descriptor with exec 3>file, all the commands afterwards will inherit it. It's probably better to do something like:
exec 3>file\n.....\n#commands that uses 3\n.....\nexec 3>&-\n\n#we don't need 3 any more\nI've seen some people using this as a way to discard, say stderr, using something like: command 2>&-. Though it might work, I'm not sure if you can expect all applications to behave correctly with a closed stderr.
When in doubt, I use 2>/dev/null.
","tags":["bash","shell","scripting","tutorial","redirection","redirect","file","descriptor"]},{"location":"howto/redirection_tutorial/#an-example","title":"An Example","text":"This example comes from this post (ffe4c2e382034ed9) on the comp.unix.shell group:
{\n {\n cmd1 3>&- |\n cmd2 2>&3 3>&-\n } 2>&1 >&4 4>&- |\n cmd3 3>&- 4>&-\n\n} 3>&2 4>&1\nThe redirections are processed from left to right, but as the file descriptors are inherited we will also have to work from the outer to the inner contexts. We will assume that we run this command in a terminal. Let's start with the outer { } 3>&2 4>&1.
--- +-------------+ --- +-------------+\n( 0 ) ---->| /dev/pts/5 | ( 3 ) ---->| /dev/pts/5 |\n --- +-------------+ --- +-------------+\n\n --- +-------------+ --- +-------------+\n( 1 ) ---->| /dev/pts/5 | ( 4 ) ---->| /dev/pts/5 |\n --- +-------------+ --- +-------------+\n\n --- +-------------+\n( 2 ) ---->| /dev/pts/5 |\n --- +-------------+\nWe only made 2 copies of stderr and stdout. 3>&1 4>&1 would have produced the same result here because we ran the command in a terminal and thus 1 and 2 go to the terminal. As an exercise, you can start with 1 pointing to file.stdout and 2 pointing to file.stderr, you will see why these redirections are very nice.
Let's continue with the right part of the second pipe: | cmd3 3>&- 4>&-
--- +-------------+\n( 0 ) ---->| 2nd pipe |\n --- +-------------+\n\n --- +-------------+\n( 1 ) ---->| /dev/pts/5 |\n --- +-------------+\n\n --- +-------------+\n( 2 ) ---->| /dev/pts/5 |\n --- +-------------+\nIt inherits the previous file descriptors, closes 3 and 4 and sets up a pipe for reading. Now for the left part of the second pipe {...} 2>&1 >&4 4>&- |
--- +-------------+ --- +-------------+\n( 0 ) ---->| /dev/pts/5 | ( 3 ) ---->| /dev/pts/5 |\n --- +-------------+ --- +-------------+\n\n --- +-------------+\n( 1 ) ---->| /dev/pts/5 |\n --- +-------------+\n\n --- +-------------+\n( 2 ) ---->| 2nd pipe |\n --- +-------------+\nFirst, The file descriptor 1 is connected to the pipe (|), then 2 is made a copy of 1 and thus is made an fd to the pipe (2>&1), then 1 is made a copy of 4 (>&4), then 4 is closed. These are the file descriptors of the inner {}. Lcet's go inside and have a look at the right part of the first pipe: | cmd2 2>&3 3>&-
--- +-------------+\n( 0 ) ---->| 1st pipe |\n --- +-------------+\n\n --- +-------------+\n( 1 ) ---->| /dev/pts/5 |\n --- +-------------+\n\n --- +-------------+\n( 2 ) ---->| /dev/pts/5 |\n --- +-------------+\nIt inherits the previous file descriptors, connects 0 to the 1st pipe, the file descriptor 2 is made a copy of 3, and 3 is closed. Finally, for the left part of the pipe:
--- +-------------+\n( 0 ) ---->| /dev/pts/5 |\n --- +-------------+\n\n --- +-------------+\n( 1 ) ---->| 1st pipe |\n --- +-------------+\n\n --- +-------------+\n( 2 ) ---->| 2nd pipe |\n --- +-------------+\nIt also inherits the file descriptor of the left part of the 2nd pipe, file descriptor 1 is connected to the first pipe, 3 is closed.
The purpose of all this becomes clear if we take only the commands:
cmd2\n\n --- +-------------+\n -->( 0 ) ---->| 1st pipe |\n / --- +-------------+\n /\n / --- +-------------+\n cmd 1 / ( 1 ) ---->| /dev/pts/5 |\n / --- +-------------+\n /\n --- +-------------+ / --- +-------------+\n( 0 ) ---->| /dev/pts/5 | / ( 2 ) ---->| /dev/pts/5 |\n --- +-------------+ / --- +-------------+\n /\n --- +-------------+ / cmd3\n( 1 ) ---->| 1st pipe | /\n --- +-------------+ --- +-------------+\n ------------>( 0 ) ---->| 2nd pipe |\n --- +-------------+ / --- +-------------+\n( 2 ) ---->| 2nd pipe |/\n --- +-------------+ --- +-------------+\n ( 1 ) ---->| /dev/pts/5 |\n --- +-------------+\n\n --- +-------------+\n ( 2 ) ---->| /dev/pts/5 |\n --- +-------------+\nAs said previously, as an exercise, you can start with 1 open on a file and 2 open on another file to see how the stdin from cmd2 and cmd3 goes to the original stdin and how the stderr goes to the original stderr.
I used to have trouble choosing between 0&<3 3&>1 3>&1 ->2 -<&0 &-<0 0<&- etc... (I think probably because the syntax is more representative of the result, i.e., the redirection, than what is done, i.e., opening, closing, or duplicating file descriptors).
If this fits your situation, then maybe the following \"rules\" will help you, a redirection is always like the following:
lhs op rhs\nlhs is always a file description, i.e., a number:
< then there is an implicit 0, if it's > or >>, there is an implicit 1.op is <, >, >>, >|, or <>:
< if the file decriptor in lhs will be read, > if it will be written, >> if data is to be appended to the file, >| to overwrite an existing file or <> if it will be both read and written.rhs is the thing that the file descriptor will describe:
&1), or, &-, which will close the file descriptor.You might not like this description, and find it a bit incomplete or inexact, but I think it really helps to easily find that, say &->0 is incorrect.
The shell is pretty loose about what it considers a valid redirect. While opinions probably differ, this author has some (strong) recommendations:
&>foo and >&foo shorthand redirects. Use the long form >foo 2>&1. (see: obsolete)# Good! This is clearly a simple commmand with two arguments and 4 redirections\ncmd arg1 arg2 <myFile 3<&1 2>/dev/null >&2\n\n# Good!\n{ cmd1 <<<'my input'; cmd2; } >someFile\n\n# Bad. Is the \"1\" a file descriptor or an argument to cmd? (answer: it's the FD). Is the space after the herestring part of the input data? (answer: No).\n# The redirects are also not delimited in any obvious way.\ncmd 2>& 1 <<< stuff\n\n# Hideously Bad. It's difficult to tell where the redirects are and whether they're even valid redirects.\n# This is in fact one command with one argument, an assignment, and three redirects.\nfoo=bar<baz bork<<< blarg>bleh\nI hope this tutorial worked for you.
I lied, I did not explain 1>&3-, go check the manual ;-)
Thanks to St\u00e9phane Chazelas from whom I stole both the intro and the example....
The intro is inspired by this introduction, you'll find a nice exercise there too:
The last example comes from this post:
The one of the simplest way to check your bash/sh scripts is run it and check it output or run it and check the result. This tutorial shows how-to use bashtest tool for testing your scripts.
"},{"location":"howto/testing-your-scripts/#write-simple-util","title":"Write simple util","text":"We have a simple stat.sh script:
#!/usr/bin/env bash\n\nif [ -z \"$1\" ]\nthen\n DIR=./\nelse\n DIR=$1\nfi\n\necho \"Evaluate *.py statistics\"\nFILES=$(find $DIR -name '*.py' | wc -l)\nLINES=$((find $DIR -name '*.py' -print0 | xargs -0 cat) | wc -l)\necho \"PYTHON FILES: $FILES\"\necho \"PYTHON LINES: $LINES\"\nThis script evaluate the number of python files and the number of python code lines in the files. We can use it like ./stat.sh <dir>
Then make test suits for stat.sh. We make a directory testsuit which contain test python files.
testsuit/main.py
import foo\nprint(foo)\ntestsuit/foo.py
BAR = 1\nBUZ = BAR + 2\nOk! Our test suit is ready! We have 2 python files which contains 4 lines of code.
"},{"location":"howto/testing-your-scripts/#write-bashtests","title":"Write bashtests","text":"Lets write tests. We just write a shell command for testing our work.
Create file tests.bashtest:
$ ./stat.sh testsuit/\nEvaluate *.py statistics\nPYTHON FILES: 2\nPYTHON LINES: 4\nThis is our test! This is simple. Try to run it.
# install bashtest if required!\n$ pip install bashtest\n\n# run tests\n$ bashtest *.bashtest\n1 items passed all tests:\n 1 tests in tests.bashtest\n1 tests in 1 items.\n1 passed and 0 failed.\nTest passed.\nThats all. We wrote one test. You can write more tests if you want.
$ ls testsuit/\nfoo.py main.py\n\n$ ./stat.sh testsuit/\nEvaluate *.py statistics\nPYTHON FILES: 2\nPYTHON LINES: 4\nAnd run tests again:
$ bashtest *.bashtest\n1 items passed all tests:\n 2 tests in tests.bashtest\n2 tests in 1 items.\n2 passed and 0 failed.\nTest passed.\nYou can find more .bashtest examples in the bashtest github repo. You can also write your question or report a bug here.
Happy testing!
"},{"location":"internals/shell_options/","title":"List of shell options","text":"This information was taken from a Bash version \"4.1\", every now and then new options are added, so likely, this list isn't complete.
The shell-options can be set with the shopt builtin command.
","tags":["bash","shell","scripting","options","runtime","variable","behaviour"]},{"location":"internals/shell_options/#shell-options","title":"Shell options","text":"","tags":["bash","shell","scripting","options","runtime","variable","behaviour"]},{"location":"internals/shell_options/#autocd","title":"autocd","text":"Option:autocd Since: 4.0-alpha Shell mode: interactive only Default: off If set, a command name that is the name of a directory is executed as if it were the argument to the cd command.
","tags":["bash","shell","scripting","options","runtime","variable","behaviour"]},{"location":"internals/shell_options/#assoc_expand_once","title":"assoc_expand_once","text":"Option:assoc_expand_once Since: 5.0-alpha Shell mode: all Default: off If set, Bash attempts to expand associative array options only once.
","tags":["bash","shell","scripting","options","runtime","variable","behaviour"]},{"location":"internals/shell_options/#cdable_vars","title":"cdable_vars","text":"Option:cdable_vars Since: unknown Shell mode: all Default: off Treat every non-directory argument to the cd-command as variable name containing a directory to cd into.
cdspell Since: unknown Shell mode: interactive only Default: off If set, minor errors in the spelling of a directory component in a cd command will be corrected. The errors checked for are transposed characters, a missing character, and one character too many. If a correction is found, the corrected file name is printed, and the command proceeds.
","tags":["bash","shell","scripting","options","runtime","variable","behaviour"]},{"location":"internals/shell_options/#checkhash","title":"checkhash","text":"Option:checkhash Since: unknown Shell mode: all Default: off If set, Bash checks that a command found in the hash table exists before trying to execute it. If a hashed command no longer exists, a normal path search is performed.
","tags":["bash","shell","scripting","options","runtime","variable","behaviour"]},{"location":"internals/shell_options/#checkjobs","title":"checkjobs","text":"Option:checkjobs Since: 4.0-alpha Shell mode: interactive only Default: off If set, Bash lists the status of any stopped and running jobs before exiting an interactive shell. If any jobs are running, this causes the exit to be deferred until a second exit is attempted without an intervening command. The shell always postpones exiting if any jobs are stopped.
","tags":["bash","shell","scripting","options","runtime","variable","behaviour"]},{"location":"internals/shell_options/#checkwinsize","title":"checkwinsize","text":"Option:checkwinsize Since: unknown Shell mode: all Default: on If set, Bash checks the window size after each command and, if necessary, updates the values of the variables LINES and COLUMNS.
","tags":["bash","shell","scripting","options","runtime","variable","behaviour"]},{"location":"internals/shell_options/#cmdhist","title":"cmdhist","text":"Option:cmdhist Since: unknown Shell mode: all Default: off If set, Bash attempts to save all lines of a multiple-line command in the same history entry. This allows easy re-editing of multi-line commands.
","tags":["bash","shell","scripting","options","runtime","variable","behaviour"]},{"location":"internals/shell_options/#compat31","title":"compat31","text":"Option:compat31 Since: 3.2 Shell mode: all Default: off Compatiblity mode for Bash 3.1
","tags":["bash","shell","scripting","options","runtime","variable","behaviour"]},{"location":"internals/shell_options/#compat32","title":"compat32","text":"Option:compat32 Since: 4.0 Shell mode: all Default: off Compatiblity mode for Bash 3.2
","tags":["bash","shell","scripting","options","runtime","variable","behaviour"]},{"location":"internals/shell_options/#compat40","title":"compat40","text":"Option:compat40 Since: 4.1-beta Shell mode: all Default: off Compatiblity mode for Bash 4.0
","tags":["bash","shell","scripting","options","runtime","variable","behaviour"]},{"location":"internals/shell_options/#compat41","title":"compat41","text":"Option:compat41 Since: 4.2-alpha Shell mode: all Default: off Compatiblity mode for Bash 4.1
","tags":["bash","shell","scripting","options","runtime","variable","behaviour"]},{"location":"internals/shell_options/#compat42","title":"compat42","text":"Option:compat42 Since: 4.3-alpha Shell mode: all Default: off Compatiblity mode for Bash 4.2
","tags":["bash","shell","scripting","options","runtime","variable","behaviour"]},{"location":"internals/shell_options/#compat43","title":"compat43","text":"Option:compat43 Since: 4.4-alpha Shell mode: all Default: off Compatiblity mode for Bash 4.3
","tags":["bash","shell","scripting","options","runtime","variable","behaviour"]},{"location":"internals/shell_options/#compat44","title":"compat44","text":"Option:compat44 Since: 5.0-alpha Shell mode: all Default: off Compatiblity mode for Bash 4.4
","tags":["bash","shell","scripting","options","runtime","variable","behaviour"]},{"location":"internals/shell_options/#direxpand","title":"direxpand","text":"Option:direxpand Since: 4.3-alpha Shell mode: all Default: off (unless changed on compile-time with --enable-direxpand-default) If set, bash replaces directory names with the results of word expansion when performing filename completion. This changes the contents of the readline editing buffer. If not set, bash attempts to preserve what the user typed.
","tags":["bash","shell","scripting","options","runtime","variable","behaviour"]},{"location":"internals/shell_options/#dirspell","title":"dirspell","text":"Option:dirspell Since: 4.0-alpha Shell mode: all Default: off If set, Bash will perform spelling corrections on directory names to match a glob.
","tags":["bash","shell","scripting","options","runtime","variable","behaviour"]},{"location":"internals/shell_options/#dotglob","title":"dotglob","text":"Option:dotglob Since: unknown Shell mode: all Default: off If set, Bash includes filenames beginning with a . (dot) in the results of pathname expansion.
execfail Since: unknown Shell mode: non-interactive Default: off If set, a non-interactive shell will not exit if it cannot execute the file specified as an argument to the exec-builtin command. An interactive shell does not exit if exec fails.
expand_aliases Since: unknown Shell mode: all Default: on (interactive), off (non-interactive) If set, aliases are expanded. This option is enabled by default for interactive shells.
","tags":["bash","shell","scripting","options","runtime","variable","behaviour"]},{"location":"internals/shell_options/#extdebug","title":"extdebug","text":"Option:extdebug Since: 3.0-alpha Shell mode: all Default: off If set, behavior intended for use by debuggers is enabled.
","tags":["bash","shell","scripting","options","runtime","variable","behaviour"]},{"location":"internals/shell_options/#extglob","title":"extglob","text":"Option:extglob Since: 2.02-alpha1 Shell mode: all Default: off If set, the extended pattern matching features are enabled. See the important note below under Parser configurations.
","tags":["bash","shell","scripting","options","runtime","variable","behaviour"]},{"location":"internals/shell_options/#extquote","title":"extquote","text":"Option:extquote Since: 3.0-alpha (?) Shell mode: all Default: on If set, $'string' and $\"string\" quoting is performed within parameter expansions enclosed in double quotes. See the important note below under Parser configurations.
failglob Since: 3.0-alpha Shell mode: all Default: off If set, patterns which fail to match filenames during pathname expansion result in an error message.
","tags":["bash","shell","scripting","options","runtime","variable","behaviour"]},{"location":"internals/shell_options/#force_fignore","title":"force_fignore","text":"Option:force_fignore Since: 3.0-alpha Shell mode: interactive Default: on If set, the suffixes specified by the FIGNORE shell variable cause words to be ignored when performing word completion even if the ignored words are the only possible completions. This option is enabled by default.
","tags":["bash","shell","scripting","options","runtime","variable","behaviour"]},{"location":"internals/shell_options/#globasciiranges","title":"globasciiranges","text":"Option:globasciiranges Since: 4.3-alpha Shell mode: all Default: on (configurable at compile time) If set, range expressions used in pattern matching behave as if in the traditional C locale when performing comparisons. That is, the current locale's collating sequence is not taken into account, so b will not collate between A and B, and upper-case and lower-case ASCII characters will collate together.
","tags":["bash","shell","scripting","options","runtime","variable","behaviour"]},{"location":"internals/shell_options/#globstar","title":"globstar","text":"Option:globstar Since: 4.0-alpha Shell mode: all Default: off If set, recursive globbing with ** is enabled.
gnu_errfmt Since: 3.0-alpha Shell mode: all Default: off If set, shell error messages are written in the \"standard GNU error message format\".
","tags":["bash","shell","scripting","options","runtime","variable","behaviour"]},{"location":"internals/shell_options/#histappend","title":"histappend","text":"Option:histappend Since: unknown Shell mode: interactive (?) Default: off If set, the history list is appended to the file named by the value of the HISTFILE variable when the shell exits, rather than overwriting the file.
","tags":["bash","shell","scripting","options","runtime","variable","behaviour"]},{"location":"internals/shell_options/#histreedit","title":"histreedit","text":"Option:histreedit Since: unknown Shell mode: interactive (?) Default: off If set, and readline is being used, a user is given the opportunity to re-edit a failed history substitution.
","tags":["bash","shell","scripting","options","runtime","variable","behaviour"]},{"location":"internals/shell_options/#histverify","title":"histverify","text":"Option:histverify Since: unknown Shell mode: interactive (?) Default: off Allow to review a history substitution result by loading the resulting line into the editing buffer, rather than directly executing it.
","tags":["bash","shell","scripting","options","runtime","variable","behaviour"]},{"location":"internals/shell_options/#hostcomplete","title":"hostcomplete","text":"Option:hostcomplete Since: 2.0-alpha3 Shell mode: interactive (?) Default: on If set, Bash completion also completes hostnames. On by default.
","tags":["bash","shell","scripting","options","runtime","variable","behaviour"]},{"location":"internals/shell_options/#huponexit","title":"huponexit","text":"Option:huponexit Since: 2.02-alpha1 Shell mode: interactive login Default: off If set, Bash will send the SIGHUP signal to all jobs when an interactive login shell exits.
interactive_comments Since: unknown Shell mode: interactive Default: on Allow commenting in interactive shells, on by default.
","tags":["bash","shell","scripting","options","runtime","variable","behaviour"]},{"location":"internals/shell_options/#lastpipe","title":"lastpipe","text":"Option:lastpipe Since: 4.2-alpha Shell mode: all Default: off If set, and job control is not active, the shell runs the last command of a pipeline not executed in the background in the current shell environment.
","tags":["bash","shell","scripting","options","runtime","variable","behaviour"]},{"location":"internals/shell_options/#lithist","title":"lithist","text":"Option:lithist Since: unknown Shell mode: interactive Default: off If set, and the #cmdhist option is enabled, multi-line commands are saved to the history with embedded newlines rather than using semicolon separators where possible.
","tags":["bash","shell","scripting","options","runtime","variable","behaviour"]},{"location":"internals/shell_options/#localvar_inherit","title":"localvar_inherit","text":"Option:localvar_inherit Since: 5.0-alpha Shell mode: all Default: off If this option is set, a local variable inherits the value of a variable with the same name at the nearest preceding scope.
","tags":["bash","shell","scripting","options","runtime","variable","behaviour"]},{"location":"internals/shell_options/#login_shell","title":"login_shell","text":"Option:login_shell Since: 2.05a-alpha1 Shell mode: all Default: n/a The option is set when Bash is a login shell. This is a readonly option.
","tags":["bash","shell","scripting","options","runtime","variable","behaviour"]},{"location":"internals/shell_options/#mailwarn","title":"mailwarn","text":"Option:mailwarn Since: unknown Shell mode: interactive (?) Default: off If set, and a file that Bash is checking for mail has been accessed since the last time it was checked, the message \"The mail in mailfile has been read\" is displayed.
","tags":["bash","shell","scripting","options","runtime","variable","behaviour"]},{"location":"internals/shell_options/#no_empty_cmd_completion","title":"no_empty_cmd_completion","text":"Option:mailwarn Since: unknown Shell mode: interactive (?) Default: off If set, and readline is being used, Bash will not attempt to search the PATH for possible completions when completion is attempted on an empty line.
","tags":["bash","shell","scripting","options","runtime","variable","behaviour"]},{"location":"internals/shell_options/#nocaseglob","title":"nocaseglob","text":"Option:nocaseglob Since: 2.02-alpha1 Shell mode: all Default: off If set, Bash matches filenames in a case-insensitive fashion when performing pathname expansion.
","tags":["bash","shell","scripting","options","runtime","variable","behaviour"]},{"location":"internals/shell_options/#nocasematch","title":"nocasematch","text":"Option:nocasematch Since: 3.1-alpha1 Shell mode: all Default: off If set, Bash matches patterns in a case-insensitive fashion when performing matching while executing case or [[ conditional commands.
nullglob Since: unknown Shell mode: all Default: off If set, Bash allows patterns which match no files to expand to a null string, rather than themselves.
","tags":["bash","shell","scripting","options","runtime","variable","behaviour"]},{"location":"internals/shell_options/#progcomp","title":"progcomp","text":"Option:progcomp Since: 2.04-alpha1 Shell mode: interactive (?) Default: on If set, the programmable completion facilities are enabled. This option is enabled by default.
","tags":["bash","shell","scripting","options","runtime","variable","behaviour"]},{"location":"internals/shell_options/#promptvars","title":"promptvars","text":"Option:promptvars Since: unknown Shell mode: interactive Default: on If set, prompt strings undergo parameter expansion, command substitution, arithmetic expansion, and quote removal after being expanded using the prompt special sequences. This option is enabled by default.
","tags":["bash","shell","scripting","options","runtime","variable","behaviour"]},{"location":"internals/shell_options/#restricted_shell","title":"restricted_shell","text":"Option:restricted_shell Since: 2.03-alpha Shell mode: interactive (?) Default: off The option is set when Bash is a restricted shell. This is a readonly option.
","tags":["bash","shell","scripting","options","runtime","variable","behaviour"]},{"location":"internals/shell_options/#shift_verbose","title":"shift_verbose","text":"Option:shift_verbose Since: unknown Shell mode: all Default: off, on in POSIX mode If set, the shift builtin prints an error message when the shift count exceeds the number of positional parameters.
","tags":["bash","shell","scripting","options","runtime","variable","behaviour"]},{"location":"internals/shell_options/#sourcepath","title":"sourcepath","text":"Option:sourcepath Since: unknown Shell mode: all Default: on If set, the source builtin command uses the value of PATH to find the directory containing the file supplied as an argument. This option is enabled by default.
","tags":["bash","shell","scripting","options","runtime","variable","behaviour"]},{"location":"internals/shell_options/#syslog_history","title":"syslog_history","text":"Option:syslog_history Since: 5.0-alpha Shell mode: unknown Default: off If set, the shell history is sent to syslog.
This option is undocumented and available only if the shell supports syslog.
","tags":["bash","shell","scripting","options","runtime","variable","behaviour"]},{"location":"internals/shell_options/#xpg_echo","title":"xpg_echo","text":"Option:xpg_echo Since: 2.04-beta1 Shell mode: all Default: off If set, the echo-builtin command expands backslash-escape sequences by default (POSIX, SUS, XPG).
Parser configurations change the way the Bash parser recognizes the syntax when parsing a line. This, of course, is impossible for a line that already was parsed.
There are two options that influence the parsing this way:
extglobextquoteConsequence: You can't use the new syntax (e.g. the extended globbing syntax) and the command to enable it in the same line.
$ shopt -s extglob; echo !(*.txt) # this is the WRONG way!\n-bash: syntax error near unexpected token `('\nYou have to configure the parser before a line with new syntax is parsed:
$ shopt -s extglob # standalone - CORRECT way!\n$ echo !(*.txt)\n...\nBashphorisms are aphorisms for the IRC channel #bash on Freenode. Keep in mind that this version is a snapshot, the bashphorisms are changed here and there. Also, another snapshot.
I think greycat was the first one who had the idea, but I'm not sure.
Our bashphorisms can be queried from greybot using !bN, where N is the bashphorism number.
And yes, these bashphorisms reflect the daily reality in #bash.
greycat gets pissed off. 4 The questioner will not read and apply the answers he is given but will instead continue to practice bashphorism #1 and bashphorism #2. 5 The ignorant will continually mis-educate the other noobies. 6 When given a choice of solutions, the newbie will always choose the wrong one. 7 The newbie will always find a reason to say, \"It doesn't work.\" 8 If you don't know to whom the bashphorism's referring, it's you. 9 All examples given by the questioner will be broken, misleading, wrong, and not representative of the actual question. 10 See B1 11 Please apply (( % 10 )) to the bashphorism value. 12 All logic is deniable; however, some logic will plonk you if you deny it. 13 Everyone ignores greycat when he is right. When he is wrong, it is !b1 14 The newbie doesn't actually know what he's asking. If he did, he wouldn't need to ask. 15 The more advanced you are, the more likely you are to be overcomplicating it. 16 The more beginner you are, the more likely you are to be overcomplicating it. 17 A newbie comes to #bash to get his script confirmed. He leaves disappointed. 18 The newbie will not accept the answer you give, no matter how right it is. 19 The newbie is a bloody loon. 20 The newbie will always have some excuse for doing it wrong. 21 When the newbie's question is ambiguous, the proper interpretation will be whichever one makes the problem the hardest to solve. 22 The newcomer will abuse the bot's factoid triggers for their own entertainment until someone gets annoyed enough to ask them to message it privately instead. 23 Everyone is a newcomer. 24 The newcomer will address greybot as if it were human. 25 The newbie won't accept any answer that uses practical or standard tools. 26 The newbie will not TELL you about this restriction until you have wasted half an hour. 27 The newbie will lie. 28 When the full horror of the newbie's true goal is revealed, the newbie will try to restate the goal to trick you into answering. Newbies are stupid. 29 It's always git. Or python virtualenv. Or docker. One of those pieces of shit. ALWAYS. 30 They won't show you the homework assignment. That would make it too easy. 31 Your teacher is a f**king idiot. 32 The more horrifyingly wrong a proposed solution is, the more likely it will be used. 33 The newbie cannot explain what he is doing, or why. He will show you incomprehensible, nonworking code instead. What? You can't read his mind?! Please feel free to correct or extend this page whenever needed.
"},{"location":"misc/readthesourceluke/","title":"Readthesourceluke","text":"Comments extracted from the bash source and therefore Copyright (C) 1987-2004 Free Software Foundation, Inc. under the terms of the GNU General Public License etc..
from mailcheck.c:
/* check_mail () is useful for more than just checking mail. Since it has\n the paranoids dream ability of telling you when someone has read your\n mail, it can just as easily be used to tell you when someones .profile\n file has been read, thus letting one know when someone else has logged\n in. Pretty good, huh? */\nFrom builtins/read.def:
/* If there are no variables, save the text of the line read to the\n variable $REPLY. ksh93 strips leading and trailing IFS whitespace,\n so that `read x ; echo \"$x\"' and `read ; echo \"$REPLY\"' behave the\n same way, but I believe that the difference in behaviors is useful\n enough to not do it. Without the bash behavior, there is no way\n to read a line completely without interpretation or modification\n unless you mess with $IFS (e.g., setting it to the empty string).\n If you disagree, change the occurrences of `#if 0' to `#if 1' below. */\nfrom variables.c:
/*\n * 24 October 2001\n *\n * I'm tired of the arguing and bug reports. Bash now leaves SSH_CLIENT\n * and SSH2_CLIENT alone. I'm going to rely on the shell_level check in\n * isnetconn() to avoid running the startup files more often than wanted.\n * That will, of course, only work if the user's login shell is bash, so\n * I've made that behavior conditional on SSH_SOURCE_BASHRC being defined\n * in config-top.h.\n */\nFrom shell.h:
/* Values that can be returned by execute_command (). */\n#define EXECUTION_FAILURE 1\n#define EXECUTION_SUCCESS 0\n\n/* Usage messages by builtins result in a return status of 2. */\n#define EX_BADUSAGE 2\n\n/* Special exit statuses used by the shell, internally and externally. */\n#define EX_RETRYFAIL 124\n#define EX_WEXPCOMSUB 125\n#define EX_BINARY_FILE 126\n#define EX_NOEXEC 126\n#define EX_NOINPUT 126\n#define EX_NOTFOUND 127\n\n#define EX_SHERRBASE 256 /* all special error values are > this. */\n\n#define EX_BADSYNTAX 257 /* shell syntax error */\n#define EX_USAGE 258 /* syntax error in usage */\n#define EX_REDIRFAIL 259 /* redirection failed */\n#define EX_BADASSIGN 260 /* variable assignment error */\n#define EX_EXPFAIL 261 /* word expansion failed */\nNothing special, just my private collection of some more or less funny shell stuff I saw during the years.
Usually Bash and/or Linux (GNU Toolset) specific.
$ %blow\n-bash: fg: %blow: no such job\n\n$ ar m god\nar: creating god\n\n$ touch /pussy\ntouch: cannot touch `/pussy': Permission denied\n\n$ mount; fsck; fsck; fsck; umount; sleep\n\n# the lover variant\n$ unzip; strip; touch; finger; grep; mount; fsck; more; yes; fsck; fsck; umount; sleep\n\n# it's not directly funny, only because it's not obvious that this is an sed command\n# for the <<<, it works only in Bash\n$ sed streetlight <<< reeter\nlighter\n\n# see above for comments\n$ sed statement <<< cat\ncement\n\n$ (-\nbash: (-: command not found\n\n$ echo '[q]sa[ln0=aln256%Pln256/snlbx]sb3135071790101768542287578439snlbxq'|dc\nGET A LIFE!\nFIXME
incomplete
","tags":["bash","shell","scripting","startup","files","dotfiles","modes","POSIX"]},{"location":"scripting/bashbehaviour/#bash-startup-modes","title":"Bash startup modes","text":"","tags":["bash","shell","scripting","startup","files","dotfiles","modes","POSIX"]},{"location":"scripting/bashbehaviour/#login-shell","title":"Login shell","text":"As a \"login shell\", Bash reads and sets (executes) the user's profile from /etc/profile and one of ~/.bash_profile, ~/.bash_login, or ~/.profile (in that order, using the first one that's readable!).
When a login shell exits, Bash reads and executes commands from the file ~/.bash_logout, if it exists.
Why an extra login shell mode? There are many actions and variable sets that only make sense for the initial user login. That's why all UNIX\u00ae shells have (should have) a \"login\" mode.
Methods to start Bash as a login shell:
argv[0] is - (a hyphen): traditional UNIX\u00ae shells start from the login binary-l option--login optionMethods to test for login shell mode:
login_shell is setRelated switches:
--noprofile disables reading of all profile filesWhen Bash starts as an interactive non-login shell, it reads and executes commands from ~/.bashrc. This file should contain, for example, aliases, since they need to be defined in every shell as they're not inherited from the parent shell.
The feature to have a system-wide /etc/bash.bashrc or a similar system-wide rc-file is specific to vendors and distributors that ship their own, patched variant of Bash. The classic way to have a system-wide rc file is to source /etc/bashrc from every user's ~/.bashrc.
Methods to test for interactive-shell mode:
$- contains the letter i (lowercase I)Related switches:
-i forces the interactive mode--norc disables reading of the startup files (e.g. /etc/bash.bashrc if supported) and ~/.bashrc--rcfile defines another startup file (instead of /etc/bash.bashrc and ~/.bashrc)When Bash starts in SH compatiblity mode, it tries to mimic the startup behaviour of historical versions of sh as closely as possible, while conforming to the POSIX\u00ae standard as well. The profile files read are /etc/profile and ~/.profile, if it's a login shell.
If it's not a login shell, the environment variable ENV is evaluated and the resulting filename is used as the name of the startup file.
After the startup files are read, Bash enters the POSIX\u00ae compatiblity mode (for running, not for starting!).
Bash starts in sh compatiblity mode when:
argv[0] is sh (:!: NB: /bin/sh may be linked to /bin/bash, but that doesn't mean it acts like /bin/bash :!:)When Bash is started in POSIX\u00ae mode, it follows the POSIX\u00ae standard for startup files. In this mode, interactive shells expand the ENV variable and commands are read and executed from the file whose name is the expanded value. No other startup files are read. Hence, a non-interactive shell doesn't read any startup files in POSIX\u00ae mode.
Bash starts in POSIX\u00ae mode when:
--posix is specified~/.bashrc would be read (at least Debian GNU/Linux behaves like that)/etc which are always read, Bash usually reads the first file found, when multiple choices are given (for user files in ~/)/etc/profile ~/.bash_profile ~/.bash_login ~/.profile ~/.bashrc ${ENV} Login shell \u2705 \u2705 \u2705 \u2705 \ud83d\udeab \ud83d\udeab Interactive shell \ud83d\udeab \ud83d\udeab \ud83d\udeab \ud83d\udeab \u2705 \ud83d\udeab SH compatible login \u2705 \ud83d\udeab \ud83d\udeab \u2705 \ud83d\udeab \ud83d\udeab SH compatible \ud83d\udeab \ud83d\udeab \ud83d\udeab \ud83d\udeab \ud83d\udeab \u2705 POSIX\u00ae compatiblity \ud83d\udeab \ud83d\udeab \ud83d\udeab \ud83d\udeab \ud83d\udeab \u2705","tags":["bash","shell","scripting","startup","files","dotfiles","modes","POSIX"]},{"location":"scripting/bashbehaviour/#bash-run-modes","title":"Bash run modes","text":"","tags":["bash","shell","scripting","startup","files","dotfiles","modes","POSIX"]},{"location":"scripting/bashbehaviour/#normal-bash","title":"Normal Bash","text":"","tags":["bash","shell","scripting","startup","files","dotfiles","modes","POSIX"]},{"location":"scripting/bashbehaviour/#posix-run-mode","title":"POSIX run mode","text":"In POSIX\u00ae mode, Bash follows the POSIX\u00ae standard regarding behaviour and parsing (excerpt from a Bash maintainer's document):
Starting Bash with the `--posix' command-line option or executing `set\n-o posix' while Bash is running will cause Bash to conform more closely\nto the POSIX standard by changing the behavior to match that specified\nby POSIX in areas where the Bash default differs.\n\nWhen invoked as `sh', Bash enters POSIX mode after reading the startup\nfiles.\n\nThe following lists what's changed when Bash is in `POSIX mode':\n\n 1. When a command in the hash table no longer exists, Bash will\n re-search `$PATH' to find the new location. This is also\n available with `shopt -s checkhash'.\n\n 2. The message printed by the job control code and builtins when a job\n exits with a non-zero status is `Done(status)'.\n\n 3. The message printed by the job control code and builtins when a job\n is stopped is `Stopped(SIGNAME)', where SIGNAME is, for example,\n `SIGTSTP'.\n\n 4. The `bg' builtin uses the required format to describe each job\n placed in the background, which does not include an indication of\n whether the job is the current or previous job.\n\n 5. Reserved words appearing in a context where reserved words are\n recognized do not undergo alias expansion.\n\n 6. The POSIX `PS1' and `PS2' expansions of `!' to the history number\n and `!!' to `!' are enabled, and parameter expansion is performed\n on the values of `PS1' and `PS2' regardless of the setting of the\n `promptvars' option.\n\n 7. The POSIX startup files are executed (`$ENV') rather than the\n normal Bash files.\n\n 8. Tilde expansion is only performed on assignments preceding a\n command name, rather than on all assignment statements on the line.\n\n 9. The default history file is `~/.sh_history' (this is the default\n value of `$HISTFILE').\n\n 10. The output of `kill -l' prints all the signal names on a single\n line, separated by spaces, without the `SIG' prefix.\n\n 11. The `kill' builtin does not accept signal names with a `SIG'\n prefix.\n\n 12. Non-interactive shells exit if FILENAME in `.' FILENAME is not\n found.\n\n 13. Non-interactive shells exit if a syntax error in an arithmetic\n expansion results in an invalid expression.\n\n 14. Redirection operators do not perform filename expansion on the word\n in the redirection unless the shell is interactive.\n\n 15. Redirection operators do not perform word splitting on the word in\n the redirection.\n\n 16. Function names must be valid shell names. That is, they may not\n contain characters other than letters, digits, and underscores, and\n may not start with a digit. Declaring a function with an invalid\n name causes a fatal syntax error in non-interactive shells.\n\n 17. POSIX special builtins are found before shell functions during\n command lookup.\n\n 18. If a POSIX special builtin returns an error status, a\n non-interactive shell exits. The fatal errors are those listed in\n the POSIX standard, and include things like passing incorrect\n options, redirection errors, variable assignment errors for\n assignments preceding the command name, etc.\n\n 19. If `CDPATH' is set, the `cd' builtin will not implicitly append\n the current directory to it. This means that `cd' will fail if no\n valid directory name can be constructed from any of the entries in\n `$CDPATH', even if the a directory with the same name as the name\n given as an argument to `cd' exists in the current directory.\n\n 20. A non-interactive shell exits with an error status if a variable\n assignment error occurs when no command name follows the assignment\n statements. A variable assignment error occurs, for example, when\n trying to assign a value to a readonly variable.\n\n 21. A non-interactive shell exits with an error status if the iteration\n variable in a `for' statement or the selection variable in a\n `select' statement is a readonly variable.\n\n 22. Process substitution is not available.\n\n 23. Assignment statements preceding POSIX special builtins persist in\n the shell environment after the builtin completes.\n\n 24. Assignment statements preceding shell function calls persist in the\n shell environment after the function returns, as if a POSIX\n special builtin command had been executed.\n\n 25. The `export' and `readonly' builtin commands display their output\n in the format required by POSIX.\n\n 26. The `trap' builtin displays signal names without the leading `SIG'.\n\n 27. The `trap' builtin doesn't check the first argument for a possible\n signal specification and revert the signal handling to the original\n disposition if it is, unless that argument consists solely of\n digits and is a valid signal number. If users want to reset the\n handler for a given signal to the original disposition, they\n should use `-' as the first argument.\n\n 28. The `.' and `source' builtins do not search the current directory\n for the filename argument if it is not found by searching `PATH'.\n\n 29. Subshells spawned to execute command substitutions inherit the\n value of the `-e' option from the parent shell. When not in POSIX\n mode, Bash clears the `-e' option in such subshells.\n\n 30. Alias expansion is always enabled, even in non-interactive shells.\n\n 31. When the `alias' builtin displays alias definitions, it does not\n display them with a leading `alias ' unless the `-p' option is\n supplied.\n\n 32. When the `set' builtin is invoked without options, it does not\n display shell function names and definitions.\n\n 33. When the `set' builtin is invoked without options, it displays\n variable values without quotes, unless they contain shell\n metacharacters, even if the result contains nonprinting characters.\n\n 34. When the `cd' builtin is invoked in LOGICAL mode, and the pathname\n constructed from `$PWD' and the directory name supplied as an\n argument does not refer to an existing directory, `cd' will fail\n instead of falling back to PHYSICAL mode.\n\n 35. When the `pwd' builtin is supplied the `-P' option, it resets\n `$PWD' to a pathname containing no symlinks.\n\n 36. The `pwd' builtin verifies that the value it prints is the same as\n the current directory, even if it is not asked to check the file\n system with the `-P' option.\n\n 37. When listing the history, the `fc' builtin does not include an\n indication of whether or not a history entry has been modified.\n\n 38. The default editor used by `fc' is `ed'.\n\n 39. The `type' and `command' builtins will not report a non-executable\n file as having been found, though the shell will attempt to\n execute such a file if it is the only so-named file found in\n `$PATH'.\n\n 40. The `vi' editing mode will invoke the `vi' editor directly when\n the `v' command is run, instead of checking `$FCEDIT' and\n `$EDITOR'.\n\n 41. When the `xpg_echo' option is enabled, Bash does not attempt to\n interpret any arguments to `echo' as options. Each argument is\n displayed, after escape characters are converted.\n\n\nThere is other POSIX behavior that Bash does not implement by default\neven when in POSIX mode. Specifically:\n\n 1. The `fc' builtin checks `$EDITOR' as a program to edit history\n entries if `FCEDIT' is unset, rather than defaulting directly to\n `ed'. `fc' uses `ed' if `EDITOR' is unset.\n\n 2. As noted above, Bash requires the `xpg_echo' option to be enabled\n for the `echo' builtin to be fully conformant.\n\n\nBash can be configured to be POSIX-conformant by default, by specifying\nthe `--enable-strict-posix-default' to `configure' when building.\nFIXME
help me to find out what breaks in POSIX\u00ae mode!
The POSIX\u00ae mode can be switched on by:
sh (the basename of argv[0] is sh)--posixset -o posixTests for the POSIX\u00ae mode:
posix in its listIn restricted mode, Bash sets up (and runs) a shell environment that's far more controlled and limited than the standard shell mode. It acts like normal Bash with the following restrictions:
cd command can't be used to change directories/ (slash) can't be called (hence you're limited to PATH)/ (slash) can't be specified as argument to the source or . builtin command/ (slash) can't be specified as argument to the -p option of the hash builtin command>, >|, <>, >&, &>, and >> redirection operators isn't allowedexec builtin command can't replace the shell with another process-f and -d options to the enable builtin command is forbiddenenable builtin command to enable disabled shell builtins doesn't work-p option to the command builtin command doesn't workset +r or set +o restricted is (of course) forbiddenThe \"-r\" restrictions are turned on after Bash has read its startup files.
When the command that is run is a shell script, then the restrictions are turned off for the (sub-)shell that runs that shell script.
The restricted shell can be switched on by:
rbash (the basename of argv[0] is rbash)-r option--restricted optionTests for restricted mode:
$- contains the letter r (lowercase R)restricted_shell is set and can be checked by the shopt builtin commandThis article is an incomplete overview of changes to Bash over time. Not all changes are listed, just the ones most likely to be useful for normal scripting. The overviews are categorized by topic and ordered by version.
A useful starting point is the NEWS file in bash sources. If you have more detailed information, or historical information about Bash versions earlier than V2, feel free to mail me, or use the discussion below.
Status: 5.1 (alpha)
"},{"location":"scripting/bashchanges/#shell-options","title":"Shell options","text":"Note that the shopt builtin command first appeared in Bash 2.0.
For this topic, see also
posix (for set -o) 1.14.0 hostcomplete 2.0-alpha3 expand_aliases 2.0 huponexit 2.02-alpha1 nocaseglob 2.02-alpha1 extglob 2.02-alpha1 together with extended globbing, KSH88 restricted_shell 2.03-alpha xpg_echo 2.04-beta1 progcomp 2.04-alpha1 no_empty_command_completion 2.04 login_shell 2.05a-alpha1 nolog (for set -o) 2.05a gnu_errfmt 3.0-alpha force_fignore 3.0-alpha failglob 3.0-alpha extquote 3.0-alpha unsure -- verify! extdebug 3.0-alpha pipefail (for set -o) 3.0 functrace (for set -o) 3.0 errtrace (for set -o) 3.0 nocasematch 3.1-alpha1 dirspell 4.0-alpha globstar 4.0-alpha checkjobs 4.0-alpha autocd 4.0-alpha set -e effects more intuitive 4.0 not directly specified by POSIX, but in consensus with POSIX WG compat40 4.1-beta lastpipe 4.2-alpha only works with job control disabled compat41 4.2-alpha globasciiranges 4.3-alpha enable \"character range globbing\" to always act as if in C locale compat42 4.3-alpha compat43 4.4-alpha compat44 5.0-alpha localvar_inherit 5.0-alpha local variables inherit preceeding scope values if they have the same name syslog_history 5.0-alpha send history lines to syslog (undocumented, default off) if syslog is supported assoc_expand_once 5.0-alpha expand associative array subscripts only one globasciiranges 5.0-beta New default: on (default may be configured at compile time) localvar_inherit 5.0-beta guard code against inheriting from an incompatible data type checkwinsize 5.0-beta2 New default: on shift_verbose 5.0-beta2 Default on when in POSIX mode"},{"location":"scripting/bashchanges/#general-allmany-builtins","title":"General (all/many builtins)","text":"Feature or change description Appeared in Bash version See also/remarks generally return 2 on usage error 2.0 generally accept -- (end of options) 2.0 (where applicable) implement a -p option to produce reusable output 2.0 shopt and umask builtins were fixed to support that in 2.02"},{"location":"scripting/bashchanges/#printf","title":"printf","text":"For this topic, see also
printf command 2.02-alpha1 respects 0.. and 0x.. prefixed numbers 2.04-beta1 consistency with arithmetic POSIX\u00ae length specifiers j, t and z 2.05a-alpha1 ISO C99 POSIX\u00ae flag ' 2.05a-alpha1 conversion a and A 2.05a-rc1 if provided by the underlying printf(3) conversion F 2.05a-rc1 conversion n 2.05a-rc1 new option -v 3.1-alpha1 escape sequences \\\" and \\? 3.0-beta1 modified option -v to assign to individual array elements 4.1-alpha conversion (...)T 4.2-alpha support stftime(3) date/time format; uses current time \\uNNNN and \\UNNNNNNNN escape sequences 4.2-alpha for: printf, echo -e, $'...'"},{"location":"scripting/bashchanges/#conditional-expressions-and-test-command","title":"Conditional expressions and test command","text":"For this topic, see also
test: -o, ==, < and > 2.0 test: -N 2.02 [[...]]: new 2.02-alpha1 KSH93 [[...]]: regex support (=~) 3.0-alpha [[...]]: quotable right-hand-side of =~ forces string matching 3.2-alpha for consistency with pattern matching [[...]]: < and > operators respect locale 4.1-alpha for consistency, since 4.1-beta: ensure you have set compatiblity to >4.0 (default) test/[/[[: -v 4.2-alpha check if a variable is set test/[/[[: -v 4.2-alpha support array syntax to check for elements test/[/[[: -N accepts nanoseconds 5.1-alpha test/[/[[: -v accepts positional parameters 5.1-alpha"},{"location":"scripting/bashchanges/#other-builtins-and-keywords","title":"Other builtins and keywords","text":"Builtin Feature or change description Appeared in Bash version See also/remarks bashbug new 1.14.0 select new 1.14.0 disown new 2.0 shopt new 2.0 shopt declare new options -a and -F 2.0 enable builtin has basic plugin support (dlopen) 2.0 exec options -l, -c and -a 2.0 read options -p, -e and -a 2.0 read readonly option -a 2.0 arrays time new keyword 2.0 shopt -p (reusable output) 2.02 umask -p (reusable output) 2.02 complete new 2.04-devel for and together with support for programmable completion compgen new 2.04-devel for and together with support for programmable completion read options -t, -n, -d, -s 2.04-devel read for ((...;...;...)) new 2.04-devel KSH93 set print shell functions in a format reusable as input 2.05-beta1 for allow an empty word list 2.05a-alpha1 read new option -u 2.05b-alpha1 read caller new 3.0 caller coproc new 4.0-alpha declare new options -l and -u 4.0-alpha together with case-changing expansion forms case new action list terminators '';;& and '';& 4.0-alpha ksh93: only ;&. zsh and mksh: ;|. mksh: all 4, (;;& is undocumented Bash compatibility) read changed -t (fractional seconds) 4.0-alpha mapfile new 4.0-alpha read new option -i 4.0-alpha compopt new 4.0-alpha read modified option -t to test for data 4.0-beta read new option -N 4.1-alpha mapfile changed behaviour regarding history spamming 4.1-alpha declare new option -g 4.2-alpha mapfile calls the callback with an additional argument: The line (data) 4.2-alpha cd new option -e 4.2-alpha echo \\uNNNN and \\UNNNNNNNN escape sequences 4.2-alpha for: printf, echo -e, $'...' exec option -a to give a argv[0] string 4.2-alpha time allowed as a command by itself to display timing values of the shell and its children 4.2-alpha POSIX change help help now searches exact topic-strings (i.e. help read won't find readonly anymore) 4.3-alpha return accept negative values as return value (e.g. return -1 will show as (8 bit) 255 in the caller) 4.3-alpha exit accept negative values as return value (e.g. return -1 will show as (8 bit) 255 in the caller) 4.3-alpha read read skips NUL (ASCII Code 0) in input 4.3-alpha declare new option -n/+n to support nameref variable type 4.3-alpha wait new option -n to wait for the next background job to finish, returning its exit status. 4.3-alpha read read checks first variable argument for validity before trying to read inout 4.3-beta help attempts substring matching (as it did through bash-4.2) if exact string matching fails 4.3-beta2 fc interprets option -0 (zero) as the current command line 4.3-beta2 cd new option -@ to browse a file's extended attributes (on systems that support O_XATTR) 4.3-rc1 kill new option -L (upper case ell) to list signals like the normal lowercase option -l (compatiblity with some standalone kill commands) 4.4-beta mapfile new option -d 4.4-alpha wait new option -f 5.0-alpha history option -d allows negative numbers to index from the end of the history list 5.0-alpha umask allows modes greater than octal 777 5.0-alpha times honors current locale settings when printing decimal points 5.0-alpha kill New options -n SIGNUMBER and -s SIGNAME 5.0-beta2 kill select Support for an empty wordlist following in 5.0-beta2 read Option -e (use ReadLine to obtain input) now works with arbitrary file descriptors (given by -u option) 5.1-alpha trap -p option prints signals with SIG_DFL/SIG_IGN on shell start (POSIX mode) 5.1-alpha unset automatically tries to unset a function if the given name is an invalid variable name 5.1-aplha wait option -n now accepts a list of jobs 5.1-alpha wait new option -p NAME to store PID/JobID (useful when waiting for a list of jobs) 5.1-alpha local new option -p to print local variables in the current scope 5.1-alpha ulimit new option -R to get/set RLIMIT_RTTIME resource 5.1-alpha"},{"location":"scripting/bashchanges/#builtin-variables","title":"Builtin variables","text":"Feature or change description Appeared in Bash version See also HISTCMD 1.14.0 interactive usage PS1, PS2, PATH, and IFS are unsettable 2.0 DIRSTACK array variable 2.0 PIPESTATUS array variable 2.0 BASH_VERSINFO array variable 2.0 HOSTNAME 2.0 SHELLOPTS 2.0 MACHTYPE 2.0 GLOBIGNORE 2.0 HISTIGNORE 2.0 respect LC_ALL 2.0 respect LC_MESSAGES 2.0 respect LC_CTYPE 2.0 respect LC_COLLATE 2.0 respect LANG 2.0 GROUPS array variable 2.01 GROUPS unsettable/takes (discarded) assignments 2.04 FUNCNAME 2.04 respect LC_NUMERIC 2.04 TMOUT 2.05b BASH_REMATCH 3.0 together with regex support in [[...]] BASH_ARGC 3.0 debugger support BASH_ARGV 3.0 debugger support BASH_SOURCE 3.0 debugger support BASH_LINENO 3.0 debugger support BASH_SUBSHELL 3.0 debugger support BASH_EXECUTION_STRING 3.0 debugger support BASH_COMMAND 3.0 debugger support HISTTIMEFORMAT 3.0 COMP_WORDBREAKS 3.0 respect LC_TIME 3.1 BASHPID 4.0-alpha Added to mksh R41. PROMPT_DIRTRIM 4.0 BASH_XTRACEFD 4.1-alpha BASHOPTS 4.1-alpha FUNCNEST 4.2-alpha HISTSIZE 4.3-alpha can be set to negative values for unlimited history length HISTFILESIZE 4.3-alpha can be set to negative values for unlimit history file size CHILD_MAX 4.3-alpha max. number of exit status of children the shell remembers BASH_COMPAT 4.3-alpha set shell compatiblity levels EPOCHSECONDS 5.0-alpha expands to the time in seconds since Unix epoch EPOCHREALTIME 5.0-alpha expands to the time in seconds since Unix epoch with microsecond granularity BASH_ARGV0 5.0-alpha get/set $0 PATH 5.0-alpha Possibility to set a static path for use in a restricted shell (at compile time) HISTSIZE 5.0-beta Default can now be set at runtime SRANDOM 5.1-alpha New random generator for 32bit numbers (using various methods in the backend) ARGV0 5.1-alpha Respected when set in initial shell environment, then initially used to set $0 BASH_REMATCH 5.1-alpha Not readonly anymore PROMPT_COMMANDS 5.1-alpha New array variable. List of commands to be executed like PROMPT_COMMAND SECONDS 5.1-alpha Assignment using arithmetic expressions (is nominally an integer variabnle) RANDOM 5.1-alpha Assignment using arithmetic expressions (is nominally an integer variabnle) LINENO 5.1-alpha Not an integer variabe"},{"location":"scripting/bashchanges/#quoting-expansions-substitutions-and-related","title":"Quoting, expansions, substitutions and related","text":"For this topic, see also
${PARAMETER//PATTERN/REPLACEMENT} 2.0 ${PARAMETER:OFFSET:LENGTH} 2.0 ${!PARAMETER} (indirection) 2.0 $\"...\" (localized strings) 2.0 $'...' (ANSI-C-like strings) 2.0 \\xNNN in $'...' (and echo -e) 2.02-alpha1 $(< FILENAME) (file content) 2.02-alpha1 globbing (fnmatch()) capable of POSIX\u00ae character classes etc. 2.02-alpha1 extended globbing 2.02-alpha1 KSH88 globbing inside array mass-assignment: ARRAY=(*.txt) 2.03-alpha $'...\\'...' escaped single quote inside ANSI-C-like strings 2.04-devel KSH93 ${!PREFIX*} (parameter name expansion) 2.04 KSH93 $'...' expands \\cx (Control-x) 2.05b [:class:] syntax for pattern matching 2.05b KSH93 ${!ARRAY[@]} (array index expansion) 3.0-alpha KSH93 {x..y} (range brace expansion) 3.0-alpha $'...' expands \\xNNN (Hexdigits) 3.0 += operator for arrays and strings 3.1-alpha1 ${PARAMETER//PATTERN/REPLACEMENT} behaviour changed 3.2-alpha anchoring for global substitution is no longer allowed, changes the way old syntax may work ${@:0:x} includes $0 4.0-alpha Support for associative arrays 4.0-alpha relevant builtins also got associative array support case modification operators for expansions 4.0-alpha {0x..0y} (zeropadding brace expansion) 4.0-alpha numerically indexed arrays can be accessed (on expansion time) from the end using negative indexes 4.1-alpha \\uNNNN and \\uNNNNNNNN in $'...' 4.2-alpha for: printf, echo -e, $'...' ${PARAMETER:OFFSET:LENGTH}: Negative LENGTH values are used as offset from the end of the string 4.2-alpha Substrings only for Bash and ksh93. Works also for argument expansions in zsh. ksh93 can use ${arr[n..-m]}. Word expansions like ${foo##bar} understand indirect variable references 4.3-beta Transformations 4.4 Process substitution now works in POSIX mode 5.1-alpha New transformations: U, u, L 5.1-alpha Case-transformation New transformation: K 5.1-alpha Display associative arrays as key/value pairs"},{"location":"scripting/bashchanges/#arithmetic","title":"Arithmetic","text":"For this topic, see also
((...)) 2.0-beta2 KSH93 ternary operator 2.0 base 64 integer constants 2.0 the max. base before is unknown. Anybody? deprecated $[...] in favor of $((...)) 2.0 exponentiaition operator (**) 2.02-alpha1 comma operator EXPR,EXPR 2.04-devel pre- and postfix operators 2.04-devel"},{"location":"scripting/bashchanges/#redirection-and-related","title":"Redirection and related","text":"For this topic, see also
/dev/tcp/, /dev/udp/) 2.04-devel OS/filesystem-independent support for /dev/std(in|out|err) and /dev/fd/* 2.04 socket redirection accepts service names 2.05 [n]<&word- and [n]>&word- FD-duplicate/closing 2.05b-alpha1 KSH93 Here strings: <<< WORD 2.05b-alpha1 |& (synonym for 2>&1|) 4.0-alpha &>> (equiv. to >>FILE 2>&1) 4.0-alpha {varname} style automatic file descriptor allocation 4.1-alpha ksh93 {varname[idx]} fd allocation accepts array subscripts and special-meaning variables 4.3-alpha ksh93"},{"location":"scripting/bashchanges/#misc","title":"Misc","text":"Feature or change description Appeared in Bash version See also/remarks DEBUG trap 2.0 ERR trap 2.05a KSH93 Support for multibyte characters: Unicode / UTF8 2.05b RETURN trap 3.0 ksh93 EXIT trap evaluates in caller scope (for function name {). Bash RETURN in same scope. command_not_found_handle handler function 4.0-alpha official introduction of switchable \"compatiblity levels\" 4.0-alpha compat31 was introduced in a 3.2 version, mainly because of the incompatibilities that were introduced by the changed =~ operator [[...]] and ((...)) conditional commands are subject to the ERR trap and set -e feature 4.1-alpha ACL support for file status checks 4.1-alpha Assignment to negative array indices 4.3-alpha ksh93, zsh declare/typeset -n 4.3-alpha Support for nameref variable type, a variable referencing another one by name shells started to run process substitutions now run any trap set on EXIT 4.3-beta process substitution does not inherit the v flag 5.0-alpha ERR trap 5.0-alpha Reports more reliable line numbers Variable assignment 5.0-beta Assignments preceeding a special builtin that chages variable attributes are not propagated back unless compatiblity mode is 44 or lower"},{"location":"scripting/basics/","title":"The basics of shell scripting","text":"","tags":["bash","shell","scripting","basics","learning","tutorial"]},{"location":"scripting/basics/#script-files","title":"Script files","text":"A shell script usually resides inside a file. The file can be executable, but you can call a Bash script with that filename as a parameter:
bash ./myfile\nThere is no need to add a boring filename extension like .bash or .sh. That is a holdover from UNIX\u00ae, where executables are not tagged by the extension, but by permissions (filemode). The file name can be any combination of legal filename characters. Adding a proper filename extension is a convention, nothing else.
chmod +x ./myfile\nIf the file is executable, and you want to use it by calling only the script name, the shebang must be included in the file.
","tags":["bash","shell","scripting","basics","learning","tutorial"]},{"location":"scripting/basics/#the-shebang","title":"The Shebang","text":"The in-file specification of the interpreter of that file, for example:
#!/bin/bash\necho \"Hello world...\"\nThis is interpreted by the kernel 1 of your system. In general, if a file is executable, but not an executable (binary) program, and such a line is present, the program specified after #! is started with the scriptname and all its arguments. These two characters # and ! must be the first two bytes in the file!
You can follow the process by using echo as a fake interpreter:
#!/bin/echo\nWe don't need a script body here, as the file will never be interpreted and executed by \"echo\". You can see what the Operating System does, it calls \"/bin/echo\" with the name of the executable file and following arguments.
$ /home/bash/bin/test testword hello\n/home/bash/bin/test testword hello\nThe same way, with #!/bin/bash the shell \"/bin/bash\" is called with the script filename as an argument. It's the same as executing \"/bin/bash /home/bash/bin/test testword hello\"
If the interpreter can be specified with arguments and how long it can be is system-specific (see #!-magic). When Bash executes a file with a #!/bin/bash shebang, the shebang itself is ignored, since the first character is a hashmark \"#\", which indicates a comment. The shebang is for the operating system, not for the shell. Programs that don't ignore such lines, may not work as shebang driven interpreters.
Warning
Attention:When the specified interpreter is unavailable or not executable (permissions), you usually get a \"bad interpreter\" error message., If you get nothing and it fails, check the shebang. Older Bash versions will respond with a \"no such file or directory\" error for a nonexistant interpreter specified by the shebang.
Additional note: When you specify #!/bin/sh as shebang and that's a link to a Bash, then Bash will run in POSIX\u00ae mode! See:
A common method is to specify a shebang like
#!/usr/bin/env bash\n...which just moves the location of the potential problem to
env utility must be located in /usr/bin/bash binary must be located in PATHWhich one you need, or whether you think which one is good, or bad, is up to you. There is no bulletproof portable way to specify an interpreter. It's a common misconception that it solves all problems. Period.
","tags":["bash","shell","scripting","basics","learning","tutorial"]},{"location":"scripting/basics/#the-standard-filedescriptors","title":"The standard filedescriptors","text":"Once Initialized, every normal UNIX\u00ae-program has at least 3 open files:
Usually, they're all connected to your terminal, stdin as input file (keyboard), stdout and stderr as output files (screen). When calling such a program, the invoking shell can change these filedescriptor connections away from the terminal to any other file (see redirection). Why two different output filedescriptors? It's convention to send error messages and warnings to stderr and only program output to stdout. This enables the user to decide if they want to see nothing, only the data, only the errors, or both - and where they want to see them.
When you write a script:
stdinstderrTo learn more about the standard filedescriptors, especially about redirection and piping, see:
It's good practice to use lowercase names for your variables, as shell and system-variable names are usually all in UPPERCASE. However, you should avoid naming your variables any of the following (incomplete list!):
BASH BASH_ARGC BASH_ARGV BASH_LINENO BASH_SOURCE BASH_VERSINFO BASH_VERSION COLUMNS DIRSTACK DISPLAY EDITOR EUID GROUPS HISTFILE HISTFILESIZE HISTSIZE HOME HOSTNAME IFS LANG LANGUAGE LC_ALL LINES LOGNAME LS_COLORS MACHTYPE MAILCHECK OLDPWD OPTERR OPTIND OSTYPE PATH PIPESTATUS PPID PROMPT_COMMAND PS1 PS2 PS4 PS3 PWD SHELL SHELLOPTS SHLVL TERM UID USER USERNAME XAUTHORITY This list is incomplete. The safest way is to use all-lowercase variable names.
","tags":["bash","shell","scripting","basics","learning","tutorial"]},{"location":"scripting/basics/#exit-codes","title":"Exit codes","text":"Every program you start terminates with an exit code and reports it to the operating system. This exit code can be utilized by Bash. You can show it, you can act on it, you can control script flow with it. The code is a number between 0 and 255. Values from 126 to 255 are reserved for use by the shell directly, or for special purposes, like reporting a termination by a signal:
The lower codes 0 to 125 are not reserved and may be used for whatever the program likes to report. A value of 0 means successful termination, a value not 0 means unsuccessful termination. This behavior (== 0, != 0) is also what Bash reacts to in some flow control statements.
An example of using the exit code of the program grep to check if a specific user is present in /etc/passwd:
if grep ^root /etc/passwd; then\n echo \"The user root was found\"\nelse\n echo \"The user root was not found\"\nfi\nA common decision making command is \"test\" or its equivalent \"[\". But note that, when calling test with the name \"[\", the square brackets are not part of the shell syntax, the left bracket is the test command!
if [ \"$mystring\" = \"Hello world\" ]; then\n echo \"Yeah dude, you entered the right words...\"\nelse\n echo \"Eeeek - go away...\"\nfi\nRead more about the test command
A common exit code check method uses the \"||\" or \"&&\" operators. This lets you execute a command based on whether or not the previous command completed successfully:
grep ^root: /etc/passwd >/dev/null || echo \"root was not found - check the pub at the corner.\"\nwhich vi && echo \"Your favourite editor is installed.\"\nPlease, when your script exits on errors, provide a \"FALSE\" exit code, so others can check the script execution.
In a larger, or complex script, it's wise to comment the code. Comments can help with debugging or tests. Comments start with the # character (hashmark) and continue to the end of the line:
#!/bin/bash\n# This is a small script to say something.\necho \"Be liberal in what you accept, and conservative in what you send\" # say something\nThe first thing was already explained, it's the so-called shebang, for the shell, only a comment. The second one is a comment from the beginning of the line, the third comment starts after a valid command. All three syntactically correct.
","tags":["bash","shell","scripting","basics","learning","tutorial"]},{"location":"scripting/basics/#block-commenting","title":"Block commenting","text":"To temporarily disable complete blocks of code you would normally have to prefix every line of that block with a # (hashmark) to make it a comment. There's a little trick, using the pseudo command : (colon) and input redirection. The : does nothing, it's a pseudo command, so it does not care about standard input. In the following code example, you want to test mail and logging, but not dump the database, or execute a shutdown:
#!/bin/bash\n# Write info mails, do some tasks and bring down the system in a safe way\necho \"System halt requested\" | mail -s \"System halt\" netadmin@example.com\nlogger -t SYSHALT \"System halt requested\"\n\n##### The following \"code block\" is effectively ignored\n: <<\"SOMEWORD\"\n/etc/init.d/mydatabase clean_stop\nmydatabase_dump /var/db/db1 /mnt/fsrv0/backups/db1\nlogger -t SYSHALT \"System halt: pre-shutdown actions done, now shutting down the system\"\nshutdown -h NOW\nSOMEWORD\n##### The ignored codeblock ends here\nWhat happened? The : pseudo command was given some input by redirection (a here-document) - the pseudo command didn't care about it, effectively, the entire block was ignored.
The here-document-tag was quoted here to avoid substitutions in the \"commented\" text! Check redirection with here-documents for more
","tags":["bash","shell","scripting","basics","learning","tutorial"]},{"location":"scripting/basics/#variable-scope","title":"Variable scope","text":"In Bash, the scope of user variables is generally global. That means, it does not matter whether a variable is set in the \"main program\" or in a \"function\", the variable is defined everywhere.
Compare the following equivalent code snippets:
myvariable=test\necho $myvariable\nmyfunction() {\n myvariable=test\n}\n\nmyfunction\necho $myvariable\nIn both cases, the variable myvariable is set and accessible from everywhere in that script, both in functions and in the \"main program\".
Attention: When you set variables in a child process, for example a subshell, they will be set there, but you will never have access to them outside of that subshell. One way to create a subshell is the pipe. It's all mentioned in a small article about Bash in the processtree!
","tags":["bash","shell","scripting","basics","learning","tutorial"]},{"location":"scripting/basics/#local-variables","title":"Local variables","text":"Bash provides ways to make a variable's scope local to a function:
local keyword, ordeclare (which will detect when it was called from within a function and make the variable(s) local).myfunc() {\nlocal var=VALUE\n\n# alternative, only when used INSIDE a function\ndeclare var=VALUE\n\n...\n}\nThe local keyword (or declaring a variable using the declare command) tags a variable to be treated completely local and separate inside the function where it was declared:
foo=external\n\nprintvalue() {\nlocal foo=internal\n\necho $foo\n}\n\n\n# this will print \"external\"\necho $foo\n\n# this will print \"internal\"\nprintvalue\n\n# this will print - again - \"external\"\necho $foo\nThe environment space is not directly related to the topic about scope, but it's worth mentioning.
Every UNIX\u00ae process has a so-called environment. Other items, in addition to variables, are saved there, the so-called environment variables. When a child process is created (in Bash e.g. by simply executing another program, say ls to list files), the whole environment including the environment variables is copied to the new process. Reading that from the other side means: Only variables that are part of the environment are available in the child process.
A variable can be tagged to be part of the environment using the export command:
# create a new variable and set it:\n# -> This is a normal shell variable, not an environment variable!\nmyvariable=\"Hello world.\"\n\n# make the variable visible to all child processes:\n# -> Make it an environment variable: \"export\" it\nexport myvariable\nRemember that the exported variable is a copy. There is no provision to \"copy it back to the parent.\" See the article about Bash in the process tree!
under specific circumstances, also by the shell itself\u00a0\u21a9
These few lines are not intended as a full-fledged debugging tutorial, but as hints and comments about debugging a Bash script.
","tags":["bash","shell","scripting","bug","debug","debugging"]},{"location":"scripting/debuggingtips/#use-a-unique-name-for-your-script","title":"Use a unique name for your script","text":"Do not name your script test, for example! Why? test is the name of a UNIX\u00ae-command, and most likely built into your shell (it's a built-in in Bash) - so you won't be able to run a script with the name test in a normal way.
Don't laugh! This is a classic mistake :-)
","tags":["bash","shell","scripting","bug","debug","debugging"]},{"location":"scripting/debuggingtips/#read-the-error-messages","title":"Read the error messages","text":"Many people come into IRC and ask something like \"Why does my script fail? I get an error!\". And when you ask them what the error message is, they don't even know. Beautiful.
Reading and interpreting error messages is 50% of your job as debugger! Error messages actually mean something. At the very least, they can give you hints as to where to start debugging. READ YOUR ERROR MESSAGES!
You may ask yourself why is this mentioned as debugging tip? Well, you would be surprised how many shell users ignore the text of error messages! When I find some time, I'll paste 2 or 3 IRC log-snips here, just to show you that annoying fact.
","tags":["bash","shell","scripting","bug","debug","debugging"]},{"location":"scripting/debuggingtips/#use-a-good-editor","title":"Use a good editor","text":"Your choice of editor is a matter of personal preference, but one with Bash syntax highlighting is highly recommended! Syntax highlighting helps you see (you guessed it) syntax errors, such as unclosed quotes and braces, typos, etc.
From my personal experience, I can suggest vim or GNU emacs.
For more complex scripts, it's useful to write to a log file, or to the system log. Nobody can debug your script without knowing what actually happened and what went wrong.
An available syslog interface is logger (online manpage).
Insert echos everywhere you can, and print to stderr:
echo \"DEBUG: current i=$i\" >&2\nIf you read input from anywhere, such as a file or command substitution, print the debug output with literal quotes, to see leading and trailing spaces!
pid=$(< fooservice.pid)\necho \"DEBUG: read from file: pid=\\\"$pid\\\"\" >&2\nBash's printf command has the %q format, which is handy for verifying whether strings are what they appear to be.
foo=$(< inputfile)\nprintf \"DEBUG: foo is |%q|\\n\" \"$foo\" >&2\n# exposes whitespace (such as CRs, see below) and non-printing characters\nThere are two useful debug outputs for that task (both are written to stderr):
set -v mode (set -o verbose)stderr as if they were read from input (script file or keyboard)set -x mode (set -o xtrace)+ (plus) sign to the displayed command)'x y'Hint: These modes can be entered when calling Bash:
bash -vx ./myscript#!/bin/bash -vxHere's a simple command (a string comparison using the classic test command) executed while in set -x mode:
set -x\nfoo=\"bar baz\"\n[ $foo = test ]\nThat fails. Why? Let's see the xtrace output:
+ '[' bar baz = test ']'\nAnd now you see that it's (\"bar\" and \"baz\") recognized as two separate words (which you would have realized if you READ THE ERROR MESSAGES ;) ). Let's check it...
# next try\n[ \"$foo\" = test ]\nxtrace now gives
+ '[' 'bar baz' = test ']'\n ^ ^\n word markers!\n(by AnMaster)
xtrace output would be more useful if it contained source file and line number. Add this assignment PS4 at the beginning of your script to enable the inclusion of that information:
export PS4='+(${BASH_SOURCE}:${LINENO}): ${FUNCNAME[0]:+${FUNCNAME[0]}(): }'\nBe sure to use single quotes here!
The output would look like this when you trace code outside a function:
+(somefile.bash:412): echo 'Hello world'\n...and like this when you trace code inside a function:
+(somefile.bash:412): myfunc(): echo 'Hello world'\nThat helps a lot when the script is long, or when the main script sources many other files.
","tags":["bash","shell","scripting","bug","debug","debugging"]},{"location":"scripting/debuggingtips/#set-flag-variables-with-descriptive-words","title":"Set flag variables with descriptive words","text":"If you test variables that flag the state of options, such as with if [[ -n $option ]];, consider using descriptive words rather than short codes, such as 0, 1, Y, N, because xtrace will show [[ -n word ]] rather than [[ -n 1 ]] when the option is set.
For general debugging purposes you can also define a function and a variable to use:
debugme() {\n [[ $script_debug = 1 ]] && \"$@\" || :\n # be sure to append || : or || true here or use return 0, since the return code\n # of this function should always be 0 to not influence anything else with an unwanted\n # \"false\" return code (for example the script's exit code if this function is used\n # as the very last command in the script)\n}\nThis function does nothing when script_debug is unset or empty, but it executes the given parameters as commands when script_debug is set. Use it like this:
script_debug=1\n# to turn it off, set script_debug=0\n\ndebugme logger \"Sorting the database\"\ndatabase_sort\ndebugme logger \"Finished sorting the database, exit code $?\"\nOf course this can be used to execute something other than echo during debugging:
debugme set -x\n# ... some code ...\ndebugme set +x\nImagine you have a script that runs FTP commands using the standard FTP client:
ftp user@host <<FTP\ncd /data\nget current.log\ndele current.log\nFTP\nA method to dry-run this with debug output is:
if [[ $DRY_RUN = yes ]]; then\n sed 's/^/DRY_RUN FTP: /'\nelse\n ftp user@host\nfi <<FTP\ncd /data\nget current.log\ndele current.log\nFTP\nThis can be wrapped in a shell function for more readable code.
","tags":["bash","shell","scripting","bug","debug","debugging"]},{"location":"scripting/debuggingtips/#common-error-messages","title":"Common error messages","text":"","tags":["bash","shell","scripting","bug","debug","debugging"]},{"location":"scripting/debuggingtips/#unexpected-end-of-file","title":"Unexpected end of file","text":"script.sh: line 100: syntax error: unexpected end of file\nUsually indicates exactly what it says: An unexpected end of file. It's unexpected because Bash waits for the closing of a compound command:
do with a done?if with a fi?case with a esac?{ with a }?( with a )?Note: It seems that here-documents (tested on versions 1.14.7, 2.05b, 3.1.17 and 4.0) are correctly terminated when there is an EOF before the end-of-here-document tag (see redirection). The reason is unknown, but it seems to be deliberate. Bash 4.0 added an extra message for this: warning: here-document at line <N> delimited by end-of-file (wanted `<MARKER>')
script.sh: line 50: unexpected EOF while looking for matching `\"'\nscript.sh: line 100: syntax error: unexpected end of file\nThis one indicates the double-quote opened in line 50 does not have a matching closing quote.
These unmatched errors occur with:
$'string'!)} with parameter expansion syntaxbash: test: too many arguments\nYou most likely forgot to quote a variable expansion somewhere. See the example for xtrace output from above. External commands may display such an error message though in our example, it was the internal test-command that yielded the error.
$ echo \"Hello world!\"\nbash: !\": event not found\nThis is not an error per se. It happens in interactive shells, when the C-Shell-styled history expansion (\"!searchword\") is enabled. This is the default. Disable it like this:
set +H\n# or\nset +o histexpand\nWhen this happens during a script function definition or on the commandline, e.g.
$ foo () { echo \"Hello world\"; }\nbash: syntax error near unexpected token `('\nyou most likely have an alias defined with the same name as the function (here: foo). Alias expansion happens before the real language interpretion, thus the alias is expanded and makes your function definition invalid.
There's a big difference in the way that UNIX\u00ae and Microsoft\u00ae (and possibly others) handle the line endings of plain text files. The difference lies in the use of the CR (Carriage Return) and LF (Line Feed) characters.
\\r\\n (ASCII CR #13 ^M, ASCII LF #10)\\n (ASCII LF #10)Keep in mind your script is a plain text file, and the CR character means nothing special to UNIX\u00ae - it is treated like any other character. If it's printed to your terminal, a carriage return will effectively place the cursor at the beginning of the current line. This can cause much confusion and many headaches, since lines containing CRs are not what they appear to be when printed. In summary, CRs are a pain.
Some possible sources of CRs:
CRs can be a nuisance in various ways. They are especially bad when present in the shebang/interpreter specified with #! in the very first line of a script. Consider the following script, written with a Windows\u00ae text editor (^M is a symbolic representation of the CR carriage return character!):
#!/bin/bash^M\n^M\necho \"Hello world\"^M\n...\nHere's what happens because of the #!/bin/bash^M in our shebang:
/bin/bash^M doesn't exist (hopefully)The error message can vary. If you're lucky, you'll get:
bash: ./testing.sh: /bin/bash^M: bad interpreter: No such file or directory\nwhich alerts you to the CR. But you may also get the following:
: bad interpreter: No such file or directory\nWhy? Because when printed literally, the ^M makes the cursor go back to the beginning of the line. The whole error message is printed, but you see only part of it!
warning
It's easy to imagine the ^M is bad in other places too. If you get weird and illogical messages from your script, rule out the possibility that^M is involved. Find and eliminate it!
To display CRs (these are only a few examples)
:set listcat(1): cat -v FILETo eliminate them (only a few examples)
tr(1): tr -d '\\r' <FILE >FILE.newrecode(1): recode MSDOS..latin1 FILEdos2unix(1): dos2unix FILE-v and -x)FIXME
tbd.
Here are some typical traps:
","tags":["bash","shell","scripting","pitfalls","traps","beginners"]},{"location":"scripting/newbie_traps/#script-execution","title":"Script execution","text":"","tags":["bash","shell","scripting","pitfalls","traps","beginners"]},{"location":"scripting/newbie_traps/#your-perfect-bash-script-executes-with-syntax-errors","title":"Your perfect Bash script executes with syntax errors","text":"If you write Bash scripts with Bash specific syntax and features, run them with Bash, and run them with Bash in native mode.
Wrong:
bash myscript#!/bin/sh shebang/bin/sh actually is, for a Bash it means compatiblity mode, not native modeSee also:
Give it another name. The executable test already exists.
In Bash it's a builtin. With other shells, it might be an executable file. Either way, it's bad name choice!
Workaround: You can call it using the pathname:
/home/user/bin/test\nThe following command line is not related to globbing (filename expansion):
# YOU EXPECT\n# -i1.vob -i2.vob -i3.vob ....\n\necho -i{*.vob,}\n\n# YOU GET\n# -i*.vob -i\nWhy? The brace expansion is simple text substitution. All possible text formed by the prefix, the postfix and the braces themselves are generated. In the example, these are only two: -i*.vob and -i. The filename expansion happens after that, so there is a chance that -i*.vob is expanded to a filename - if you have files like -ihello.vob. But it definitely doesn't do what you expected.
Please see:
if [ $foo ] ...if [-d $dir] ...Please see:
There is no $ (dollar-sign) when you reference the name of a variable! Bash is not PHP!
# THIS IS WRONG!\n$myvar=\"Hello world!\"\nA variable name preceeded with a dollar-sign always means that the variable gets expanded. In the example above, it might expand to nothing (because it wasn't set), effectively resulting in...
=\"Hello world!\"\n...which definitely is wrong!
When you need the name of a variable, you write only the name, for example
picture=/usr/share/images/foo.pngread builtin command: read pictureunset pictureWhen you need the content of a variable, you prefix its name with a dollar-sign, like
echo \"The used picture is: $picture\"\nPutting spaces on either or both sides of the equal-sign (=) when assigning a value to a variable will fail.
# INCORRECT 1\nexample = Hello\n\n# INCORRECT 2\nexample= Hello\n\n# INCORRECT 3\nexample =Hello\nThe only valid form is no spaces between the variable name and assigned value:
# CORRECT 1\nexample=Hello\n\n# CORRECT 2\nexample=\" Hello\"\nA typical beginner's trap is quoting.
As noted above, when you want to expand a variable i.e. \\\"get the content\\\", the variable name needs to be prefixed with a dollar-sign. But, since Bash knows various ways to quote and does word-splitting, the result isn't always the same.
Let's define an example variable containing text with spaces:
example=\"Hello world\"\n$example Hello world 2 \"$example\" Hello world 1 \\$example $example 1 '$example' $example 1 If you use parameter expansion, you must use the name (PATH) of the referenced variables/parameters. i.e. not ($PATH):
# WRONG!\necho \"The first character of PATH is ${$PATH:0:1}\"\n\n# CORRECT\necho \"The first character of PATH is ${PATH:0:1}\"\nNote that if you are using variables in arithmetic expressions, then the bare name is allowed:
((a=$a+7)) # Add 7 to a\n((a = a + 7)) # Add 7 to a. Identical to the previous command.\n((a += 7)) # Add 7 to a. Identical to the previous command.\n\na=$((a+7)) # POSIX-compatible version of previous code.\nPlease see:
Exporting a variable means giving newly created (child-)processes a copy of that variable. It does not copy a variable created in a child process back to the parent process. The following example does not work, since the variable hello is set in a child process (the process you execute to start that script ./script.sh):
$ cat script.sh\nexport hello=world\n\n$ ./script.sh\n$ echo $hello\n$\nExporting is one-way. The direction is from parent process to child process, not the reverse. The above example will work, when you don't execute the script, but include (\\\"source\\\") it:
$ source ./script.sh\n$ echo $hello\nworld\n$\nIn this case, the export command is of no use.
Please see:
If you just want to react to an exit code, regardless of its specific value, you don't need to use $? in a test command like this:
grep ^root: /etc/passwd >/dev/null 2>&1\n\nif [ $? -ne 0 ]; then\n echo \"root was not found - check the pub at the corner\"\nfi\nThis can be simplified to:
if ! grep ^root: /etc/passwd >/dev/null 2>&1; then\n echo \"root was not found - check the pub at the corner\"\nfi\nOr, simpler yet:
grep ^root: /etc/passwd >/dev/null 2>&1 || echo \"root was not found - check the pub at the corner\"\nIf you need the specific value of $?, there's no other choice. But if you need only a \\\"true/false\\\" exit indication, there's no need for $?.
See also:
It's important to remember the different ways to run a child command, and whether you want the output, the return value, or neither.
When you want to run a command (or a pipeline) and save (or print) the output, whether as a string or an array, you use Bash's $(command) syntax:
$(ls -l /tmp)\nnewvariable=$(printf \"foo\")\nWhen you want to use the return value of a command, just use the command, or add ( ) to run a command or pipeline in a subshell:
if grep someuser /etc/passwd ; then\n # do something\nfi\n\nif ( w | grep someuser | grep sqlplus ) ; then\n # someuser is logged in and running sqlplus\nfi\nMake sure you\\'re using the form you intended:
# WRONG!\nif $(grep ERROR /var/log/messages) ; then\n # send alerts\nfi\nPlease see:
The script programming language of BASH is based on the Bourne Shell syntax, with some extensions and derivations.
If scripts need to be portable, some of the BASH-specific syntax elements should be avoided. Others should be avoided for all scripts, e.g. if there is a corresponding POSIX\u00ae-compatible syntax (see obsolete).
Some syntax elements have a BASH-specific, and a portable1) pendant. In these cases the portable syntax should be preferred.
construct portable equivalent Description Portabilitysource FILE . FILE include a script file Bourne shell (bash, ksh, POSIX\u00ae, zsh, \u2026) declare keyword typeset keyword define local variables (or variables with special attributes) ksh, zsh, \u2026, not POSIX! command <<< WORD command <<MARKER\\nWORD\\nMARKER a here-string, a special form of the here-document, avoid it in portable scripts! POSIX\u00ae export VAR=VALUE VAR=VALUE\\nexport VAR Though POSIX\u00ae allows it, some shells don't want the assignment and the exporting in one command POSIX\u00ae, zsh, ksh, \u2026 (( MATH )) : $(( MATH )) POSIX\u00ae does't define an arithmetic compund command, many shells don't know it. Using the pseudo-command : and the arithmetic expansion $(( )) is a kind of workaround here. Attention: Not all shell support assignment like $(( a = 1 + 1 ))! Also see below for a probably more portable solution. all POSIX\u00ae compatible shells [[ EXPRESSION ]] [ EXPRESSION ]\\nor\\ntest EXPRESSION The Bashish test keyword is reserved by POSIX\u00ae, but not defined. Use the old fashioned way with the test command. See [[web/20230315170826/https://wiki.bash-hackers.org/commands/classictest]] POSIX\u00ae and others COMMAND < <( \u2026INPUTCOMMANDS\u2026 ) INPUTCOMMANDS > TEMPFILE\\nCOMMAND < TEMPFILE Process substitution (here used with redirection); use the old fashioned way (tempfiles) POSIX\u00ae and others ((echo X);(echo Y)) ( (echo X); (echo Y) ) Nested subshells (separate the inner () from the outer () by spaces, to not confuse the shell regarding arithmetic control operators) POSIX\u00ae and others","tags":["bash","shell","scripting","portability","POSIX","portable"]},{"location":"scripting/nonportable/#portability-rationale","title":"Portability rationale","text":"Here is some assorted portability information. Take it as a small guide to make your scripts a bit more portable. It's not complete (it never will be!) and it's not very detailed (e.g. you won't find information about how which shell technically forks off which subshell). It's just an assorted small set of portability guidelines. -Thebonsai
FIXME
UNIX shell gurus out there, please be patient with a newbie like me and give comments and hints instead of flames.
","tags":["bash","shell","scripting","portability","POSIX","portable"]},{"location":"scripting/nonportable/#environment-exported-variables","title":"Environment (exported) variables","text":"When a new value is assigned to an existing environment variable, there are two possibilities:
The new value is seen by subsequent programs
export VARIABLE (e.g. Sun's /bin/sh)Since an extra export doesn't hurt, the safest and most portable way is to always (re-)export a changed variable if you want it to be seen by subsequent processes.
Bash has a special compound command to do arithmetic without expansion. However, POSIX has no such command. In the table at the top, there's the : $((MATH)) construct mentioned as possible alternative. Regarding the exit code, a 100% equivalent construct would be:
# Bash (or others) compound command\nif ((MATH)); then\n...\n\n# portable equivalent command\nif [ \"$((MATH))\" -ne 0 ]; then\n...\nQuotes around the arithmetic expansion $((MATH)) should not be necessary as per POSIX, but Bash and AT&T-KSH perform word-splitting on aritrhmetic expansions, so the most portable is with quotes.
The overall problem with echo is, that there are 2 (maybe more) mainstream flavours around. The only case where you may safely use an echo on all systems is: Echoing non-variable arguments that don't start with a - (dash) and don't contain a \\ (backslash).
Why? (list of known behaviours)
-n)--)echo -n and echo -e are neither portable nor standard (even within the same shell, depending on the version or environment variables or the build options, especially KSH93 and Bash)For these, and possibly other, reasons, POSIX (SUS) standardized the existance of the printf command.
${var:x:x} is KSH93/Bash specific${var/../..} and ${var//../..} are KSH93/Bash specificvar=$* and var=$@ are not handled the same in all shells if the first char of IFS is not \" \" (space). var=\"$*\" should work (except the Bourne shell always joins the expansions with space)PWD is POSIX but not Bourne. Most shells are not POSIX in that they don't ignore the value of the PWD environment variable. Workaround to fix the value of PWD at the start of your script:
pwd -P > dev/null\nRANDOM is Bash/KSH/ZSH specific variable that will give you a random number up to 32767 (2^15-1). Among many other available external options, you can use awk to generate a random number. There are multiple implementations of awk and which version your system uses will depend. Most modern systems will call 'gawk' (i.e. GNU awk) or 'nawk'. 'oawk' (i.e. Original/Old awk) does not have the rand() or srand() functions, so is best avoided.
# 'gawk' can produce random numbers using srand(). In this example, 10 integers between 1 and 500:\nrandpm=$(gawk -v min=1 -v max=500 -v nNum=10 'BEGIN { srand(systime() + PROCINFO[\"pid\"]); for (i = 0; i < nNum; ++i) {print int(min + rand() * (max - min)} }')\n\n# 'nawk' and 'mawk' does the same, but needs a seed to be provided for its rand() function. In this example we use $(date)\nrandpm=$(mawk -v min=1 -v max=500 -v nNum=10 -v seed=\"$(date +%Y%M%d%H%M%S)\" 'BEGIN { srand(seed); for (i = 0; i < nNum; ++i) {print int(min + rand() * (max - min)} }')\nYes, I'm not an awk expert, so please correct it, rather than complaining about possible stupid code!
# Well, seeing how this //is// BASH-hackers.org I kinda missed the bash way of doing the above ;-)\n# print a number between 0 and 500 :-)\n printf $(( 500 * RANDOM / 32767 ))\n\n# Or print 30 random numbers between 0 and 10 ;)\n X=0; while (( X++ < 30 )); do echo $(( 10 * RANDOM / 32767 )); done\nSECONDS is KSH/ZSH/Bash specific. Avoid it. Find another method.
","tags":["bash","shell","scripting","portability","POSIX","portable"]},{"location":"scripting/nonportable/#check-for-a-command-in-path","title":"Check for a command in PATH","text":"The PATH variable is a colon-delimited list of directory names, so it's basically possible to run a loop and check every PATH component for the command you're looking for and for executability.
However, this method doesn't look nice. There are other ways of doing this, using commands that are not directly related to this task.
","tags":["bash","shell","scripting","portability","POSIX","portable"]},{"location":"scripting/nonportable/#hash","title":"hash","text":"The hash command is used to make the shell store the full pathname of a command in a lookup-table (to avoid re-scanning the PATH on every command execution attempt). Since it has to do a PATH search, it can be used for this check.
For example, to check if the command ls is available in a location accessible by PATH:
if hash ls >/dev/null 2>&1; then\n echo \"ls is available\"\nfi\nSomewhat of a mass-check:
for name in ls grep sed awk; do\n if ! hash \"$name\" >/dev/null 2>&1; then\n echo \"FAIL: Missing command '$name'\"\n exit 1\n fi\ndone\nHere (bash 3), hash also respects builtin commands. I don't know if this works everywhere, but it seems logical.
The command command is used to explicitly call an external command, rather than a builtin with the same name. For exactly this reason, it has to do a PATH search, and can be used for this check.
For example, to check if the command sed is available in a location accessible by PATH:
if command -v sed >/dev/null 2>&1; then\n echo \"sed is available\"\nfi\n\"portable\" doesn't necessarily mean it's POSIX, it can also mean it's \"widely used and accepted\", and thus maybe more portable than POSIX(r\u00a0\u21a9
This (incomplete) page describes some syntax and commands considered obsolete by some measure. A thorough discussion of the rationale is beyond the scope of this page. See the portability page for a discussion on portability issues.
","tags":["bash","shell","scripting","obsolete","deprecated","outdated"]},{"location":"scripting/obsolete/#tolerated-but-few-ligitimate-uses","title":"Tolerated but few ligitimate uses","text":"This first table lists syntax that is tolerated by Bash but has few if any legitimate uses. These features exist mostly for Bourne, csh, or some other backward compatibility with obsolete shells, or were Bash-specific features considered failed experiments and deprecated or replaced with a better alternative. These should be irrelevant to most everyone except maybe code golfers. New scripts should never use them. None of the items on this list are specified by the most current version of POSIX, and some may be incompatible with POSIX.
Syntax Replacement Description&>FILE and >&FILE >FILE 2>&1 This redirection syntax is short for >FILE 2>&1 and originates in the C Shell. The latter form is especially uncommon and should never be used, and the explicit form using separate redirections is preferred over both. These shortcuts contribute to confusion about the copy descriptor because the syntax is unclear. They also introduce parsing ambiguity, and conflict with POSIX. Shells without this feature treat cmd1 &>file cmd2 as: \"background cmd1 and then execute cmd2 with its stdout redirected to file\", which is the correct interpretation of this expression. See: redirection $ { bash; dash </dev/fd/0; } <<<'echo foo>/dev/null&>/dev/fd/2 echo bar'foo echo barbar $[EXPRESSION] $((EXPRESSION)) This undocumented syntax is completely replaced by the POSIX-conforming arithmetic expansion $((EXPRESSION)). It is unimplemented almost everywhere except Bash and Zsh. See arithmetic expansion. Some discussion. COMMAND|& COMMAND COMMAND 2>&1| COMMAND This is an alternate pipeline operator derived from Zsh. Officially, it is not considered deprecated by Bash, but I highly discourage it. It conflicts with the list operator used for coprocess creation in most Korn shells. It also has confusing behavior. The stdout is redirected first like an ordinary pipe, while the stderr is actually redirected last \u2013 after other redirects preceding the pipe operator. Overall, it's pointless syntax bloat. Use an explicit redirect instead. function NAME() COMPOUND-CMD NAME() COMPOUND-CMD or function NAME { CMDS; } This is an amalgamation between the Korn and POSIX style function definitions - using both the function keyword and parentheses. It has no useful purpose and no historical basis or reason to exist. It is not specified by POSIX. It is accepted by Bash, mksh, zsh, and perhaps some other Korn shells, where it is treated as identical to the POSIX-style function. It is not accepted by AT&T ksh. It should never be used. See the next table for the function keyword. Bash doesn't have this feature documented as expressly deprecated. for x; { ...;} do, done, in, esac, etc. This undocumented syntax replaces the do and done reserved words with braces. Many Korn shells support various permutations on this syntax for certain compound commands like for, case, and while. Which ones and certain details like whether a newline or semicolon are required vary. Only for works in Bash. Needless to say, don't use it.","tags":["bash","shell","scripting","obsolete","deprecated","outdated"]},{"location":"scripting/obsolete/#syntax-superseded-by-superior-alternatives","title":"Syntax superseded by superior alternatives","text":"This table lists syntax that is specified by POSIX (unless otherwise specified below), but has been superseded by superior alternatives (either in POSIX, Bash, or both), or is highly discouraged for other reasons such as encouraging bad practices or dangerous code. Those that are specified by POSIX may be badly designed and unchangeable for historical reasons.
Syntax Replacement Description Unquoted expansions, wordsplitting, and Pathname expansion (globbing) Proper quoting, Ksh/Bash-style arrays, The \"$@\" expansion, the read builtin command Quoting errors are a broad category of common mistakes brought about by a few unintuitive features carried over from the Bourne shell due to complaints of broken scripts and changes in previously documented behavior. Most of the important expansions are performed at the same time from left to right. However, a few expansions, most notably word-splitting and globbing, and in shells other than Bash, brace expansion, are performed on the results of previous expansions, by default, unless they are quoted. This means that the act of expanding an unquoted variable in an ordinary argument context, depending on the value of the variable, can yield different results depending on possibly uncontrolled side-effects like the value ofIFS, and the names of files in the current working directory. You can't get globbing without word-splitting, or vice versa (without set -f). You can't store a command or character-delimited list in a variable and safely evaluate it with unquoted expansion. If possible, always choose a shell that supports Korn shell arrays such as Bash. They are a vital but non-standard feature for writing clean, safe scripts. Well-written scripts don't use word-splitting. A few exceptions are listed on the word splitting page. A significant proportion of the issues on the famous Pitfalls list fall under this category. See also: Don't read lines with for! COMMANDS $(COMMANDS) his is the older Bourne-compatible form of the command substitution. Both the `COMMANDS` and $(COMMANDS) syntaxes are specified by POSIX, but the latter is [greatly]{.underline} preferred, though the former is unfortunately still very prevalent in scripts. New-style command substitutions are widely implemented by every modern shell (and then some). The only reason for using backticks is for compatibility with a real Bourne shell (like Heirloom). Backtick command substitutions require special escaping when nested, and examples found in the wild are improperly quoted more often than not. See: Why is $(...) preferred over ... (backticks)?. [ EXPRESSION ] and test EXPRESSION [[ EXPRESSION ]] test and [ are the Bourne/POSIX commands for evaluating test expressions (they are almost identical, and [ is somewhat more common). The expressions consist of regular arguments, unlike the Ksh/Bash [[ command. While the issue is analogous to let vs ((, the advantages of [[ vs [ are even more important because the arguments/expansions aren't just concatenated into one expression. With the classic [ command, the number of arguments is significant. If at all possible, use the conditional expression (\"new test command\") [[ EXPRESSION ]]. Unless there is a need for POSIX compatibility, there are only a few reasons to use [. [[ is one of the most portable and consistent non-POSIX ksh extensions available. See: conditional_expression and What is the difference between test, [ and [[ ? set -e, set -o errexit and the ERR trap proper control flow and error handling set -e causes untested non-zero exit statuses to be fatal. It is a debugging feature intended for use only during development and should not be used in production code, especially init scripts and other high-availability scripts. Do not be tempted to think of this as \"error handling\"; it's not, it's just a way to find the place you've forgotten to put error handling.Think of it as akin to use strict in Perl or throws in C++: tough love that makes you write better code. Many guides recommend avoiding it entirely because of the apparently-complex rules for when non-zero statuses cause the script to abort. Conversely, large software projects with experienced coders may recommend or even mandate its use.Because it provides no notification of the location of the error, it's more useful combined with set -x or the DEBUG trap and other Bash debug features, and both flags are normally better set on the command line rather than within the script itself.Most of this also applies to the ERR trap, though I've seen it used in a few places in shells that lack pipefail or PIPESTATUS. The ERR trap is not POSIX, but set -e is. failglob is another Bash feature that falls into this category (mainly useful for debugging).The set -e feature generates more questions and false bug reports on the Bash mailing list than all other features combined! Please do not rely on set -e for logic in scripts. If you still refuse to take this advice, make sure you understand exactly how it works. See: Why doesn't set -e (or set -o errexit, or trap ERR) do what I expected? and http://www.fvue.nl/wiki/Bash:_Error_handling set -u or set -o nounset Proper control flow and error handling set -u causes attempts to expand unset variables or parameters as fatal errors. Like set -e, it bypasses control flow and exits immediately from the current shell environment. Like non-zero statuses, unset variables are a normal part of most non-trivial shell scripts. Living with set -u requires hacks like ${1+\"$1\"} for each expansion that might possibly be unset. Only very current shells guarantee that expanding @ or * won't trigger an error when no parameters are set (http://austingroupbugs.net/view.php?id=155, http://www.in-ulm.de/~mascheck/various/bourne_args/). Apparently some find it useful for debugging. See How do I determine whether a variable is already defined? Or a function? for how to properly test for defined variables. Don't use set -u. ${var?msg} or ${var:?msg} Proper control flow and error handling Like set -u, this expansion causes a fatal error which immediately exits the current shell environment if the given parameter is unset or is null. It prints the error message given, to the right of the operator. If a value is expected and you'd like to create an assertion or cause errors, it's better to test for undefined variables using one of these techniques and handle the error manually, or call a die function. This expansion is defined by POSIX. It's better than set -u, because it's explicit, but not by much. It also allows you to accidentally construct hilariously deceptive error messages: bash -c 'f() { definitely_not_printf \"${printf:?\"$1\" - No such option}\"; }; f -v'bash: printf: -v - No such option","tags":["bash","shell","scripting","obsolete","deprecated","outdated"]},{"location":"scripting/obsolete/#frequently-misued","title":"Frequently misued","text":"This table lists features that are used only if you have a specific reason to prefer it over another alternative. These have some legitimate uses if you know what you're doing, such as for those with specific portability requirements, or in order to make use of some subtle behavioral differences. These are frequently (mis)used for no reason. Writing portable scripts that go outside of POSIX features requires knowing how to account for many (often undocumented) differences across many shells. If you do happen to know what you're doing, don't be too surprised if you run across someone telling you not to use these.
Syntax Replacement Descriptionfunction NAME { CMDS; } NAME() COMPOUND-CMD This is the ksh form of function definition created to extend the Bourne and POSIX form with modified behaviors and additional features like local variables. The idea was for new-style functions to be analogous to regular builtins with their own environment and scope, while POSIX-style functions are more like special builtins. function is supported by almost every ksh-derived shell including Bash and Zsh, but isn't specified by POSIX. Bash treats all function styles the same, but this is unusual. function has some preferable characteristics in many ksh variants, making it more portable for scripts that use non-POSIX extensions by some measures. If you're going to use the function keyword, it implies that you're either targeting Ksh specifically, or that you have detailed knowledge of how to compensate for differences across shells. It should always be used consistently with typeset, but never used with declare or local. Also in ksh93, the braces are not a command group, but a required part of the syntax (unlike Bash and others). See shell function definitions typeset declare, local, export, readonly This is closely related to the above, and should often be used together. typeset exists primarily for ksh compatibility, but is marked as \"deprecated\" in Bash (though I don't entirely agree with this). This makes some sense, because future compatibility can't be guaranteed, and any compatibility at all, requires understanding the non-POSIX features of other shells and their differences. Using declare instead of typeset emphasizes your intention to be \"Bash-only\", and definitely breaks everywhere else (except possibly zsh if you're lucky). The issue is further complicated by Dash and the Debian policy requirement for a local builtin, which is itself not entirely compatible with Bash and other shells. let 'EXPR' ((EXPR)) or [ $((EXPR)) -ne 0 ] let is the \"simple command\" variant of arithmetic evaluation command, which takes regular arguments. Both let and ((expr)) were present in ksh88, and everything that supports one should support the other. Neither are POSIX. The compound variant is preferable because it doesn't take regular arguments for wordsplitting and globbing, which makes it safer and clearer. It is also usually faster, especially in Bash, where compound commands are typically significantly faster. Some of the (few) reasons for using let are detailed on the let page. See arithmetic evaluation compound command eval Depends. Often code can be restructured to use better alternatives. eval is thrown in here for good measure, as sadly it is so often misused that any use of eval (even the rare clever one) is immediately dismissed as wrong by experts, and among the most immediate solutions abused by beginners. In reality, there are correct ways to use eval, and even cases in which it's necessary, even in sophisticated shells like Bash and Ksh. eval is unusual in that it is less frequently appropriate in more feature-rich shells than in more minimal shells like Dash, where it is used to compensate for more limitations. If you find yourself needing eval too frequently, it might be a sign that you're either better off using a different language entirely, or trying to borrow an idiom from some other paradigm that isn't well suited to the shell language. By the same token, there are some cases in which working too hard to avoid eval ends up adding a lot of complexity and sacrificing all portability. Don't substitute a clever eval for something that's a bit \"too clever\", just to avoid the eval, yet, take reasonable measures to avoid it where it is sensible to do so. See: The eval builtin command and Eval command and security issues.","tags":["bash","shell","scripting","obsolete","deprecated","outdated"]},{"location":"scripting/obsolete/#see-also","title":"See also","text":"The day will come when you want to give arguments to your scripts. These arguments are known as positional parameters. Some relevant special parameters are described below:
Parameter(s) Description$0 the first positional parameter, equivalent to argv[0] in C, see the first argument $FUNCNAME the function name (attention: inside a function, $0 is still the $0 of the shell, not the function name) $1 ... $9 the argument list elements from 1 to 9 ${10} ... ${N} the argument list elements beyond 9 (note the parameter expansion syntax!) $* all positional parameters except $0, see mass usage $@ all positional parameters except $0, see mass usage $# the number of arguments, not counting $0 These positional parameters reflect exactly what was given to the script when it was called.
Option-switch parsing (e.g. -h for displaying help) is not performed at this point.
See also the dictionary entry for \"parameter\".
","tags":["bash","shell","scripting","arguments","positional","parameters","options"]},{"location":"scripting/posparams/#the-first-argument","title":"The first argument","text":"The very first argument you can access is referenced as $0. It is usually set to the script's name exactly as called, and it's set on shell initialization:
Testscript - it just echos $0:
#!/bin/bash\necho \"$0\"\nYou see, $0 is always set to the name the script is called with (> is the prompt...):
> ./testscript\n./testscript\n\n> /usr/bin/testscript\n/usr/bin/testscript\nHowever, this isn't true for login shells:
> echo \"$0\"\n-bash\nIn other terms, $0 is not a positional parameter, it's a special parameter independent from the positional parameter list. It can be set to anything. In the ideal case it's the pathname of the script, but since this gets set on invocation, the invoking program can easily influence it (the login program does that for login shells, by prefixing a dash, for example).
Inside a function, $0 still behaves as described above. To get the function name, use $FUNCNAME.
The builtin command shift is used to change the positional parameter values:
$1 will be discarded$2 will become $1$3 will become $2$N will become $N-1The command can take a number as argument: Number of positions to shift. e.g. shift 4 shifts $5 to $1.
Enough theory, you want to access your script-arguments. Well, here we go.
","tags":["bash","shell","scripting","arguments","positional","parameters","options"]},{"location":"scripting/posparams/#one-by-one","title":"One by one","text":"One way is to access specific parameters:
#!/bin/bash\necho \"Total number of arguments: $#\"\necho \"Argument 1: $1\"\necho \"Argument 2: $2\"\necho \"Argument 3: $3\"\necho \"Argument 4: $4\"\necho \"Argument 5: $5\"\nWhile useful in another situation, this way is lacks flexibility. The maximum number of arguments is a fixedvalue - which is a bad idea if you write a script that takes many filenames as arguments.
=> forget that one
","tags":["bash","shell","scripting","arguments","positional","parameters","options"]},{"location":"scripting/posparams/#loops","title":"Loops","text":"There are several ways to loop through the positional parameters.
You can code a C-style for-loop using $# as the end value. On every iteration, the shift-command is used to shift the argument list:
numargs=$#\nfor ((i=1 ; i <= numargs ; i++))\ndo\n echo \"$1\"\n shift\ndone\nNot very stylish, but usable. The numargs variable is used to store the initial value of $# because the shift command will change it as the script runs.
Another way to iterate one argument at a time is the for loop without a given wordlist. The loop uses the positional parameters as a wordlist:
for arg\ndo\n echo \"$arg\"\ndone\nAdvantage: The positional parameters will be preserved
The next method is similar to the first example (the for loop), but it doesn't test for reaching $#. It shifts and checks if $1 still expands to something, using the test command:
while [ \"$1\" ]\ndo\n echo \"$1\"\n shift\ndone\nLooks nice, but has the disadvantage of stopping when $1 is empty (null-string). Let's modify it to run as long as $1 is defined (but may be null), using parameter expansion for an alternate value:
while [ \"${1+defined}\" ]; do\n echo \"$1\"\n shift\ndone\nThere is a small tutorial dedicated to getopts (under construction).
Sometimes it's necessary to just \"relay\" or \"pass\" given arguments to another program. It's very inefficient to do that in one of these loops, as you will destroy integrity, most likely (spaces!).
The shell developers created $* and $@ for this purpose.
As overview:
Syntax Effective result$* $1 $2 $3 ... ${N} $@ $1 $2 $3 ... ${N} \"$*\" \"$1c$2c$3c...c${N}\" \"$@\" \"$1\" \"$2\" \"$3\" ... \"${N}\" Without being quoted (double quotes), both have the same effect: All positional parameters from $1 to the last one used are expanded without any special handling.
When the $* special parameter is double quoted, it expands to the equivalent of: \"$1c$2c$3c$4c........$N\", where 'c' is the first character of IFS.
But when the $@ special parameter is used inside double quotes, it expands to the equivanent of...
\"$1\" \"$2\" \"$3\" \"$4\" ..... \"$N\"
...which reflects all positional parameters as they were set initially and passed to the script or function. If you want to re-use your positional parameters to call another program (for example in a wrapper-script), then this is the choice for you, use double quoted \"$@\".
Well, let's just say: You almost always want a quoted \"$@\"!
Another way to mass expand the positional parameters is similar to what is possible for a range of characters using substring expansion on normal parameters and the mass expansion range of arrays.
${@:START:COUNT}
${*:START:COUNT}
\"${@:START:COUNT}\"
\"${*:START:COUNT}\"
The rules for using @ or * and quoting are the same as above. This will expand COUNT number of positional parameters beginning at START. COUNT can be omitted (${@:START}), in which case, all positional parameters beginning at START are expanded.
If START is negative, the positional parameters are numbered in reverse starting with the last one.
COUNT may not be negative, i.e. the element count may not be decremented.
Example: START at the last positional parameter:
echo \"${@: -1}\"\nAttention: As of Bash 4, a START of 0 includes the special parameter $0, i.e. the shell name or whatever $0 is set to, when the positional parameters are in use. A START of 1 begins at $1. In Bash 3 and older, both 0 and 1 began at $1.
Setting positional parameters with command line arguments, is not the only way to set them. The builtin command, set may be used to \"artificially\" change the positional parameters from inside the script or function:
set \"This is\" my new \"set of\" positional parameters\n\n# RESULTS IN\n# $1: This is\n# $2: my\n# $3: new\n# $4: set of\n# $5: positional\n# $6: parameters\nIt's wise to signal \"end of options\" when setting positional parameters this way. If not, the dashes might be interpreted as an option switch by set itself:
# both ways work, but behave differently. See the article about the set command!\nset -- ...\nset - ...\nAlternately this will also preserve any verbose (-v) or tracing (-x) flags, which may otherwise be reset by set
set -$- ...\nFIXME
continue
","tags":["bash","shell","scripting","arguments","positional","parameters","options"]},{"location":"scripting/posparams/#production-examples","title":"Production examples","text":"","tags":["bash","shell","scripting","arguments","positional","parameters","options"]},{"location":"scripting/posparams/#using-a-while-loop","title":"Using a while loop","text":"To make your program accept options as standard command syntax:
COMMAND [options] <params> # Like 'cat -A file.txt'
See simple option parsing code below. It's not that flexible. It doesn't auto-interpret combined options (-fu USER) but it works and is a good rudimentary way to parse your arguments.
#!/bin/sh\n# Keeping options in alphabetical order makes it easy to add more.\n\nwhile :\ndo\n case \"$1\" in\n -f | --file)\n file=\"$2\" # You may want to check validity of $2\n shift 2\n ;;\n -h | --help)\n display_help # Call your function\n # no shifting needed here, we're done.\n exit 0\n ;;\n -u | --user)\n username=\"$2\" # You may want to check validity of $2\n shift 2\n ;;\n -v | --verbose)\n # It's better to assign a string, than a number like \"verbose=1\"\n # because if you're debugging the script with \"bash -x\" code like this:\n #\n # if [ \"$verbose\" ] ...\n #\n # You will see:\n #\n # if [ \"verbose\" ] ...\n #\n # Instead of cryptic\n #\n # if [ \"1\" ] ...\n #\n verbose=\"verbose\"\n shift\n ;;\n --) # End of all options\n shift\n break;\n -*)\n echo \"Error: Unknown option: $1\" >&2\n exit 1\n ;;\n *) # No more options\n break\n ;;\n esac\ndone\n\n# End of file\nThis simple wrapper enables filtering unwanted options (here: -a and --all for ls) out of the command line. It reads the positional parameters and builds a filtered array consisting of them, then calls ls with the new option set. It also respects the -- as \"end of options\" for ls and doesn't change anything after it:
#!/bin/bash\n\n# simple ls(1) wrapper that doesn't allow the -a option\n\noptions=() # the buffer array for the parameters\neoo=0 # end of options reached\n\nwhile [[ $1 ]]\ndo\n if ! ((eoo)); then\n case \"$1\" in\n -a)\n shift\n ;;\n --all)\n shift\n ;;\n -[^-]*a*|-a?*)\n options+=(\"${1//a}\")\n shift\n ;;\n --)\n eoo=1\n options+=(\"$1\")\n shift\n ;;\n *)\n options+=(\"$1\")\n shift\n ;;\n esac\n else\n options+=(\"$1\")\n\n # Another (worse) way of doing the same thing:\n # options=(\"${options[@]}\" \"$1\")\n shift\n fi\ndone\n\n/bin/ls \"${options[@]}\"\nThere is a small tutorial dedicated to getopts (under construction).
The processes in UNIX\u00ae are - unlike other systems - organized as a tree. Every process has a parent process that started, or is responsible, for it. Every process has its own context memory (Not the memory where the process stores its data, rather, the memory where data is stored that doesn't directly belong to the process, but is needed to run the process) i.e. The environment.
Every process has its own environment space.
The environment stores, among other things, data that's useful to us, the environment variables. These are strings in common NAME=VALUE form, but they are not related to shell variables. A variable named LANG, for example, is used by every program that looks it up in its environment to determinate the current locale.
Attention: A variable that is set, like with MYVAR=Hello, is not automatically part of the environment. You need to put it into the environment with the bash builtin command export:
export MYVAR\nCommon system variables like PATH or HOME are usually part of the environment (as set by login scripts or programs).
","tags":["bash","shell","scripting","processes","pipes","variables","environment"]},{"location":"scripting/processtree/#executing-programs","title":"Executing programs","text":"All the diagrams of the process tree use names like \"xterm\" or \"bash\", but that's just to make it easier to understand what's going on, it doesn't mean those processes are actually executed.
Let's take a short look at what happens when you \"execute a program\" from the Bash prompt, a program like \"ls\":
$ ls\nBash will now perform two steps:
The copy of Bash will inherit the environment from the \"main Bash\" process: All environment variables will also be copied to the new process. This step is called forking.
For a short moment, you have a process tree that might look like this...
xterm ----- bash ----- bash(copy)\n...and after the \"second Bash\" (the copy) replaces itself with the ls program (the copy execs it), it might look like
xterm ----- bash ----- ls\nIf everything was okay, the two steps resulted in one program being run. The copy of the environment from the first step (forking) becomes the environment for the final running program (in this case, ls).
What is so important about it? In our example, what the program ls does inside its own environment, it can't affect the environment of its parent process (in this case, bash). The environment was copied when ls was executed. Nothing is \"copied back\" to the parent environment when ls terminates.
Pipes are a very powerful tool. You can connect the output of one process to the input of another process. We won't delve into piping at this point, we just want to see how it looks in the process tree. Again, we execute some commands, this time, we'll run ls and grep:
$ ls | grep myfile\nIt results in a tree like this:
+-- ls\nxterm ----- bash --|\n +-- grep\nNote once again, ls can't influence the grep environment, grep can't influence the ls environment, and neither grep nor ls can influence the bash environment.
How is that related to shell programming?!?
Well, imagine some Bash code that reads data from a pipe. For example, the internal command read, which reads data from stdin and puts it into a variable. We run it in a loop here to count input lines:
counter=0\n\ncat /etc/passwd | while read; do ((counter++)); done\necho \"Lines: $counter\"\nWhat? It's 0? Yes! The number of lines might not be 0, but the variable $counter still is 0. Why? Remember the diagram from above? Rewriting it a bit, we have:
+-- cat /etc/passwd\nxterm ----- bash --|\n +-- bash (while read; do ((counter++)); done)\nSee the relationship? The forked Bash process will count the lines like a charm. It will also set the variable counter as directed. But if everything ends, this extra process will be terminated - your \"counter\" variable is gone. You see a 0 because in the main shell it was 0, and wasn't changed by the child process!
So, how do we count the lines? Easy: Avoid the subshell. The details don't matter, the important thing is the shell that sets the counter must be the \"main shell\". For example:
counter=0\n\nwhile read; do ((counter++)); done </etc/passwd\necho \"Lines: $counter\"\nIt's nearly self-explanatory. The while loop runs in the current shell, the counter is incremented in the current shell, everything vital happens in the current shell, also the read command sets the variable REPLY (the default if nothing is given), though we don't use it here.
Bash creates subshells or subprocesses on various actions it performs:
","tags":["bash","shell","scripting","processes","pipes","variables","environment"]},{"location":"scripting/processtree/#executing-commands","title":"Executing commands","text":"As shown above, Bash will create subprocesses everytime it executes commands. That's nothing new.
But if your command is a subprocess that sets variables you want to use in your main script, that won't work.
For exactly this purpose, there's the source command (also: the dot . command). Source doesn't execute the script, it imports the other script's code into the current shell:
source ./myvariables.sh\n# equivalent to:\n. ./myvariables.sh\nThe last big section was about pipes, so no example here.
","tags":["bash","shell","scripting","processes","pipes","variables","environment"]},{"location":"scripting/processtree/#explicit-subshell","title":"Explicit subshell","text":"If you group commands by enclosing them in parentheses, these commands are run inside a subshell:
(echo PASSWD follows; cat /etc/passwd; echo GROUP follows; cat /etc/group) >output.txt\nWith command substitution you re-use the output of another command as text in your command line, for example to set a variable. The other command is run in a subshell:
number_of_users=$(cat /etc/passwd | wc -l)\nNote that, in this example, a second subshell was created by using a pipe in the command substitution:
+-- cat /etc/passwd\nxterm ----- bash ----- bash (cmd. subst.) --|\n +-- wc -l\nFIXME
to be continued
","tags":["bash","shell","scripting","processes","pipes","variables","environment"]},{"location":"scripting/style/","title":"Scripting with style","text":"FIXME
continue
These are some coding guidelines that helped me to read and understand my own code over the years. They also will help to produce code that will be a bit more robust than \"if something breaks, I know how to fix it\".
This is not a bible, of course. But I have seen so much ugly and terrible code (not only in shell) during all the years, that I'm 100% convinced there needs to be some code layout and style. No matter which one you use, use it throughout your code (at least don't change it within the same shellscript file); don't change your code layout with your mood.
Some good code layout helps you to read your own code after a while. And of course it helps others to read the code.
"},{"location":"scripting/style/#indentation-guidelines","title":"Indentation guidelines","text":"Indentation is nothing that technically influences a script, it's only for us humans.
I'm used to seeing/using indentation of two space characters (though many may prefer 4 spaces, see below in the discussion section):
Speaking of hard-tabs: Avoid them if possible. They only make trouble. I can imagine one case where they're useful: Indenting here-documents.
"},{"location":"scripting/style/#breaking-up-lines","title":"Breaking up lines","text":"Whenever you need to break lines of long code, you should follow one of these two rules:
Indention using command width:
activate some_very_long_option \\\n some_other_option\nIndention using two spaces:
activate some_very_long_option \\\n some_other_option\nPersonally, with some exceptions, I prefer the first form because it supports the visual impression of \"these belong together\".
"},{"location":"scripting/style/#breaking-compound-commands","title":"Breaking compound commands","text":"Compound commands form the structures that make a shell script different from a stupid enumeration of commands. Usually they contain a kind of \"head\" and a \"body\" that contains command lists. This type of compound command is relatively easy to indent.
I'm used to (not all points apply to all compound commands, just pick the basic idea):
What?! Well, here again:
"},{"location":"scripting/style/#symbolic","title":"Symbolic","text":"HEAD_KEYWORD parameters; BODY_BEGIN\n BODY_COMMANDS\nBODY_END\nThis construct is a bit special, because it has keywords (elif, else) \"in the middle\". The visually appealing way is to indent them like this:
if ...; then\n ...\nelif ...; then\n ...\nelse\n ...\nfi\nfor f in /etc/*; do\n ...\ndone\nwhile [[ $answer != [YyNn] ]]; do\n ...\ndone\nThe case construct might need a bit more discussion here, since its structure is a bit more complex.
In general, every new \"layer\" gets a new indentation level:
case $input in\n hello)\n echo \"You said hello\"\n ;;\n bye)\n echo \"You said bye\"\n if foo; then\n bar\n fi\n ;;\n *)\n echo \"You said something weird...\"\n ;;\nesac\nSome notes:
hello)) and the corresponding action terminator (;;) are indented at the same levelCryptic constructs, we all know them, we all love them. If they are not 100% needed, avoid them, since nobody except you may be able to decipher them.
It's - just like in C - the middle ground between smart, efficient and readable.
If you need to use a cryptic construct, include a comment that explains what your \"monster\" does.
"},{"location":"scripting/style/#variable-names","title":"Variable names","text":"Since all reserved variables are UPPERCASE, the safest way is to only use lowercase variable names. This is true for reading user input, loop counting variables, etc., ... (in the example: file)
lowercase variablesUPPERCASE names, do not use reserved variable names (see SUS for an incomplete list)if you use UPPERCASE names, prepend the name with a unique prefix (MY_ in the example below)
MY_LOG_DIRECTORY=/var/adm/
for file in \"$MY_LOG_DIRECTORY\"/*; do echo \"Found Logfile: $file\" done
"},{"location":"scripting/style/#variable-initialization","title":"Variable initialization","text":"As in C, it's always a good idea to initialize your variables, though, the shell will initialize fresh variables itself (better: Unset variables will generally behave like variables containing a null string).
It's no problem to pass an environment variable to the script. If you blindly assume that all variables you use for the first time are empty, anybody can inject content into a variable by passing it via the environment.
The solution is simple and effective: Initialize them
my_input=\"\"\nmy_array=()\nmy_number=0\nIf you do that for every variable you use, then you also have some in-code documentation for them.
"},{"location":"scripting/style/#parameter-expansion","title":"Parameter expansion","text":"Unless you are really sure what you're doing, quote every parameter expansion.
There are some cases where this isn't needed from a technical point of view, e.g.
[[ ... ]] (other than the RHS of the ==, !=, and =~ operators)WORD) in case $WORD in ....VAR=$WORDBut quoting these is never a mistake. If you quote every parameter expansion, you'll be safe.
If you need to parse a parameter as a list of words, you can't quote, of course, e.g.
list=\"one two three\"\n\n# you MUST NOT quote $list here\nfor word in $list; do\n ...\ndone\nFunction names should be all lowercase and meaningful. The function names should be human readable. A function named f1 may be easy and quick to write down, but for debugging and especially for other people, it reveals nothing. Good names help document your code without using extra comments.
do not use command names for your functions. e.g. naming a script or function test, will collide with the UNIX test command.
Unless absolutely necessary, only use alphanumeric characters and the underscore for function names. /bin/ls is a valid function name in Bash, but is not a good idea.
As noted in the article about command substitution, you should use the $( ... ) form.
If portability is a concern, use the backquoted form ` ... `.
In any case, if other expansions and word splitting are not wanted, you should quote the command substitution!
"},{"location":"scripting/style/#eval","title":"Eval","text":"Well, like Greg says: \"If eval is the answer, surely you are asking the wrong question.\"
Avoid it, unless absolutely neccesary:
eval can be your neckshoteval with your current methodThe basic structure of a script simply reads:
#!SHEBANG\n\nCONFIGURATION_VARIABLES\n\nFUNCTION_DEFINITIONS\n\nMAIN_CODE\nIf possible (I know it's not always possible!), use a shebang.
Be careful with /bin/sh: The argument that \"on Linux /bin/sh is Bash\" is a lie (and technically irrelevant)
The shebang serves two purposes for me:
bash!bash when you write a Bash-script, use sh when you write a general Bourne/POSIX script, ...)I call variables that are meant to be changed by the user \"configuration variables\" here.
Make them easy to find (directly at the top of the script), give them meaningful names and maybe a short comment. As noted above, use UPPERCASE for them only when you're sure about what you're doing. lowercase will be the safest.
Unless there are reasons not to, all function definitions should be declared before the main script code runs. This gives a far better overview and ensures that all function names are known before they are used.
Since a function isn't parsed before it is executed, you usually don't have to ensure they're in a specific order.
The portable form of the function definition should be used, without the function keyword (here using the grouping compound command):
getargs() {\n ...\n}\nSpeaking about the command grouping in function definitions using { ...; }: If you don't have a good reason to use another compound command directly, you should always use this one.
Fail early, this sounds bad, but usually is good. Failing early means to error out as early as possible when checks indicate an error or unmet condition. Failing early means to error out before your script begins its work in a potentially broken state.
"},{"location":"scripting/style/#availability-of-commands","title":"Availability of commands","text":"If you use external commands that may not be present on the path, or not installed, check for their availability, then tell the user they're missing.
Example:
my_needed_commands=\"sed awk lsof who\"\n\nmissing_counter=0\nfor needed_command in $my_needed_commands; do\n if ! hash \"$needed_command\" >/dev/null 2>&1; then\n printf \"Command not found in PATH: %s\\n\" \"$needed_command\" >&2\n ((missing_counter++))\n fi\ndone\n\nif ((missing_counter > 0)); then\n printf \"Minimum %d commands are missing in PATH, aborting\\n\" \"$missing_counter\" >&2\n exit 1\nfi\nThe exit code is your only way to directly communicate with the calling process without any special provisions.
If your script exits, provide a meaningful exit code. That minimally means:
exit 0 (zero) if everything is okayexit 1 - in general non-zero - if there was an errorThis, and only this, will enable the calling component to check the operation status of your script.
You know: \"One of the main causes of the fall of the Roman Empire was that, lacking zero, they had no way to indicate successful termination of their C programs.\" -- Robert Firth
"},{"location":"scripting/style/#misc","title":"Misc","text":""},{"location":"scripting/style/#output-and-appearance","title":"Output and appearance","text":"STDOUT. write error, warning and diagnostic messages to STDERR-? or -h or --help arguments), it should go to STDOUTTerminal (control) codes are used to issue specific commands to your terminal. This can be related to switching colors or positioning the cursor, i.e. anything that can't be done by the application itself.
","tags":["bash","shell","scripting","colors","cursor","control","vt100","ansi"]},{"location":"scripting/terminalcodes/#how-it-technically-works","title":"How it technically works","text":"A terminal control code is a special sequence of characters that is printed (like any other text). If the terminal understands the code, it won't display the character-sequence, but will perform some action. You can print the codes with a simple echo command.
Note: I see codes referenced as \"Bash colors\" sometimes (several \"Bash tutorials\" etc...): That's a completely incorrect definition.
","tags":["bash","shell","scripting","colors","cursor","control","vt100","ansi"]},{"location":"scripting/terminalcodes/#the-tput-command","title":"The tput command","text":"Because there's a large number of different terminal control languages, usually a system has an intermediate communication layer. The real codes are looked up in a database for the currently detected terminal type and you give standardized requests to an API or (from the shell) to a command.
One of these commands is tput. Tput accepts a set of acronyms called capability names and any parameters, if appropriate, then looks up the correct escape sequences for the detected terminal in the terminfo database and prints the correct codes (the terminal hopefully understands).
In this list I'll focus on ANSI/VT100 control codes for the most common actions - take it as quick reference. The documentation of your terminal or the terminfo database is always the preferred source when something is unclear! Also the tput acronyms are usually the ones dedicated for ANSI escapes!
I listed only the most relevant codes, of course, any ANSI terminal understands many more! But let's keep the discussion centered on common shell scripting ;-)
If I couldn't find a matching ANSI escape, you'll see a :?: as the code. Feel free to mail me or fix it.
The ANSI codes always start with the ESC character. (ASCII 0x1B or octal 033) This isn't part of the list, but you should avoid using the ANSI codes directly - use the tput command!
All codes that can be used with tput can be found in terminfo(5). (on OpenBSD at least) See OpenBSD's terminfo(5) under the Capabilities section. The cap-name is the code to use with tput. A description of each code is also provided.
The Ctrl-Key representation is simply associating the non-printable characters from ASCII code 1 with the printable (letter) characters from ASCII code 65 (\"A\"). ASCII code 1 would be ^A (Ctrl-A), while ASCII code 7 (BEL) would be ^G (Ctrl-G). This is a common representation (and input method) and historically comes from one of the VT series of terminals.
BEL 7 007 0x07 \\a ^G Terminal bell BS 8 010 0x08 \\b ^H Backspace HT 9 011 0x09 \\t ^I Horizontal TAB LF 10 012 0x0A \\n ^J Linefeed (newline) VT 11 013 0x0B \\v ^K Vertical TAB FF 12 014 0x0C \\f ^L Formfeed (also: New page NP) CR 13 015 0x0D \\r ^M Carriage return ESC 27 033 0x1B <none> ^[ Escape character DEL 127 177 0x7F <none> <none> Delete character","tags":["bash","shell","scripting","colors","cursor","control","vt100","ansi"]},{"location":"scripting/terminalcodes/#cursor-handling","title":"Cursor handling","text":"ANSI terminfo equivalent Description [ <X> ; <Y> H[ <X> ; <Y> f cup <X> <Y> Home-positioning to X and Y coordinates:!: it seems that ANSI uses 1-1 as home while tput uses 0-0 [ H home Move cursor to home position (0-0) 7 sc Save current cursor position 8 rc Restore saved cursor position :?: most likely a normal code like \\b cub1 move left one space (backspace) VT100 [ ? 25 l civis make cursor invisible VT100 [ ? 25 h cvvis make cursor visible","tags":["bash","shell","scripting","colors","cursor","control","vt100","ansi"]},{"location":"scripting/terminalcodes/#erasing-text","title":"Erasing text","text":"ANSI terminfo equivalent Description [ K[ 0 K el Clear line from current cursor position to end of line [ 1 K el1 Clear line from beginning to current cursor position [ 2 K el2 :?: Clear whole line (cursor position unchanged)","tags":["bash","shell","scripting","colors","cursor","control","vt100","ansi"]},{"location":"scripting/terminalcodes/#general-text-attributes","title":"General text attributes","text":"ANSI terminfo equivalent Description [ 0 m sgr0 Reset all attributes [ 1 m bold Set \"bright\" attribute [ 2 m dim Set \"dim\" attribute [ 3 m smso Set \"standout\" attribute [ 4 m set smul unset rmul :?: Set \"underscore\" (underlined text) attribute [ 5 m blink Set \"blink\" attribute [ 7 m rev Set \"reverse\" attribute [ 8 m invis Set \"hidden\" attribute","tags":["bash","shell","scripting","colors","cursor","control","vt100","ansi"]},{"location":"scripting/terminalcodes/#foreground-coloring","title":"Foreground coloring","text":"ANSI terminfo equivalent Description [ 3 0 m setaf 0 Set foreground to color #0 - black [ 3 1 m setaf 1 Set foreground to color #1 - red [ 3 2 m setaf 2 Set foreground to color #2 - green [ 3 3 m setaf 3 Set foreground to color #3 - yellow [ 3 4 m setaf 4 Set foreground to color #4 - blue [ 3 5 m setaf 5 Set foreground to color #5 - magenta [ 3 6 m setaf 6 Set foreground to color #6 - cyan [ 3 7 m setaf 7 Set foreground to color #7 - white [ 3 9 m setaf 9 Set default color as foreground color","tags":["bash","shell","scripting","colors","cursor","control","vt100","ansi"]},{"location":"scripting/terminalcodes/#background-coloring","title":"Background coloring","text":"ANSI terminfo equivalent Description [ 4 0 m setab 0 Set background to color #0 - black [ 4 1 m setab 1 Set background to color #1 - red [ 4 2 m setab 2 Set background to color #2 - green [ 4 3 m setab 3 Set background to color #3 - yellow [ 4 4 m setab 4 Set background to color #4 - blue [ 4 5 m setab 5 Set background to color #5 - magenta [ 4 6 m setab 6 Set background to color #6 - cyan [ 4 7 m setab 7 Set background to color #7 - white [ 4 9 m setab 9 Set default color as background color","tags":["bash","shell","scripting","colors","cursor","control","vt100","ansi"]},{"location":"scripting/terminalcodes/#misc-codes","title":"Misc codes","text":"","tags":["bash","shell","scripting","colors","cursor","control","vt100","ansi"]},{"location":"scripting/terminalcodes/#saverestore-screen","title":"Save/restore screen","text":"Used capabilities: smcup, rmcup
You've undoubtedly already encountered programs that restore the terminal contents after they do their work (like vim). This can be done by the following commands:
# save, clear screen\ntput smcup\nclear\n\n# example \"application\" follows...\nread -n1 -p \"Press any key to continue...\"\n# example \"application\" ends here\n\n# restore\ntput rmcup\nThese features require that certain capabilities exist in your termcap/terminfo. While xterm and most of its clones (rxvt, urxvt, etc) will support the instructions, your operating system may not include references to them in its default xterm profile. (FreeBSD, in particular, falls into this category.) If tput smcup appears to do nothing for you, and you don't want to modify your system termcap/terminfo data, and you KNOW that you are using a compatible xterm application, the following may work for you:
echo -e '\\033[?47h' # save screen\necho -e '\\033[?47l' # restore screen\nCertain software uses these codes (via their termcap capabilities) as well. You may have seen the screen save/restore in less, vim, top, screen and others. Some of these applications may also provide configuration options to disable this behaviour. For example, less has a -X option for this, which can also be set in an environment variable:
export LESS=X\nless /path/to/file\nSimilarly, vim can be configured not to \"restore\" the screen by adding the following to your ~/.vimrc:
set t_ti= t_te=\nSome terminal emulators support additional colors. The most common extension used by xterm-compatible terminals supports 256 colors. These can be generated by tput with seta{f,b} [0-255] when the TERM value has a -256color suffix. Some terminals also support full 24-bit colors, and any X11 color code can be written directly into a special escape sequence. (More infos) Only a few programs make use of anything beyond 256 colors, and tput doesn't know about them. Colors beyond 16 usually only apply to modern terminal emulators running in graphical environments.
The Virtual Terminal implemented in the Linux kernel supports only 16 colors, and the usual default terminfo entry for TERM=linux defines only 8. There is sometimes an alternate \"linux-16color\" that you can switch to, to get the other 8 colors.
printf '%b\\n' 'It is \\033[31mnot\\033[39m intelligent to use \\033[32mhardcoded ANSI\\033[39m codes!'\nDirectly inside the echo:
echo \"TPUT is a $(tput setaf 2)nice$(tput setaf 9) and $(tput setaf 5)user friendly$(tput setaf 9) terminal capability database.\"\nWith preset variables:
COL_NORM=\"$(tput setaf 9)\"\nCOL_RED=\"$(tput setaf 1)\"\nCOL_GREEN=\"$(tput setaf 2)\"\necho \"It's ${COL_RED}red${COL_NORM} and ${COL_GREEN}green${COL_NORM} - have you seen?\"\nHOME function
home() {\n # yes, actually not much shorter ;-)\n tput home\n}\n#!/bin/bash\n\nDATA[0]=\" _/ _/ _/ _/ \"\nDATA[1]=\" _/_/_/_/_/ _/_/_/ _/_/_/ _/_/_/ _/_/_/ \"\nDATA[2]=\" _/ _/ _/ _/ _/ _/ _/_/ _/ _/\"\nDATA[3]=\"_/_/_/_/_/ _/ _/ _/ _/ _/_/ _/ _/ \"\nDATA[4]=\" _/ _/ _/_/_/ _/_/_/ _/_/_/ _/ _/ \"\n\n# virtual coordinate system is X*Y ${#DATA} * 5\n\nREAL_OFFSET_X=0\nREAL_OFFSET_Y=0\n\ndraw_char() {\n V_COORD_X=$1\n V_COORD_Y=$2\n\n tput cup $((REAL_OFFSET_Y + V_COORD_Y)) $((REAL_OFFSET_X + V_COORD_X))\n\n printf %c ${DATA[V_COORD_Y]:V_COORD_X:1}\n}\n\n\ntrap 'exit 1' INT TERM\ntrap 'tput setaf 9; tput cvvis; clear' EXIT\n\ntput civis\nclear\n\nwhile :; do\n\nfor ((c=1; c <= 7; c++)); do\n tput setaf $c\n for ((x=0; x<${#DATA[0]}; x++)); do\n for ((y=0; y<=4; y++)); do\n draw_char $x $y\n done\n done\ndone\n\ndone\nThis is a slightly modified version of Charles Cooke's colorful Mandelbrot plot scripts (original w/ screenshot) -- ungolfed, optimized a bit, and without hard-coded terminal escapes. The colorBox function is memoized to collect tput output only when required and output a new escape only when a color change is needed. This limits the number of tput calls to at most 16, and reduces raw output by more than half. The doBash function uses integer arithmetic, but is still ksh93-compatible (run as e.g. bash ./mandelbrot to use it). The ksh93-only floating-point doKsh is almost 10x faster than doBash (thus the ksh shebang by default), but uses only features that don't make the Bash parser crash.
#!/usr/bin/env ksh\n\n# Charles Cooke's 16-color Mandelbrot\n# http://earth.gkhs.net/ccooke/shell.html\n# Combined Bash/ksh93 flavors by Dan Douglas (ormaaj)\n\nfunction doBash {\n typeset P Q X Y a b c i v x y\n for ((P=10**8,Q=P/100,X=320*Q/cols,Y=210*Q/lines,y=-105*Q,v=-220*Q,x=v;y<105*Q;x=v,y+=Y)); do\n for ((;x<P;a=b=i=c=0,x+=X)); do\n for ((;a**2+b**2<4*P**2&&i++<99;a=((c=a)**2-b**2)/P+x,b=2*c*b/P+y)); do :\n done\n colorBox $((i<99?i%16:0))\n done\n echo\n done\n}\n\nfunction doKsh {\n integer i\n float a b c x=2.2 y=-1.05 X=3.2/cols Y=2.1/lines\n while\n for ((a=b=i=0;(c=a)**2+b**2<=2&&i++<99&&(a=a**2-b**2+x,b=2*c*b+y);)); do :\n done\n . colorBox $((i<99?i%16:0))\n if ((x<1?!(x+=X):(y+=Y,x=-2.2))); then\n print\n ((y<1.05))\n fi\n do :\n done\n}\n\nfunction colorBox {\n (($1==lastclr)) || printf %s \"${colrs[lastclr=$1]:=$(tput setaf \"$1\")}\"\n printf '\\u2588'\n}\n\nunset -v lastclr\n((cols=$(tput cols)-1, lines=$(tput lines)))\ntypeset -a colrs\ntrap 'tput sgr0; echo' EXIT\n${KSH_VERSION+. doKsh} ${BASH_VERSION+doBash}\nA much more sophisticated version by Roland Mainz can be found here
","tags":["bash","shell","scripting","colors","cursor","control","vt100","ansi"]},{"location":"snipplets/","title":"Small code snipplets","text":"These snipplets are not meant as HowTo or Tutorial or FAQ. Mostly they are only a line of code and a short comment.
See it more like an initial idea to give you a start.
","tags":["bash","shell","scripting","code","download","snipplet","example"]},{"location":"snipplets/add_color_to_your_scripts/","title":"Add Color to your scripts","text":"---- dataentry snipplet ---- snipplet_tags : terminal, color LastUpdate_dt : 2013-03-23 Contributors : Frank Lazzarini, Dan Douglas type : snipplet
Make your scripts output more readable using bash colors. Simply add these variables to your script, and you will be able to echo in color. (I haven't added all the colors available, just some basics)
# Colors\nESC_SEQ=\"\\x1b[\"\nCOL_RESET=$ESC_SEQ\"39;49;00m\"\nCOL_RED=$ESC_SEQ\"31;01m\"\nCOL_GREEN=$ESC_SEQ\"32;01m\"\nCOL_YELLOW=$ESC_SEQ\"33;01m\"\nCOL_BLUE=$ESC_SEQ\"34;01m\"\nCOL_MAGENTA=$ESC_SEQ\"35;01m\"\nCOL_CYAN=$ESC_SEQ\"36;01m\"\nNow if you want to output some text in color use echo -e instead of just echo. And always remember to use the \\$COL_RESET variable to reset the color changes in bash. Like so ....
echo -e \"$COL_RED This is red $COL_RESET\"\necho -e \"$COL_BLUE This is blue $COL_RESET\"\necho -e \"$COL_YELLOW This is yellow $COL_RESET\"\nBut also see the notes in the article about using terminalcodes about generating codes and hardwiring codes.
This snipplet sets up associative arrays for basic color codes using tput for Bash, ksh93 or zsh. You can pass it variable names to correspond with a collection of codes. There's a main function with example usage.
#!/usr/bin/env bash\n\n${ZSH_VERSION+false} || emulate ksh\n${BASH_VERSION+shopt -s lastpipe extglob}\n\n# colorSet [ --setaf | --setab | --misc ] var\n# Assigns the selected set of escape mappings to the given associative array names.\nfunction colorSet {\n typeset -a clrs msc\n typeset x\n clrs=(black red green orange blue magenta cyan grey darkgrey ltred ltgreen yellow ltblue ltmagenta ltcyan white)\n msc=(sgr0 bold dim smul blink rev invis)\n\n while ! ${2:+false}; do\n ${KSH_VERSION:+eval typeset -n \"$2\"=\\$2}\n case ${1#--} in\n setaf|setab)\n for x in \"${!clrs[@]}\"; do\n eval \"$2\"'[${clrs[x]}]=$(tput \"${1#--}\" \"$x\")'\n done\n ;;\n misc)\n for x in \"${msc[@]}\"; do\n eval \"$2\"'[$x]=$(tput \"$x\")'\n done\n ;;\n *)\n return 1\n esac\n shift 2\n done\n}\n\n# Example code\nfunction main {\n typeset -A fgColors bgColors miscEscapes\n if colorSet --setaf fgColors --setab bgColors --misc miscEscapes; then\n if ! ${1:+${fgColors[$1]:+false}}; then\n printf '%s%s%s\\n' \"${fgColors[$1]}\" \"this text is ${1}\" \"${miscEscapes[sgr0]}\" >&3\n else\n printf '%s, %s\\n' \"${1:-Empty}\" 'no such color.'\n typeset x y\n for x in fgColors bgColors miscEscapes; do\n typeset -a keys\n eval 'keys=(\"${!'\"$x\"'[@]}\")'\n printf '%s=( ' \"$x\"\n for y in \"${keys[@]}\"; do\n eval 'printf \"[%q]=%q \" \"$y\" \"${'\"$x\"'[$y]}\"'\n done\n printf ')\\n'\n done\n return 1\n fi\n else\n echo 'Failed setting color arrays.'\n return 1\n fi 3>&1 >&2\n}\n\nmain \"$@\"\n\n# vim: set fenc=utf-8 ff=unix ft=sh :\nawk to deal with CSV that uses quoted/unquoted delimiters","text":"CSV files are a mess, yes.
Assume you have CSV files that use the comma as delimiter and quoted data fields that can contain the delimiter.
\"first\", \"second\", \"last\"\n\"fir,st\", \"second\", \"last\"\n\"firtst one\", \"sec,ond field\", \"final,ly\"\nSimply using the comma as separator for awk won't work here, of course.
Solution: Use the field separator \", \"|^\"|\"$ for awk.
This is an OR-ed list of 3 possible separators:
\", \" matches the area between the datafields ^\" matches the area left of the first datafield \"$ matches the area right of the last data field You can tune these delimiters if you have other needs (for example if you don't have a space after the commas).
Test:
The awk command used for the CSV above just prints the fileds separated by ### to see what's going on:
$ awk -v FS='\", \"|^\"|\"$' '{print $2\"###\"$3\"###\"$4}' data.csv\nfirst###second###last\nfir,st###second###last\nfirtst one###sec,ond field###final,ly\nATTENTION If the CSV data changes its format every now and then (for example it only quotes the data fields if needed, not always), then this way will not work.
","tags":["awk","csv"]},{"location":"snipplets/filesize/","title":"Show size of a file","text":"---- dataentry snipplet ---- snipplet_tags: files, file size LastUpdate_dt: 2010-07-31 Contributors: Frank Lazzarini type: snipplet
This is a simple snippet to echo the size of a file in bytes.
#!/bin/bash\nFILENAME=/home/heiko/dummy/packages.txt\nFILESIZE=$(wc -c < \"$FILENAME\")\n# non standard way (GNU stat): FILESIZE=$(stat -c%s \"$FILENAME\")\n\necho \"Size of $FILENAME = $FILESIZE bytes.\"\n---- dataentry snipplet ---- snipplet_tags: kill, process management, jobs LastUpdate_dt: 2010-07-31 Contributors: Jan Schampera type: snipplet
When you start background jobs from within a script (non-interactive shell) and kill it afterwards, you will get a message from the shell that the process was terminated.
Example:
#!/bin/bash\n\n# example background process\nsleep 300 &\n\n# get the PID\nBG_PID=$!\n\n# kill it, hard and mercyless\nkill -9 $BG_PID\n\necho \"Yes, we killed it\"\nYou will get something like this:
$ ./bg_kill1.sh\n./bg_kill1.sh: line 11: 3413 Killed sleep 300\nYes, we killed it\nThis is more or less a normal message. And it can't be easily redirected since it's the shell itself that yells this message, not the command kill or something else. You would have to redirect the whole script's output.
It's also useless to temporarily redirect stderr when you call the kill command, since the successful termination of the job, the termination of the kill command and the message from the shell may not happen at the same time. And a blind sleep after the kill would be just a workaround.
The solution is relatively easy: The shell spits that message because it controls the background job, and when it terminates, the shell will tell you whenever possible. Now you just need to tell your shell that it is no longer responsible for that background process. This is done by the disown command, which can take an internal shell job number (like %1) or a process ID as argument.
#!/bin/bash\n\n# example background process\nsleep 300 &\n\n# get the PID\nBG_PID=$!\n\n### HERE, YOU TELL THE SHELL TO NOT CARE ANY MORE ###\ndisown $BG_PID\n###\n\n\n# kill it, hard and mercyless, now without a trace\nkill -9 $BG_PID\n\necho \"Yes, we killed it\"\nThat way, you can run and kill background processes without disturbing messages.
"},{"location":"snipplets/largestfile/","title":"Get largest file","text":"---- dataentry snipplet ---- snipplet_tags: directory, recursive, find, crawl LastUpdate_dt: 2013-03-23 Contributors: Dan Douglas type: snipplet
One basic pattern for recursive directory traversal with operations on files at each node. This gets the largest file in each subdirectory. Toggling some small details will make it return the smallest, or traverse breadth-first instead of depth-first.
#!/usr/bin/env bash\n# GNU find + bash4 / ksh93v / zsh\n# Get the largest file matching pattern in the given directories recursively\n${ZSH_VERSION+false} || emulate ksh\n${BASH_VERSION+shopt -s lastpipe extglob}\n\nfunction getLargest {\n typeset -A cur top || return\n typeset dir x\n for dir in \"$2\"/*/; do\n [[ -d $dir ]] || return 0\n getLargest \"$1\" \"${dir%/}\" || return\n top[size]=-1\n find \"$dir\" -maxdepth 1 -type f -name \"$1\" -printf '%s\\0%f\\0' | {\n while :; do\n for x in cur\\[{size,name}\\]; do\n IFS= read -rd '' \"$x\" || break 2\n done\n if (( cur[size] > top[size] )); then\n top[size]=${cur[size]} top[name]=${cur[name]}\n fi\n done\n printf '%q\\n' \"${dir}${top[name]}\"\n }\n done\n}\n\n# main pattern dir [ dir ... ]\nfunction main {\n if [[ -n $1 ]]; then\n typeset dir pattern=$1\n shift\n for dir; do\n [[ -d $dir ]] || return\n getLargest \"$pattern\" \"$dir\"\n done\n else\n return 1\n fi\n}\n\nmain \"$@\"\n\n# vim: set fenc=utf-8 ff=unix ft=sh :\n---- dataentry snipplet ---- snipplet_tags: terminal, pause, input LastUpdate_dt: 2010-07-31 Contributors: Jan Schampera type: snipplet
From the example section of the read command, something that acts similar to the MSDOS pause command:
pause() {\n local dummy\n read -s -r -p \"Press any key to continue...\" -n 1 dummy\n}\n---- dataentry snipplet ---- snipplet_tags: debug, arguments LastUpdate_dt: 2013-03-23 Contributors: Snappy (IRC), Dan Douglas type: snipplet
Sometimes you might find it useful to see how arguments passed to a program arrive there.
Check this script (save it as script file or make a function):
printf '\"%b\"\\n' \"$0\" \"$@\" | nl -v0 -s\": \"\nIt uses the printf command to generate a list of arguments, even with escape sequences interpreted. This list is shown formatted by the nl(1) utility.
Another alternative with colorized output. If run in Bash, it temporarily disables all debug output for itself, including the test that determines whether to hide debug output. In ksh, tracing would have to be enabled on the function to show debug output, so it works out to being equivalent.
# Bash or ksh93 debugging function for colored display of argv.\n# Optionally set OFD to the desired output file descriptor.\nfunction args {\n { BASH_XTRACEFD=3 command eval ${BASH_VERSION+\"$(</dev/fd/0)\"}; } <<-'EOF' 3>/dev/null\n case $- in *x*)\n set +x\n trap 'trap RETURN; set -x' RETURN\n esac\nEOF\n\n [[ ${OFD-1} == +([0-9]) ]] || return\n\n if [[ -t ${OFD:-2} ]]; then\n typeset -A clr=([green]=$(tput setaf 2) [sgr0]=$(tput sgr0))\n else\n typeset clr\n fi\n\n if ! ${1+false}; then\n printf -- \"${clr[green]}<${clr[sgr0]}%s${clr[green]}>${clr[sgr0]} \" \"$@\"\n echo\n else\n echo 'no args.'\n fi >&\"${OFD:-2}\"\n}\n---- dataentry snipplet ---- snipplet_tags: terminal, line LastUpdate_dt: 2010-07-31 Contributors: Jan Schampera, prince_jammys, ccsalvesen, others type: snipplet
The purpose of this small code collection is to show some code that draws a horizontal line using as less external tools as possible (it's not a big deal to do it with AWK or Perl, but with pure or nearly-pure Bash it gets more interesting).
In general, you should be able to use this code to repeat any character or character sequence.
"},{"location":"snipplets/print_horizontal_line/#the-simple-way-just-print-it","title":"The simple way: Just print it","text":"Not a miracle, just to be complete here.
printf '%s\\n' --------------------\nThis one simply loops 20 times, always draws a dash, finally a newline
for ((x = 0; x < 20; x++)); do\n printf %s -\ndone\necho\nThis one uses the printf command to print an empty field with a minimum field width of 20 characters. The text is padded with spaces, since there is no text, you get 20 spaces. The spaces are then converted to - by the tr command.
printf '%20s\\n' | tr ' ' -\nwhitout an external command, using the (non-POSIX) substitution expansion and -v option:
printf -v res %20s\nprintf '%s\\n' \"${res// /-}\"\nThis is a variant of the above that uses tput cols to find the width of the terminal and set that number as the minimum field witdh.
printf '%*s\\n' \"${COLUMNS:-$(tput cols)}\" '' | tr ' ' -\nThis one is a bit tricky. The format for the printf command is %.0s, which specified a field with the maximum length of zero. After this field, printf is told to print a dash. You might remember that it's the nature of printf to repeat, if the number of conversion specifications is less than the number of given arguments. With brace expansion {1..20}, 20 arguments are given (you could easily write 1 2 3 4 ... 20, of course!). Following happens: The zero-length field plus the dash is repeated 20 times. A zero length field is, naturally, invisible. What you see is the dash, repeated 20 times.
# Note: you might see that as ''%.s'', which is a (less documented) shorthand for ''%.0s''\nprintf '%.0s-' {1..20}; echo\nIf the 20 is variable, you can use eval to insert the expansion (take care that using eval is potentially dangerous if you evaluate external data):
eval printf %.0s- '{1..'\"${COLUMNS:-$(tput cols)}\"\\}; echo\nOr restrict the length to 1 and prefix the arguments with the desired character.
eval printf %.1s '-{1..'\"${COLUMNS:-$(tput cols)}\"\\}; echo\nYou can also do it the crazy ormaaj way\u2122 following basically the same principle as this string reverse example. It completely depends on Bash due to its brace expansion evaluation order and array parameter parsing details. As above, the eval only inserts the COLUMNS expansion into the expression and isn't involved in the rest, other than to put the _ value into the environment of the _[0] expansion. This works well since we\\'re not creating one set of arguments and then editing or deleting them to create another as in the previous examples.
_=- command eval printf %s '\"${_[0]\"{0..'\"${COLUMNS:-$(tput cols)}\"'}\"}\"'; echo\nPreparing enough dashes in advance, we can then use a non-POSIX subscript expansion:
hr=---------------------------------------------------------------\\\n----------------------------------------------------------------\nprintf '%s\\n' \"${hr:0:${COLUMNS:-$(tput cols)}}\"\nA more flexible approach, and also using modal terminal line-drawing characters instead of hyphens:
hr() {\n local start=$'\\e(0' end=$'\\e(B' line='qqqqqqqqqqqqqqqq'\n local cols=${COLUMNS:-$(tput cols)}\n while ((${#line} < cols)); do line+=\"$line\"; done\n printf '%s%s%s\\n' \"$start\" \"${line:0:cols}\" \"$end\"\n}\n---- dataentry snipplet ---- snipplet_tags: terminal, line LastUpdate_dt: 2013-04-30 Contributors: Dan Douglas (ormaaj) type: snipplet
First off, here is a fast / reliable random string function for scripts or libraries which can optionally assign directly to a variable.
# Print or assign a random alphanumeric string of a given length.\n# rndstr len [ var ]\nfunction rndstr {\n if [[ $FUNCNAME == \"${FUNCNAME[1]}\" ]]; then\n unset -v a\n printf \"$@\"\n elif [[ $1 != +([[:digit:]]) ]]; then\n return 1\n elif (( $1 )); then\n typeset -a a=({a..z} {A..Z} {0..9})\n eval '${2:+\"$FUNCNAME\" -v} \"${2:-printf}\" -- %s \"${a[RANDOM%'\"${#a[@]}\"']\"{1..'\"$1\"'}\"}\"'\n fi\n}\nThis example prints 10 random positional parameters and operates on basically the same principle as the rndstr function above.
~ $ ( set -- foo bar baz bork; printf '%s ' \"${!_[_=RANDOM%$#+1,0]\"{0..10}\"}\"; echo )\nbork bar baz baz foo baz baz baz baz baz bork\n# Print or assign a random alphanumeric string of a given length.\n# rndstr [ -v var ] len\n# Bash-only\nrndstr()\n if [[ $FUNCNAME == \"${FUNCNAME[1]}\" ]]; then\n # On recursion, this branch unsets the outer scope's locals and assigns the result.\n unset -v a b\n printf -v \"$1\" %s \"${@:2}\"\n elif ! { [[ $1 == -v ]] && shift; }; [[ $?+1 -ne $# || ${!#} != +([[:digit:]]) || ( $? -gt 0 && -z $1 ) ]]; then\n # This branch does input validation, strips -v, and guarantees we're left with either 1 or 2 args.\n return 1\n elif (( ! ${!#} )); then\n # If a zero-length string is requested, return success.\n return\n else\n # This line generates the string and assigns it to \"b\".\n local -a a=({a..z} {A..Z} {0..9}) 'b=(\"${a[RANDOM%'\"${#a[@]}\"']\"{1..'\"${!#}\"'}\"}\")'\n if (( $# == 2 )); then\n # If -v, then pass a variable name and value to assign and recurse once.\n \"$FUNCNAME\" \"$1\" \"${b[@]}\"\n else\n # If no -v, write to stdout.\n printf %s \"${b[@]}\"\n fi\n fi\nThe remaining examples don't use quite the same tricks, which will hopefully be explained elsewhere eventually. See unset for why doing assignments in this way works well.
This next example is a variation on print_horizontal_line. We\\'re using the printf field width specifier to truncate the values of a sequence expansion to one character.
a=({a..z} {A..Z} {0..9})\nprintf '%.1s' \"${a[RANDOM%${#a[@]}]}\"{0..9} $'\\n'\nThe extra detail that makes this work is to notice that in Bash, brace expansion is usually the very first type of expansion to be processed, always before parameter expansion. Bash is unique in this respect -- all other shells with a brace expansion feature perform it almost last, just before pathname expansion. First the sequence expansion generates ten parameters, then the parameters are expanded left-to-right causing the arithmetic for each to be evaluated individually, resulting in independent selection of random element of a. To get ten of the same element, put the array selection inside the format string where it will only be evaluated once, just like the dashed-line trick:
printf \"%.s${a[RANDOM%${#a[@]}]}\" {0..9} \nSelecting random elements whose lengths are not fixed is harder.
a=(one two three four five six seven eight nine ten)\nprintf '%.*s ' $(printf '%s ' \"${#a[x=RANDOM%${#a[@]}]} ${a[x]}\"{1..10})\nThis generates each parameter and it's length in pairs. The \\'*\\' modifier instructs printf to use the value preceding each parameter as the field width. Note the space between the parameters. This example unfortunately relies upon the unquoted command substitution to perform unsafe wordsplitting so that the outer printf gets each argument. Values in the array can't contain characters in IFS, or anything that might be interpreted as a pattern without using set -f.
Lastly, empty brace expansions can be used which don't generate any output that would need to be filtered. The disadvantage of course is that you must construct the brace expansion syntax to add up to the number of arguments to be generated, where the most optimal solution is its set of prime factors.
a=(one two three)\necho \"${a[RANDOM%${#a[@]}]}\"{,}{,,,,}\n---- dataentry snipplet ---- snipplet_tags: terminal, restore screen LastUpdate_dt: 2010-07-31 Contributors: Greg Wooledge type: snipplet
This cool hack uses the terminal capabilities (see terminfo(5) manual) smcup and rmcup to save and restore the terminal content.
For sure, you've already seen those programs that restore the terminal contents after they did their work (like vim).
# save, clear screen\ntput smcup\nclear\n\n# example \"application\" follows...\nread -n1 -p \"Press any key to continue...\"\n# example \"application\" ends here\n\n# restore\ntput rmcup\n---- dataentry snipplet ---- snipplet_tags: ssh, ssh-keys LastUpdate_dt: 2010-07-31 Contributors: Jan Schampera
Applies at least to openssh.
To get the hostkeys for a server, and write them to known_hosts-file (to avoid that yes/no query when the key isn't known), you can do:
ssh-keyscan -t rsa foo foo.example.com 1.2.3.4 >> ~/.ssh/known_host\nThis example queries the hostkeys for the very same machine, but under 3 different \\\"names\\\" (hostname, FQDN, IP) and redirects the output to the known_hosts-file.
Notes:
known_host-file may grow very large. It might be wise to check for key existance firstknown_hosts, the first one is taken (which might be an old or wrong one)---- dataentry snipplet ---- snipplet_tags: ssh, variables LastUpdate_dt: 2010-07-31 Contributors: cweiss type: snipplet
In this example, we want to make sure a certain file exists on the remote server:
file=/tmp/file.log\nssh ${options} ${login} \"if [ ! -e '$file' ] ; then touch '$file' ; fi\"\nNotice the command is surrounded by double quotes, and the \\$file variable is surrounded by single quotes. That has the effect to be wordsplit-proof in the local shell (due to the double-quotes) and in the remote shell (due to the single-quotes).
"},{"location":"snipplets/wrapperargs/","title":"Generate code with own arguments properly quoted","text":"There are situations where Bash code needs to generate Bash code. A script that writes out another script the user or cron may start, for example.
The general issue is easy, just write out text to the file.
A specific detail of it is tricky: If the generated script needs to call a command using the arguments the first original script got, you have problem in writing out the correct code.
I.e. if you run your generator script like
./myscript \"give me 'some' water\"\nthen this script should generate code that looks like
echo give me 'some' water\"\nyou need correct escapes or quotes to not generate shell special characters out of normal text (like embedded dollar signs $).
Solution:
A loop over the own arguments that writes out properly quoted/escaped code to the generated script file
There are two (maybe more) easy options:
%q format specification, which will print a string (like %s does), but with all shell special characters escaped#!/bin/bash\n\n# first option:\n# generate singlequoted strings out of your own arguments and handle embedded singlequotes\n# here to call 'echo' in the generated script\n\n{\nprintf \"#!/bin/bash\\n\\n\"\nprintf \"echo \"\nfor arg; do\n arg=${arg/\\'/\\'\\\\\\'\\'}\n printf \"'%s' \" \"${arg}\"\ndone\n\nprintf \"\\n\"\n} >s2\nThe generated script will look like:
#!/bin/bash\n\necho 'fir$t' 'seco \"ond\"' 'thir'\\''d'\nThe second method is easier, though more or less Bash-only (due to the %q in printf):
#!/bin/bash\n\n{\nprintf \"#!/bin/bash\\n\\n\"\nprintf \"echo \"\nfor arg; do\n printf '%q ' \"$arg\"\ndone\n\nprintf \"\\n\"\n} >s2\nThe generated script will look like:
#!/bin/bash\n\necho fir\\$t seco\\ \\\"ond\\\" thir\\'d\n---- dataentry snipplet ---- snipplet_tags: clipboard, x11, xclip, readline LastUpdate_dt: 2010-07-31 Contributors: Josh Triplett type: snipplet
# Make Control-v paste, if in X and if xclip available - Josh Triplett\nif [ -n \"$DISPLAY\" ] && [ -x /usr/bin/xclip ] ; then\n # Work around a bash bug: \\C-@ does not work in a key binding\n bind '\"\\C-x\\C-m\": set-mark'\n # The '#' characters ensure that kill commands have text to work on; if\n # not, this binding would malfunction at the start or end of a line.\n bind 'Control-v: \"#\\C-b\\C-k#\\C-x\\C-?\\\"$(xclip -o -selection c)\\\"\\e\\C-e\\C-x\\C-m\\C-a\\C-y\\C-?\\C-e\\C-y\\ey\\C-x\\C-x\\C-d\"'\nfi\nThe behaviour is a bit tricky to explain:
\"$(xclip -o -selection c)\"\"$(xclip -o -selection c)\"Of course you can use any other command, you\\'re not limited to xclip here.
Note: C-@ as well as M-SPC both works and set the mark for me -- pgas
"},{"location":"syntax/arith_expr/","title":"Arithmetic expressions","text":"Arithmetic expressions are used in several situations:
let builtin commandThese expressions are evaluated following some rules described below. The operators and rules of arithmetic expressions are mainly derived from the C programming language.
This article describes the theory of the used syntax and the behaviour. To get practical examples without big explanations, see this page on Greg's wiki.
","tags":["bash","shell","scripting","math","arithmetic","C","calculation","integer"]},{"location":"syntax/arith_expr/#constants","title":"Constants","text":"Mathematical constants are simply fixed values you write: 1, 3567, or 4326. Bash interprets some notations specially:
0... (leading zero) is interpreted as an octal value0x... is interpreted as a hex value0X... also interpreted as a hex<BASE>#... is interpreted as a number according to the specified base <BASE>, e.g., 2#00111011 (see below)If you have a constant set in a variable, like,
x=03254\nthis is interpreted as an octal value. If you want it to be interpreted as a decimal value, you need to expand the parameter and specify base 10:
# this is interpreted as a decimal:\necho $(( 10#$x ))\n\n# this is interpreted as an octal:\necho $(( x ))\n\n# this is an invalid digit for base 10 (the \"x\")...:\necho $(( 10#x ))\nFor a constant, the base can be specified using the form
<BASE>#<DIGITS...>\nRegardless of the specified base, the arithmetic expressions will, if ever displayed, be displayed in decimal!
When no base is specified, the base 10 (decimal) is assumed, except when the prefixes as mentioned above (octals, hexadecimals) are present. The specified base can range from 2 to 64. To represent digits in a specified base greater than 10, characters other than 0 to 9 are needed (in this order, low => high):
0 ... 9a ... zA ... Z@_Let's quickly invent a new number system with base 43 to show what I mean:
$ echo $((43#1))\n1\n\n$ echo $((43#a))\n10\n\n$echo $((43#A))\n36\n\n$ echo $((43#G))\n42\n\n$ echo $((43#H))\nbash: 43#H: value too great for base (error token is \"43#H\")\nIf you have no clue what a base is and why there might be other bases, and what numbers are and how they are built, then you don't need different bases.
If you want to convert between the usual bases (octal, decimal, hex), use the printf command and its format strings.
","tags":["bash","shell","scripting","math","arithmetic","C","calculation","integer"]},{"location":"syntax/arith_expr/#shell-variables","title":"Shell variables","text":"Shell variables can of course be used as operands, even when the integer attribute is not turned on (by declare -i <NAME>). If the variable is empty (null) or unset, its reference evaluates to 0. If the variable doesn't hold a value that looks like a valid expression (numbers or operations), the expression is re-used to reference, for example, the named parameters, e.g.:
test=string\nstring=3\n\necho $((test))\n# will output \"3\"!\nOf course, in the end, when it finally evaluates to something that is not a valid arithmetic expression (newlines, ordinary text, ...) then you'll get an error.
When variables are referenced, the notation 1 + $X is equivalent to the notation 1 + X, both are allowed.
When variables are referenced like $X, the rules of parameter expansion apply and are performed before the expression is evaluated. Thus, a construct like ${MYSTRING:4:3} is valid inside an arithmetic expression.
Unlike command exit and return codes, arithmetic expressions evaluate to logical \"true\" when they are not 0. When they are 0, they evaluate to \"false\". The arithmetic evaluation compound command reverses the \"truth\" of an arithmetic expression to match the \"truth\" of command exit codes:
That means, the following if-clause will execute the else-thread:
if ((0)); then\n echo \"true\"\nelse\n echo \"false\"\nfi\n<ID> = <EXPR> normal assignment <ID> *= <EXPR> equivalent to <ID> = <ID> * <EXPR>, see calculation operators <ID> /= <EXPR> equivalent to <ID> = <ID> / <EXPR>, see calculation operators <ID> %= <EXPR> equivalent to <ID> = <ID> % <EXPR>, see calculation operators <ID> += <EXPR> equivalent to <ID> = <ID> + <EXPR>, see calculation operators <ID> -= <EXPR> equivalent to <ID> = <ID> - <EXPR>, see calculation operators <ID> <<= <NUMBER> equivalent to <ID> = <ID> << <NUMBER>, see bit operations <ID> >>= <NUMBER> equivalent to <ID> = <ID> >> <NUMBER>, see bit operations <ID> &= <EXPR> equivalent to <ID> = <ID> & <EXPR>, see bit operations <ID> ^= <EXPR> equivalent to <ID> = <ID> ^ <EXPR>, see bit operations <ID>|= <EXPR> equivalent to <ID> = <ID>|<EXPR>, see bit operations","tags":["bash","shell","scripting","math","arithmetic","C","calculation","integer"]},{"location":"syntax/arith_expr/#calculations","title":"Calculations","text":"Operator Description * multiplication / division % remainder (modulo) + addition - subtraction ** exponentiation","tags":["bash","shell","scripting","math","arithmetic","C","calculation","integer"]},{"location":"syntax/arith_expr/#comparisons","title":"Comparisons","text":"Operator Description < comparison: less than > comparison: greater than <= comparison: less than or equal >= comparison: greater than or equal == equality != inequality","tags":["bash","shell","scripting","math","arithmetic","C","calculation","integer"]},{"location":"syntax/arith_expr/#bit-operations","title":"Bit operations","text":"Operator Description ~ bitwise negation << bitwise shifting (left) >> bitwise shifting (right) & bitwise AND ^ bitwise exclusive OR (XOR) | bitwise OR","tags":["bash","shell","scripting","math","arithmetic","C","calculation","integer"]},{"location":"syntax/arith_expr/#logical","title":"Logical","text":"Operator Description ! logical negation && logical AND || logical OR","tags":["bash","shell","scripting","math","arithmetic","C","calculation","integer"]},{"location":"syntax/arith_expr/#misc","title":"Misc","text":"Operator Description id++ post-increment of the variable id (not required by POSIX\u00ae) id-- post-decrement of the variable id (not required by POSIX\u00ae) ++id pre-increment of the variable id (not required by POSIX\u00ae) --id pre-decrement of the variable id (not required by POSIX\u00ae) + unary plus - unary minus <EXPR> ? <EXPR> : <EXPR> conditional (ternary) operator <condition> ? <result-if-true> : <result-if-false> <EXPR> , <EXPR> expression list ( <EXPR> ) subexpression (to force precedence)","tags":["bash","shell","scripting","math","arithmetic","C","calculation","integer"]},{"location":"syntax/arith_expr/#precedence","title":"Precedence","text":"The operator precedence is as follows (highest -> lowest):
id++, id--)++id, --id)-, +)!, ~)**)*, /, %)+, -)<<, >>)<, >, <=, >=)==, !=)&)^)|)&&)||)<EXPR> ? <EXPR> : <EXPR>)=, *=, /=, %=, +=, -=, <<=, >>=, &=, ^=, |=)<EXPR> , <EXPR>)The precedence can be adjusted using subexpressions of the form ( <EXPR> ) at any time. These subexpressions are always evaluated first.
Bash's overall language construct is based on exit codes or return codes of commands or functions to be executed. if statements, while loops, etc., they all take the return codes of commands as conditions.
Now the problem is: The return codes (0 means \"TRUE\" or \"SUCCESS\", not 0 means \"FALSE\" or \"FAILURE\") don't correspond to the meaning of the result of an arithmetic expression (0 means \"FALSE\", not 0 means \"TRUE\").
That's why all commands and keywords that do arithmetic operations attempt to translate the arithmetical meaning into an equivalent return code. This simply means:
This way, you can easily use arithmetic expressions (along with the commands or keywords that operate them) as conditions for if, while and all the others, including set -e for autoexit on error:
MY_TEST_FLAG=0\n\nif ((MY_TEST_FLAG)); then\n echo \"MY_TEST_FLAG is ON\"\nelse\n echo \"MY_TEST_FLAG is OFF\"\nfi\nBeware that set -e can change the runtime behavior of scripts.
For example,
This non-equivalence of code behavior deserves some attention. Consider what happens if v happens to be zero in the expression below:
((v += 0))\necho $?\n1
(\"FAILURE\")
v=$((v + 0))\necho $?\n0
(\"SUCCESS\")
The return code behavior is not equivalent to the arithmetic behavior, as has been noted.
A workaround is to use a list operation that returns True, or use the second assignment style.
((v += 0)) || :\necho $?\n0
(\"SUCCESS\")
This change in code behavior was discovered once the script was run under set -e.
","tags":["bash","shell","scripting","math","arithmetic","C","calculation","integer"]},{"location":"syntax/arith_expr/#arithmetic-expressions-in-bash","title":"Arithmetic expressions in Bash","text":"An array is a parameter that holds mappings from keys to values. Arrays are used to store a collection of parameters into a parameter. Arrays (in any programming language) are a useful and common composite data structure, and one of the most important scripting features in Bash and other shells.
Here is an abstract representation of an array named NAMES. The indexes go from 0 to 3.
NAMES\n 0: Peter\n 1: Anna\n 2: Greg\n 3: Jan\nInstead of using 4 separate variables, multiple related variables are grouped grouped together into elements of the array, accessible by their key. If you want the second name, ask for index 1 of the array NAMES.
Bash supports two different types of ksh-like one-dimensional arrays. Multidimensional arrays are not implemented.
-a attribute.-A attribute, and unlike indexed arrays, Bash requires that they always be declared explicitly (as indexed arrays are the default, see declaration). Associative arrays were first introduced in ksh93, and similar mechanisms were later adopted by Zsh and Bash version 4. These three are currently the only POSIX-compatible shells with any associative array support.To accommodate referring to array variables and their individual elements, Bash extends the parameter naming scheme with a subscript suffix. Any valid ordinary scalar parameter name is also a valid array name: [[:alpha:]_][[:alnum:]_]*. The parameter name may be followed by an optional subscript enclosed in square brackets to refer to a member of the array.
The overall syntax is arrname[subscript] - where for indexed arrays, subscript is any valid arithmetic expression, and for associative arrays, any nonempty string. Subscripts are first processed for parameter and arithmetic expansions, and command and process substitutions. When used within parameter expansions or as an argument to the unset builtin, the special subscripts * and @ are also accepted which act upon arrays analogously to the way the @ and * special parameters act upon the positional parameters. In parsing the subscript, bash ignores any text that follows the closing bracket up to the end of the parameter name.
With few exceptions, names of this form may be used anywhere ordinary parameter names are valid, such as within arithmetic expressions, parameter expansions, and as arguments to builtins that accept parameter names. An array is a Bash parameter that has been given the -a (for indexed) or -A (for associative) attributes. However, any regular (non-special or positional) parameter may be validly referenced using a subscript, because in most contexts, referring to the zeroth element of an array is synonymous with referring to the array name without a subscript.
# \"x\" is an ordinary non-array parameter.\n$ x=hi; printf '%s ' \"$x\" \"${x[0]}\"; echo \"${_[0]}\"\nhi hi hi\nThe only exceptions to this rule are in a few cases where the array variable's name refers to the array as a whole. This is the case for the unset builtin (see destruction) and when declaring an array without assigning any values (see declaration).
The following explicitly give variables array attributes, making them arrays:
Syntax DescriptionARRAY=() Declares an indexed array ARRAY and initializes it to be empty. This can also be used to empty an existing array. ARRAY[0]= Generally sets the first element of an indexed array. If no array ARRAY existed before, it is created. declare -a ARRAY Declares an indexed array ARRAY. An existing array is not initialized. declare -A ARRAY Declares an associative array ARRAY. This is the one and only way to create associative arrays. As an example, and for use below, let's declare our NAMES array as described above:
declare -a NAMES=('Peter' 'Anna' 'Greg' 'Jan')\nStoring values in arrays is quite as simple as storing values in normal variables.
Syntax DescriptionARRAY[N]=VALUE Sets the element N of the indexed array ARRAY to VALUE. N can be any valid arithmetic expression. ARRAY[STRING]=VALUE Sets the element indexed by STRING of the associative array ARRAY. ARRAY=VALUE As above. If no index is given, as a default the zeroth element is set to VALUE. Careful, this is even true of associative arrays - there is no error if no key is specified, and the value is assigned to string index \"0\". ARRAY=(E1\\ E2\\ ...) Compound array assignment - sets the whole array ARRAY to the given list of elements indexed sequentially starting at zero. The array is unset before assignment unless the += operator is used. When the list is empty (ARRAY=()), the array will be set to an empty array. This method obviously does not use explicit indexes. An associative array can not be set like that! Clearing an associative array using ARRAY=() works. ARRAY=([X]=E1\\ [Y]=E2\\ ...) Compound assignment for indexed arrays with index-value pairs declared individually (here for example X and Y). X and Y are arithmetic expressions. This syntax can be combined with the above - elements declared without an explicitly specified index are assigned sequentially starting at either the last element with an explicit index, or zero. ARRAY=([S1]=E1\\ [S2]=E2\\ ...) Individual mass-setting for associative arrays. The named indexes (here: S1 and S2) are strings. ARRAY+=(E1\\ E2\\ ...) Append to ARRAY. ARRAY=(\"${ANOTHER_ARRAY[@]}\") Copy ANOTHER_ARRAY to ARRAY, copying each element. As of now, arrays can't be exported.
"},{"location":"syntax/arrays/#getting-values","title":"Getting values","text":"For completeness and details on several parameter expansion variants, see the article about parameter expansion and check the notes about arrays.
Syntax Description${ARRAY[N]} Expands to the value of the index N in the indexed array ARRAY. If N is a negative number, it's treated as the offset from the maximum assigned index (can't be used for assignment) - 1 ${ARRAY[S]} Expands to the value of the index S in the associative array ARRAY. \"${ARRAY[@]}\" ${ARRAY[@]} \"${ARRAY[*]}\" ${ARRAY[*]} Similar to mass-expanding positional parameters, this expands to all elements. If unquoted, both subscripts * and @ expand to the same result, if quoted, @ expands to all elements individually quoted, * expands to all elements quoted as a whole. \"${ARRAY[@]:N:M}\" ${ARRAY[@]:N:M} \"${ARRAY[*]:N:M}\" ${ARRAY[*]:N:M} Similar to what this syntax does for the characters of a single string when doing substring expansion, this expands to M elements starting with element N. This way you can mass-expand individual indexes. The rules for quoting and the subscripts * and @ are the same as above for the other mass-expansions. For clarification: When you use the subscripts @ or * for mass-expanding, then the behaviour is exactly what it is for $@ and $* when mass-expanding the positional parameters. You should read this article to understand what's going on.
${#ARRAY[N]} Expands to the length of an individual array member at index N (stringlength) ${#ARRAY[STRING]} Expands to the length of an individual associative array member at index STRING (stringlength) ${#ARRAY[@]} \\ ${#ARRAY[*]} Expands to the number of elements in ARRAY ${!ARRAY[@]} \\ ${!ARRAY[*]} Expands to the indexes in ARRAY since BASH 3.0"},{"location":"syntax/arrays/#destruction","title":"Destruction","text":"The unset builtin command is used to destroy (unset) arrays or individual elements of arrays.
Syntax Descriptionunset -v ARRAY \\ unset -v ARRAY[@] \\ unset -v ARRAY[*] Destroys a complete array unset -v ARRAY[N] Destroys the array element at index N unset -v ARRAY[STRING] Destroys the array element of the associative array at index STRING It is best to explicitly specify -v when unsetting variables with unset.
warning
Specifying unquoted array elements as arguments to any command, such as with the syntax above may cause pathname expansion to occur due to the presence of glob characters.
Example: You are in a directory with a file named x1, and you want to destroy an array element x[1], with
unset x[1]\nthen pathname expansion will expand to the filename x1 and break your processing!
Even worse, if nullglob is set, your array/index will disappear.
To avoid this, always quote the array name and index:
unset -v 'x[1]'\nThis applies generally to all commands which take variable names as arguments. Single quotes preferred.
"},{"location":"syntax/arrays/#usage","title":"Usage","text":""},{"location":"syntax/arrays/#numerical-index","title":"Numerical Index","text":"Numerical indexed arrays are easy to understand and easy to use. The Purpose and Indexing chapters above more or less explain all the needed background theory.
Now, some examples and comments for you.
Let's say we have an array sentence which is initialized as follows:
sentence=(Be liberal in what you accept, and conservative in what you send)\nSince no special code is there to prevent word splitting (no quotes), every word there will be assigned to an individual array element. When you count the words you see, you should get 12. Now let's see if Bash has the same opinion:
$ echo ${#sentence[@]}\n12\nYes, 12. Fine. You can take this number to walk through the array. Just subtract 1 from the number of elements, and start your walk at 0 (zero):
((n_elements=${#sentence[@]}, max_index=n_elements - 1))\n\nfor ((i = 0; i <= max_index; i++)); do\n echo \"Element $i: '${sentence[i]}'\"\ndone\nYou always have to remember that, it seems newbies have problems sometimes. Please understand that numerical array indexing begins at 0 (zero)!
The method above, walking through an array by just knowing its number of elements, only works for arrays where all elements are set, of course. If one element in the middle is removed, then the calculation is nonsense, because the number of elements doesn't correspond to the highest used index anymore (we call them \"sparse arrays\").
Now, suppose that you want to replace your array sentence with the values in the previously-declared array NAMES . You might think you could just do
$ unset sentence ; declare -a sentence=NAMES\n$ echo ${#sentence[@]}\n1\n# omit calculating max_index as above, and iterate as one-liner\n$ for ((i = 0; i < ${#sentence[@]}; i++)); do echo \"Element $i: '${sentence[i]}'\" ; done\nElement 0: 'NAMES'\nObviously that's wrong. What about
$ unset sentence ; declare -a sentence=${NAMES}\n? Again, wrong:
$ echo ${#sentence[*]}\n1\n$ for ((i = 0; i < ${#sentence[@]}; i++)); do echo \"Element $i: '${sentence[i]}'\" ; done\nElement 0: 'Peter'\nSo what's the right way? The (slightly ugly) answer is, reuse the enumeration syntax:
$ unset sentence ; declare -a sentence=(\"${NAMES[@]}\")\n$ echo ${#sentence[@]}\n4\n$ for ((i = 0; i < ${#sentence[@]}; i++)); do echo \"Element $i: '${sentence[i]}'\" ; done\nElement 0: 'Peter'\nElement 1: 'Anna'\nElement 2: 'Greg'\nElement 3: 'Jan'\nAssociative arrays (or hash tables) are not much more complicated than numerical indexed arrays. The numerical index value (in Bash a number starting at zero) just is replaced with an arbitrary string:
# declare -A, introduced with Bash 4 to declare an associative array\ndeclare -A sentence\n\nsentence[Begin]='Be liberal in what'\nsentence[Middle]='you accept, and conservative'\nsentence[End]='in what you send'\nsentence['Very end']=...\nBeware: don't rely on the fact that the elements are ordered in memory like they were declared, it could look like this:
# output from 'set' command\nsentence=([End]=\"in what you send\" [Middle]=\"you accept, and conservative \" [Begin]=\"Be liberal in what \" [\"Very end\"]=\"...\")\nThis effectively means, you can get the data back with \"${sentence[@]}\", of course (just like with numerical indexing), but you can't rely on a specific order. If you want to store ordered data, or re-order data, go with numerical indexes. For associative arrays, you usually query known index values:
for element in Begin Middle End \"Very end\"; do\n printf \"%s\" \"${sentence[$element]}\"\ndone\nprintf \"\\n\"\nA nice code example: Checking for duplicate files using an associative array indexed with the SHA sum of the files:
# Thanks to Tramp in #bash for the idea and the code\n\nunset flist; declare -A flist;\nwhile read -r sum fname; do\n if [[ ${flist[$sum]} ]]; then\n printf 'rm -- \"%s\" # Same as >%s<\\n' \"$fname\" \"${flist[$sum]}\"\n else\n flist[$sum]=\"$fname\"\n fi\ndone < <(find . -type f -exec sha256sum {} +) >rmdups\nAny type attributes applied to an array apply to all elements of the array. If the integer attribute is set for either indexed or associative arrays, then values are considered as arithmetic for both compound and ordinary assignment, and the += operator is modified in the same way as for ordinary integer variables.
~ $ ( declare -ia 'a=(2+4 [2]=2+2 [a[2]]=\"a[2]\")' 'a+=(42 [a[4]]+=3)'; declare -p a )\ndeclare -ai a='([0]=\"6\" [2]=\"4\" [4]=\"7\" [5]=\"42\")'\na[0] is assigned to the result of 2+4. a[2] gets the result of 2+2. The last index in the first assignment is the result of a[2], which has already been assigned as 4, and its value is also given a[2].
This shows that even though any existing arrays named a in the current scope have already been unset by using = instead of += to the compound assignment, arithmetic variables within keys can self-reference any elements already assigned within the same compound-assignment. With integer arrays this also applies to expressions to the right of the =. (See evaluation order, the right side of an arithmetic assignment is typically evaluated first in Bash.)
The second compound assignment argument to declare uses +=, so it appends after the last element of the existing array rather than deleting it and creating a new array, so a[5] gets 42.
Lastly, the element whose index is the value of a[4] (4), gets 3 added to its existing value, making a[4] == 7. Note that having the integer attribute set this time causes += to add, rather than append a string, as it would for a non-integer array.
The single quotes force the assignments to be evaluated in the environment of declare. This is important because attributes are only applied to the assignment after assignment arguments are processed. Without them the += compound assignment would have been invalid, and strings would have been inserted into the integer array without evaluating the arithmetic. A special-case of this is shown in the next section.
info
Bash declaration commands are really keywords in disguise. They magically parse arguments to determine whether they are in the form of a valid assignment. If so, they are evaluated as assignments. If not, they are undergo normal argument expansion before being passed to the builtin which evaluates the resulting string as an assignment (somewhat like eval, but there are differences.) 'Todo:\\' Discuss this in detail.
Arrays can be expanded indirectly using the indirect parameter expansion syntax. Parameters whose values are of the form: name[index], name[@], or name[*] when expanded indirectly produce the expected results. This is mainly useful for passing arrays (especially multiple arrays) by name to a function.
This example is an \"isSubset\"-like predicate which returns true if all key-value pairs of the array given as the first argument to isSubset correspond to a key-value of the array given as the second argument. It demonstrates both indirect array expansion and indirect key-passing without eval using the aforementioned special compound assignment expansion.
isSubset() {\n local -a 'xkeys=(\"${!'\"$1\"'[@]}\")' 'ykeys=(\"${!'\"$2\"'[@]}\")'\n set -- \"${@/%/[key]}\"\n\n (( ${#xkeys[@]} <= ${#ykeys[@]} )) || return 1\n\n local key\n for key in \"${xkeys[@]}\"; do\n [[ ${!2+_} && ${!1} == ${!2} ]] || return 1\n done\n}\n\nmain() {\n # \"a\" is a subset of \"b\"\n local -a 'a=({0..5})' 'b=({0..10})'\n isSubset a b\n echo $? # true\n\n # \"a\" contains a key not in \"b\"\n local -a 'a=([5]=5 {6..11})' 'b=({0..10})'\n isSubset a b\n echo $? # false\n\n # \"a\" contains an element whose value != the corresponding member of \"b\"\n local -a 'a=([5]=5 6 8 9 10)' 'b=({0..10})'\n isSubset a b\n echo $? # false\n}\n\nmain\nThis script is one way of implementing a crude multidimensional associative array by storing array definitions in an array and referencing them through indirection. The script takes two keys and dynamically calls a function whose name is resolved from the array.
callFuncs() {\n # Set up indirect references as positional parameters to minimize local name collisions.\n set -- \"${@:1:3}\" ${2+'a[\"$1\"]' \"$1\"'[\"$2\"]'}\n\n # The only way to test for set but null parameters is unfortunately to test each individually.\n local x\n for x; do\n [[ $x ]] || return 0\n done\n\n local -A a=(\n [foo]='([r]=f [s]=g [t]=h)'\n [bar]='([u]=i [v]=j [w]=k)'\n [baz]='([x]=l [y]=m [z]=n)'\n ) ${4+${a[\"$1\"]+\"${1}=${!3}\"}} # For example, if \"$1\" is \"bar\" then define a new array: bar=([u]=i [v]=j [w]=k)\n\n ${4+${a[\"$1\"]+\"${!4-:}\"}} # Now just lookup the new array. for inputs: \"bar\" \"v\", the function named \"j\" will be called, which prints \"j\" to stdout.\n}\n\nmain() {\n # Define functions named {f..n} which just print their own names.\n local fun='() { echo \"$FUNCNAME\"; }' x\n\n for x in {f..n}; do\n eval \"${x}${fun}\"\n done\n\n callFuncs \"$@\"\n}\n\nmain \"$@\"\ntypeset -A in Bash 4, Zsh, and Ksh93.= compound assignment operator unsets the array, including any attributes that have been set on the array prior to assignment. In order to preserve attributes, you must use the += operator. However, declaring an associative array, then attempting an a=(...) style compound assignment without specifying indexes is an error. I can't explain this inconsistency.$ ksh -c 'function f { typeset -a a; a=([0]=foo [1]=bar); typeset -p a; }; f' # Attribute is lost, and since subscripts are given, we default to associative. typeset -A a=([0]=foo [1]=bar) $ ksh -c 'function f { typeset -a a; a+=([0]=foo [1]=bar); typeset -p a; }; f' # Now using += gives us the expected results. typeset -a a=(foo bar) $ ksh -c 'function f { typeset -A a; a=(foo bar); typeset -p a; }; f' # On top of that, the reverse does NOT unset the attribute. No idea why. ksh: f: line 1: cannot append index array to associative array a$ ksh -c 'function f { typeset -a a; a+=(foo bar baz); a+=([3]=blah [0]=bork [1]=blarg [2]=zooj); typeset -p a; }; f' # ksh93 forces appending to the array, disregarding subscripts typeset -a a=(foo bar baz '[3]=blah' '[0]=bork' '[1]=blarg' '[2]=zooj') $ bash -c 'function f { typeset -a a; a+=(foo bar baz); a+=(blah [0]=bork blarg zooj); typeset -p a; }; f' # Bash applies += to every individual subscript. declare -a a='([0]=\"foobork\" [1]=\"barblarg\" [2]=\"bazzooj\" [3]=\"blah\")' $ mksh -c 'function f { typeset -a a; a+=(foo bar baz); a+=(blah [0]=bork blarg zooj); typeset -p a; }; f' # Mksh does like Bash, but clobbers previous values rather than appending. set -A a typeset a[0]=bork typeset a[1]=blarg typeset a[2]=zooj typeset a[3]=blah${arr[idx]:=foo}) evaluates the subscript twice, first to determine whether to expand the alternate, and second to determine the index to assign the alternate to. See evaluation order. $ : ${_[$(echo $RANDOM >&2)1]:=$(echo hi >&2)} 13574 hi 14485typeset.set -A name arg1 arg2 ... syntax. This is supported by almost all shells that support ksh-like arrays except for Bash. Additionally, these shells usually support an optional -s argument to set which performs lexicographic sorting on either array elements or the positional parameters. Bash has no built-in sorting ability other than the usual comparison operators. $ ksh -c 'set -A arr -- foo bar bork baz; typeset -p arr' # Classic array assignment syntax typeset -a arr=(foo bar bork baz) $ ksh -c 'set -sA arr -- foo bar bork baz; typeset -p arr' # Native sorting! typeset -a arr=(bar baz bork foo) $ mksh -c 'set -sA arr -- foo \"[3]=bar\" \"[2]=baz\" \"[7]=bork\"; typeset -p arr' # Probably a bug. I think the maintainer is aware of it. set -A arr typeset arr[2]=baz typeset arr[3]=bar typeset arr[7]=bork typeset arr[8]=foo\"${arr[@]:(-n):1}\", to expand the nth-last element (or the next-highest indexed after n if arr[n] is unset). In Bash 4.2, you may expand (but not assign to) a negative index. In Bash 4.3, ksh93, and zsh, you may both assign and expand negative offsets.\"${arr[n..m]}\" where n and m are arithmetic expressions. These are needed for use with multi-dimensional arrays.UINT_MAX, which would be addressed by arr[-1].-v var test doesn't support individual array subscripts. You may supply an array name to test whether an array is defined, but can't check an element. ksh93's -v supports both. Other shells lack a -v test.[...]= will be clumped with the subscript and counted as a glob. Therefore, you must quote anything on the right of the = sign. This is fixed in 4.3, so that each subscript assignment statement is expanded following the same rules as an ordinary assignment. This also works correctly in ksh93. $ touch '[1]=a'; bash -c 'a=([1]=*); echo \"${a[@]}\"' [1]=a mksh has a similar but even worse problem in that the entire subscript is considered a glob. $ touch 1=a; mksh -c 'a=([123]=*); print -r -- \"${a[@]}\"' 1=aFixed in 4.3 In addition to the above globbing issue, assignments preceding \"declare\" have an additional effect on brace and pathname expansion. `$ set -x; foo=bar declare arr=( {1..10} )
$ touch xy=foo + touch xy=foo $ declare x[y]= + declare 'x[y]=' $ foo=bar declare x[y]=* + foo=bar + declare xy=foo Each word (the entire assignment) is subject to globbing and brace expansion. This appears to trigger the same strange expansion mode aslet,eval, other declaration commands, and maybe more. - **Fixed in 4.3** Indirection combined with another modifier expands arrays to a single word.$ a=({a..c}) b=a[@]; printf '<%s> ' \"${!b}\"; echo; printf '<%s> ' \"${!b/%/foo}\"; echo - **Fixed in 4.3** Process substitutions are evaluated within array indexes. Zsh and ksh don't do this in any arithmetic context.# print \"moo\" dev=fd=1 _[1<(echo moo >&2)]="},{"location":"syntax/arrays/#fork-bomb","title":"Fork bomb","text":"
${dev[${dev='dev[1>(${dev[dev]})]'}]} `
"},{"location":"syntax/arrays/#evaluation-order","title":"Evaluation order","text":"Here are some of the nasty details of array assignment evaluation order. You can use this testcase code to generate these results.
Each testcase prints evaluation order for indexed array assignment\ncontexts. Each context is tested for expansions (represented by digits) and\narithmetic (letters), ordered from left to right within the expression. The\noutput corresponds to the way evaluation is re-ordered for each shell:\n\na[ $1 a ]=${b[ $2 b ]:=${c[ $3 c ]}} No attributes\na[ $1 a ]=${b[ $2 b ]:=c[ $3 c ]} typeset -ia a\na[ $1 a ]=${b[ $2 b ]:=c[ $3 c ]} typeset -ia b\na[ $1 a ]=${b[ $2 b ]:=c[ $3 c ]} typeset -ia a b\n(( a[ $1 a ] = b[ $2 b ] ${c[ $3 c ]} )) No attributes\n(( a[ $1 a ] = ${b[ $2 b ]:=c[ $3 c ]} )) typeset -ia b\na+=( [ $1 a ]=${b[ $2 b ]:=${c[ $3 c ]}} [ $4 d ]=$(( $5 e )) ) typeset -a a\na+=( [ $1 a ]=${b[ $2 b ]:=c[ $3 c ]} [ $4 d ]=${5}e ) typeset -ia a\n\nbash: 4.2.42(1)-release\n2 b 3 c 2 b 1 a\n2 b 3 2 b 1 a c\n2 b 3 2 b c 1 a\n2 b 3 2 b c 1 a c\n1 2 3 c b a\n1 2 b 3 2 b c c a\n1 2 b 3 c 2 b 4 5 e a d\n1 2 b 3 2 b 4 5 a c d e\n\nksh93: Version AJM 93v- 2013-02-22\n1 2 b b a\n1 2 b b a\n1 2 b b a\n1 2 b b a\n1 2 3 c b a\n1 2 b b a\n1 2 b b a 4 5 e d\n1 2 b b a 4 5 d e\n\nmksh: @(#)MIRBSD KSH R44 2013/02/24\n2 b 3 c 1 a\n2 b 3 1 a c\n2 b 3 c 1 a\n2 b 3 c 1 a\n1 2 3 c a b\n1 2 b 3 c a\n1 2 b 3 c 4 5 e a d\n1 2 b 3 4 5 a c d e\n\nzsh: 5.0.2\n2 b 3 c 2 b 1 a\n2 b 3 2 b 1 a c\n2 b 1 a\n2 b 1 a\n1 2 3 c b a\n1 2 b a\n1 2 b 3 c 2 b 4 5 e\n1 2 b 3 2 b 4 5\nBash builds its features on top of a few basic grammar rules. The code you see everywhere, the code you use, is based on those rules. However, this is a very theoretical view, but if you're interested, it may help you understand why things look the way they look.
If you don't know the commands used in the following examples, just trust the explanation.
","tags":["bash","shell","scripting","grammar","syntax","language"]},{"location":"syntax/basicgrammar/#simple-commands","title":"Simple Commands","text":"Bash manual says:
A simple command is a sequence of optional variable assignments followed by blank-separated words and redirections,\nand terminated by a control operator. The first word specifies the command to be executed, and is passed as argument\nzero. The remaining words are passed as arguments to the invoked command.\nSounds harder than it actually is. It is what you do daily. You enter simple commands with parameters, and the shell executes them.
Every complex Bash operation can be split into simple commands:
ls\nls > list.txt\nls -l\nLC_ALL=C ls\nThe last one might not be familiar. That one simply adds \"LC_ALL=C\" to the environment of the ls program. It doesn't affect your current shell. This also works while calling functions, unless Bash runs in POSIX\u00ae mode (in which case it affects your current shell).
Every command has an exit code. It's a type of return status. The shell can catch it and act on it. Exit code range is from 0 to 255, where 0 means success, and the rest mean either something failed, or there is an issue to report back to the calling program.
info
The simple command construct is the base for all higher constructs. Everything you execute, from pipelines to functions, finally ends up in (many) simple commands. That's why Bash only has one method to expand and execute a simple command.
","tags":["bash","shell","scripting","grammar","syntax","language"]},{"location":"syntax/basicgrammar/#pipelines","title":"Pipelines","text":"FIXME
Missing an additional article about pipelines and pipelining
[time [-p]] [ ! ] command [ | command2 ... ]
Don't get confused about the name \"pipeline.\" It's a grammatic name for a construct. Such a pipeline isn't necessarily a pair of commands where stdout/stdin is connected via a real pipe.
Pipelines are one or more simple commands (separated by the | symbol connects their input and output), for example:
ls /etc | wc -l\nwill execute ls on /etc and pipe the output to wc, which will count the lines generated by the ls command. The result is the number of directory entries in /etc.
The last command in the pipeline will set the exit code for the pipeline. This exit code can be \"inverted\" by prefixing an exclamation mark to the pipeline: An unsuccessful pipeline will exit \"successful\" and vice versa. In this example, the commands in the if stanza will be executed if the pattern \"^root:\" is not found in /etc/passwd:
if ! grep '^root:' /etc/passwd; then\n echo \"No root user defined... eh?\"\nfi\nYes, this is also a pipeline (although there is no pipe!), because the exclamation mark to invert the exit code can only be used in a pipeline. If grep's exit code is 1 (FALSE) (the text was not found), the leading ! will \"invert\" the exit code, and the shell sees (and acts on) exit code 0 (TRUE) and the then part of the if stanza is executed. One could say we checked for \"not grep \"^root\" /etc/passwd\".
The set option pipefail determines the behavior of how bash reports the exit code of a pipeline. If it's set, then the exit code ($?) is the last command that exits with non zero status, if none fail, it's zero. If it's not set, then $? always holds the exit code of the last command (as explained above).
The shell option lastpipe will execute the last element in a pipeline construct in the current shell environment, i.e. not a subshell.
There's also an array PIPESTATUS[] that is set after a foreground pipeline is executed. Each element of PIPESTATUS[] reports the exit code of the respective command in the pipeline. Note: (1) it's only for foreground pipe and (2) for higher level structure that is built up from a pipeline. Like list, PIPESTATUS[] holds the exit status of the last pipeline command executed.
Another thing you can do with pipelines is log their execution time. Note that time is not a command, it is part of the pipeline syntax:
# time updatedb\nreal 3m21.288s\nuser 0m3.114s\nsys 0m4.744s\nFIXME
Missing an additional article about list operators
A list is a sequence of one or more pipelines separated by one of the operators ;, &, &&, or \u2502\u2502, and optionally terminated by one of ;, &, or <newline>.
=> It's a group of pipelines separated or terminated by tokens that all have different meanings for Bash.
Your whole Bash script technically is one big single list!
Operator Description<PIPELINE1> <newline> <PIPELINE2> Newlines completely separate pipelines. The next pipeline is executed without any checks. (You enter a command and press <RETURN>!) <PIPELINE1> ; <PIPELINE2> The semicolon does what <newline> does: It separates the pipelines <PIPELINE> & <PIPELINE> The pipeline in front of the & is executed asynchronously (\"in the background\"). If a pipeline follows this, it is executed immediately after the async pipeline starts <PIPELINE1> && <PIPELINE2> <PIPELINE1> is executed and only if its exit code was 0 (TRUE), then <PIPELINE2> is executed (AND-List) <PIPELINE1>||<PIPELINE2> <PIPELINE1> is executed and only if its exit code was not 0 (FALSE), then <PIPELINE2> is executed (OR-List) Note: POSIX calls this construct a \"compound lists\".
","tags":["bash","shell","scripting","grammar","syntax","language"]},{"location":"syntax/basicgrammar/#compound-commands","title":"Compound Commands","text":"See also the list of compound commands.
There are two forms of compound commands:
Essentially, everything else that's not described in this article. Compound commands have the following characteristics:
for ... done)See the following table for a short overview (no details - just an overview):
Compound command syntax Description( <LIST> ) Execute <LIST> in an extra subshell => article { <LIST> ; } Execute <LIST> as separate group (but not in a subshell) => article (( <EXPRESSION> )) Evaluate the arithmetic expression <EXPRESSION> => article [[ <EXPRESSION> ]] Evaluate the conditional expression <EXPRESSION> (aka \"the new test command\") => article for <NAME> in <WORDS> ; do <LIST> ; done Executes <LIST> while setting the variable <NAME> to one of <WORDS> on every iteration (classic for-loop) => article for (( <EXPR1> ; <EXPR2> ; <EXPR3> )) ; do <LIST> ; done C-style for-loop (driven by arithmetic expressions) => article select <NAME> in <WORDS> ; do <LIST> ; done Provides simple menus => article case <WORD> in <PATTERN>) <LIST> ;; ... esac Decisions based on pattern matching - executing <LIST> on match => article if <LIST> ; then <LIST> ; else <LIST> ; fi The if clause: makes decisions based on exit codes => article while <LIST1> ; do <LIST2> ; done Execute <LIST2> while <LIST1> returns TRUE (exit code) => article until <LIST1> ; do <LIST2> ; done Execute <LIST2> until <LIST1> returns TRUE (exit code) => article","tags":["bash","shell","scripting","grammar","syntax","language"]},{"location":"syntax/basicgrammar/#shell-function-definitions","title":"Shell Function Definitions","text":"FIXME
Missing an additional article about shell functions
A shell function definition makes a compound command available via a new name. When the function runs, it has its own \"private\" set of positional parameters and I/O descriptors. It acts like a script-within-the-script. Simply stated: You've created a new command.
The definition is easy (one of many possibilities):
<NAME> () <COMPOUND_COMMAND> <REDIRECTIONS>
which is usually used with the {...; } compound command, and thus looks like:
print_help() { echo \"Sorry, no help available\"; }\nAs above, a function definition can have any compound command as a body. Structures like
countme() for ((x=1;x<=9;x++)); do echo $x; done\nare unusual, but perfectly valid, since the for loop construct is a compound command!
If redirection is specified, the redirection is not performed when the function is defined. It is performed when the function runs:
# this will NOT perform the redirection (at definition time)\nf() { echo ok ; } > file\n\n# NOW the redirection will be performed (during EXECUTION of the function)\nf\nBash allows three equivalent forms of the function definition:
NAME () <COMPOUND_COMMAND> <REDIRECTIONS>\nfunction NAME () <COMPOUND_COMMAND> <REDIRECTIONS>\nfunction NAME <COMPOUND_COMMAND> <REDIRECTIONS>\nThe space between NAME and () is optional, usually you see it without the space.
I suggest using the first form. It's specified in POSIX and all Bourne-like shells seem to support it.
Note: Before version 2.05-alpha1, Bash only recognized the definition using curly braces (name() { ... }), other shells allow the definition using any command (not just the compound command set).
To execute a function like a regular shell script you put it together like this:
#!/bin/bash\n# Add shebang\n\nmycmd()\n{\n # this $1 belongs to the function!\n find / -iname \"$1\"\n}\n\n# this $1 belongs the script itself!\nmycmd \"$1\" # Execute command immediately after defining function\n\nexit 0\nJust informational(1):
Internally, for forking, Bash stores function definitions in environment variables. Variables with the content \"() ....\".
Something similar to the following works without \"officially\" declaring a function:
$ export testfn=\"() { echo test; }\"\n$ bash -c testfn\ntest\n$\nJust informational(2):
It is possible to create function names containing slashes:
/bin/ls() {\n echo LS FAKE\n}\nThe elements of this name aren't subject to a path search.
Weird function names should not be used. Quote from the maintainer:
unset doesn't work to unset them without forcing -f, for instance). We're stuck with them for backwards compatibility, but I don't have to encourage their use. *FIXME
more...
A (very) simple command
echo \"Hello world...\"\nAll of the following are simple commands
x=5\n\n>tmpfile\n\n{x}<\"$x\" _=${x=<(echo moo)} <&0$(cat <&\"$x\" >&2)\nA common compound command
if [ -d /data/mp3 ]; then\n cp mymusic.mp3 /data/mp3\nfi\nif clauseif checks actually contains the simple command [ -d /data/mp3 ]if executes contains a simple command (cp mymusic.mp3 /data/mp3)Let's invert test command exit code, only one thing changes:
if ! [ -d /data/mp3 ]; then\n cp mymusic.mp3 /data/mp3\nfi\nif checks contains a pipeline now (because of the !)A pattern is a string description. Bash uses them in various ways:
The pattern description language is relatively easy. Any character that's not mentioned below matches itself. The NUL character may not occur in a pattern. If special characters are quoted, they're matched literally, i.e., without their special meaning.
Do not confuse patterns with regular expressions, because they share some symbols and do similar matching work.
","tags":["bash","shell","scripting","glob","globbing","wildcards","filename","pattern","matching"]},{"location":"syntax/pattern/#normal-pattern-language","title":"Normal pattern language","text":"Sequence Description* Matches any string, including the null string (empty string) ? Matches any single character X Matches the character X which can be any character that has no special meaning \\X Matches the character X, where the character's special meaning is stripped by the backslash \\\\ Matches a backslash [...] Defines a pattern bracket expression (see below). Matches any of the enclosed characters at this position.","tags":["bash","shell","scripting","glob","globbing","wildcards","filename","pattern","matching"]},{"location":"syntax/pattern/#bracket-expressions","title":"Bracket expressions","text":"The bracket expression [...] mentioned above has some useful applications:
[XYZ] The \"normal\" bracket expression, matching either X, Y or Z [X-Z] A range expression: Matching all the characters from X to Y (your current locale, defines how the characters are sorted!) [[:class:]] Matches all the characters defined by a POSIX\u00ae character class: alnum, alpha, ascii, blank, cntrl, digit, graph, lower, print, punct, space, upper, word and xdigit [^...] A negating expression: It matches all the characters that are not in the bracket expression [!...] Equivalent to [^...] []...] or [-...] Used to include the characters ] and - into the set, they need to be the first characters after the opening bracket [=C=] Matches any character that is eqivalent to the collation weight of C (current locale!) [[.SYMBOL.]] Matches the collating symbol SYMBOL","tags":["bash","shell","scripting","glob","globbing","wildcards","filename","pattern","matching"]},{"location":"syntax/pattern/#examples","title":"Examples","text":"Some simple examples using normal pattern matching:
\"Hello world\" matchesHello world[Hh]\"ello world\" matchesHello worldhello worldHello* matches (for example)Hello worldHelloworldHelloWoRlDHelloHello world[[:punct:]] matches (for example)Hello world!Hello world.Hello world+Hello world?[[.backslash.]]Hello[[.vertical-line.]]world[[.exclamation-mark.]] matches (using collation symbols)\\Hello|world!If you set the shell option extglob, Bash understands some powerful patterns. A <PATTERN-LIST> is one or more patterns, separated by the pipe-symbol (PATTERN|PATTERN).
?(<PATTERN-LIST>) Matches zero or one occurrence of the given patterns *(<PATTERN-LIST>) Matches zero or more occurrences of the given patterns +(<PATTERN-LIST>) Matches one or more occurrences of the given patterns @(<PATTERN-LIST>) Matches one of the given patterns !(<PATTERN-LIST>) Matches anything except one of the given patterns","tags":["bash","shell","scripting","glob","globbing","wildcards","filename","pattern","matching"]},{"location":"syntax/pattern/#examples_1","title":"Examples","text":"Delete all but one specific file
rm -f !(survivior.txt)\ndotglob globbing see Pathname expansion customization extglob global enable/disable extended pattern matching language, as described above failglob globbing see Pathname expansion customization nocaseglob globbing see Pathname expansion customization nocasematch pattern/string matching perform pattern matching without regarding the case of individual letters nullglob globbing see Pathname expansion customization globasciiranges globbing see Pathname expansion customization","tags":["bash","shell","scripting","glob","globbing","wildcards","filename","pattern","matching"]},{"location":"syntax/pattern/#bugs-and-portability-considerations","title":"Bugs and Portability considerations","text":"* Counter-intuitively, only the [!chars] syntax for negating a character class is specified by POSIX for shell pattern matching. [^chars] is merely a commonly-supported extension. Even dash supports [^chars], but not posh.
* All of the extglob quantifiers supported by bash were supported by ksh88. The set of extglob quantifiers supported by ksh88 are identical to those supported by Bash, mksh, ksh93, and zsh.
* mksh does not support POSIX character classes. Therefore, character ranges like [0-9] are somewhat more portable than an equivalent POSIX class like [:digit:].
* Bash uses a custom runtime interpreter for pattern matching. (at least) ksh93 and zsh translate patterns into regexes and then use a regex compiler to emit and cache optimized pattern matching code. This means Bash may be an order of magnitude or more slower in cases that involve complex back-tracking (usually that means extglob quantifier nesting). You may wish to use Bash's regex support (the =~ operator) if performance is a problem, because Bash will use your C library regex implementation rather than its own pattern matcher.
TODO: describe the pattern escape bug https://gist.github.com/ormaaj/6195070
","tags":["bash","shell","scripting","glob","globbing","wildcards","filename","pattern","matching"]},{"location":"syntax/pattern/#ksh93-extras","title":"ksh93 extras","text":"ksh93 supports some very powerful pattern matching features in addition to those described above.
* ksh93 supports arbitrary quantifiers just like ERE using the {from,to}(pattern-list) syntax. {2,4}(foo)bar matches between 2-4 \"foo\"'s followed by \"bar\". {2,}(foo)bar matches 2 or more \"foo\"'s followed by \"bar\". You can probably figure out the rest. So far, none of the other shells support this syntax.
* In ksh93, a pattern-list may be delimited by either & or |. & means \"all patterns must be matched\" instead of \"any pattern\". For example, [[ fo0bar == @(fo[0-9]&+([[:alnum:]]))bar ]] would be true while [[ f00bar == @(fo[0-9]&+([[:alnum:]]))bar ]] is false, because all members of the and-list must be satisfied. No other shell supports this so far, but you can simulate some cases in other shells using double extglob negation. The aforementioned ksh93 pattern is equivalent in Bash to: [[ fo0bar == !(!(fo[0-9])|!(+([[:alnum:]])))bar ]], which is technically more portable, but ugly.
* ksh93's printf builtin can translate from shell patterns to ERE and back again using the %R and %P format specifiers respectively.
TODO: ~() (and regex), .sh.match, backrefs, special ${var/.../...} behavior, %()
One core functionality of Bash is to manage parameters. A parameter is an entity that stores values and is referenced by a name, a number or a special symbol.
Parameter expansion is the procedure to get the value from the referenced entity, like expanding a variable to print its value. On expansion time you can do very nasty things with the parameter or its value. These things are described here.
If you saw some parameter expansion syntax somewhere, and need to check what it can be, try the overview section below!
Arrays can be special cases for parameter expansion, every applicable description mentions arrays below. Please also see the article about arrays.
For a more technical view what a parameter is and which types exist, see the dictionary entry for \"parameter\".
","tags":["bash","shell","scripting","expansion","substitution","text","variable","parameter","mangle","substitute","change","check","defined","null","array","arrays"]},{"location":"syntax/pe/#overview","title":"Overview","text":"Looking for a specific syntax you saw, without knowing the name?
$PARAMETER${PARAMETER}${!PARAMETER}${PARAMETER^}${PARAMETER^^}${PARAMETER,}${PARAMETER,,}${PARAMETER~}${PARAMETER~~}${!PREFIX*}${!PREFIX@}${PARAMETER#PATTERN}${PARAMETER##PATTERN}${PARAMETER%PATTERN}${PARAMETER%%PATTERN}${PARAMETER/PATTERN/STRING}${PARAMETER//PATTERN/STRING}${PARAMETER/PATTERN}${PARAMETER//PATTERN}${#PARAMETER}${PARAMETER:OFFSET}${PARAMETER:OFFSET:LENGTH}${PARAMETER:-WORD}${PARAMETER-WORD}${PARAMETER:=WORD}${PARAMETER=WORD}${PARAMETER:+WORD}${PARAMETER+WORD}${PARAMETER:?WORD}${PARAMETER?WORD}$PARAMETER
${PARAMETER}
The easiest form is to just use a parameter's name within braces. This is identical to using $FOO like you see it everywhere, but has the advantage that it can be immediately followed by characters that would be interpreted as part of the parameter name otherwise. Compare these two expressions (WORD=\"car\" for example), where we want to print a word with a trailing \"s\":
echo \"The plural of $WORD is most likely $WORDs\"\necho \"The plural of $WORD is most likely ${WORD}s\"\nWhy does the first one fail? It prints nothing, because a parameter (variable) named \"WORDs\" is undefined and thus printed as \"\" (nothing). Without using braces for parameter expansion, Bash will interpret the sequence of all valid characters from the introducing \"$\" up to the last valid character as name of the parameter. When using braces you just force Bash to only interpret the name inside your braces.
Also, please remember, that parameter names are (like nearly everything in UNIX\u00ae) case sensitive!
The second form with the curly braces is also needed to access positional parameters (arguments to a script) beyond $9:
echo \"Argument 1 is: $1\"\necho \"Argument 10 is: ${10}\"\nSee also the article about general array syntax
For arrays you always need the braces. The arrays are expanded by individual indexes or mass arguments. An individual index behaves like a normal parameter, for the mass expansion, please read the article about arrays linked above.
${array[5]}${array[*]}${array[@]}${!PARAMETER}
In some cases, like for example
${PARAMETER}\n\n${PARAMETER:0:3}\nyou can instead use the form
${!PARAMETER}\nto enter a level of indirection. The referenced parameter is not PARAMETER itself, but the parameter whose name is stored as the value of PARAMETER. If the parameter PARAMETER has the value \"TEMP\", then ${!PARAMETER} will expand to the value of the parameter named TEMP:
read -rep 'Which variable do you want to inspect? ' look_var\n\nprintf 'The value of \"%s\" is: \"%s\"\\n' \"$look_var\" \"${!look_var}\"\nOf course the indirection also works with special variables:
# set some fake positional parameters\nset one two three four\n\n# get the LAST argument (\"#\" stores the number of arguments, so \"!#\" will reference the LAST argument)\necho ${!#}\nYou can think of this mechanism as being roughly equivalent to taking any parameter expansion that begins with the parameter name, and substituting the !PARAMETER part with the value of PARAMETER.
echo \"${!var^^}\"\n# ...is equivalent to\neval 'echo \"${'\"$var\"'^^}\"'\nIt was an unfortunate design decision to use the ! prefix for indirection, as it introduces parsing ambiguity with other parameter expansions that begin with !. Indirection is not possible in combination with any parameter expansion whose modifier requires a prefix to the parameter name. Specifically, indirection isn't possible on the ${!var@}, ${!var*}, ${!var[@]}, ${!var[*]}, and ${#var} forms. This means the ! prefix can't be used to retrieve the indices of an array, the length of a string, or number of elements in an array indirectly (see syntax/arrays#indirection for workarounds). Additionally, the !-prefixed parameter expansion conflicts with ksh-like shells which have the more powerful \"name-reference\" form of indirection, where the exact same syntax is used to expand to the name of the variable being referenced.
Indirect references to array names are also possible since the Bash 3 series (exact version unknown), but undocumented. See syntax/arrays#indirection for details.
Chet has added an initial implementation of the ksh nameref declaration command to the git devel branch. (declare -n, local -n, etc, will be supported). This will finally address many issues around passing and returning complex datatypes to/from functions.
${PARAMETER^}
${PARAMETER^^}
${PARAMETER,}
${PARAMETER,,}
${PARAMETER~}
${PARAMETER~~}
These expansion operators modify the case of the letters in the expanded text.
The ^ operator modifies the first character to uppercase, the , operator to lowercase. When using the double-form (^^ and ,,), all characters are converted.
Info
The (currently undocumented) operators ~ and ~~ reverse the case of the given text (in PARAMETER).~ reverses the case of first letter of words in the variable while ~~ reverses case for all. Thanks to Bushmills and geirha on the Freenode IRC channel for this finding.
Example: Rename all *.txt filenames to lowercase
for file in *.txt; do\n mv \"$file\" \"${file,,}\"\ndone\nNote: Case modification is a handy feature you can apply to a name or a title. Or is it? Case modification was an important aspect of the Bash 4 release. Bash version 4, RC1 would perform word splitting, and then case modification, resulting in title case (where every word is capitalized). It was decided to apply case modification to values, not words, for the Bash 4 release. Thanks Chet.
","tags":["bash","shell","scripting","expansion","substitution","text","variable","parameter","mangle","substitute","change","check","defined","null","array","arrays"]},{"location":"syntax/pe/#case-modification-arrays","title":"Case modification: Arrays","text":"Case modification can be used to create the proper capitalization for names or titles. Just assign it to an array:
declare -a title=(my hello world john smith)
For array expansion, the case modification applies to every expanded element, no matter if you expand an individual index or mass-expand the whole array using @ or * subscripts. Some examples:
Assume: array=(This is some Text)
echo \"${array[@],}\"this is some textecho \"${array[@],,}\"this is some textecho \"${array[@]^}\"This Is Some Textecho \"${array[@]^^}\"THIS IS SOME TEXTecho \"${array[2]^^}\"SOME${!PREFIX*}
${!PREFIX@}
This expands to a list of all set variable names beginning with the string PREFIX. The elements of the list are separated by the first character in the IFS-variable ( by default).
This will show all defined variable names (not values!) beginning with \"BASH\":
$ echo ${!BASH*}\nBASH BASH_ARGC BASH_ARGV BASH_COMMAND BASH_LINENO BASH_SOURCE BASH_SUBSHELL BASH_VERSINFO BASH_VERSION\nThis list will also include array names.
","tags":["bash","shell","scripting","expansion","substitution","text","variable","parameter","mangle","substitute","change","check","defined","null","array","arrays"]},{"location":"syntax/pe/#substring-removal","title":"Substring removal","text":"${PARAMETER#PATTERN}
${PARAMETER##PATTERN}
${PARAMETER%PATTERN}
${PARAMETER%%PATTERN}
This one can expand only a part of a parameter's value, given a pattern to describe what to remove from the string. The pattern is interpreted just like a pattern to describe a filename to match (globbing). See Pattern matching for more.
Example string (just a quote from a big man):
MYSTRING=\"Be liberal in what you accept, and conservative in what you send\"\n${PARAMETER#PATTERN} and ${PARAMETER##PATTERN}
This form is to remove the described pattern trying to match it from the beginning of the string. The operator \"#\" will try to remove the shortest text matching the pattern, while \"##\" tries to do it with the longest text matching. Look at the following examples to get the idea (matched text ~~marked striked~~, remember it will be removed!):
${MYSTRING#*in} ~~Be liberal in~~ what you accept, and conservative in what you send ${MYSTRING##*in} ~~Be liberal in what you accept, and conservative in~~ what you send","tags":["bash","shell","scripting","expansion","substitution","text","variable","parameter","mangle","substitute","change","check","defined","null","array","arrays"]},{"location":"syntax/pe/#from-the-end","title":"From the end","text":"${PARAMETER%PATTERN} and ${PARAMETER%%PATTERN}
In the second form everything will be the same, except that Bash now tries to match the pattern from the end of the string:
Syntax Result${MYSTRING%in*} Be liberal in what you accept, and conservative ~~in what you send~~ ${MYSTRING%%in*} Be liberal ~~in what you accept, and conservative in what you send~~ The second form nullifies variables that begin with in, by working from the end.
How the heck does that help to make my life easier?
Well, maybe the most common use for it is to extract parts of a filename. Just look at the following list with examples:
${FILENAME%.*}bash_hackers.txt${FILENAME##*.}bash_hackers.txt${PATHNAME%/*}/home/bash/bash_hackers.txt${PATHNAME##*/}/home/bash/bash_hackers.txtThese are the syntaxes for filenames with a single extension. Depending on your needs, you might need to adjust shortest/longest match.
","tags":["bash","shell","scripting","expansion","substitution","text","variable","parameter","mangle","substitute","change","check","defined","null","array","arrays"]},{"location":"syntax/pe/#substring-removal-arrays","title":"Substring removal: Arrays","text":"As for most parameter expansion features, working on arrays will handle each expanded element, for individual expansion and also for mass expansion.
Simple example, removing a trailing is from all array elements (on expansion):
Assume: array=(This is a text)
echo \"${array[@]%is}\"Th a textThis is a text)All other variants of this expansion behave the same.
","tags":["bash","shell","scripting","expansion","substitution","text","variable","parameter","mangle","substitute","change","check","defined","null","array","arrays"]},{"location":"syntax/pe/#search-and-replace","title":"Search and replace","text":"${PARAMETER/PATTERN/STRING}
${PARAMETER//PATTERN/STRING}
${PARAMETER/PATTERN}
${PARAMETER//PATTERN}
This one can substitute (replace) a substring matched by a pattern, on expansion time. The matched substring will be entirely removed and the given string will be inserted. Again some example string for the tests:
MYSTRING=\"Be liberal in what you accept, and conservative in what you send\"\nThe two main forms only differ in the number of slashes after the parameter name: ${PARAMETER/PATTERN/STRING} and ${PARAMETER//PATTERN/STRING}
The first one (one slash) is to only substitute the first occurrence of the given pattern, the second one (two slashes) is to substitute all occurrences of the pattern.
First, let's try to say \"happy\" instead of \"conservative\" in our example string:
${MYSTRING//conservative/happy}\n=> Be liberal in what you accept, and conservativehappy in what you send
Since there is only one \"conservative\" in that example, it really doesn't matter which of the two forms we use.
Let's play with the word \"in\", I don't know if it makes any sense, but let's substitute it with \"by\".
First form: Substitute first occurrence
${MYSTRING/in/by}\nBe liberal inby what you accept, and conservative by what you send
Second form: Substitute all occurrences
${MYSTRING//in/by}\n=> Be liberal inby what you accept, and conservative inby what you send
Anchoring Additionally you can \"anchor\" an expression: A # (hashmark) will indicate that your expression is matched against the beginning portion of the string, a % (percent-sign) will do it for the end portion.
MYSTRING=xxxxxxxxxx\necho ${MYSTRING/#x/y} # RESULT: yxxxxxxxxx\necho ${MYSTRING/%x/y} # RESULT: xxxxxxxxxy\nIf the replacement part is completely omitted, the matches are replaced by the nullstring, i.e., they are removed. This is equivalent to specifying an empty replacement:
echo ${MYSTRING//conservative/}\n# is equivalent to\necho ${MYSTRING//conservative}\nThis parameter expansion type applied to arrays applies to all expanded elements, no matter if an individual element is expanded, or all elements using the mass expansion syntaxes.
A simple example, changing the (lowercase) letter t to d:
Assume: array=(This is a text)
echo \"${array[@]/t/d}\"This is a dextecho \"${array[@]//t/d}\"This is a dexd${#PARAMETER}
When you use this form, the length of the parameter's value is expanded. Again, a quote from a big man, to have a test text:
MYSTRING=\"Be liberal in what you accept, and conservative in what you send\"\nUsing echo ${#MYSTRING}...
=> 64
The length is reported in characters, not in bytes. Depending on your environment this may not always be the same (multibyte-characters, like in UTF8 encoding).
There's not much to say about it, mh?
","tags":["bash","shell","scripting","expansion","substitution","text","variable","parameter","mangle","substitute","change","check","defined","null","array","arrays"]},{"location":"syntax/pe/#string-length-arrays","title":"(String) length: Arrays","text":"For arrays, this expansion type has two meanings:
@ and * it reports the number of set elements in the arrayExample:
Assume: array=(This is a text)
echo ${#array[1]}echo ${#array[@]}Attention: The number of used elements does not need to conform to the highest index. Sparse arrays are possible in Bash, that means you can have 4 elements, but with indexes 1, 7, 20, 31. You can't loop through such an array with a counter loop based on the number of elements!
","tags":["bash","shell","scripting","expansion","substitution","text","variable","parameter","mangle","substitute","change","check","defined","null","array","arrays"]},{"location":"syntax/pe/#substring-expansion","title":"Substring expansion","text":"${PARAMETER:OFFSET}
${PARAMETER:OFFSET:LENGTH}
This one can expand only a part of a parameter's value, given a position to start and maybe a length. If LENGTH is omitted, the parameter will be expanded up to the end of the string. If LENGTH is negative, it's taken as a second offset into the string, counting from the end of the string.
OFFSET and LENGTH can be any arithmetic expression. Take care: The OFFSET starts at 0, not at 1!
Example string (a quote from a big man): MYSTRING=\"Be liberal in what you accept, and conservative in what you send\"
In the first form, the expansion is used without a length value, note that the offset 0 is the first character:
echo ${MYSTRING:35}\n=> Be liberal in what you accept, and conservative in what you send
In the second form we also give a length value:
echo ${MYSTRING:35:12}\n=> Be liberal in what you accept, and conservative in what you send
If the given offset is negative, it's counted from the end of the string, i.e. an offset of -1 is the last character. In that case, the length still counts forward, of course. One special thing is to do when using a negative offset: You need to separate the (negative) number from the colon:
${MYSTRING: -10:5}\n${MYSTRING:(-10):5}\nWhy? Because it's interpreted as the parameter expansion syntax to use a default value.
","tags":["bash","shell","scripting","expansion","substitution","text","variable","parameter","mangle","substitute","change","check","defined","null","array","arrays"]},{"location":"syntax/pe/#negative-length-value","title":"Negative Length Value","text":"If the LENGTH value is negative, it's used as offset from the end of the string. The expansion happens from the first to the second offset then:
echo \"${MYSTRING:11:-17}\"\n=> Be liberal in what you accept, and conservative in what you send
This works since Bash 4.2-alpha, see also bashchanges.
","tags":["bash","shell","scripting","expansion","substitution","text","variable","parameter","mangle","substitute","change","check","defined","null","array","arrays"]},{"location":"syntax/pe/#substringelement-expansion-arrays","title":"Substring/Element expansion: Arrays","text":"For arrays, this expansion type has again 2 meanings:
@ and * it mass-expands individual array elements denoted by the 2 numbers given (starting element, number of elements)Example:
Assume: array=(This is a text)
echo ${array[0]:2:2}is (the \"is\" in \"This\", array element 0)echo ${array[@]:1:2}is a (from element 1 inclusive, 2 elements are expanded, i.e. element 1 and 2)${PARAMETER:-WORD}
${PARAMETER-WORD}
If the parameter PARAMETER is unset (never was defined) or null (empty), this one expands to WORD, otherwise it expands to the value of PARAMETER, as if it just was ${PARAMETER}. If you omit the : (colon), like shown in the second form, the default value is only used when the parameter was unset, not when it was empty.
echo \"Your home directory is: ${HOME:-/home/$USER}.\"\necho \"${HOME:-/home/$USER} will be used to store your personal data.\"\nIf HOME is unset or empty, everytime you want to print something useful, you need to put that parameter syntax in.
#!/bin/bash\n\nread -p \"Enter your gender (just press ENTER to not tell us): \" GENDER\necho \"Your gender is ${GENDER:-a secret}.\"\nIt will print \"Your gender is a secret.\" when you don't enter the gender. Note that the default value is used on expansion time, it is not assigned to the parameter.
","tags":["bash","shell","scripting","expansion","substitution","text","variable","parameter","mangle","substitute","change","check","defined","null","array","arrays"]},{"location":"syntax/pe/#use-a-default-value-arrays","title":"Use a default value: Arrays","text":"For arrays, the behaviour is very similar. Again, you have to make a difference between expanding an individual element by a given index and mass-expanding the array using the @ and * subscripts.
NULL or unset (watch the :- and - variants), the default text is expanded:- and - variants mean the same here):- variant)In other words: The basic meaning of this expansion type is applied as consistent as possible to arrays.
Example code (please try the example cases yourself):
####\n# Example cases for unset/empty arrays and nullstring elements\n####\n\n\n### CASE 1: Unset array (no array)\n\n# make sure we have no array at all\nunset array\n\necho ${array[@]:-This array is NULL or unset}\necho ${array[@]-This array is NULL or unset}\n\n### CASE 2: Set but empty array (no elements)\n\n# declare an empty array\narray=()\n\necho ${array[@]:-This array is NULL or unset}\necho ${array[@]-This array is NULL or unset}\n\n\n### CASE 3: An array with only one element, a nullstring\narray=(\"\")\n\necho ${array[@]:-This array is NULL or unset}\necho ${array[@]-This array is NULL or unset}\n\n\n### CASE 4: An array with only two elements, a nullstring and a normal word\narray=(\"\" word)\n\necho ${array[@]:-This array is NULL or unset}\necho ${array[@]-This array is NULL or unset}\n${PARAMETER:=WORD}
${PARAMETER=WORD}
This one works like the using default values, but the default text you give is not only expanded, but also assigned to the parameter, if it was unset or null. Equivalent to using a default value, when you omit the : (colon), as shown in the second form, the default value will only be assigned when the parameter was unset.
echo \"Your home directory is: ${HOME:=/home/$USER}.\"\necho \"$HOME will be used to store your personal data.\"\nAfter the first expansion here (${HOME:=/home/$USER}), HOME is set and usable.
Let's change our code example from above:
#!/bin/bash\n\nread -p \"Enter your gender (just press ENTER to not tell us): \" GENDER\necho \"Your gender is ${GENDER:=a secret}.\"\necho \"Ah, in case you forgot, your gender is really: $GENDER\"\nFor arrays this expansion type is limited. For an individual index, it behaves like for a \"normal\" parameter, the default value is assigned to this one element. The mass-expansion subscripts @ and * can not be used here because it's not possible to assign to them!
${PARAMETER:+WORD}
${PARAMETER+WORD}
This form expands to nothing if the parameter is unset or empty. If it is set, it does not expand to the parameter's value, but to some text you can specify:
echo \"The Java application was installed and can be started.${JAVAPATH:+ NOTE: JAVAPATH seems to be set}\"\nThe above code will simply add a warning if JAVAPATH is set (because it could influence the startup behaviour of that imaginary application).
Some more unrealistic example... Ask for some flags (for whatever reason), and then, if they were set, print a warning and also print the flags:
#!/bin/bash\n\nread -p \"If you want to use special flags, enter them now: \" SPECIAL_FLAGS\necho \"The installation of the application is finished${SPECIAL_FLAGS:+ (NOTE: there are special flags set: $SPECIAL_FLAGS)}.\"\nIf you omit the colon, as shown in the second form (${PARAMETER+WORD}), the alternate value will be used if the parameter is set (and it can be empty)! You can use it, for example, to complain if variables you need (and that can be empty) are undefined:
# test that with the three stages:\n\n# unset foo\n# foo=\"\"\n# foo=\"something\"\n\nif [[ ${foo+isset} = isset ]]; then\n echo \"foo is set...\"\nelse\n echo \"foo is not set...\"\nfi\nSimilar to the cases for arrays to expand to a default value, this expansion behaves like for a \"normal\" parameter when using individual array elements by index, but reacts differently when using the mass-expansion subscripts @ and *:
* For individual elements, it's the very same: If the expanded element is **not** NULL or unset (watch the :+ and + variants), the alternate text is expanded\n * For mass-expansion syntax, the alternate text is expanded if the array\n * contains elements where min. one element is **not** a nullstring (the :+ and + variants mean the same here)\n * contains **only** elements that are **not** the nullstring (the :+ variant)\nFor some cases to play with, please see the code examples in the description for using a default value.
","tags":["bash","shell","scripting","expansion","substitution","text","variable","parameter","mangle","substitute","change","check","defined","null","array","arrays"]},{"location":"syntax/pe/#display-error-if-null-or-unset","title":"Display error if null or unset","text":"${PARAMETER:?WORD}
${PARAMETER?WORD}
If the parameter PARAMETER is set/non-null, this form will simply expand it. Otherwise, the expansion of WORD will be used as appendix for an error message:
$ echo \"The unset parameter is: ${p_unset?not set}\"\nbash: p_unset: not set\nAfter printing this message,
$? to a non-zero valueThe meaning of the colon (:) is the same as for the other parameter expansion syntaxes: It specifies if
are taken into account.
","tags":["bash","shell","scripting","expansion","substitution","text","variable","parameter","mangle","substitute","change","check","defined","null","array","arrays"]},{"location":"syntax/pe/#code-examples","title":"Code examples","text":"","tags":["bash","shell","scripting","expansion","substitution","text","variable","parameter","mangle","substitute","change","check","defined","null","array","arrays"]},{"location":"syntax/pe/#substring-removal_1","title":"Substring removal","text":"Removing the first 6 characters from a text string:
STRING=\"Hello world\"\n\n# only print 'Hello'\necho \"${STRING%??????}\"\n\n# only print 'world'\necho \"${STRING#??????}\"\n\n# store it into the same variable\nSTRING=${STRING#??????}\nFixed in 4.2.36 (patch). Bash doesn't follow either POSIX or its own documentation when expanding either a quoted \"$@\" or \"${arr[@]}\" with an adjacent expansion. \"$@$x\" expands in the same way as \"$*$x\" - i.e. all parameters plus the adjacent expansion are concatenated into a single argument. As a workaround, each expansion needs to be quoted separately. Unfortunately, this bug took a very long time to notice. ~ $ set -- a b c; x=foo; printf '<%s> ' \"$@$x\" \"$*\"\"$x\" \"$@\"\"$x\" <a b cfoo> <a b cfoo> <a> <b> <cfoo>
Almost all shells disagree about the treatment of an unquoted $@, ${arr[@]}, $*, and ${arr[*]} when IFS is set to null. POSIX is unclear about the expected behavior. A null IFS causes both word splitting and pathname expansion to behave randomly. Since there are few good reasons to leave IFS set to null for more than the duration of a command or two, and even fewer to expand $@ and $* unquoted, this should be a rare issue. Always quote them!
touch x 'y z'\nfor sh in bb {{d,b}a,{m,}k,z}sh; do\n echo \"$sh\"\n \"$sh\" -s a 'b c' d \\* </dev/fd/0\ndone <<\\EOF\n${ZSH_VERSION+:} false && emulate sh\nIFS=\nprintf '<%s> ' $*\necho\nprintf \"<%s> \" $@\necho\nEOF\nbb\n<ab cd*>\n<ab cd*>\ndash\n<ab cd*>\n<ab cd*>\nbash\n<a> <b c> <d> <x> <y z>\n<a> <b c> <d> <x> <y z>\nmksh\n<a b c d *>\n<a b c d *>\nksh\n<a> <b c> <d> <x> <y z>\n<a> <b c> <d> <x> <y z>\nzsh\n<a> <b c> <d> <x> <y z>\n<a> <b c> <d> <x> <y z>\nIFS is set to a non-null value, or unset, all shells behave the same - first expanding into separate args, then applying pathname expansion and word-splitting to the results, except for zsh, which doesn't do pathname expansion in its default mode. Additionally, shells disagree about various wordsplitting behaviors, the behavior of inserting delimiter characters from IFS in $*, and the way adjacent arguments are concatenated, when IFS is modified in the middle of expansion through side-effects.
for sh in bb {{d,b}a,po,{m,}k,z}sh; do\n printf '%-4s: ' \"$sh\"\n \"$sh\" </dev/fd/0\ndone <<\\EOF\n${ZSH_VERSION+:} false && emulate sh\nset -f -- a b c\nunset -v IFS\nprintf '<%s> ' ${*}${IFS=}${*}${IFS:=-}\"${*}\"\necho\nEOF\nbb : <a b cabc> <a-b-c>\ndash: <a b cabc> <a-b-c>\nbash: <a> <b> <ca> <b> <c-a b c>\nposh: <a> <b> <ca b c> <a-b-c>\nmksh: <a> <b> <ca b c> <a-b-c>\nksh : <a> <b> <ca> <b> <c> <a b c>\nzsh : <a> <b> <ca> <b> <c> <a-b-c>\n${ cmds;} expansion. I haven't yet tested every possible side-effect that can affect expansion halfway through expansion that way. As previously mentioned, the Bash form of indirection by prefixing a parameter expansion with a ! conflicts with the same syntax used by mksh, zsh, and ksh93 for a different purpose. Bash will \"slightly\" modify this expansion in the next version with the addition of namerefs.
Bash (and most other shells) don't allow .'s in identifiers. In ksh93, dots in variable names are used to reference methods (i.e. \"Discipline Functions\"), attributes, special shell variables, and to define the \"real value\" of an instance of a class.
In ksh93, the _ parameter has even more uses. It is used in the same way as self in some object-oriented languages; as a placeholder for some data local to a class; and also as the mechanism for class inheritance. In most other contexts, _ is compatible with Bash.
Bash only evaluates the subscripts of the slice expansion (${x:y:z}) if the parameter is set (for both nested expansions and arithmetic). For ranges, Bash evaluates as little as possible, i.e., if the first part is out of range, the second won't be evaluated. ksh93 and mksh always evaluate the subscript parts even if the parameter is unset.
$ bash -c 'n=\"y[\\$(printf yo >&2)1]\" m=\"y[\\$(printf jo >&2)1]\"; x=(); echo \"${x[@]:n,6:m}\"' # No output\n $ bash -c 'n=\"y[\\$(printf yo >&2)1]\" m=\"y[\\$(printf jo >&2)1]\"; x=([5]=hi); echo \"${x[@]:n,6:m}\"'\nyo\n $ bash -c 'n=\"y[\\$(printf yo >&2)1]\" m=\"y[\\$(printf jo >&2)1]\"; x=([6]=hi); echo \"${x[@]:n,6:m}\"'\nyojo\n $ bash -c 'n=\"y[\\$(printf yo >&2)1]\" m=\"y[\\$(printf jo >&2)1]\"; x=12345; echo \"${x:n,5:m}\"'\nyojo\n $ bash -c 'n=\"y[\\$(printf yo >&2)1]\" m=\"y[\\$(printf jo >&2)1]\"; x=12345; echo \"${x:n,6:m}\"'\nyo\n# Bash\n $ typeset -a a=(meh bleh blerg) b\n $ IFS=e\n $ printf \"<%s> \" \"${b[@]-\"${a[@]}\" \"${a[@]}\"}\"; echo # The entire PE is quoted so Bash considers the inner quotes redundant.\n<meh> <bleh> <blerg meh> <bleh> <blerg>\n $ printf \"<%s> \" \"${b[@]-${a[@]} ${a[@]}}\"; echo # The outer quotes cause the inner expansions to be considered quoted.\n<meh> <bleh> <blerg meh> <bleh> <blerg>\n $ b=(meep beep)\n $ printf \"<%s> \" \"${b[@]-\"${a[@]}\" \"${a[@]}\"}\" \"${b[@]-${a[@]} ${a[@]}}\"; echo # Again no surprises. Outer quotes quote everything recursively.\n<meep> <beep> <meep> <beep>\nNow lets see what can happen if we leave the outside unquoted.
# Bash\n $ typeset -a a=(meh bleh blerg) b\n $ IFS=e\n $ printf \"<%s> \" ${b[@]-\"${a[@]}\" \"${a[@]}\"}; echo # Inner quotes make inner expansions quoted.\n<meh> <bleh> <blerg meh> <bleh> <blerg>\n $ printf \"<%s> \" ${b[@]-${a[@]} ${a[@]}}; echo' # No quotes at all wordsplits / globs, like you'd expect.\n<m> <h> <bl> <h> <bl> <rg m> <h> <bl> <h> <bl> <rg>\nThis all might be intuitive, and is the most common implementation, but this design sucks for a number of reasons. For one, it means Bash makes it absolutely impossible to expand any part of the inner region unquoted while leaving the outer region quoted. Quoting the outer forces quoting of the inner regions recursively (except nested command substitutions of course). Word-splitting is necessary to split words of the inner region, which cannot be done together with outer quoting. Consider the following (only slightly far-fetched) code:
# Bash (non-working example)\n\nunset -v IFS # make sure we have a default IFS\n\nif some crap; then\n typeset -a someCmd=(myCmd arg1 'arg2 yay!' 'third*arg*' 4)\nfi\n\nsomeOtherCmd=mycommand\ntypeset -a otherArgs=(arg3 arg4)\n\n# What do you think the programmer expected to happen here?\n# What do you think will actually happen...\n\n\"${someCmd[@]-\"$someOtherCmd\" arg2 \"${otherArgs[@]}\"}\" arg5\nThis final line is perhaps not the most obvious, but I've run into cases were this type of logic can be desirable and realistic. We can deduce what was intended:
someCmd is set, then the resulting expansion should run the command: \"myCmd\" \"arg1\" \"arg2 yay!\" \"third*arg*\" \"4\" \"arg5\"someCmd is not set, expand $someOtherCmd and the inner args, to run a different command: \"mycommand\" \"arg2\" \"arg3\" \"arg4\" \"arg5\".Unfortunately, it is impossible to get the intended result in Bash (and most other shells) without taking a considerably different approach. The only way to split the literal inner parts is through word-splitting, which requires that the PE be unquoted. But, the only way to expand the outer expansion correctly without word-splitting or globbing is to quote it. Bash will actually expand the command as one of these:
# The quoted PE produces a correct result here...\n $ bash -c 'typeset -a someCmd=(myCmd arg1 \"arg2 yay!\" \"third*arg*\" 4); printf \"<%s> \" \"${someCmd[@]-\"$someOtherCmd\" arg2 \"${otherArgs[@]}\"}\" arg5; echo'\n<myCmd> <arg1> <arg2 yay!> <third*arg*> <4> <arg5>\n\n# ...but in the opposite case the first 3 arguments are glued together. There are no workarounds.\n $ bash -c 'typeset -a otherArgs=(arg3 arg4); someOtherCmd=mycommand; printf \"<%s> \" \"${someCmd[@]-\"$someOtherCmd\" arg2 \"${otherArgs[@]}\"}\" arg5; echo'\n<mycommand arg2 arg3> <arg4> <arg5>\n\n# UNLESS! we unquote the outer expansion allowing the inner quotes to\n# affect the necessary parts while allowing word-splitting to split the literals:\n $ bash -c 'typeset -a otherArgs=(arg3 arg4); someOtherCmd=mycommand; printf \"<%s> \" ${someCmd[@]-\"$someOtherCmd\" arg2 \"${otherArgs[@]}\"} arg5; echo'\n<mycommand> <arg2> <arg3> <arg4> <arg5>\n\n# Success!!!\n $ bash -c 'typeset -a someCmd=(myCmd arg1 \"arg2 yay!\" \"third*arg*\" 4); printf \"<%s> \" ${someCmd[@]-\"$someOtherCmd\" arg2 \"${otherArgs[@]}\"} arg5; echo'\n<myCmd> <arg1> <arg2> <yay!> <third*arg*> <4> <arg5>\n\n# ...Ah f^^k. (again, no workaround possible.)\nTo the best of my knowledge, ksh93 is the only shell that acts differently. Rather than forcing nested expansions into quoting, a quote at the beginning and end of the nested region will cause the quote state to reverse itself within the nested part. I have no idea whether it's an intentional or documented effect, but it does solve the problem and consequently adds a lot of potential power to these expansions.
All we need to do is add two extra double-quotes:
# ksh93 passing the two failed tests from above:\n\n $ ksh -c 'otherArgs=(arg3 arg4); someOtherCmd=\"mycommand\"; printf \"<%s> \" \"${someCmd[@]-\"\"$someOtherCmd\" arg2 \"${otherArgs[@]}\"\"}\" arg5; echo'\n<mycommand> <arg2> <arg3> <arg4> <arg5>\n\n $ ksh -c 'typeset -a someCmd=(myCmd arg1 \"arg2 yay!\" \"third*arg*\" 4); printf \"<%s> \" \"${someCmd[@]-\"\"$someOtherCmd\" arg2 \"${otherArgs[@]}\"\"}\" arg5; echo'\n<myCmd> <arg1> <arg2 yay!> <third*arg*> <4> <arg5>\nThis can be used to control the quote state of any part of any expansion to an arbitrary depth. Sadly, it is the only shell that does this and the difference may introduce a possible compatibility problem.
","tags":["bash","shell","scripting","expansion","substitution","text","variable","parameter","mangle","substitute","change","check","defined","null","array","arrays"]},{"location":"syntax/pe/#see-also","title":"See also","text":"Quoting and escaping are important, as they influence the way Bash acts upon your input. There are three recognized types:
\\$stuff\"stuff\"'stuff'All three forms have the very same purpose: They give you general control over parsing, expansion and expansion results.
Besides these basic variants, there are some special quoting methods (like interpreting ANSI-C escapes in a string) you'll meet below.
:!: ATTENTION :!: The quote characters (\", double quote and ', single quote) are a syntax element that influence parsing. It is not related to the quote characters passed as text to the command line! The syntax quotes are removed before the command is called! Example:
### NO NO NO: this passes three strings:\n### (1) \"my\n### (2) multiword\n### (3) argument\"\nMYARG=\"\\\"my multiword argument\\\"\"\nsomecommand $MYARG\n\n### THIS IS NOT (!) THE SAME AS ###\ncommand \"my multiword argument\"\n\n### YOU NEED ###\nMYARG=\"my multiword argument\"\ncommand \"$MYARG\"\nPer-character escaping is useful in on expansions and substitutions. In general, a character that has a special meaning to Bash, like the dollar-sign ($) can be masked to not have a special meaning using the backslash:
echo \\$HOME is set to \\\"$HOME\\\"\n\\$HOME won't expand because it's not in variable-expansion syntax anymoreThe sequence <newline> (an unquoted backslash, followed by a <newline> character) is interpreted as line continuation. It is removed from the input stream and thus effectively ignored. Use it to beautify your code:
# escapestr_sed()\n# read a stream from stdin and escape characters in text that could be interpreted as\n# special characters by sed\nescape_sed() {\n sed \\\n -e 's/\\//\\\\\\//g' \\\n -e 's/\\&/\\\\\\&/g'\n}\nThe backslash can be used to mask every character that has a special meaning to bash. Exception: Inside a single-quoted string (see below).
","tags":["bash","shell","scripting","quoting","quotes","escape","backslash","marks","singlequotes","doublequotes","single","double"]},{"location":"syntax/quoting/#weak-quoting","title":"Weak quoting","text":"Inside a weak-quoted string there's no special interpretion of:
Everything else, especially parameter expansion, is performed!
ls -l \"*\"\nWill not be expanded. ls gets the literal * as argument. It will, unless you have a file named *, spit out an error.
echo \"Your PATH is: $PATH\"\nWill work as expected. $PATH is expanded, because it's double (weak) quoted.
If a backslash in double quotes (\"weak quoting\") occurs, there are 2 ways to deal with it
In particuar this means that \"\\$\" will become $, but \"\\x\" will become \\x.
Strong quoting is very easy to explain:
Inside a single-quoted string nothing is interpreted, except the single-quote that closes the string.
echo 'Your PATH is: $PATH'\n$PATH won't be expanded, it's interpreted as ordinary text because it's surrounded by strong quotes.
In practise that means, to produce a text like Here's my test... as a single-quoted string, you have to leave and re-enter the single quoting to get the character \"'\" as literal text:
# WRONG\necho 'Here's my test...'\n\n# RIGHT\necho 'Here'\\''s my test...'\n\n# ALTERNATIVE: It's also possible to mix-and-match quotes for readability:\necho \"Here's my test\"\nBash provides another quoting mechanism: Strings that contain ANSI C-like escape sequences. The Syntax is:
$'string'\nwhere the following escape sequences are decoded in string:
\\\" double-quote \\' single-quote \\\\ backslash \\a terminal alert character (bell) \\b backspace \\e escape (ASCII 033) \\E escape (ASCII 033) \\E is non-standard \\f form feed \\n newline \\r carriage return \\t horizontal tab \\v vertical tab \\cx a control-x character, for example, $'\\cZ' to print the control sequence composed of Ctrl-Z (^Z) \\uXXXX Interprets XXXX as a hexadecimal number and prints the corresponding character from the character set (4 digits) (Bash 4.2-alpha) \\UXXXXXXXX Interprets XXXX as a hexadecimal number and prints the corresponding character from the character set (8 digits) (Bash 4.2-alpha) \\nnn the eight-bit character whose value is the octal value nnn (one to three digits) \\xHH the eight-bit character whose value is the hexadecimal value HH (one or two hex digits) This is especially useful when you want to pass special characters as arguments to some programs, like passing a newline to sed.
The resulting text is treated as if it were single-quoted. No further expansion happens.
The $'...' syntax comes from ksh93, but is portable to most modern shells including pdksh. A specification for it was accepted for SUS issue 7. There are still some stragglers, such as most ash variants including dash, (except busybox built with \"bash compatibility\" features).
A dollar-sign followed by a double-quoted string, for example
echo $\"generating database...\"\nmeans I18N. If there is a translation available for that string, it is used instead of the given text. If not, or if the locale is C/POSIX, the dollar sign is simply ignored, which results in a normal double quoted string.
If the string was replaced (translated), the result is double quoted.
In case you're a C programmer: The purpose of $\"...\" is the same as for gettext() or _().
For useful examples to localize your scripts, please see Appendix I of the Advanced Bash Scripting Guide.
Attention: There is a security hole. Please read the gettext documentation
","tags":["bash","shell","scripting","quoting","quotes","escape","backslash","marks","singlequotes","doublequotes","single","double"]},{"location":"syntax/quoting/#common-mistakes","title":"Common mistakes","text":"","tags":["bash","shell","scripting","quoting","quotes","escape","backslash","marks","singlequotes","doublequotes","single","double"]},{"location":"syntax/quoting/#string-lists-in-for-loops","title":"String lists in for-loops","text":"The classic for loop uses a list of words to iterate through. The list can also be in a variable:
mylist=\"DOG CAT BIRD HORSE\"\nWRONG way to iterate through this list:
for animal in \"$mylist\"; do\n echo $animal\ndone\nWhy? Due to the double-quotes, technically, the expansion of $mylist is seen as one word. The for loop iterates exactly one time, with animal set to the whole list.
RIGHT way to iterate through this list:
for animal in $mylist; do\n echo $animal\ndone\nThe command test or [ ... ] (the classic test command) is an ordinary command, so ordinary syntax rules apply. Let's take string comparison as an example:
[ WORD = WORD ]\nThe ] at the end is a convenience; if you type which [ you will see that there is in fact a binary file with that name. So if we were writing this as a test command it would be:
test WORD = WORD\nWhen you compare variables, it's wise to quote them. Let's create a test string with spaces:
mystring=\"my string\"\nAnd now check that string against the word \"testword\":
[ $mystring = testword ] # WRONG!\nThis fails! These are too many arguments for the string comparison test. After expansion is performed, you really execute:
[ my string = testword ]\ntest my string = testword\nWhich is wrong, because my and string are two separate arguments.
So what you really want to do is:
[ \"$mystring\" = testword ] # RIGHT!\n\ntest 'my string' = testword\nNow the command has three parameters, which makes sense for a binary (two argument) operator.
Hint: Inside the conditional expression ([[ ]]) Bash doesn't perform word splitting, and thus you don't need to quote your variable references - they are always seen as \"one word\".
FIXME
To be continued
Redirection makes it possible to control where the output of a command goes to, and where the input of a command comes from. It's a mighty tool that, together with pipelines, makes the shell powerful. The redirection operators are checked whenever a simple command is about to be executed.
Under normal circumstances, there are 3 files open, accessible by the file descriptors 0, 1 and 2, all connected to your terminal:
Name FD Descriptionstdin 0 standard input stream (e.g. keyboard) stdout 1 standard output stream (e.g. monitor) stderr 2 standard error output stream (usually also on monitor) INFO
The terms \"monitor\" and \"keyboard\" refer to the same device, the terminal here. Check your preferred UNIX\u00ae-FAQ for details, I'm too lazy to explain what a terminal is ;-)
Both, stdout and stderr are output file descriptors. Their difference is the convention that a program outputs payload on stdout and diagnostic- and error-messages on stderr. If you write a script that outputs error messages, please make sure you follow this convention!
Whenever you name such a filedescriptor, i.e. you want to redirect this descriptor, you just use the number:
# this executes the cat-command and redirects its error messages (stderr) to the bit bucket\ncat some_file.txt 2>/dev/null\nWhenever you reference a descriptor, to point to its current target file, then you use a \"&\" followed by a the descriptor number:
# this executes the echo-command and redirects its normal output (stdout) to the standard error target\necho \"There was an error\" 1>&2\nThe redirection operation can be anywhere in a simple command, so these examples are equivalent:
cat foo.txt bar.txt >new.txt\ncat >new.txt foo.txt bar.txt\n>new.txt cat foo.txt bar.txt\nimportant
Every redirection operator takes one or two words as operands. If you have to use operands (e.g. filenames to redirect to) that contain spaces you must quote them!
"},{"location":"syntax/redirection/#valid-redirection-targets-and-sources","title":"Valid redirection targets and sources","text":"This syntax is recognized whenever a TARGET or a SOURCE specification (like below in the details descriptions) is used.
FILENAME references a normal, ordinary filename from the filesystem (which can of course be a FIFO, too. Simply everything you can reference in the filesystem) &N references the current target/source of the filedescriptor N (\"duplicates\" the filedescriptor) &- closes the redirected filedescriptor, useful instead of > /dev/null constructs (> &-) /dev/fd/N duplicates the filedescriptor N, if N is a valid integer /dev/stdin duplicates filedescriptor 0 (stdin) /dev/stdout duplicates filedescriptor 1 (stdout) /dev/stderr duplicates filedescriptor 2 (stderr) /dev/tcp/HOST/PORT assuming HOST is a valid hostname or IP address, and PORT is a valid port number or service name: redirect from/to the corresponding TCP socket /dev/udp/HOST/PORT assuming HOST is a valid hostname or IP address, and PORT is a valid port number or service name: redirect from/to the corresponding UDP socket If a target/source specification fails to open, the whole redirection operation fails. Avoid referencing file descriptors above 9, since you may collide with file descriptors Bash uses internally.
"},{"location":"syntax/redirection/#redirecting-output","title":"Redirecting output","text":"N > TARGET\nThis redirects the file descriptor number N to the target TARGET. If N is omitted, stdout is assumed (FD 1). The TARGET is truncated before writing starts.
If the option noclobber is set with the set builtin, with cause the redirection to fail, when TARGET names a regular file that already exists. You can manually override that behaviour by forcing overwrite with the redirection operator >| instead of >.
N >> TARGET\nThis redirects the file descriptor number N to the target TARGET. If N is omitted, stdout is assumed (FD 1). The TARGET is not truncated before writing starts.
&> TARGET\n\n>& TARGET\nThis special syntax redirects both, stdout and stderr to the specified target. It's equivalent to
> TARGET 2>&1\nSince Bash4, there's &>>TARGET, which is equivalent to >> TARGET 2>&1.
important
This syntax is deprecated and should not be used. See the page about obsolete and deprecated syntax.
"},{"location":"syntax/redirection/#appending-redirected-output-and-error-output","title":"Appending redirected output and error output","text":"To append the cumulative redirection of stdout and stderr to a file you simply do
>> FILE 2>&1\n\n&>> FILE\nCOMMAND1 2>&1 | COMMAND2\n\nCOMMAND1 |& COMMAND2\nN < SOURCE\nThe input descriptor N uses SOURCE as its data source. If N is omitted, filedescriptor 0 (stdin) is assumed.
<<TAG\n...\nTAG\n\n<<-TAG\n...\nTAG\nA here-document is an input redirection using source data specified directly at the command line (or in the script), no \"external\" source. The redirection-operator << is used together with a tag TAG that's used to mark the end of input later:
# display help\n\ncat <<EOF\nSorry...\nNo help available yet for $PROGRAM.\nHehe...\nEOF\nAs you see, substitutions are possible. To be precise, the following substitutions and expansions are performed in the here-document data:
You can avoid that by quoting the tag:
cat <<\"EOF\"\nThis won't be expanded: $PATH\nEOF\nLast but not least, if the redirection operator << is followed by a - (dash), all leading TAB from the document data will be ignored. This might be useful to have optical nice code also when using here-documents.
The tag you use must be the only word in the line, to be recognized as end-of-here-document marker.
info
It seems that here-documents (tested on versions 1.14.7, 2.05b and 3.1.17) are correctly terminated when there is an EOF before the end-of-here-document tag. The reason is unknown, but it seems to be done on purpose. Bash 4 introduced a warning message when end-of-file is seen before the tag is reached.
<<< WORD\nThe here-strings are a variation of the here-documents. The word WORD is taken for the input redirection:
cat <<< \"Hello world... $NAME is here...\"\nJust beware to quote the WORD if it contains spaces. Otherwise the rest will be given as normal parameters.
The here-string will append a newline (\\n) to the data.
More redirection operations can occur in a line of course. The order is important! They're evaluated from left to right. If you want to redirect both, stderr and stdout to the same file (like /dev/null, to hide it), this is the wrong way:
# { echo OUTPUT; echo ERRORS >&2; } is to simulate something that outputs to STDOUT and STDERR\n# you can test with it\n{ echo OUTPUT; echo ERRORS >&2; } 2>&1 1>/dev/null\nWhy? Relatively easy:
stdout points to your terminal (you read it)stderr, it's connected to your terminal2>&1 redirects stderr away from the terminal to the target for stdout: the terminal (again...)1>/dev/null redirects stdout away from your terminal to the file /dev/nullWhat remains? stdout goes to /dev/null, stderr still (or better: \"again\") goes to the terminal. You have to swap the order to make it do what you want:
{ echo OUTPUT; echo ERRORS >&2; } 1>/dev/null 2>&1\nHow to make a program quiet (assuming all output goes to STDOUT and STDERR?
command >/dev/null 2>&1\n* asterisk The positional parameters starting from the first. When used inside doublequotes (see quoting), like \"$*\", it expands to all positional parameters as one word, delimited by the first character of the IFS variable (a space in this example): \"$1 $2 $3 $4\".If IFS is unset, the delimiter used will be always a space, if IFS is NULL, the delimiter will be nothing, which effectively concatenates all the positional parameters without any delimiter.When used unquoted, it will just expand to the strings, one by one, not preserving the word boundaries (i.e. word splitting will split the text again, if it contains IFS characters.See also the scripting article about handling positional parameters. @ at-sign The positional parameters starting from the first. When used inside doublequotes (see quoting), like \"$@\", it expands all positional parameters as separate words: \"$1\" \"$2\" \"$3\" \"$4\"Without doublequotes, the behaviour is like the one of * without doublequotes.See also the scripting article about handling positional parameters. # hash mark Number of positional parameters (decimal)See also the scripting article about handling positional parameters. ? question mark Status of the most recently executed foreground-pipeline (exit/return code) - dash Current option flags set by the shell itself, on invocation, or using the set builtin command. It\\'s just a set of characters, like himB for h, i, m and B. $ dollar-sign The process ID (PID) of the shell. In an explicit subshell it expands to the PID of the current \\\"main shell\\\", not the subshell. This is different from $BASHPID! ! exclamation mark The process ID (PID) of the most recently executed background pipeline (like started with command &) 0 zero The name of the shell or the shell script (filename). Set by the shell itself.If Bash is started with a filename to execute (script), it\\'s set to this filename. If started with the -c <CMDLINE> option (commandline given as argument), then $0 will be the first argument after the given <CMDLINE>. Otherwise, it is set to the string given on invocation for argv[0].Unlike popular belief, $0 is not a positional parameter. _ underscore A kind of catch-all parameter. Directly after shell invocation, it\\'s set to the filename used to invoke Bash, or the absolute or relative path to the script, just like $0 would show it. Subsequently, expands to the last argument to the previous command. Placed into the environment when executing commands, and set to the full pathname of these commands. When checking mail, this parameter holds the name of the mail file currently being checked."},{"location":"syntax/shellvars/#shell-variables","title":"Shell Variables","text":""},{"location":"syntax/shellvars/#bash","title":"BASH","text":"Variable: BASH Since: unknown Type: normal variable Read-only: no Set by Bash: yes Default: n/a Expands to the full file name used to invoke the current instance of Bash.
"},{"location":"syntax/shellvars/#bashopts","title":"BASHOPTS","text":"Variable:BASHOPTS Since: 4.1-alpha Type: normal variable Read-only: yes Set by Bash: yes Default: n/a A colon-separated list of enabled shell options.
Each word in the list is a valid argument for the -s option to the shopt builtin command. The options appearing in BASHOPTS are those reported as on by shopt. If this variable is in the environment when Bash starts up, each shell option in the list will be enabled before reading any startup files.
Example content:
cmdhist:expand_aliases:extquote:force_fignore:hostcomplete:interactive_comments:progcomp:promptvars:sourcepath\nThis variable is read-only.
"},{"location":"syntax/shellvars/#bashpid","title":"BASHPID","text":"Variable:BASHPID Since: 4.0-alpha Type: integer variable Read-only: yes Set by Bash: yes Default: n/a Always expands to the process ID of the current Bash process. This differs from the special parameter $ under certain circumstances, such as subshells that do not require Bash to be re-initialized.
BASH_ALIASES Since: unknown Type: associative array Read-only: no Set by Bash: yes Default: n/a An associative array variable whose members correspond to the internal list of aliases as maintained by the alias builtin. Elements added to this array appear in the alias list; unsetting array elements cause aliases to be removed from the alias list.
The associative key is the name of the alias as used with the alias builtin command.
BASH_ARGC Since: 3.0 Type: integer indexed array Read-only: no Set by Bash: only in extended debugging mode Default: n/a An array variable whose values are the number of parameters in each frame of the current Bash execution call stack.
The number of parameters to the current subroutine (shell function or script executed with . or source builtin command) is at the top of the stack. When a subroutine is executed, the number of parameters passed is pushed onto BASH_ARGC.
BASH_ARGV Since: 3.0 Type: integer indexed array Read-only: no Set by Bash: only in extended debugging mode Default: n/a An array variable containing all of the parameters in the current Bash execution call stack.
The final parameter of the last subroutine call is at the top of the stack; the first parameter of the initial call is at the bottom. When a subroutine is executed, the parameters supplied are pushed onto BASH_ARGV.
BASH_ARGV0 Since: 5.0-alpha Type: string Read-only: no Set by Bash: yes Default: same as $0 Expands to the name of the shell or shell script - as the special parameter $0 does. Assignments to BASH_ARGV0 causes the value to be assigned to $0.
If this parameter is unset, it loses its special properties, even if subsequently reset.
"},{"location":"syntax/shellvars/#bash_cmds","title":"BASH_CMDS","text":"Variable:BASH_CMDS Since: unknown Type: associative array Read-only: no Set by Bash: yes Default: n/a An associative array variable whose members correspond to the internal hash table of commands as maintained by the hash builtin command. Elements added to this array appear in the hash table; unsetting array elements cause commands to be removed from the hash table.
The associative key is the name of the command as used with the hash builtin command.
BASH_COMMAND Since: 3.0 Type: normal variable Read-only: no Set by Bash: yes Default: n/a The command currently being executed or about to be executed, unless the shell is executing a command as the result of a trap, in which case it is the command executing at the time of the trap.
"},{"location":"syntax/shellvars/#bash_compat","title":"BASH_COMPAT","text":"Variable:BASH_COMPAT Since: 4.3-alpha Type: normal variable Read-only: no Set by Bash: no Default: n/a The value is used to set the shell's compatibility level. The value may be a decimal number (e.g., 4.2) or an integer (e.g., 42) corresponding to the desired compatibility level. If BASH_COMPAT is unset or set to the empty string, the compatibility level is set to the default for the current version. If BASH_COMPAT is set to a value that is not one of the valid compatibility levels, the shell prints an error message and sets the compatibility level to the default for the current version. The valid compatibility levels correspond to the compatibility options accepted by the shopt builtin. The current version is also a valid value.
BASH_EXECUTION_STRING Since: 3.0 Type: normal variable Read-only: no Set by Bash: yes Default: n/a The command argument to the -c invocation option.
BASH_LINENO Since: 3.0 Type: integer indexed array Read-only: no Set by Bash: yes Default: n/a An array variable whose members are the line numbers in source files corresponding to each member of FUNCNAME.
${BASH_LINENO[$i]} is the line number in the source file where ${FUNCNAME[$ifP]} was called. The corresponding source file name is ${BASH_SOURCE[$i]}. Use LINENO to obtain the current line number.
BASH_REMATCH Since: 3.0 Type: integer indexed array Read-only: no Set by Bash: yes Default: n/a An array variable whose members are assigned by the =~ binary operator to the [[ conditional command.
The element with index 0 is the portion of the string matching the entire regular expression. The element with index n is the portion of the string matching the nth parenthesized subexpression.
Before Bash version 5.1-alpha this variable was readonly.
"},{"location":"syntax/shellvars/#bash_source","title":"BASH_SOURCE","text":"Variable:BASH_SOURCE Since: 3.0 Type: integer indexed array Read-only: no Set by Bash: yes Default: n/a An array variable whose members are the source filenames corresponding to the elements in the FUNCNAME array variable.
BASH_SUBSHELL Since: 3.0 Type: normal variable Read-only: no Set by Bash: yes Default: n/a Incremented by one each time a subshell or subshell environment is spawned. The initial value is 0.
"},{"location":"syntax/shellvars/#bash_versinfo","title":"BASH_VERSINFO","text":"Variable:BASH_VERSINFO Since: 2.0 Type: integer indexed array Read-only: yes Set by Bash: yes Default: n/a A readonly array variable whose members hold version information for this instance of Bash. The values assigned to the array members are as follows:
-- -- BASH_VERSINFO[0] The major version number (the release) BASH_VERSINFO[1] The minor version number (the version) BASH_VERSINFO[2] The patch level BASH_VERSINFO[3] The build version BASH_VERSINFO[4] The release status (e.g., beta1) BASH_VERSINFO[5] The value ofMACHTYPE"},{"location":"syntax/shellvars/#bash_version","title":"BASH_VERSION","text":"Variable: BASH_VERSION Since: unknown Type: normal variable Read-only: no Set by Bash: yes Default: n/a Expands to a string describing the version of this instance of Bash.
Since Bash 2.0 it includes the shell's \"release status\" (alpha[N], beta[N], release).
"},{"location":"syntax/shellvars/#child_max","title":"CHILD_MAX","text":"Variable:CHILD_MAX Since: 4.3-alpha Type: normal variable Read-only: no Set by Bash: no Default: n/a Set the number of exited child status values for the shell to remember. Bash will not allow this value to be decreased below a POSIX-mandated minimum, and there is a maximum value (currently 8192) that this may not exceed. The minimum value is system-dependent.
"},{"location":"syntax/shellvars/#comp_cword","title":"COMP_CWORD","text":"Variable:COMP_CWORD Since: unknown Type: normal variable Read-only: no Set by Bash: only for programmable completion facilities Default: n/a An index into COMP_WORDS of the word containing the current cursor position.
COMP_KEY Since: unknown Type: normal variable Read-only: no Set by Bash: only for programmable completion facilities Default: n/a The key (or final key of a key sequence) used to invoke the current completion function.
"},{"location":"syntax/shellvars/#comp_line","title":"COMP_LINE","text":"Variable:COMP_LINE Since: unknown Type: normal variable Read-only: no Set by Bash: only for programmable completion facilities Default: n/a The current command line.
"},{"location":"syntax/shellvars/#comp_point","title":"COMP_POINT","text":"Variable:COMP_POINT Since: unknown Type: normal variable Read-only: no Set by Bash: only for programmable completion facilities Default: n/a The index of the current cursor position relative to the beginning of the current command. If the current cursor position is at the end of the current command, the value of this variable is equal to ${#COMP_LINE}.
COMP_TYPET Since: unknown Type: normal variable Read-only: no Set by Bash: only for programmable completion facilities Default: n/a Set to an integer value corresponding to the type of completion attempted that caused a completion function to be called:
-- --TAB normal completion ? listing completions after successive tabs ! listing alternatives on partial word completion @ to list completions if the word is not unmodified % for menu completion FIXME
where are the integer values?
"},{"location":"syntax/shellvars/#comp_wordbreaks","title":"COMP_WORDBREAKS","text":"Variable:COMP_WORDBREAKS Since: unknown Type: normal variable Read-only: no Set by Bash: yes Default: n/a Reports the set of characters that the readline library treats as word separators when performing word completion.
If this parameter is unset, it loses its special properties, even if it is subsequently reset.
"},{"location":"syntax/shellvars/#comp_words","title":"COMP_WORDS","text":"Variable:COMP_WORDS Since: unknown Type: integer indexed array Read-only: no Set by Bash: only for programmable completion facilities Default: n/a An array variable consisting of the individual words in the current command line. The line is split into words as readline would split it, using COMP_WORDBREAKS as described above.
COPROC Since: unknown Type: integer indexed array Read-only: no Set by Bash: yes Default: n/a An array variable created to hold the file descriptors for output from and input to an unnamed coprocess.
"},{"location":"syntax/shellvars/#dirstack","title":"DIRSTACK","text":"Variable:DIRSTACK Since: unknown Type: integer indexed array Read-only: no Set by Bash: yes Default: n/a An array variable containing the current contents of the directory stack.
Directories appear in the stack in the order they are displayed by the dirs builtin. Assigning to members of this array variable may be used to modify directories already in the stack, but the pushd and popd builtins must be used to add and remove directories.
Assignment to this variable will not change the current directory.
If this parameter is unset, it loses its special properties, even if it is subsequently reset.
"},{"location":"syntax/shellvars/#epochrealtime","title":"EPOCHREALTIME","text":"Variable:EPOCHREALTIME Since: 5.0-alpha Type: integer variable Read-only: no Set by Bash: yes Default: n/a Expands to the number of seconds since Unix expoch as a floating point value with micro-second granularity.
Assignments to this parameter are ignored. If this parameter is unset, it loses its special properties, even if it is subsequently reset.
"},{"location":"syntax/shellvars/#epochseconds","title":"EPOCHSECONDS","text":"Variable:EPOCHSECONDS Since: 5.0-alpha Type: integer variable Read-only: no Set by Bash: yes Default: n/a Expands to the number of seconds since Unix expoch.
Assignments to this parameter are ignored. If this parameter is unset, it loses its special properties, even if it is subsequently reset.
"},{"location":"syntax/shellvars/#euid","title":"EUID","text":"Variable:EUID Since: unknown Type: integer variable Read-only: yes Set by Bash: yes Default: n/a Expands to the effective user ID of the current user, initialized at shell startup.
!
Do not rely on this variable when security is a concern.
"},{"location":"syntax/shellvars/#funcname","title":"FUNCNAME","text":"Variable:FUNCNAME Since: 2.04 Type: integer indexed array Read-only: no Set by Bash: only inside shell functions Default: n/a An array variable containing the names of all shell functions currently in the execution call stack.
The element with index 0 is the name of any currently-executing shell function. The bottom-most element (the one with the highest index) is \"main\".
This variable can be used with BASH_LINENO and BASH_SOURCE: Each element of FUNCNAME has corresponding elements in BASH_LINENO and BASH_SOURCE to describe the call stack. For instance, ${FUNCNAME[$i]} was called from the file ${BASH_SOURCE[$i+1]} at line number ${BASH_LINENO[$i]}. The caller builtin command displays the current call stack using this information.
This variable exists only when a shell function is executing.
Assignments to this parameter have no effect and return an error status.
If this parameter is unset, it loses its special properties, even if it is subsequently reset.
"},{"location":"syntax/shellvars/#groups","title":"GROUPS","text":"Variable:GROUPS Since: 2.01 Type: integer indexed array Read-only: no Set by Bash: yes Default: n/a An array variable containing the list of groups of which the current user is a member.
Assignments to this parameter have no effect and return an error status.
If this parameter is unset, it loses its special properties, even if it is subsequently reset.
"},{"location":"syntax/shellvars/#histcmd","title":"HISTCMD","text":"Variable:HISTCMD Since: 1.14.0 Type: integer variable Read-only: no Set by Bash: yes Default: n/a Expands to the history number (index in the history list) of the current command.
If this parameter is unset, it loses its special properties, even if it is subsequently reset.
"},{"location":"syntax/shellvars/#hostname","title":"HOSTNAME","text":"Variable:HOSTNAME Since: unknown Type: normal variable Read-only: no Set by Bash: yes Default: n/a Automatically set to the name of the current host.
"},{"location":"syntax/shellvars/#hosttype","title":"HOSTTYPE","text":"Variable:HOSTTYPE Since: unknown Type: normal variable Read-only: no Set by Bash: yes Default: system-dependent Automatically set to a string that uniquely describes the type of machine on which Bash is executing.
Example content:
x86_64\nLINENO Since: unknown Type: integer variable Read-only: no Set by Bash: yes Default: n/a Each time this parameter is referenced, the shell substitutes a decimal number representing the current sequential line number (starting with 1) within a script or function.
When not in a script or function, the value substituted is not guaranteed to be meaningful.
If this parameter is unset, it loses its special properties, even if it is subsequently reset.
"},{"location":"syntax/shellvars/#machtype","title":"MACHTYPE","text":"Variable:MACHTYPE Since: unknown Type: normal variable Read-only: no Set by Bash: yes Default: system-dependent Automatically set to a string that fully describes the system type on which Bash is executing, in the standard GNU \"cpu-company-system\" format.
Example content:
x86_64-unknown-linux-gnu\nMAPFILE Since: unknown Type: integer indexed array Read-only: no Set by Bash: yes Default: n/a An array variable created to hold the text read by the mapfile builtin command when no variable name is supplied.
"},{"location":"syntax/shellvars/#oldpwd","title":"OLDPWD","text":"Variable:OLDPWD Since: unknown Type: normal variable Read-only: no Set by Bash: yes Default: n/a The previous working directory as set by the cd command.
"},{"location":"syntax/shellvars/#optarg","title":"OPTARG","text":"Variable:OPTARG Since: unknown Type: normal variable Read-only: no Set by Bash: yes Default: n/a The value of the last option argument processed by the getopts builtin command.
OPTIND Since: unknown Type: integer variable Read-only: no Set by Bash: yes Default: n/a The index of the next argument to be processed by the getopts builtin command.
OSTYPE Since: unknown Type: normal variable Read-only: no Set by Bash: yes Default: system-dependent Automatically set to a string that describes the operating system on which Bash is executing.
Example content:
linux-gnu\nPIPESTATUS Since: 2.0 Type: integer indexed array Read-only: no Set by Bash: yes Default: n/a An array variable containing a list of exit status values from the processes in the most-recently-executed foreground pipeline (which may contain only a single command).
"},{"location":"syntax/shellvars/#ppid","title":"PPID","text":"Variable:PPID Since: unknown Type: integer variable Read-only: yes Set by Bash: yes Default: n/a The process ID of the shell's parent process.
"},{"location":"syntax/shellvars/#pwd","title":"PWD","text":"Variable:PWD Since: unknown Type: normal variable Read-only: no Set by Bash: yes Default: n/a The current working directory as set by the cd builtin command.
"},{"location":"syntax/shellvars/#random","title":"RANDOM","text":"Variable:RANDOM Since: unknown Type: integer variable Read-only: no Set by Bash: yes Default: n/a Each time this parameter is referenced, a random integer between 0 and 32767 is generated. The sequence of random numbers may be initialized by assigning a value to RANDOM.
If this parameter is unset, it loses its special properties, even if it is subsequently reset.
"},{"location":"syntax/shellvars/#readline_line","title":"READLINE_LINE","text":"Variable:READLINE_LINE Since: unknown Type: normal variable Read-only: no Set by Bash: yes Default: n/a The contents of the readline line buffer, for use with bind -x.
READLINE_POINT Since: unknown Type: normal variable Read-only: no Set by Bash: yes Default: n/a The position of the insertion point in the readline line buffer, for use with bind -x.
REPLY Since: unknown Type: normal variable Read-only: no Set by Bash: only by the read builtin command Default: n/a Set to the line of input read by the read builtin command when no arguments are supplied that name target variables.
"},{"location":"syntax/shellvars/#seconds","title":"SECONDS","text":"Variable:SECONDS Since: unknown Type: integer variable Read-only: no Set by Bash: yes Default: n/a Each time this parameter is referenced, the number of seconds since shell invocation is returned. If a value is assigned to SECONDS, the value returned upon subsequent references is the number of seconds since the assignment plus the value assigned.
If this parameter is unset, it loses its special properties, even if it is subsequently reset.
"},{"location":"syntax/shellvars/#shellopts","title":"SHELLOPTS","text":"Variable:SHELLOPTS Since: unknown Type: normal variable Read-only: yes Set by Bash: yes Default: n/a A colon-separated list of enabled shell options. Each word in the list is a valid argument for the -o option to the set builtin command. The options appearing in SHELLOPTS are those reported as on by set -o.
If this variable is in the environment when Bash starts up, each shell option in the list will be enabled before reading any startup files.
"},{"location":"syntax/shellvars/#shlvl","title":"SHLVL","text":"Variable:SHLVL Since: unknown Type: normal variable Read-only: no Set by Bash: yes Default: n/a Incremented by one each time an instance of Bash is started.
"},{"location":"syntax/shellvars/#uid","title":"UID","text":"Variable:UID Since: unknown Type: integer variable Read-only: yes Set by Bash: yes Default: n/a Expands to the user ID of the current user, initialized at shell startup.
:!: Do not rely on this variable when security is a concern.
"},{"location":"syntax/shellvars/#bash_env","title":"BASH_ENV","text":"Variable:BASH_ENV Since: unknown Type: normal variable Read-only: no Set by Bash: no Default: n/a If this parameter is set when Bash is executing a shell script, its value is interpreted as a filename containing commands to initialize the shell, as in ~/.bashrc. The value of BASH_ENV is subjected to
before being interpreted as a file name.
PATH is not used to search for the resultant file name.
BASH_XTRACEFD Since: 4.1-alpha Type: normal variable Read-only: no Set by Bash: no Default: n/a If set to an integer corresponding to a valid file descriptor, Bash will write the trace output generated when set -x is enabled to that file descriptor.
The file descriptor is closed when BASH_XTRACEFD is unset or assigned a new value.
Unsetting BASH_XTRACEFD or assigning it the empty string causes the trace output to be sent to the standard error. Note that setting BASH_XTRACEFD to 2 (the standard error file descriptor) and then unsetting it will result in the standard error being closed.
CDPATH Since: unknown Type: normal variable Read-only: no Set by Bash: no Default: n/a The search path for the cd builtin command.
This is a colon-separated list of directories in which the shell looks for destination directories specified by the cd command.
Example content:
.:~:/usr\nCOLUMNS Since: unknown Type: normal variable Read-only: no Set by Bash: on |SIGWINCH Default: n/a Used by the select compound command to determine the terminal width when printing selection lists. Automatically set upon receipt of a SIGWINCH.
COMPREPLY Since: unknown Type: integer indexed array Read-only: no Set by Bash: no Default: n/a An array variable from which Bash reads the possible completions generated by a shell function invoked by the programmable completion facility.
"},{"location":"syntax/shellvars/#emacs","title":"EMACS","text":"Variable:EMACS Since: unknown Type: normal variable Read-only: no Set by Bash: no Default: n/a If Bash finds this variable in the environment when the shell starts with value \"t\", it assumes that the shell is running in an Emacs shell buffer and disables line editing.
"},{"location":"syntax/shellvars/#env","title":"ENV","text":"Variable:ENV Since: unknown Type: normal variable Read-only: no Set by Bash: no Default: n/a Similar to BASH_ENV: Used when the shell is invoked in POSIX\u00ae mode.
FCEDIT Since: unknown Type: normal variable Read-only: no Set by Bash: no Default: n/a The default editor for the fc builtin command.
FIGNORE Since: unknown Type: normal variable Read-only: no Set by Bash: no Default: n/a A colon-separated list of suffixes to ignore when performing filename completion. A filename whose suffix matches one of the entries in FIGNORE is excluded from the list of matched filenames.
Example content:
.o:~\nFUNCNEST Since: 4.2-alpha Type: normal variable Read-only: no Set by Bash: no Default: n/a If set to a numeric value greater than 0, defines a maximum function nesting level. Function invocations that exceed this nesting level will cause the current command to abort.
Negative values, 0 or non-numeric assignments have the effect as if FUNCNEST was unset or empty: No nest control
GLOBIGNORE Since: 2.0 Type: normal variable Read-only: no Set by Bash: no Default: n/a A colon-separated list of patterns defining the set of filenames to be ignored by pathname expansion. If a filename matched by a pathname expansion pattern also matches one of the patterns in GLOBIGNORE, it is removed from the list of matches.
HISTCONTROL Since: unknown Type: normal variable Read-only: no Set by Bash: no Default: n/a A colon-separated list of values controlling how commands are saved on the history list:
ignorespace lines which begin with a space character are not saved in the history list ignoredups don't save lines matching the previous history entry ignoreboth short for ignorespace:ignoredups erasedups remove all previous lines matching the current line from the history list before the current line is saved Any value not in the above list is ignored.
If HISTCONTROL is unset, or does not include a valid value, all lines read by the shell parser are saved on the history list, subject to the value of HISTIGNORE. The second and subsequent lines of a multi-line compound command are not tested, and are added to the history regardless of the value of HISTCONTROL.
HISTFILE Since: unknown Type: normal variable Read-only: no Set by Bash: if unset Default: \\'\\' \\~/.bash_history\\'\\' The |name of the file in which command history is saved.|
If unset, the command history is not saved when an interactive shell exits.
"},{"location":"syntax/shellvars/#histfilesize","title":"HISTFILESIZE","text":"Variable:HISTFILESIZE Since: unknown Type: normal variable Read-only: no Set by Bash: if unset Default: HISTSIZE The |maximum number of lines contained in the history file.|
When this variable is assigned a value, the history file is truncated, if necessary, by removing the oldest entries, to contain no more than the given number of lines. If the given number of lines is 0 (zero), the file is truncated to zero size. Non-numeric values and numeric values less than zero inhibit truncation.
The history file is also truncated to this size after writing it when an interactive shell exits.
"},{"location":"syntax/shellvars/#histignore","title":"HISTIGNORE","text":"Variable:HISTIGNORE Since: 2.0 Type: normal variable Read-only: no Set by Bash: no Default: n/a A colon-separated list of patterns used to decide which command lines should be saved on the history list. Each pattern is anchored at the beginning of the line and must match the complete line (no implicit '*' is appended).
Each pattern is tested against the line after the checks specified by HISTCONTROL are applied.
In addition to the normal shell pattern matching characters, \"&\" matches the previous history line. \"&\" may be escaped using a backslash; the backslash is removed before attempting a match.
The second and subsequent lines of a multi-line compound command are not tested, and are added to the history regardless of the value of HISTIGNORE.
HISTSIZE Since: unknown Type: normal variable Read-only: no Set by Bash: if unset Default: set at compile time (default 500) The number of commands to remember in the command history.
If the number is set to 0 (zero), then the history list is disabled. If the number is set to any negative number, then the history list is unlimited.
"},{"location":"syntax/shellvars/#histtimeformat","title":"HISTTIMEFORMAT","text":"Variable:HISTTIMEFORMAT Since: unknown Type: normal variable Read-only: no Set by Bash: no Default: n/a If this variable is set and not null, its value is used as a format string for strftime(3) to print the time stamp associated with each history entry displayed by the history builtin.
If this variable is set, time stamps are written to the history file so they may be preserved across shell sessions. This uses the history comment character to distinguish timestamps from other history lines.
"},{"location":"syntax/shellvars/#home","title":"HOME","text":"Variable:HOME Since: unknown Type: normal variable Read-only: no Set by Bash: no Default: n/a The home directory of the current user.
The default argument for the cd builtin command.
The value of this variable is also used when performing tilde expansion.
"},{"location":"syntax/shellvars/#hostfile","title":"HOSTFILE","text":"Variable:HOSTFILE Since: unknown Type: normal variable Read-only: no Set by Bash: no Default: n/a Contains the name of a file in the same format as /etc/hosts that should be read when the shell needs to complete a hostname.
The list of possible hostname completions may be changed while the shell is running. the next time hostname completion is attempted after the value is changed, Bash adds the contents of the new file to the existing list.
If HOSTFILE is set, but has no value, or does not name a readable file, Bash attempts to read /etc/hosts to obtain the list of possible hostname completions.
When HOSTFILE is unset, the hostname list is cleared.
IFS Since: unknown Type: normal variable Read-only: no Set by Bash: no Default: <space><tab><newline> The Internal Field Separator |that is used for word splitting after| expansion and to split lines into words with the read builtin command.
"},{"location":"syntax/shellvars/#ignoreeof","title":"IGNOREEOF","text":"Variable:IGNOREEOF Since: unknown Type: normal variable Read-only: no Set by Bash: no Default: 10 (when invalid) Controls the action of an interactive shell on receipt of an EOF character (e.g. by Ctrl-D) as the sole input.
If set, the value is the number of consecutive EOF characters which must be typed as the first characters on an input line before Bash exits.
If the variable exists but does not have a numeric value, or has no value, the default value is 10.
If it does not exist, EOF signifies the end of input to the shell.
INPUTRC Since: unknown Type: normal variable Read-only: no Set by Bash: no Default: n/a The filename for the readline startup file, overriding the default of ~/.inputrc.
LANG Since: unknown Type: normal variable Read-only: no Set by Bash: no Default: n/a Used to determine the locale category for any category not specifically selected with a variable starting with LC_.
LC_ALL Since: unknown Type: normal variable Read-only: no Set by Bash: no Default: n/a This variable overrides the value of LANG and any other LC_ variable specifying a locale category.
LC_COLLATE Since: unknown Type: normal variable Read-only: no Set by Bash: no Default: n/a This variable determines the collation order used when sorting the results of pathname expansion, and determines the behavior of range expressions, equivalence classes, and collating sequences within pathname expansion and pattern matching.
"},{"location":"syntax/shellvars/#lc_ctype","title":"LC_CTYPE","text":"Variable:LC_CTYPE Since: unknown Type: normal variable Read-only: no Set by Bash: no Default: n/a This variable determines the interpretation of characters and the behavior of character classes within pathname expansion and pattern matching.
"},{"location":"syntax/shellvars/#lc_messages","title":"LC_MESSAGES","text":"Variable:LC_MESSAGES Since: unknown Type: normal variable Read-only: no Set by Bash: no Default: n/a This variable determines the locale used to translate double- quoted strings preceded by a $.
LC_NUMERIC Since: unknown Type: normal variable Read-only: no Set by Bash: no Default: n/a This variable determines the locale category used for number formatting.
"},{"location":"syntax/shellvars/#lines","title":"LINES","text":"Variable:LINES Since: unknown Type: normal variable Read-only: no Set by Bash: on |SIGWINCH Default: n/a Used by the select compound command to determine the column length for printing selection lists. Automatically set upon receipt of a SIGWINCH.
MAIL Since: unknown Type: normal variable Read-only: no Set by Bash: no Default: system-dependent If this parameter is set to a file or directory name and the MAILPATH variable is not set, Bash informs the user of the arrival of mail in the specified file or Maildir-format direc\u2010 tory.
MAILCHECK Since: unknown Type: normal variable Read-only: no Set by Bash: no Default: 60 Specifies |how often (in seconds) Bash checks for mail.|
When it is time to check for mail, the shell does so before displaying the primary prompt.
If this variable is unset, or set to a value that is not a number greater than or equal to zero, the shell disables mail checking.
"},{"location":"syntax/shellvars/#mailpath","title":"MAILPATH","text":"Variable:MAILPATH Since: unknown Type: normal variable Read-only: no Set by Bash: no Default: system-dependent A colon-separated list of file names to be checked for mail.
The message to be printed when mail arrives in a particular file may be specified by separating the file name from the message with a '?' (question mark).
When used in the text of the message, $_ expands to the name of the current mailfile.
Example content:
/var/mail/bfox?\"You have mail\":~/shell-mail?\"$_ has mail!\"\nOPTERR Since: unknown Type: normal variable Read-only: no Set by Bash: yes Default: 1 (set on startup) If set to the value 1, Bash displays error messages generated by the getopts builtin command.
OPTERR is initialized to 1 each time the shell is invoked or a shell script is executed.
PATH Since: unknown Type: normal variable Read-only: no Set by Bash: no Default: system-dependent (set on compile time) The search path for commands. This is a colon-separated list of directories in which the shell looks for commands.
A zero-length (null) directory name in the value of PATH indicates the current directory.
A null directory name may appear as two adjacent colons, or as an initial or trailing colon.
There can be a static path compiled in for use in a restricted shell.
"},{"location":"syntax/shellvars/#posixly_correct","title":"POSIXLY_CORRECT","text":"Variable:POSIXLY_CORRECT Since: unknown Type: normal variable Read-only: no Set by Bash: no Default: n/a If this variable is in the environment when Bash starts, the shell enters posix mode before reading the startup files, as if the --posix invocation option had been supplied.
If it is set while the shell is running, Bash enables posix mode, as if the command set -o posix had been executed.
PROMPT_COMMAND Since: unknown Type: normal variable Read-only: no Set by Bash: no Default: n/a If set, the value is executed as a command prior to issuing each primary prompt.
"},{"location":"syntax/shellvars/#prompt_commands","title":"PROMPT_COMMANDS","text":"Variable:PROMPT_COMMANDS Since: 5.1-alpha Type: integer indexed array Read-only: no Set by Bash: no Default: n/a If set, each element is executed as a command prior to issuing each primary prompt (like PROMPT_COMMAND, just as array).
PROMPT_DIRTRIM Since: unknown Type: normal variable Read-only: no Set by Bash: no Default: n/a If set to a number greater than zero, the value is used as the number of trailing directory components to retain when expanding the \\w and \\W prompt string escapes.
Characters removed are replaced with an ellipsis.
"},{"location":"syntax/shellvars/#ps0","title":"PS0","text":"Variable:PS0 Since: 4.4.0 Type: normal variable Read-only: no Set by Bash: if unset Default: \\\"\\'\\'\\'\\'\\\" Expanded |and displayed by interactive shells after reading a complete| command but before executing it.
"},{"location":"syntax/shellvars/#ps1","title":"PS1","text":"Variable:PS1 Since: unknown Type: normal variable Read-only: no Set by Bash: if unset Default: \\\"\\'\\'\\s-\\v\\\\$ \\'\\'\\\" The |value of this parameter is expanded and used as the primary prompt| string. See Controlling the Prompt.
"},{"location":"syntax/shellvars/#ps2","title":"PS2","text":"Variable:PS2 Since: unknown Type: normal variable Read-only: no Set by Bash: if unset Default: \\\"\\'\\'> \\'\\'\\\" The |value of this parameter is expanded as with PS1 and used as the| secondary prompt string.
"},{"location":"syntax/shellvars/#ps3","title":"PS3","text":"Variable:PS3 Since: unknown Type: normal variable Read-only: no Set by Bash: no Default: n/a The value of this parameter is used as the prompt for the select command.
"},{"location":"syntax/shellvars/#ps4","title":"PS4","text":"Variable:PS4 Since: unknown Type: normal variable Read-only: no Set by Bash: if unset Default: \\\"\\'\\'+ \\'\\'\\\" The |value of this parameter is expanded as with PS1 and the value is| printed before each command Bash displays during an execution trace. The first character of PS4 is replicated multiple times, as necessary, to indicate multiple levels of indirection.
SHELL Since: unknown Type: normal variable Read-only: no Set by Bash: no Default: n/a The full pathname to the shell is kept in this environment variable. If it is not set when the shell starts, Bash assigns the full pathname of the current user's login shell.
"},{"location":"syntax/shellvars/#srandom","title":"SRANDOM","text":"Variable:SRANDOM Since: 5.1-alpha Type: normal variable Read-only: no Set by Bash: yes Default: n/a A variable that delivers a 32bit random number. The random number generation uses platform specific generators in the background and a builtin fallback generator.
"},{"location":"syntax/shellvars/#timeformat","title":"TIMEFORMAT","text":"Variable:TIMEFORMAT Since: unknown Type: normal variable Read-only: no Set by Bash: no Default: n/a The value of this parameter is used as a format string specifying how the timing information for pipelines prefixed with the time reserved word should be displayed.
The % character introduces an escape sequence that is expanded to a time value or other information. The escape sequences and their meanings are as follows, the braces denote optional portions:
%% a literal % (percent sign) %[p][l]R elapsed time in seconds %[p][l]U number of CPU seconds spent in user mode %[p][l]S number of CPU seconds spent in system mode %P CPU percentage, computed as (%U + %S) / %R The optional modifiers (p and l) are:
p A digit specifying the precision. A value of 0 causes no decimal point or fraction to be output. At most three digits after the decimal point are shown. If not specified, the value 3 is used. l A longer format, including minutes, of the form MMmSS.FFs. The value of p determines whether or not the fraction is included. If this variable is not set, Bash acts as if it had the value
$'\\nreal\\t%3lR\\nuser\\t%3lU\\nsys%3lS'\nIf the value is null, no timing information is displayed.
A trailing newline is added when the format string is displayed.
"},{"location":"syntax/shellvars/#tmout","title":"TMOUT","text":"Variable:TMOUT Since: 2.05b Type: normal variable Read-only: no Set by Bash: no Default: n/a If set to a value greater than zero, TMOUT is treated as the default timeout for the read builtin command.
The select command terminates if input does not arrive after TMOUT seconds when input is coming from a terminal.
In an interactive shell, the value is interpreted as the number of seconds to wait for input after issuing the primary prompt. Bash terminates after waiting for that number of seconds if input does not arrive.
"},{"location":"syntax/shellvars/#tmpdir","title":"TMPDIR","text":"Variable:TMPDIR Since: unknown Type: normal variable Read-only: no Set by Bash: no Default: n/a If set, Bash uses its value as the name of a directory in which Bash creates temporary files for the shell's use.
"},{"location":"syntax/shellvars/#auto_resume","title":"auto_resume","text":"Variable:auto_resume Since: unknown Type: normal variable Read-only: no Set by Bash: no Default: n/a This variable controls how the shell interacts with the user and job control. If this variable is set, single word simple commands without redirections are treated as candidates for resumption of an existing stopped job. There is no ambiguity allowed; if there is more than one job beginning with the string typed, the job most recently accessed is selected. The name of a stopped job, in this context, is the command line used to start it. If set to the value exact, the string supplied must match the name of a stopped job exactly; if set to substring, the string supplied needs to match a substring of the name of a stopped job. The substring value provides functionality analogous to the %? job identifier.
If set to any other value, the supplied string must be a prefix of a stopped job's name; this provides functionality analogous to the %string job identifier.
histchars Since: unknown Type: normal variable Read-only: no Set by Bash: no Default: n/a The two or three characters which control history expansion and tokenization.
The first character is the history expansion character, the character which signals the start of a history expansion, normally '!' (exlamation mark).
The second character is the quick substitution character, which is used as shorthand for re-running the previous command entered, substi tuting one string for another in the command. The default is '^' (carret).
The optional third character is the character which indicates that the remainder of the line is a comment when found as the first character of a word, normally '#' (hash mark). The history comment character causes history substitution to be skipped for the remaining words on the line. It does not necessarily cause the shell parser to treat the rest of the line as a comment.
"},{"location":"syntax/words/","title":"Words...","text":"FIXME
This article needs a review, it covers two topics (command line splitting and word splitting) and mixes both a bit too much. But in general, it's still usable to help understand this behaviour, it's \"wrong but not wrong\".
One fundamental principle of Bash is to recognize words entered at the command prompt, or under other circumstances like variable-expansion.
","tags":["bash","shell","scripting","token","words","split","splitting","recognition"]},{"location":"syntax/words/#splitting-the-commandline","title":"Splitting the commandline","text":"Bash scans the command line and splits it into words, usually to put the parameters you enter for a command into the right C-memory (the argv vector) to later correctly call the command. These words are recognized by splitting the command line at the special character position, Space or Tab (the manual defines them as blanks). For example, take the echo program. It displays all its parameters separated by a space. When you enter an echo command at the Bash prompt, Bash will look for those special characters, and use them to separate the parameters.
You don't know what I'm talking about? I'm talking about this:
$ echo Hello little world\nHello little world\nIn other words, something you do (and Bash does) everyday. The characters where Bash splits the command line (SPACE, TAB i.e. blanks) are recognized as delimiters. There is no null argument generated when you have 2 or more blanks in the command line. A sequence of more blank characters is treated as a single blank. Here's an example:
$ echo Hello little world\nHello little world\nBash splits the command line at the blanks into words, then it calls echo with each word as an argument. In this example, echo is called with three arguments: \"Hello\", \"little\" and \"world\"!
Does that mean we can't echo more than one Space? Of course not! Bash treats blanks as special characters, but there are two ways to tell Bash not to treat them special: Escaping and quoting.
Escaping a character means, to take away its special meaning. Bash will use an escaped character as text, even if it's a special one. Escaping is done by preceeding the character with a backslash:
$ echo Hello\\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ little \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ \\ world\nHello little world\nNone of the escaped spaces will be used to perform word splitting. Thus, echo is called with one argument: \"Hello little world\".
Bash has a mechanism to \"escape\" an entire string: Quoting. In the context of command-splitting, which this section is about, it doesn't matter which kind of quoting you use: weak quoting or strong quoting, both cause Bash to not treat spaces as special characters:
$ echo \"Hello little world\"\nHello little world\n\n$ echo 'Hello little world'\nHello little world\nWhat is it all about now? Well, for example imagine a program that expects a filename as an argument, like cat. Filenames can have spaces in them:
$ ls -l\ntotal 4\n-rw-r--r-- 1 bonsai bonsai 5 Apr 18 18:16 test file\n\n$ cat test file\ncat: test: No such file or directory\ncat: file: No such file or directory\n\n$ cat test\\ file\nm00!\n\n$ cat \"test file\"\nm00!\nIf you enter that on the command line with Tab completion, that will take care of the spaces. But Bash also does another type of splitting.
","tags":["bash","shell","scripting","token","words","split","splitting","recognition"]},{"location":"syntax/words/#word-splitting","title":"Word splitting","text":"For a more technical description, please read the article about word splitting!
The first kind of splitting is done to parse the command line into separate tokens. This is what was described above, it's a pure command line parsing.
After the command line has been split into words, Bash will perform expansion, if needed - variables that occur in the command line need to be expanded (substituted by their value), for example. This is where the second type of word splitting comes in - several expansions undergo word splitting (but others do not).
Imagine you have a filename stored in a variable:
MYFILE=\"test file\"\nWhen this variable is used, its occurance will be replaced by its content.
$ cat $MYFILE\ncat: test: No such file or directory\ncat: file: No such file or directory\nThough this is another step where spaces make things difficult, quoting is used to work around the difficulty. Quotes also affect word splitting:
$ cat \"$MYFILE\"\nm00!\nLet's follow an unquoted command through these steps, assuming that the variable is set:
MYFILE=\"THE FILE.TXT\"\nand the first review is:
echo The file is named $MYFILE\nThe parser will scan for blanks and mark the relevant words (\"splitting the command line\"):
Initial command line splitting: Word 1 Word 2 Word 3 Word 4 Word 5 Word 6echo The file is named $MYFILE A parameter/variable expansion is part of that command line, Bash will perform the substitution, and the word splitting on the results:
Word splitting after substitution: Word 1 Word 2 Word 3 Word 4 Word 5 Word 6 Word 7echo The file is named THE FILE.TXT Now let's imagine we quoted $MYFILE, the command line now looks like:
echo The file is named \"$MYFILE\"\necho The file is named THE FILE.TXT","tags":["bash","shell","scripting","token","words","split","splitting","recognition"]},{"location":"syntax/words/#see-also","title":"See also","text":"(( <EXPRESSION> ))\nThis command evaluates the arithmetic expression <EXPRESSION>.
If the expression evaluates to 0 then the exit code of the expression is set to 1 (FALSE). If the expression evaluates to something else than 0, then the exit code of the expression is set to 0 (TRUE). For this return code mapping, please see this section.
The functionality basically is equivalent to what the let builtin command does. The arithmetic evaluation compound command should be preferred.
let builtin commandfor (( <EXPR1> ; <EXPR2> ; <EXPR3> )); do\n <LIST>\ndone\n\n# as a special case: without semicolon after ((...))\nfor (( <EXPR1> ; <EXPR2> ; <EXPR3> )) do\n <LIST>\ndone\n\n# alternative, historical and undocumented syntax\nfor (( <EXPR1> ; <EXPR2> ; <EXPR3> )) {\n <LIST>\n}\nThe C-style for-loop is a compound command derived from the equivalent ksh88 feature, which is in turn derived from the C \"for\" keyword. Its purpose is to provide a convenient way to evaluate arithmetic expressions in a loop, plus initialize any required arithmetic variables. It is one of the main \"loop with a counter\" mechanisms available in the language.
The ((;;)) syntax at the top of the loop is not an ordinary arithmetic compound command, but is part of the C-style for-loop's own syntax. The three sections separated by semicolons are arithmetic expression contexts. Each time one of the sections is to be evaluated, the section is first processed for: brace, parameter, command, arithmetic, and process substitution/expansion as usual for arithmetic contexts. When the loop is entered for the first time, <EXPR1> is evaluated, then <EXPR2> is evaluated and checked. If <EXPR2> is true, then the loop body is executed. After the first and all subsequent iterations, <EXPR1> is skipped, <EXPR3> is evaluated, then <EXPR2> is evaluated and checked again. This process continues until <EXPR2> is false.
<EXPR1> is to initialize variables before the first run.<EXPR2> is to check for a termination condition. This is always the last section to evaluate prior to leaving the loop.<EXPR3> is to change conditions after every iteration. For example, incrementing a counter.:!: If one of these arithmetic expressions in the for-loop is empty, it behaves as if it would be 1 (TRUE in arithmetic context).
:!: Like all loops (Both types of for-loop, while and until), this loop can be:
break builtin, optionally as break N to break out of N levels of nested loops.continue builtin, optionally as the continue N analog to break N.The equivalent construct using a while loop and the arithmetic expression compound command would be structured as:
(( <EXPR1> ))\nwhile (( <EXPR2> )); do\n <LIST>\n (( <EXPR3> ))\ndone\nThe equivalent while construct isn't exactly the same, because both, the for and the while loop behave differently in case you use the continue command.
Bash, Ksh93, Mksh, and Zsh also provide an alternate syntax for the for loop - enclosing the loop body in {...} instead of do ... done:
for ((x=1; x<=3; x++))\n{\n echo $x\n}\nThis syntax is not documented and shouldn't be used. I found the parser definitions for it in 1.x code, and in modern 4.x code. My guess is that it's there for compatibility reasons. Unlike the other aforementioned shells, Bash does not support the analogous syntax for case..esac.
"},{"location":"syntax/ccmd/c_for/#return-status","title":"Return status","text":"The return status is that of the last command executed from <LIST>, or FALSE if any of the arithmetic expressions failed.
A simple counter, the loop iterates 101 times (\"0\" to \"100\" are 101 numbers -> 101 runs!), and everytime the variable x is set to the current value.
x = 0x \u2264 100After every iteration it changes x++
for ((x = 0 ; x <= 100 ; x++)); do echo \"Counter: $x\" done
This is the very same counter (compare it to the simple counter example above), but the change that is made is a x += 10. That means, it will count from 0 to 100, but with a step of 10.
for ((x = 0 ; x <= 100 ; x += 10)); do\n echo \"Counter: $x\"\ndone\nThis example loops through the bit-values of a Byte, beginning from 128, ending at 1. If that bit is set in the testbyte, it prints \"1\", else \"0\" => it prints the binary representation of the testbyte value (8 bits).
#!/usr/bin/env bash\n# Example written for http://wiki.bash-hackers.org/syntax/ccmd/c_for#bits_analyzer\n# Based on TheBonsai's original.\n\nfunction toBin {\n typeset m=$1 n=2 x='x[(n*=2)>m]'\n for ((x = x; n /= 2;)); do\n printf %d $(( m & n && 1))\n done\n}\n\nfunction main {\n [[ $1 == +([0-9]) ]] || return\n typeset result\n if (( $(ksh -c 'printf %..2d $1' _ \"$1\") == ( result = $(toBin \"$1\") ) )); then\n printf '%s is %s in base 2!\\n' \"$1\" \"$result\"\n else\n echo 'Oops, something went wrong with our calculation.' >&2\n exit 1\n fi\n}\n\nmain \"${1:-123}\"\n\n# vim: set fenc=utf-8 ff=unix ft=sh :\n\n<div hide>\n\ntestbyte=123\nfor (( n = 128 ; n >= 1 ; n /= 2 )); do\n if (( testbyte & n )); then\n printf %d 1\n else\n printf %s 0\n fi\ndone\necho\n\n</div>\nWhy that one begins at 128 (highest value, on the left) and not 1 (lowest value, on the right)? It's easier to print from left to right...
We arrive at 128 for n through the recursive arithmetic expression stored in x, which calculates the next-greatest power of 2 after m. To show that it works, we use ksh93 to double-check the answer, because it has a built-in feature for printf to print a representation of any number in an arbitrary base (up to 64). Very few languages have that ability built-in, even things like Python.
This counts up and down from 0 to ${1:-5}, ${2:-4} times, demonstrating more complicated arithmetic expressions with multiple variables.
for (( incr = 1, n=0, times = ${2:-4}, step = ${1:-5}; (n += incr) % step || (incr *= -1, --times);)); do\n printf '%*s\\n' \"$((n+1))\" \"$n\"\ndone\n~ \\$ bash <(xclip -o) 1\n\n 2\n 3\n 4\n 5\n 4\n 3\n 2\n\n1 0 1\n\n 2\n 3\n 4\n 5\n 4\n 3\n 2\n\n1\ncase <WORD> in\n [(] <PATTERN1> ) <LIST1> ;; # or ;& or ;;& in Bash 4\n [(] <PATTERN2> ) <LIST2> ;;\n [(] <PATTERN3> | <PATTERN4> ) <LIST3-4> ;;\n ...\n [(] <PATTERNn>) <LISTn> [;;]\nesac\nThe case-statement can execute commands based on a pattern matching decision. The word <WORD> is matched against every pattern <PATTERNn> and on a match, the associated list <LISTn> is executed. Every commandlist is terminated by ;;. This rule is optional for the very last commandlist (i.e., you can omit the ;; before the esac). Every <PATTERNn> is separated from it's associated <LISTn> by a ), and is optionally preceded by a (.
Bash 4 introduces two new action terminators. The classic behavior using ;; is to execute only the list associated with the first matching pattern, then break out of the case block. The ;& terminator causes case to also execute the next block without testing its pattern. The ;;& operator is like ;;, except the case statement doesn't terminate after executing the associated list - Bash just continues testing the next pattern as though the previous pattern didn't match. Using these terminators, a case statement can be configured to test against all patterns, or to share code between blocks, for example.
The word <WORD> is expanded using tilde, parameter and variable expansion; arithmetic, command and process substitution; and quote removal. No word splitting, brace, or pathname expansion is done, which means you can leave expansions unquoted without problems:
var=\"test word\"\n\ncase $var in\n ...\nesac\nThis is similar to the behavior of the conditional expression command (\"new test command\") (also no word splitting for expansions).
Unlike the C-case-statement, only the matching list and nothing else is executed. If more patterns match the word, only the first match is taken. (Note the comment about Bash v4 changes above.)
Multiple |-delimited patterns can be specified for a single block. This is a POSIX-compatable equivalent to the @(pattern-list) extglob construct.
The case statement is one of the most difficult commands to indent clearly, and people frequently ask about the most \"correct\" style. Just do your best - there are many variations of indenting style for case and no real agreed-upon best practice.
Another one of my stupid examples...
printf '%s ' 'Which fruit do you like most?'\nread -${BASH_VERSION+e}r fruit\n\ncase $fruit in\n apple)\n echo 'Mmmmh... I like those!'\n ;;\n banana)\n echo 'Hm, a bit awry, no?'\n ;;\n orange|tangerine)\n echo $'Eeeks! I don't like those!\\nGo away!'\n exit 1\n ;;\n *)\n echo \"Unknown fruit - sure it isn't toxic?\"\nesac\nHere's a practical example showing a common pattern involving a case statement. If the first argument is one of a valid set of alternatives, then perform some sysfs operations under Linux to control a video card's power profile. Otherwise, show a usage synopsis, and print the current power profile and GPU temperature.
# Set radeon power management\nfunction clk {\n typeset base=/sys/class/drm/card0/device\n [[ -r ${base}/hwmon/hwmon0/temp1_input && -r ${base}/power_profile ]] || return 1\n\n case $1 in\n low|high|default)\n printf '%s\\n' \"temp: $(<${base}/hwmon/hwmon0/temp1_input)C\" \"old profile: $(<${base}/power_profile)\"\n echo \"$1\" >${base}/power_profile\n echo \"new profile: $(<${base}/power_profile)\"\n ;;\n *)\n echo \"Usage: $FUNCNAME [ low | high | default ]\"\n printf '%s\\n' \"temp: $(<${base}/hwmon/hwmon0/temp1_input)C\" \"current profile: $(<${base}/power_profile)\"\n esac\n}\nA template for experiments with case logic, showing shared code between blocks using ;&, and the non-short-circuiting ;;& operator:
#!/usr/bin/env bash\n\nf() {\n local -a \"$@\"\n local x\n\n for x; do\n case $x in\n $1)\n local \"$x\"'+=(1)' ;;&\n $2)\n local \"$x\"'+=(2)' ;&\n $3)\n local \"$x\"'+=(3)' ;;\n $1|$2)\n local \"$x\"'+=(4)'\n esac\n IFS=, local -a \"$x\"'=(\"${x}: ${'\"$x\"'[*]}\")'\n done\n\n for x; do\n echo \"${!x}\"\n done\n}\n\nf a b c\n\n# output:\n# a: 1,4\n# b: 2,3\n# c: 3\n;; delimiter is specified by POSIX.;| control operator instead of Bash's ;;&. Mksh has ;;& for Bash compatability (undocumented).;& operator, but no ;;& or equivalent.in and esac: case word { x) ...; };. This is similar to the alternate form Bash supports for its for loops, but Bash doesn't support this syntax for case..esac.for <NAME>; do\n <LIST>\ndone\n\nfor <NAME> in <WORDS>; do\n <LIST>\ndone\nalternative, historical and undocumented syntax 1
for <NAME>; {\n <LIST>\n}\n\nfor <NAME> in <WORDS>; {\n <LIST>\n}\nFor every word in <WORDS>, one iteration of the loop is performed and the variable <NAME> is set to the current word. If no \"in <WORDS>\" is present to give an own word-list, then the positional parameters (\"$@\") are used (the arguments to the script or function). In this case (and only in this case), the semicolon between the variable name and the do is optional.
If you use the loop-variable inside the for-loop and it can contain spaces, you need to quote it, since normal word-splitting procedures apply.
:!: Like all loops (both for-loops, while and until), this loop can be
break command, optionally as break N to break N levels of nested loopscontinue command, optionally as continue N analog to break NBash knows an alternative syntax for the for loop, enclosing the loop body in {...} instead of do ... done:
for x in 1 2 3\n{\n echo $x\n}\nThis syntax is not documented and should not be used. I found the parser definitions for it in 1.x code, and in modern 4.x code. My guess is that it's there for compatiblity reasons. This syntax is not specified by POSIX\u00ae.
"},{"location":"syntax/ccmd/classic_for/#return-status","title":"Return status","text":"The return status is the one of the last command executed in <LIST> or 0 (TRUE), if the item list <WORDS> evaluates to nothing (i.e.: \"is empty\"!).
With some array syntax (see arrays) you can easily \"feed\" the for-loop to iterate over all elements in an array (by mass-expanding all elements):
for element in \"${myarray[@]}\"; do\n echo \"Element: $element\"\ndone\nAnother way is to mass-expand all used indexes and access the array by index:
for index in \"${!myarray[@]}\"; do\n echo \"Element[$index]: ${myarray[$index]}\"\ndone\nYou can use this function to test how arguments to a command will be interpreted and parsed, and finally used:
argtest() {\n n=1\n for arg; do\n echo \"Argument $((n++)): \\\"$arg\\\"\"\n done\n}\nSince pathname expansion will expand all filenames to separate words, regardless of spaces, you can use the for-loop to iterate through filenames in a directory:
for fn in *; do\n if [ -h \"$fn\" ]; then\n echo -n \"Symlink: \"\n elif [ -d \"$fn\" ]; then\n echo -n \"Dir: \"\n elif [ -f \"$fn\" ]; then\n echo -n \"File: \"\n else\n echo -n \"Unknown: \"\n fi\n echo \"$fn\"\ndone\nStupid example, I know ;-)
"},{"location":"syntax/ccmd/classic_for/#loop-over-lines-of-output","title":"Loop over lines of output","text":"To be complete: You can change the internal field separator (IFS) to a newline and thus make a for-loop iterating over lines instead of words:
IFS=$'\\n'\nfor f in $(ls); do\n echo $f\ndone\nThis is just an example. In general
ls(1) outputread command) is a better joice to iterate over linesIt's of course possible to use another for-loop as <LIST>. Here, counting from 0 to 99 in a weird way:
for x in 0 1 2 3 4 5 6 7 8 9; do\n for y in 0 1 2 3 4 5 6 7 8 9; do\n echo $x$y\n done\ndone\nBeginning in Bash 4, you can also use \"sequence expression\" form of brace expansion syntax when looping over numbers, and this form does not create leading zeroes unless you ask for them:
# 100 numbers, no leading zeroes\nfor x in {0..99}; do\n echo $x\ndone\n# Every other number, width 3\nfor x in {000..99..2}; do\n echo $x\ndone\nWARNING: the entire list is created before looping starts. If your list is huge this may be an issue, but no more so than for a glob that expands to a huge list.
"},{"location":"syntax/ccmd/classic_for/#portability-considerations","title":"Portability considerations","text":""},{"location":"syntax/ccmd/classic_for/#see-also","title":"See also","text":"http://pubs.opengroup.org/onlinepubs/9699919799/xrat/V4_xcu_chap02.html#tag_23_02_09_12 \u21a9
[[ <EXPRESSION> ]]\nThe conditional expression is meant as the modern variant of the classic test command. Since it is not a normal command, Bash doesn't need to apply the normal commandline parsing rules like recognizing && as command list operator.
The testing features basically are the same (see the lists for classic test command), with some additions and extensions.
Operator Description( <EXPRESSION> ) Used to group expressions, to influence precedence of operators <EXPRESSION1> && <EXPRESSION2> TRUE if <EXPRESSION1>and<EXPRESSION2> are TRUE (do not use -a!) <EXPRESSION1>||<EXPRESSION2> TRUE if <EXPRESSION1>or<EXPRESSION2> is TRUE (do not use -o!) <STRING> == <PATTERN> <STRING> is checked against the pattern <PATTERN> - TRUE on a matchBut note\u00b9, quoting the pattern forces a literal comparison. <STRING> = <PATTERN> equivalent to the == operator <STRING> != <PATTERN> <STRING> is checked against the pattern <PATTERN> - TRUE on no match <STRING> =~ <ERE> <STRING> is checked against the extended regular expression <ERE> - TRUE on a match See the classic test operators Do not use the test-typical operators -a and -o for AND and OR. See also arithmetic comparisons Using (( <EXPRESSION> )), the arithmetic expression compound command When the == and != operators are used, the string to the right of the operator is considered a pattern and matched according to the rules of Pattern Matching. If the shell option nocasematch is enabled, the match is performed without regard to the case of alphabetic characters.
\u00b9Any part of the pattern may be quoted to force it to be matched as a literal string.
When the operators < and > are used (string collation order), the test happens using the current locale when the compat level is greater than \"40\".
Operator precedence (highest => lowest):
( <EXPRESSION> )! <EXPRESSION><EXPRESSION1> && <EXPRESSION2><EXPRESSION1> || <EXPRESSION2>Do not use the test-typical operators -a and -o for AND and OR, they are not known to the conditional expression. Instead, use the operators && and ||.
Word splitting and pathname expansion are not performed in the expression you give. That means, a variable containing spaces can be used without quoting:
sentence=\"Be liberal in what you accept, and conservative in what you send\"\ncheckme=\"Be liberal in what you accept, and conservative in what you send\"\nif [[ $sentence == $checkme ]]; then\n echo \"Matched...!\"\nelse\n echo \"Sorry, no match :-(\"\nfi\nCompare that to the classic test command, where word splitting is done (because it's a normal command, not something special):
sentence=\"Be liberal in what you accept, and conservative in what you send\"\ncheckme=\"Be liberal in what you accept, and conservative in what you send\"\nif [ \"$sentence\" == \"$checkme\" ]; then\n echo \"Matched...!\"\nelse\n echo \"Sorry, no match :-(\"\nfi\nYou need to quote that variable reference in the classic test command, since (due to the spaces) the word splitting will break it otherwise!
"},{"location":"syntax/ccmd/conditional_expression/#regular-expression-matching","title":"Regular Expression Matching","text":"Using the operator =~, the left hand side operand is matched against the extended regular expression (ERE) on the right hand side.
This is consistent with matching against patterns: Every quoted part of the regular expression is taken literally, even if it contains regular expression special characters.
Best practise is to put the regular expression to match against into a variable. This is to avoid shell parsing errors on otherwise valid regular expressions.
REGEX=\"^[[:upper:]]{2}[[:lower:]]*$\"\n\n# Test 1\nSTRING=Hello\nif [[ $STRING =~ $REGEX ]]; then\n echo \"Match.\"\nelse\n echo \"No match.\"\nfi\n# ==> \"No match.\"\n\n# Test 2\nSTRING=HEllo\nif [[ $STRING =~ $REGEX ]]; then\n echo \"Match.\"\nelse\n echo \"No match.\"\nfi\n# ==> \"Match.\"\nThe interpretation of quoted regular expression special characters can be influenced by setting the compat31 and compat32 shell options (compat* in general). See shell_options.
An array variable whose members are assigned by the =~ binary operator to the [[ conditional command.
The element with index 0 is the portion of the string matching the entire regular expression. The element with index n is the portion of the string matching the nth parenthesized subexpression.
See BASH_REMATCH.
Example:
if [[ \"The quick, red fox\" =~ ^The\\ (.*),\\ (.*)\\ fox$ ]]; then\n echo \"${BASH_REMATCH[0]} is ${BASH_REMATCH[1]} and ${BASH_REMATCH[2]}.\";\nfi\n\n==> The quick, red fox is quick and red.\nAs of Bash 4.1 alpha, the test primaries '<' and '>' (compare strings lexicographically) use the current locale settings, while the same primitives for the builtin test command don't. This leads to the following situation where they behave differently:
$ ./cond.sh\n[[ ' 4' < '1' ]] --> exit 1\n[[ 'step+' < 'step-' ]] --> exit 1\n[ ' 4' < '1' ] --> exit 0\n[ 'step+' < 'step-' ] --> exit 0\nIt won't be aligned. The conditional expression continues to respect the locate, as introduced with 4.1-alpha, the builtin test/[ command continues to behave differently.
When you use a numeric comparison, the arguments are evaluated as an arithmetic expression. The arithmetic expression must be quoted if it both contains whitespace and is not the result of an expansion.
[[ 'i=5, i+=2' -eq 3+4 ]] && echo true # prints true.\n[[ ... ]] functionality isn't specified by POSIX(R), though it's a reserved word[[, the test expression compound command is one of the very most portable non-POSIX features. Aside from the =~ operator, almost every major feature is consistent between Ksh88, Ksh93, mksh, Zsh, and Bash. Ksh93 also adds a large number of unique pattern matching features not supported by other shells including support for several different regex dialects, which are invoked using a different syntax from Bash's =~, though =~ is still supported by ksh and defaults to ERE.$(m='(abc(def))(\\1)(\\2)'; [[ abcdefabcdefdef =~ $m ]]; printf '<%s> ' $? \"${BASH_REMATCH[@]}\" ) will give <0> <abcdefabcdefdef> <abcdef> <def> <abcdef> <def>.=~ (regex) operator was introduced in Bash 3.0, and its behaviour changed in Bash 3.2: since 3.2, quoted strings and substrings are matched as literals by default.< and > operators (string collation order) has changed since Bash 4.0{ <LIST>; }\n\n{\n<LIST>\n}\nThe list <LIST> is simply executed in the current shell environment. The list must be terminated with a newline or semicolon. For parsing reasons, the curly braces must be separated from <LIST> by a semicolon and blanks if they're in the same line! 12
This is known as a group command. The return status is the exit status (exit code) of the list.
The input and output filedescriptors are cumulative:
{\n echo \"PASSWD follows\"\n cat /etc/passwd\n echo\n echo \"GROUPS follows\"\n cat /etc/group\n} >output.txt\nThis compound command also usually is the body of a function definition, though not the only compound command that's valid there:
print_help() {\n echo \"Options:\"\n echo \"-h This help text\"\n echo \"-f FILE Use config file FILE\"\n echo \"-u USER Run as user USER\"\n}\n try_catch() {\n { # Try-block:\n eval \"$@\"\n } ||\n { # Catch-block:\n echo \"An error occurred\"\n return -1\n }\n }\n * [[syntax:ccmd:grouping_subshell | grouping commands in a subshell]]\nActually any properly terminated compound command will work without extra separator (also in some other shells), example: { while sleep 1; do echo ZzZzzZ; done } is valid. But this is not documented, infact the documentation explicitly says that a semicolon or a newline must separate the enclosed list. -- thanks geirha at Freenode\u00a0\u21a9
The main reason is the fact that in shell grammar, the curly braces are not control operators but reserved words -- TheBonsai\u00a0\u21a9
( <LIST> )\nThe list <LIST> is executed in a separate shell - a subprocess. No changes to the environment (variables etc...) are reflected in the \"main shell\".
Execute a command in a different directory.
echo \"$PWD\"\n( cd /usr; echo \"$PWD\" )\necho \"$PWD\" # Still in the original directory.\n(((1+1))) # Equivalent to: (( (1+1) ))\nif <LIST>; then\n <LIST>\nfi\n\nif <LIST>; then\n <LIST>\nelse\n <LIST>\nfi\n\nif <LIST>; then\n <LIST>\nelif <LIST>; then\n <LIST>\nelse\n <LIST>\nfi\nThe if-clause can control the script's flow (what's executed) by looking at the exit codes of other commands.
All commandsets <LIST> are interpreted as command lists, thus they can contain the whole palette from simple commands over pipelines to compound commands (and their combination) as condition.
The if <LIST> commands are executed. If the exit code was 0 (TRUE) then the then <LIST> commands are executed, otherwise the elif <LIST> commands and their then <LIST> statements are executed in turn, if all down to the last one fails, the else <LIST> commands are executed, if one of the elif succeeds, its then thread is executed, and the if-clause finishes.
Basically, the elif clauses are just additional conditions to test (like a chain of conditions) if the very first condition failed. If one of the conditions fails, the else commands are executed, otherwise the commands of the condition that succeeded.
Check if a specific user exists in /etc/passwd :-)
if grep ^myuser: /etc/passwd >/dev/null 2>&1; then\n echo \"Yes, it seems I'm real\"\nelse\n echo \"Uh - am I a ghost?\"\nfi\nMount with check
if ! mount /mnt/backup >/dev/null 2>&1; then\n echo \"FATAL: backup mount failed\" >&2\n exit 1\nfi\nMultiple commands as condition
It's perfectly valid to do:
if echo \"I'm testing!\"; [ -e /some/file ]; then\n ...\nfi\nThe exit code that dictates the condition's value is the exit code of the very last command executed in the condition-list (here: The [ -e /some/file ])
A complete pipe as condition
A complete pipe can also be used as condition. It's very similar to the example above (multiple commands):
if echo \"Hello world!\" | grep -i hello >/dev/null 2>&1; then\n echo \"You just said 'hello', yeah?\"\nfi\nThe main part of Bash's syntax are the so-called compound commands. They're called like that because they use \"real\" commands (simple commands or lists) and knit some intelligence around them. That is what the essential \"Bash language\" is made of.
"},{"location":"syntax/ccmd/intro/#command-grouping","title":"Command grouping","text":"Note that conditionals can also be scripted using list, which are syntax elements, not commands.
until <LIST1> ; do\n <LIST2>\ndone\nThe until-loop is relatively simple in what it does: it executes the command list <LIST1> and if the exit code of it was not 0 (FALSE) it executes <LIST2>. This happens again and again until <LIST1> returns TRUE.
This is exactly the opposite of the while loop.
:!: Like all loops (both for-loops, while and until), this loop can be
break command, optionally as break N to break N levels of nested loopscontinue command, optionally as continue N analog to break NThe return status is the one of the last command executed in <LIST2>, or 0 (TRUE) if none was executed.
select <NAME>; do\n <LIST>\ndone\n\nselect <NAME> in <WORDS>; do\n <LIST>\ndone\n\n# alternative, historical and undocumented syntax\n\nselect <NAME>\n{\n <LIST>\n}\n\nselect <NAME> in <WORDS>\n{\n <LIST>\n}\nThis compound command provides a kind of menu. The user is prompted with a numbered list of the given words, and is asked to input the index number of the word. If a word was selected, the variable <NAME> is set to this word, and the list <LIST> is executed.
If no in <WORDS> is given, then the positional parameters are taken as words (as if in \"$@\" was written).
Regardless of the functionality, the number the user entered is saved in the variable REPLY.
Bash knows an alternative syntax for the select command, enclosing the loop body in {...} instead of do ... done:
select x in 1 2 3\n{\n echo $x\n}\nThis syntax is not documented and should not be used. I found the parser definitions for it in 1.x code, and in modern 4.x code. My guess is that it's there for compatiblity reasons. This syntax is not specified by POSIX(R).
"},{"location":"syntax/ccmd/user_select/#examples","title":"Examples","text":"# select <NAME> in <WORDS>; do\n# <LIST>\n# done\n\n\n# meaning e.g.:\n\nclear\necho\necho hit number key 1 2 or 3 then ENTER-key\necho ENTER alone is an empty choice and will loop endlessly until Ctrl-C or Ctrl-D\necho\n\nselect OPTIONX in beer whiskey wine liquor ; do\n\n echo you ordered a $OPTIONX\n break # break avoids endless loop -- second line to be executed always\n\ndone\n\n# place some if else fi business here\n# and explain how it makes sense that $OPTIONX is red but OPTIONX is black \n# even though both are variables\nwhile <LIST1> ; do\n <LIST2>\ndone\nThe while-loop is relatively simple in what it does: it executes the command list <LIST1> and if the exit code of it was 0 (TRUE) it executes <LIST2>. This happens again and again until <LIST1> returns FALSE.
This is exactly the opposite of the until loop.
:!: Like all loops (both for-loops, while and until), this loop can be
break command, optionally as break N to break N levels of nested loopscontinue command, optionally as continue N analog to break NThe return status is the one of the last command executed in <LIST2>, or 0 (TRUE) if none was executed.
$(( <EXPRESSION> ))\n\n$[ <EXPRESSION> ]\nThe arithmetic expression <EXPRESSION> is evaluated and expands to the result. The output of the arithmetic expansion is guaranteed to be one word and a digit in Bash.
Please do not use the second form $[ ... ]! It's deprecated. The preferred and standardized form is $(( ... ))!
Example
function printSum {\n typeset -A args\n typeset name\n for name in first second; do\n [[ -t 0 ]] && printf 'Enter %s positive integer: ' \"$name\" >&2\n read -r ${BASH_VERSION+-e} \"args[$name]\"\n [[ ${args[$name]} == +([[:digit:]]) ]] || return 1 # Validation is extremely important whenever user input is used in arithmetic.\n done\n printf 'The sum is %d.' $((${args[first]} + ${args[second]}))\n}\nNote that in Bash you don't need the arithmetic expansion to check for the boolean value of an arithmetic expression. This can be done using the arithmetic evaluation compound command:
printf %s 'Enter a number: ' >&2\nread -r number\nif ((number == 1234)); then\n echo 'Good guess'\nelse\n echo 'Haha... :-P'\nfi\nVariables used inside the arithmetic expansion, as in all arithmetic contexts, can be used with or without variable expansion:
x=1\n\necho $((x)) # Good.\necho $(($x)) # Ok. Avoid expansions within arithmetic. Use variables directly.\necho $((\"$x\")) # Error. There is no quote-removal in arithmetic contexts. It expands to $((\"1\")), which is an invalid arithmetic expression.\necho $((x[0])) # Good.\necho $((${x[0]})) # Ok. Nested expansion again.\necho $((${x[$((${x[!$x]}-$x))]})) # Same as above but more ridiculous.\necho $(($x[0])) # Error. This expands to $((1[0])), an invalid expression.\nexpr(1) within backticks instead. Since expr is horrible (as are backticks), and arithmetic expansion is required by POSIX, you should not worry about this, and preferably fix any code you find that's still using expr.{string1,string2,...,stringN}\n{<START>..<END>}\n\n{<START>..<END>..<INCR>} (Bash 4)\n\n<PREFIX>{........}\n\n{........}<SUFFIX>\n\n<PREFIX>{........}<SUFFIX>\nBrace expansion is used to generate arbitrary strings. The specified strings are used to generate all possible combinations with the optional surrounding prefixes and suffixes.
Usually it's used to generate mass-arguments for a command, that follow a specific naming-scheme.
:!: It is the very first step in expansion-handling, it's important to understand that. When you use
echo {a,b}$PATH\nthen the brace expansion does not expand the variable - this is done in a later step. Brace expansion just makes it being:
echo a$PATH b$PATH\nAnother common pitfall is to assume that a range like {1..200} can be expressed with variables using {$a..$b}. Due to what I described above, it simply is not possible, because it's the very first step in doing expansions. A possible way to achieve this, if you really can't handle this in another way, is using the eval command, which basically evaluates a commandline twice: eval echo {$a..$b} For instance, when embedded inside a for loop : for i in $(eval echo {$a..$b}) This requires that the entire command be properly escaped to avoid unexpected expansions. If the sequence expansion is to be assigned to an array, another method is possible using declaration commands: declare -a 'pics=(img{'\"$a..$b\"'}.png)'; mv \"${pics[@]}\" ../imgs This is significantly safer, but one must still be careful to control the values of $a and $b. Both the exact quoting, and explicitly including \"-a\" are important.
The brace expansion is present in two basic forms, string lists and ranges.
It can be switched on and off under runtime by using the set builtin and the option -B and +B or the long option braceexpand. If brace expansion is enabled, the stringlist in SHELLOPTS contains braceexpand.
{string1,string2,...,stringN}\nWithout the optional prefix and suffix strings, the result is just a space-separated list of the given strings:
$ echo {I,want,my,money,back}\nI want my money back\nWith prefix or suffix strings, the result is a space-separated list of all possible combinations of prefix or suffix specified strings:
$ echo _{I,want,my,money,back}\n_I _want _my _money _back\n\n$ echo {I,want,my,money,back}_\nI_ want_ my_ money_ back_\n\n$ echo _{I,want,my,money,back}-\n_I- _want- _my- _money- _back-\nThe brace expansion is only performed, if the given string list is really a list of strings, i.e., if there is a minimum of one \",\" (comma)! Something like {money} doesn't expand to something special, it's really only the text \"{money}\".
{<START>..<END>}\nBrace expansion using ranges is written giving the startpoint and the endpoint of the range. This is a \"sequence expression\". The sequences can be of two types
characters
$ echo {5..12} 5 6 7 8 9 10 11 12
$ echo {c..k} c d e f g h i j k
When you mix these both types, brace expansion is not performed:
$ echo {5..k}\n{5..k}\nWhen you zero pad one of the numbers (or both) in a range, then the generated range is zero padded, too:
$ echo {01..10}\n01 02 03 04 05 06 07 08 09 10\nThere's a chapter of Bash 4 brace expansion changes at the end of this article.
Similar to the expansion using stringlists, you can add prefix and suffix strings:
$ echo 1.{0..9}\n1.0 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9\n\n$ echo ---{A..E}---\n---A--- ---B--- ---C--- ---D--- ---E---\nWhen you combine more brace expansions, you effectively use a brace expansion as prefix or suffix for another one. Let's generate all possible combinations of uppercase letters and digits:
$ echo {A..Z}{0..9}\nA0 A1 A2 A3 A4 A5 A6 A7 A8 A9 B0 B1 B2 B3 B4 B5 B6 B7 B8 B9 C0 C1 C2 C3 C4 C5 C6\nC7 C8 C9 D0 D1 D2 D3 D4 D5 D6 D7 D8 D9 E0 E1 E2 E3 E4 E5 E6 E7 E8 E9 F0 F1 F2 F3\nF4 F5 F6 F7 F8 F9 G0 G1 G2 G3 G4 G5 G6 G7 G8 G9 H0 H1 H2 H3 H4 H5 H6 H7 H8 H9 I0\nI1 I2 I3 I4 I5 I6 I7 I8 I9 J0 J1 J2 J3 J4 J5 J6 J7 J8 J9 K0 K1 K2 K3 K4 K5 K6 K7\nK8 K9 L0 L1 L2 L3 L4 L5 L6 L7 L8 L9 M0 M1 M2 M3 M4 M5 M6 M7 M8 M9 N0 N1 N2 N3 N4\nN5 N6 N7 N8 N9 O0 O1 O2 O3 O4 O5 O6 O7 O8 O9 P0 P1 P2 P3 P4 P5 P6 P7 P8 P9 Q0 Q1\nQ2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 R0 R1 R2 R3 R4 R5 R6 R7 R8 R9 S0 S1 S2 S3 S4 S5 S6 S7 S8\nS9 T0 T1 T2 T3 T4 T5 T6 T7 T8 T9 U0 U1 U2 U3 U4 U5 U6 U7 U8 U9 V0 V1 V2 V3 V4 V5\nV6 V7 V8 V9 W0 W1 W2 W3 W4 W5 W6 W7 W8 W9 X0 X1 X2 X3 X4 X5 X6 X7 X8 X9 Y0 Y1 Y2\nY3 Y4 Y5 Y6 Y7 Y8 Y9 Z0 Z1 Z2 Z3 Z4 Z5 Z6 Z7 Z8 Z9\nHey.. that saves you writing 260 strings!
Brace expansions can be nested, but too much of it usually makes you losing overview a bit ;-)
Here's a sample to generate the alphabet, first the uppercase letters, then the lowercase ones:
$ echo {{A..Z},{a..z}}\nA B C D E F G H I J K L M N O P Q R S T U V W X Y Z a b c d e f g h i j k l m n o p q r s t u v w x y z\nIn this example, wget is used to download documentation that is split over several numbered webpages.
wget won't see your braces. It will see 6 different URLs to download.
wget http://docs.example.com/documentation/slides_part{1,2,3,4,5,6}.html\nOf course it's possible, and even easier, to do that with a sequence:
wget http://docs.example.com/documentation/slides_part{1..6}.html\nYour life is hard? Let's ease it a bit - that's what shells are here for.
mkdir /home/bash/test/{foo,bar,baz,cat,dog}\nUsing a prefix:
for i in 0{1..9} 10; do printf \"%s\\n\" \"$i\";done
If you need to create words with the number embedded, you can use nested brace:
printf \"%s\\n\" img{00{1..9},0{10..99},{100..999}}.png\nFormatting the numbers with printf:
echo $(printf \"img%02d.png \" {1..99})
See the text below for a new Bash 4 method.
","tags":["bash","shell","scripting","expansion","substitution","text","list","brace"]},{"location":"syntax/expansion/brace/#repeating-arguments-or-words","title":"Repeating arguments or words","text":"somecommand -v -v -v -v -v\nCan be written as
somecommand -v{,,,,}\n...which is a kind of a hack, but hey, it works.
More fun
The most optimal possible brace expansion to expand n arguments of course consists of n's prime factors. We can use the \"factor\" program bundled with GNU coreutils to emit a brace expansion that will expand any number of arguments.
function braceify {\n [[ $1 == +([[:digit:]]) ]] || return\n typeset -a a\n read -ra a < <(factor \"$1\")\n eval \"echo $(printf '{$(printf ,%%.s {1..%s})}' \"${a[@]:1}\")\"\n}\n\nprintf 'eval printf \"$arg\"%s' \"$(braceify 1000000)\"\n\"Braceify\" generates the expansion code itself. In this example we inject that output into a template which displays the most terse brace expansion code that would expand \"$arg\" 1,000,000 times if evaluated. In this case, the output is:
eval printf \"$arg\"{,,}{,,}{,,}{,,}{,,}{,,}{,,,,,}{,,,,,}{,,,,,}{,,,,,}{,,,,,}{,,,,,}\nPrefix either of the numbers in a numeric range with 0 to pad the expanded numbers with the correct amount of zeros:
$ echo {0001..5}\n0001 0002 0003 0004 0005\nIt is now possible to specify an increment using ranges:
{<START>..<END>..<INCR>}\n<INCR> is numeric, you can use a negative integer but the correct sign is deduced from the order of <START> and <END> anyways.
$ echo {1..10..2}\n1 3 5 7 9\n$ echo {10..1..2}\n10 8 6 4 2\nInteresting feature: The increment specification also works for letter-ranges:
$ echo {a..z..3}\na d g j m p s v y\n$( <COMMANDS> )\n\n` <COMMANDS> `\nThe command substitution expands to the output of commands. These commands are executed in a subshell, and their stdout data is what the substitution syntax expands to.
All trailing newlines are removed (below is an example for a workaround).
In later steps, if not quoted, the results undergo word splitting and pathname expansion. You have to remember that, because the word splitting will also remove embedded newlines and other IFS characters and break the results up into several words. Also you'll probably get unexpected pathname matches. If you need the literal results, quote the command substitution!
The second form `COMMAND` is more or less obsolete for Bash, since it has some trouble with nesting (\"inner\" backticks need to be escaped) and escaping characters. Use $(COMMAND), it's also POSIX!
When you call an explicit subshell (COMMAND) inside the command substitution $(), then take care, this way is wrong:
$((COMMAND))\nWhy? because it collides with the syntax for arithmetic expansion. You need to separate the command substitution from the inner (COMMAND):
$( (COMMAND) )\nWhen the inner command is only an input redirection, and nothing else, for example
$( <FILE )\n# or\n` <FILE `\nthen Bash attempts to read the given file and act just if the given command was cat FILE.
In general you really should only use the form $(), it's escaping-neutral, it's nestable, it's also POSIX. But take a look at the following code snips to decide yourself which form you need under specific circumstances:
Nesting
Backtick form `...` is not directly nestable. You will have to escape the \"inner\" backticks. Also, the deeper you go, the more escape characters you need. Ugly.
echo `echo `ls`` # INCORRECT\necho `echo \\`ls\\`` # CORRECT\necho $(echo $(ls)) # CORRECT\nParsing
All is based on the fact that the backquote-form is simple character substitution, while every $()-construct opens an own, subsequent parsing step. Everything inside $() is interpreted as if written normal on a commandline. No special escaping of nothing is needed:
echo \"$(echo \"$(ls)\")\" # nested double-quotes - no problem\nConstructs you should avoid
It's not all shiny with $(), at least for my current Bash (3.1.17(1)-release. :!: Update: Fixed since 3.2-beta together with a misinterpretion of '))' being recognized as arithmetic expansion by redduck666). This command seems to incorrectly close the substitution step and echo prints \"ls\" and \")\":
echo $(\n# some comment ending with a )\nls\n)\nIt seems that every closing \")\" confuses this construct. Also a (very uncommon ;-)) construct like:
echo $(read VAR; case \"$var\" in foo) blah ;; esac) # spits out some error, when it sees the \";;\"\n\n# fixes it:\necho $(read VAR; case \"$var\" in (foo) blah ;; esac) # will work, but just let it be, please ;-)\nConclusion:
In general, the $() should be the preferred method:
To get the date:
DATE=\"$(date)\"\nTo copy a file and get cp error output:
COPY_OUTPUT=\"$(cp file.txt /some/where 2>&1)\"\nAttention: Here, you need to redirect cp STDERR to its STDOUT target, because command substitution only catches STDOUT!
Catch stdout and preserve trailing newlines:
var=$(echo -n $'\\n'); echo -n \"$var\"; # $var == \"\"\nvar=$(echo -n $'\\n'; echo -n x); var=\"${var%x}\"; echo -n \"$var\" # $var == \"\\n\"\nThis adds \"x\" to the output, which prevents the trailing newlines of the previous commands' output from being deleted by $().
By removing this \"x\" later on, we are left with the previous commands' output with its trailing newlines.
","tags":["bash","shell","scripting","expansion","substitution","text","variable","output","execute","stdout","save","result","return","value"]},{"location":"syntax/expansion/cmdsubst/#see-also","title":"See also","text":"Unlike on other platforms you may have seen, on UNIX\u00ae, the shell is responsible for interpreting and expanding globs (\"filename wildcards\"). A called program will never see the glob itself; it will only see the expanded filenames as its arguments (here, all filenames matching *.log):
grep \"changes:\" *.log\nThe base syntax for the pathname expansion is the pattern matching syntax. The pattern you describe is matched against all existing filenames and the matching ones are substituted. Since this substitution happens after word splitting, all resulting filenames are literal and treated as separate words, no matter how many spaces or other IFS-characters they contain.
-f, noglob) you can entirely disable pathname expansion/) always needs to be matched explicitly/ in the glob)nullglob is set, non-matching globs are removed, rather than preservedfailglob is set, non-matching globs produce an error message and the current command is not executednocaseglob is set, the match is performed case-insensitivedotglob is set, wildcard-characters can match a dot at the beginning of a filenamedirspell is set, Bash performs spelling corrections when matching directory namesglobstar is set, the glob ** will recursively match all files and directories. This glob isn't \"configurable\", i.e. you can't do something like **.c to recursively get all *.c filenames.globasciiranges is set, the bracket-range globs (e.g. [A-Z]) use C locale order rather than the configured locale's order (i.e. ABC...abc... instead of e.g. AaBbCc...) - since 4.3-alphaNormally, when no glob specified matches an existing filename, no pathname expansion is performed, and the globs are not removed:
$ echo \"Textfiles here:\" *.txt\nTextfiles here: *.txt\nIn this example, no files matched the pattern, so the glob was left intact (a literal asterisk, followed by dot-txt).
This can be very annoying, for example when you drive a for-loop using the pathname expansion:
for filename in *.txt; do\n echo \"=== BEGIN: $filename ===\"\n cat \"$filename\"\n echo \"=== END: $filename ===\"\ndone\nWhen no file name matches the glob, the loop will not only output stupid text (\"BEGIN: *.txt\"), but also will make the cat-command fail with an error, since no file named *.txt exists.
Now, when the shell option nullglob is set, Bash will remove the entire glob from the command line. In case of the for-loop here, not even one iteration will be done. It just won't run.
So in our first example:
$ shopt -s nullglob\n$ echo \"Textfiles here:\" *.txt\nTextfiles here:\nand the glob is gone.
"},{"location":"syntax/expansion/globs/#glob-characters","title":"Glob characters","text":"* - means \\'match any number of characters\\'. \\'/\\' is not matched (and depending on your settings, things like \\'.\\' may or may not be matched, see above)? - means \\'match any single character\\'[abc] - match any of the characters listed. This syntax also supports ranges, like [0-9]For example, to match something beginning with either \\'S\\' or \\'K\\' followed by two numbers, followed by at least 3 more characters:
[SK][0-9][0-9]???*\nBefore executing your commands, Bash checks whether there are any syntax elements in the command line that should be interpreted rather than taken literally. After splitting the command line into tokens (words), Bash scans for these special elements and interprets them, resulting in a changed command line: the elements are said to be expanded to or substituted to new text and maybe new tokens (words).
The most simple example of this behaviour is a referenced variable:
mystring=\"Hello world\"\necho \"$mystring\"\nThe echo program definitely doesn't care about what a shell variable is. It is Bash's job to deal with the variable. Bash expands the string \"$mystring\" to \"Hello world\", so that echo will only see Hello world, not the variable or anything else!
After all these expansions and substitutions are done, all quotes that are not meant literally (i.e., the quotes that marked contiguous words, as part of the shell syntax) are removed from the commandline text, so the called program won't see them. This step is called quote-removal.
","tags":["bash","shell","scripting","expansion","substitution","text","variable","filename","macro","wildcard"]},{"location":"syntax/expansion/intro/#overview","title":"Overview","text":"Saw a possible expansion syntax but don't know what it is? Here's a small list.
$WORD${STUFF...}*.txtpage_1?.html$(( EXPRESSION ))$[ EXPRESSION ]$( COMMAND )` COMMAND `~~+~-{X,Y,Z}{X..Y}{X..Y..Z}<( COMMAND )>( COMMAND )Bash performs expansions and substitutions in a defined order. This explains why globbing (pathname expansion), for example, is safe to use on filenames with spaces (because it happens after the final word splitting!).
The order is (from first to last):
Process substitution is performed simultaneously with parameter expansion, command substitution and arithmetic expansion. It is only performed when the underlying operating system supports it.
The 3 steps parameter expansion, arithmetic expansion and command substitution happen at the same time in a left-to-right fashion on nthe commandline. This means
i=1\necho $i $((i++)) $i\nwill output 1 1 2 and not 1 1 1.
Process substitution is a form of redirection where the input or output of a process (some sequence of commands) appear as a temporary file.
<( <LIST> )\n\n>( <LIST> )\nProcess substitution is performed simultaneously with parameter expansion, command substitution and arithmetic expansion.
The command list <LIST> is executed and its
<( ... ) form or>( ... ) formis connected to a FIFO or a file in /dev/fd/. The filename (where the filedescriptor is connected) is then used as a substitution for the <(...)-construct.
That, for example, allows to give data to a command that can't be reached by pipelining (that doesn't expect its data from stdin but from a file).
Note
According to multiple comments and sources, the scope of process substitution file descriptors is not stable, guaranteed, or specified by bash. Newer versions of bash (5.0+) seem to have shorter scope, and substitutions scope seems to be shorter than function scope. See stackexchange and stackoverflow; the latter discussion contains a script that can test the scoping behavior case-by-case
If a process substitution is expanded as an argument to a function, expanded to an environment variable during calling of a function, or expanded to any assignment within a function, the process substitution will be \"held open\" for use by any command within the function or its callees, until the function in which it was set returns. If the same variable is set again within a callee, unless the new variable is local, the previous process substitution is closed and will be unavailable to the caller when the callee returns.
In essence, process substitutions expanded to variables within functions remain open until the function in which the process substitution occured returns - even when assigned to locals that were set by a function's caller. Dynamic scope doesn't protect them from closing.
","tags":["bash","shell","scripting","expansion","substitution","text","stdin","stdout","save","capture"]},{"location":"syntax/expansion/proc_subst/#examples","title":"Examples","text":"This code is useless, but it demonstrates how it works:
$ echo <(ls)\n/dev/fd/63\nThe output of the ls-program can then be accessed by reading the file /dev/fd/63.
Consider the following:
diff <(ls \"$first_directory\") <(ls \"$second_directory\")\nThis will compare the contents of each directory. In this command, each process is substituted for a file, and diff doesn't see <(bla), it sees two files, so the effective command is something like
diff /dev/fd/63 /dev/fd/64\nwhere those files are written to and destroyed automatically.
","tags":["bash","shell","scripting","expansion","substitution","text","stdin","stdout","save","capture"]},{"location":"syntax/expansion/proc_subst/#avoiding-subshells","title":"Avoiding subshells","text":"See Also
See Also: BashFAQ/024 -- I set variables in a loop that's in a pipeline. Why do they disappear after the loop terminates? Or, why can't I pipe data to read?
One of the most common uses for process substitutions is to avoid the final subshell that results from executing a pipeline. The following is a wrong piece of code to count all files in /etc is:
counter=0\n\nfind /etc -print0 | while IFS= read -rd '' _; do\n ((counter++))\ndone\n\necho \"$counter files\" # prints \"0 files\"\nDue to the pipe, the while read; do ... done part is executed in a subshell (in Bash, by default), which means counter is only incremented within the subshell. When the pipeline finishes, the subshell is terminated, and the counter visible to echo is still at \"0\"!
Process substitution helps us avoid the pipe operator (the reason for the subshell):
counter=0\n\nwhile IFS= read -rN1 _; do\n ((counter++))\ndone < <(find /etc -printf ' ')\n\necho \"$counter files\"\nThis is the normal input file redirection < FILE, just that the FILE in this case is the result of process substitution. It's important to note that the space is required in order to disambiguate the syntax from here documents.
: < <(COMMAND) # Good.\n: <<(...) # Wrong. Will be parsed as a heredoc. Bash fails when it comes across the unquoted metacharacter ''(''\n: ><(...) # Technically valid but pointless syntax. Bash opens the pipe for writing, while the commands within the process substitution have their stdout connected to the pipe.\nThis example demonstrates how process substitutions can be made to resemble \"passable\" objects. This results in converting the output of f's argument to uppercase.
f() {\n cat \"$1\" >\"$x\"\n}\n\nx=>(tr '[:lower:]' '[:upper:]') f <(echo 'hi there')\nSee the above section on #scope
","tags":["bash","shell","scripting","expansion","substitution","text","stdin","stdout","save","capture"]},{"location":"syntax/expansion/proc_subst/#bugs-and-portability-considerations","title":"Bugs and Portability Considerations","text":"/dev/fd/* method for accessing open files. If the system doesn't support /dev/fd/*, Bash falls back to creating named pipes. Note that not all shells that support process substitution have that fallback.# print \"moo\"\ndev=fd=1 _[1<(echo moo >&2)]=\n# fork bomb\n${dev[${dev='dev[1>(${dev[dev]})]'}]}\n~\n~/...\n\n~NAME\n~NAME/...\n\n~+\n~+/...\n\n~-\n~-/...\nThe tilde expansion is used to expand to several specific pathnames:
Tilde expansion is only performed, when the tilde-construct is at the beginning of a word, or a separate word.
If there's nothing to expand, i.e., in case of a wrong username or any other error condition, the tilde construct is not replaced, it stays what it is.
Tilde expansion is also performed everytime a variable is assigned:
=: TARGET=~moonman/share: (colon) in the assigned value: TARGET=file:~moonman/shareNote
As of now (Bash 4.3-alpha) the following constructs also works, though it's not a variable assignment:
echo foo=~\necho foo=:~\nI don't know yet, if this is a bug or intended.
This way you can correctly use the tilde expansion in your PATH:
PATH=~/mybins:~peter/mybins:$PATH\nSpaces in the referenced pathes? A construct like...
~/\"my directory\"\n...is perfectly valid and works!
","tags":["bash","shell","scripting","expansion","substitution","tilde","home","homedir","shortcut"]},{"location":"syntax/expansion/tilde/#home-directory","title":"Home directory","text":"~\n~<NAME>\nThis form expands to the home-directory of the current user (~) or the home directory of the given user (~<NAME>).
If the given user doesn't exist (or if his home directory isn't determinable, for some reason), it doesn't expand to something else, it stays what it is. The requested home directory is found by asking the operating system for the associated home directory for <NAME>.
To find the home directory of the current user (~), Bash has a precedence:
That means, the variable HOME can override the \"real\" home directory, at least regarding tilde expansion.
~+\nThis expands to the value of the PWD variable, which holds the currect working directory:
echo \"CWD is $PWD\"\nis equivalent to (note it must be a separate word!):
echo \"CWD is\" ~+\n~-\nThis expands to the value of the OLDPWD variable, which holds the previous working directory (the one before the last cd). If OLDPWD is unset (never changed the directory), it is not expanded.
$ pwd\n/home/bash\n$ cd /etc\n$ echo ~-\n/home/bash\nFIXME
to be continued!
Word splitting occurs once any of the following expansions are done (and only then!)
Bash will scan the results of these expansions for special IFS characters that mark word boundaries. This is only done on results that are not double-quoted!
The IFS variable holds the characters that Bash sees as word boundaries in this step. The default contains the characters
These characters are also assumed when IFS is unset. When IFS is empty (nullstring), no word splitting is performed at all.
The results of the expansions mentioned above are scanned for IFS-characters. If one or more (in a sequence) of them is found, the expansion result is split at these positions into multiple words.
This doesn't happen when the expansion results were double-quoted.
When a null-string (e.g., something that before expanded to >>nothing<<) is found, it is removed, unless it is quoted ('' or \"\").
Again note: Without any expansion beforehand, Bash won't perform word splitting! In this case, the initial token parsing is solely responsible.
"},{"location":"syntax/expansion/wordsplit/#see-also","title":"See also","text":"FIXME
work in progress...
","tags":["bash","shell","scripting","syntax","language","behaviour","executing","execution"]},{"location":"syntax/grammar/parser_exec/#parsing-and-execution","title":"Parsing and execution","text":"Nearly everything in Bash grammar can be broken down to a \"simple command\". The only thing Bash has to expand, evaluate and execute is the simple command.
","tags":["bash","shell","scripting","syntax","language","behaviour","executing","execution"]},{"location":"syntax/grammar/parser_exec/#simple-command-expansion","title":"Simple command expansion","text":"info
This step happens after the initial command line splitting.
The expansion of a simple command is done in four steps (interpreting the simple command from left to right):
WORD=WORD= in each variable assignment undergoes tilde expansion, parameter expansion, command substitution, arithmetic expansion, and quote removal before being assigned to the variable.If no command name results after expansion:
> FILE without any command will be performed: the FILE will be created!Otherwise, if a command name results:
The behavior regarding the variable assignment errors can be tested:
This one exits the script completely
#!/bin/sh\n# This shell runs in POSIX mode!\n\necho PRE\n\n# The following is an assignment error, since there is no digit '9'\n# for a base eight number!\nfoo=$((8#9))\n\necho POST\nThis one terminates only the enclosing compound command (the { ...; }):
#!/bin/bash\n# This shell runs in native Bash-mode!\n\necho PRE\n\n# The following is an assignment error!\n# The \"echo TEST\" won't be executed, since the { ...; } is terminated\n{ foo=$((8#9)); echo TEST; }\n\necho POST\nIf a parsed simple command contains no slashes, the shell attempts to locate and execute it:
PATHAs of Bash Version 4, when a command search fails, the shell executes a shell function named command_not_found_handle() using the failed command as arguments. This can be used to provide user friendly messages or install software packages etc. Since this function runs in a separate execution environment, you can't really influence the main shell with it (changing directory, setting variables).
FIXME
to be continued
","tags":["bash","shell","scripting","syntax","language","behaviour","executing","execution"]},{"location":"syntax/grammar/parser_exec/#see-also","title":"See also","text":" coproc [NAME] command [redirections]\nBash 4.0 introduced coprocesses, a feature certainly familiar to ksh users. The coproc keyword starts a command as a background job, setting up pipes connected to both its stdin and stdout so that you can interact with it bidirectionally. Optionally, the co-process can have a name NAME. If NAME is given, the command that follows must be a compound command. If no NAME is given, then the command can be either simple or compound.
The process ID of the shell spawned to execute the coprocess is available through the value of the variable named by NAME followed by a _PID suffix. For example, the variable name used to store the PID of a coproc started with no NAME given would be COPROC_PID (because COPROC is the default NAME). wait may be used to wait for the coprocess to terminate. Additionally, coprocesses may be manipulated through their jobspec.
The return status of a coprocess is the exit status of its command.
"},{"location":"syntax/keywords/coproc/#redirections","title":"Redirections","text":"The optional redirections are applied after the pipes have been set up. Some examples:
# redirecting stderr in the pipe\n$ coproc { ls thisfiledoesntexist; read; } 2>&1\n[2] 23084\n$ IFS= read -ru ${COPROC[0]} x; printf '%s\\n' \"$x\"\nls: cannot access thisfiledoesntexist: No such file or directory\n#let the output of the coprocess go to stdout\n$ { coproc mycoproc { awk '{print \"foo\" $0;fflush()}'; } >&3; } 3>&1\n[2] 23092\n$ echo bar >&${mycoproc[1]}\n$ foobar\nHere we need to save the previous file descriptor of stdout, because by the time we redirect the fds of the coprocess, stdout has already been redirected to the pipe.
"},{"location":"syntax/keywords/coproc/#pitfalls","title":"Pitfalls","text":""},{"location":"syntax/keywords/coproc/#avoid-the-final-pipeline-subshell","title":"Avoid the final pipeline subshell","text":"The traditional Ksh workaround to avoid the subshell when doing command | while read is to use a coprocess. Unfortunately, Bash's behavior differs.
In Ksh you would do:
# ksh93 or mksh/pdksh derivatives\nls |& # start a coprocess\nwhile IFS= read -rp file; do print -r -- \"$file\"; done # read its output\nIn bash:
#DOESN'T WORK\n$ coproc ls\n[1] 23232\n$ while IFS= read -ru ${COPROC[0]} line; do printf '%s\\n' \"$line\"; done\nbash: read: line: invalid file descriptor specification\n[1]+ Done coproc COPROC ls\nBy the time we start reading from the output of the coprocess, the file descriptor has been closed.
See this FAQ entry on Greg's wiki for other pipeline subshell workarounds.
"},{"location":"syntax/keywords/coproc/#buffering","title":"Buffering","text":"In the first example, we GNU awk's fflush() command. As always, when you use pipes the I/O operations are buffered. Let's see what happens with sed:
$ coproc sed s/^/foo/\n[1] 22981\n$ echo bar >&${COPROC[1]}\n$ read -t 3 -ru ${COPROC[0]} _; (( $? > 127 )) && echo \"nothing read\"\nnothing read\nEven though this example is the same as the first awk example, the read doesn't return because the output is waiting in a buffer.
See this faq entry on Greg's wiki for some workarounds and more information on buffering issues.
"},{"location":"syntax/keywords/coproc/#background-processes","title":"background processes","text":"A coprocess' file descriptors are accessible only to the process from which the coproc was started. They are not inherited by subshells.
Here is a not-so-meaningful illustration. Suppose we want to continuously read the output of a coprocess and echo the result:
#NOT WORKING\n$ coproc awk '{print \"foo\" $0;fflush()}'\n[2] 23100\n$ while IFS= read -ru ${COPROC[0]} x; do printf '%s\\n' \"$x\"; done &\n[3] 23104\nbash: line 243: read: 61: invalid file descriptor: Bad file descriptor\nThis fails because the file descriptors created by the parent are not available to the subshell created by &.
A possible workaround:
#WARNING: for illustration purpose ONLY\n# this is not the way to make the coprocess print its output\n# to stdout, see the redirections above.\n$ coproc awk '{print \"foo\" $0;fflush()}'\n[2] 23109\n$ exec 3<&${COPROC[0]}\n$ while IFS= read -ru 3 x; do printf '%s\\n' \"$x\"; done &\n[3] 23110\n$ echo bar >&${COPROC[1]}\n$ foobar\nHere, fd 3 is inherited.
"},{"location":"syntax/keywords/coproc/#examples","title":"Examples","text":""},{"location":"syntax/keywords/coproc/#anonymous-coprocess","title":"Anonymous Coprocess","text":"Unlike ksh, Bash doesn't have true anonymous coprocesses. Instead, Bash assigns FDs to a default array named COPROC if no NAME is supplied. Here's an example:
$ coproc awk '{print \"foo\" $0;fflush()}'\n[1] 22978\nThis command starts in the background, and coproc returns immediately. Two new file descriptors are now available via the COPROC array. We can send data to our command:
$ echo bar >&${COPROC[1]}\nAnd then read its output:
$ IFS= read -ru ${COPROC[0]} x; printf '%s\\n' \"$x\"\nfoobar\nWhen we don't need our command anymore, we can kill it via its pid:
$ kill $COPROC_PID\n$\n[1]+ Terminated coproc COPROC awk '{print \"foo\" $0;fflush()}'\nUsing a named coprocess is simple. We just need a compound command (like when defining a function), and the resulting FDs will be assigned to the indexed array NAME we supply instead.
$ coproc mycoproc { awk '{print \"foo\" $0;fflush()}' ;}\n[1] 23058\n$ echo bar >&${mycoproc[1]}\n$ IFS= read -ru ${mycoproc[0]} x; printf '%s\\n' \"$x\"\nfoobar\n$ kill $mycoproc_PID\n$\n[1]+ Terminated coproc mycoproc { awk '{print \"foo\" $0;fflush()}'; }\n#!/bin/bash\n# we start tee in the background\n# redirecting its output to the stdout of the script\n{ coproc tee { tee logfile ;} >&3 ;} 3>&1\n# we redirect stding and stdout of the script to our coprocess\nexec >&${tee[1]} 2>&1\ncoproc keyword is not specified by POSIX(R)coproc keyword appeared in Bash version 4.0-alpha-p option to Bash's print loadable is a NOOP and not connected to Bash coprocesses in any way. It is only recognized as an option for ksh compatibility, and has no effect.-p option to Bash's read builtin conflicts with that of all kshes and zsh. The equivalent in those shells is to add a \\?prompt suffix to the first variable name argument to read. i.e., if the first variable name given contains a ? character, the remainder of the argument is used as the prompt string. Since this feature is pointless and redundant, I suggest not using it in either shell. Simply precede the read command with a printf %s prompt >&2.ksh93, mksh, zsh, and Bash all support something called \"coprocesses\" which all do approximately the same thing. ksh93 and mksh have virtually identical syntax and semantics for coprocs. A list operator: |& is added to the language which runs the preceding pipeline as a coprocess (This is another reason not to use the special |& pipe operator in Bash -- its syntax is conflicting). The -p option to the read and print builtins can then be used to read and write to the pipe of the coprocess (whose FD isn't yet known). Special redirects are added to move the last spawned coprocess to a different FD: <&p and >&p, at which point it can be accessed at the new FD using ordinary redirection, and another coprocess may then be started, again using |&.
zsh coprocesses are very similar to ksh except in the way they are started. zsh adds the shell reserved word coproc to the pipeline syntax (similar to the way Bash's time keyword works), so that the pipeline that follows is started as a coproc. The coproc's input and output FDs can then be accessed and moved using the same read/print -p and redirects used by the ksh shells.
It is unfortunate that Bash chose to go against existing practice in their coproc implementation, especially considering it was the last of the major shells to incorporate this feature. However, Bash's method accomplishes the same without requiring nearly as much additional syntax. The coproc keyword is easy enough to wrap in a function such that it takes Bash code as an ordinary argument and/or stdin like eval. Coprocess functionality in other shells can be similarly wrapped to create a COPROC array automatically.
The title says it all, complain to the bug-bash mailing list if you want more. See http://lists.gnu.org/archive/html/bug-bash/2011-04/msg00056.html for more details
The ability to use multiple coprocesses in Bash is considered \"experimental\". Bash will throw an error if you attempt to start more than one. This may be overridden at compile-time with the MULTIPLE_COPROCS option. However, at this time there are still issues -- see the above mailing list discussion.
Following is a list of all existing tags:
"},{"location":"tags/#c","title":"C","text":"