mirror of
https://github.com/flokoe/bash-hackers-wiki.git
synced 2024-11-01 14:53:06 +01:00
62 lines
57 KiB
HTML
62 lines
57 KiB
HTML
|
<!doctype html><html lang=en class=no-js> <head><meta charset=utf-8><meta name=viewport content="width=device-width,initial-scale=1"><link href=https://flokoe.github.io/bash-hackers-wiki/syntax/expansion/proc_subst/ rel=canonical><link href=../intro/ rel=prev><link href=../tilde/ rel=next><link rel=icon href=../../../assets/images/favicon.png><meta name=generator content="mkdocs-1.6.1, mkdocs-material-9.5.40"><title>Process substitution - The Bash Hackers Wiki</title><link rel=stylesheet href=../../../assets/stylesheets/main.8c3ca2c6.min.css><link rel=stylesheet href=../../../assets/stylesheets/palette.06af60db.min.css><link rel=preconnect href=https://fonts.gstatic.com crossorigin><link rel=stylesheet href="https://fonts.googleapis.com/css?family=Roboto:300,300i,400,400i,700,700i%7CRoboto+Mono:400,400i,700,700i&display=fallback"><style>:root{--md-text-font:"Roboto";--md-code-font:"Roboto Mono"}</style><script>__md_scope=new URL("../../..",location),__md_hash=e=>[...e].reduce(((e,_)=>(e<<5)-e+_.charCodeAt(0)),0),__md_get=(e,_=localStorage,t=__md_scope)=>JSON.parse(_.getItem(t.pathname+"."+e)),__md_set=(e,_,t=localStorage,a=__md_scope)=>{try{t.setItem(a.pathname+"."+e,JSON.stringify(_))}catch(e){}}</script></head> <body dir=ltr data-md-color-scheme=default data-md-color-primary=indigo data-md-color-accent=indigo> <input class=md-toggle data-md-toggle=drawer type=checkbox id=__drawer autocomplete=off> <input class=md-toggle data-md-toggle=search type=checkbox id=__search autocomplete=off> <label class=md-overlay for=__drawer></label> <div data-md-component=skip> <a href=#process-substitution class=md-skip> Skip to content </a> </div> <div data-md-component=announce> </div> <header class=md-header data-md-component=header> <nav class="md-header__inner md-grid" aria-label=Header> <a href=../../.. title="The Bash Hackers Wiki" class="md-header__button md-logo" aria-label="The Bash Hackers Wiki" data-md-component=logo> <svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M12 8a3 3 0 0 0 3-3 3 3 0 0 0-3-3 3 3 0 0 0-3 3 3 3 0 0 0 3 3m0 3.54C9.64 9.35 6.5 8 3 8v11c3.5 0 6.64 1.35 9 3.54 2.36-2.19 5.5-3.54 9-3.54V8c-3.5 0-6.64 1.35-9 3.54"/></svg> </a> <label class="md-header__button md-icon" for=__drawer> <svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M3 6h18v2H3zm0 5h18v2H3zm0 5h18v2H3z"/></svg> </label> <div class=md-header__title data-md-component=header-title> <div class=md-header__ellipsis> <div class=md-header__topic> <span class=md-ellipsis> The Bash Hackers Wiki </span> </div> <div class=md-header__topic data-md-component=header-topic> <span class=md-ellipsis> Process substitution </span> </div> </div> </div> <form class=md-header__option data-md-component=palette> <input class=md-option data-md-color-media data-md-color-scheme=default data-md-color-primary=indigo data-md-color-accent=indigo aria-label="Switch to dark mode" type=radio name=__palette id=__palette_0> <label class="md-header__button md-icon" title="Switch to dark mode" for=__palette_1 hidden> <svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M12 8a4 4 0 0 0-4 4 4 4 0 0 0 4 4 4 4 0 0 0 4-4 4 4 0 0 0-4-4m0 10a6 6 0 0 1-6-6 6 6 0 0 1 6-6 6 6 0 0 1 6 6 6 6 0 0 1-6 6m8-9.31V4h-4.69L12 .69 8.69 4H4v4.69L.69 12 4 15.31V20h4.69L12 23.31 15.31 20H20v-4.69L23.31 12z"/></svg> </label> <input class=md-option data-md-color-media data-md-color-scheme=slate data-md-color-primary=indigo data-md-color-accent=indigo aria-label="Switch to light mode" type=radio name=__palette id=__palette_1> <label class="md-header__button md-icon" title="Switch to light mode" for=__palette_0 hidden> <svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M12 18c-.89 0-1.74-.2-2.5-.55C11.56 16.5 13 14.42 13 12s-1.44-4.5-3.5-5.45C10.26 6.2 11.11 6 12 6a6 6 0 0 1 6 6 6 6 0 0 1-6 6m8-9.31V4h-4.69L12 .69 8.69 4H4v4.69L.69 12 4 15.31V20h4.69L12 23.31 15.31 20H20v-4.69L23.31 12z"/></svg> </label> </form> <script>var palette=__md_get("__palette");if(palette&&palette.color){if("(prefers-color-scheme)"===palette.color.media){var media=match
|
||
|
|
||
|
>( <LIST> )
|
||
|
</code></pre></div> <p>Process substitution is performed <strong>simultaneously</strong> with <a href=../../pe/ >parameter expansion</a>, <a href=../cmdsubst/ >command substitution</a> and <a href=../arith/ >arithmetic expansion</a>.</p> <p>The <a href=../../basicgrammar/#lists>command list</a> <code><LIST></code> is executed and its</p> <ul> <li>standard output filedescriptor in the <code><( ... )</code> form or</li> <li>standard input filedescriptor in the <code>>( ... )</code> form</li> </ul> <p>is connected to a FIFO or a file in <code>/dev/fd/</code>. The filename (where the filedescriptor is connected) is then used as a substitution for the <code><(...)</code>-construct.</p> <p>That, for example, allows to give data to a command that can't be reached by pipelining (that doesn't expect its data from <code>stdin</code> but from a file).</p> <h3 id=scope>Scope<a class=headerlink href=#scope title="Permanent link">¶</a></h3> <div class="admonition note"> <p class=admonition-title>Note</p> <p>According to multiple comments and sources, the scope of process substitution file descriptors is <strong>not</strong> stable, guaranteed, or specified by bash. Newer versions of bash (5.0+) seem to have shorter scope, and substitutions scope seems to be shorter than function scope. See <a href=https://unix.stackexchange.com/questions/425456/conditional-process-substitution>stackexchange</a> and <a href=https://stackoverflow.com/questions/46660020/bash-what-is-the-scope-of-the-process-substitution>stackoverflow</a>; the latter discussion contains a script that can test the scoping behavior case-by-case</p> </div> <p>If a process substitution is expanded as an argument to a function, expanded to an environment variable during calling of a function, or expanded to any assignment within a function, the process substitution will be "held open" for use by any command within the function or its callees, until the function in which it was set returns. If the same variable is set again within a callee, unless the new variable is local, the previous process substitution is closed and will be unavailable to the caller when the callee returns.</p> <p>In essence, process substitutions expanded to variables within functions remain open until the function in which the process substitution occured returns - even when assigned to locals that were set by a function's caller. Dynamic scope doesn't protect them from closing.</p> <h2 id=examples>Examples<a class=headerlink href=#examples title="Permanent link">¶</a></h2> <p>This code is useless, but it demonstrates how it works:</p> <div class=highlight><pre><span></span><code><a id=__codelineno-0-1 name=__codelineno-0-1 href=#__codelineno-0-1></a>$<span class=w> </span><span class=nb>echo</span><span class=w> </span><<span class=o>(</span>ls<span class=o>)</span>
|
||
|
<a id=__codelineno-0-2 name=__codelineno-0-2 href=#__codelineno-0-2></a>/dev/fd/63
|
||
|
</code></pre></div> <p>The <strong>output</strong> of the <code>ls</code>-program can then be accessed by reading the file <code>/dev/fd/63</code>.</p> <p>Consider the following:</p> <div class=highlight><pre><span></span><code><a id=__codelineno-1-1 name=__codelineno-1-1 href=#__codelineno-1-1></a>diff<span class=w> </span><<span class=o>(</span>ls<span class=w> </span><span class=s2>"</span><span class=nv>$first_directory</span><span class=s2>"</span><span class=o>)</span><span class=w> </span><<span class=o>(</span>ls<span class=w> </span><span class=s2>"</span><span class=nv>$second_directory</span><span class=s2>"</span><span class=o>)</span>
|
||
|
</code></pre></div> <p>This will compare the contents of each directory. In this command, each <em>process</em> is <em>substituted</em> for a <em>file</em>, and diff doesn't see <(bla), it sees two files, so the effective command is something like</p> <div class=highlight><pre><span></span><code><a id=__codelineno-2-1 name=__codelineno-2-1 href=#__codelineno-2-1></a>diff<span class=w> </span>/dev/fd/63<span class=w> </span>/dev/fd/64
|
||
|
</code></pre></div> <p>where those files are written to and destroyed automatically.</p> <h3 id=avoiding-subshells>Avoiding subshells<a class=headerlink href=#avoiding-subshells title="Permanent link">¶</a></h3> <div class="admonition info"> <p class=admonition-title>See Also</p> <p>See Also: <a href=http://mywiki.wooledge.org/BashFAQ/024>BashFAQ/024</a> -- <em>I set variables in a loop that's in a pipeline. Why do they disappear after the loop terminates? Or, why can't I pipe data to read?</em></p> </div> <p>One of the most common uses for process substitutions is to avoid the final subshell that results from executing a pipeline. The following is a <strong>wrong</strong> piece of code to count all files in <code>/etc</code> is:</p> <div class=highlight><pre><span></span><code><a id=__codelineno-3-1 name=__codelineno-3-1 href=#__codelineno-3-1></a><span class=nv>counter</span><span class=o>=</span><span class=m>0</span>
|
||
|
<a id=__codelineno-3-2 name=__codelineno-3-2 href=#__codelineno-3-2></a>
|
||
|
<a id=__codelineno-3-3 name=__codelineno-3-3 href=#__codelineno-3-3></a>find<span class=w> </span>/etc<span class=w> </span>-print0<span class=w> </span><span class=p>|</span><span class=w> </span><span class=k>while</span><span class=w> </span><span class=nv>IFS</span><span class=o>=</span><span class=w> </span><span class=nb>read</span><span class=w> </span>-rd<span class=w> </span><span class=s1>''</span><span class=w> </span>_<span class=p>;</span><span class=w> </span><span class=k>do</span>
|
||
|
<a id=__codelineno-3-4 name=__codelineno-3-4 href=#__codelineno-3-4></a><span class=w> </span><span class=o>((</span>counter++<span class=o>))</span>
|
||
|
<a id=__codelineno-3-5 name=__codelineno-3-5 href=#__codelineno-3-5></a><span class=k>done</span>
|
||
|
<a id=__codelineno-3-6 name=__codelineno-3-6 href=#__codelineno-3-6></a>
|
||
|
<a id=__codelineno-3-7 name=__codelineno-3-7 href=#__codelineno-3-7></a><span class=nb>echo</span><span class=w> </span><span class=s2>"</span><span class=nv>$counter</span><span class=s2> files"</span><span class=w> </span><span class=c1># prints "0 files"</span>
|
||
|
</code></pre></div> <p>Due to the pipe, the <code>while read; do ... done</code> part is executed in a subshell (in Bash, by default), which means <code>counter</code> is only incremented within the subshell. When the pipeline finishes, the subshell is terminated, and the <code>counter</code> visible to <code>echo</code> is still at "0"!</p> <p>Process substitution helps us avoid the pipe operator (the reason for the subshell):</p> <div class=highlight><pre><span></span><code><a id=__codelineno-4-1 name=__codelineno-4-1 href=#__codelineno-4-1></a><span class=nv>counter</span><span class=o>=</span><span class=m>0</span>
|
||
|
<a id=__codelineno-4-2 name=__codelineno-4-2 href=#__codelineno-4-2></a>
|
||
|
<a id=__codelineno-4-3 name=__codelineno-4-3 href=#__codelineno-4-3></a><span class=k>while</span><span class=w> </span><span class=nv>IFS</span><span class=o>=</span><span class=w> </span><span class=nb>read</span><span class=w> </span>-rN1<span class=w> </span>_<span class=p>;</span><span class=w> </span><span class=k>do</span>
|
||
|
<a id=__codelineno-4-4 name=__codelineno-4-4 href=#__codelineno-4-4></a><span class=w> </span><span class=o>((</span>counter++<span class=o>))</span>
|
||
|
<a id=__codelineno-4-5 name=__codelineno-4-5 href=#__codelineno-4-5></a><span class=k>done</span><span class=w> </span><<span class=w> </span><<span class=o>(</span>find<span class=w> </span>/etc<span class=w> </span>-printf<span class=w> </span><span class=s1>' '</span><span class=o>)</span>
|
||
|
<a id=__codelineno-4-6 name=__codelineno-4-6 href=#__codelineno-4-6></a>
|
||
|
<a id=__codelineno-4-7 name=__codelineno-4-7 href=#__codelineno-4-7></a><span class=nb>echo</span><span class=w> </span><span class=s2>"</span><span class=nv>$counter</span><span class=s2> files"</span>
|
||
|
</code></pre></div> <p>This is the normal input file redirection <code>< FILE</code>, just that the <code>FILE</code> in this case is the result of process substitution. It's important to note that the space is required in order to disambiguate the syntax from <a href=../../redirection/#here_documents>here documents</a>.</p> <div class=highlight><pre><span></span><code><a id=__codelineno-5-1 name=__codelineno-5-1 href=#__codelineno-5-1></a>:<span class=w> </span><<span class=w> </span><<span class=o>(</span>COMMAND<span class=o>)</span><span class=w> </span><span class=c1># Good.</span>
|
||
|
<a id=__codelineno-5-2 name=__codelineno-5-2 href=#__codelineno-5-2></a>:<span class=w> </span><<<span class=o>(</span>...<span class=o>)</span><span class=w> </span><span class=c1># Wrong. Will be parsed as a heredoc. Bash fails when it comes across the unquoted metacharacter ''(''</span>
|
||
|
<a id=__codelineno-5-3 name=__codelineno-5-3 href=#__codelineno-5-3></a>:<span class=w> </span>><<span class=o>(</span>...<span class=o>)</span><span class=w> </span><span class=c1># Technically valid but pointless syntax. Bash opens the pipe for writing, while the commands within the process substitution have their stdout connected to the pipe.</span>
|
||
|
</code></pre></div> <h3 id=process-substitution-assigned-to-a-parameter>Process substitution assigned to a parameter<a class=headerlink href=#process-substitution-assigned-to-a-parameter title="Permanent link">¶</a></h3> <p>This example demonstrates how process substitutions can be made to resemble "passable" objects. This results in converting the output of <code>f</code>'s argument to uppercase.</p> <div class=highlight><pre><span></span><code><a id=__codelineno-6-1 name=__codelineno-6-1 href=#__codelineno-6-1></a>f<span class=o>()</span><span class=w> </span><span class=o>{</span>
|
||
|
<a id=__codelineno-6-2 name=__codelineno-6-2 href=#__codelineno-6-2></a><span class=w> </span>cat<span class=w> </span><span class=s2>"</span><span class=nv>$1</span><span class=s2>"</span><span class=w> </span>><span class=s2>"</span><span class=nv>$x</span><span class=s2>"</span>
|
||
|
<a id=__codelineno-6-3 name=__codelineno-6-3 href=#__codelineno-6-3></a><span class=o>}</span>
|
||
|
<a id=__codelineno-6-4 name=__codelineno-6-4 href=#__codelineno-6-4></a>
|
||
|
<a id=__codelineno-6-5 name=__codelineno-6-5 href=#__codelineno-6-5></a><span class=nv>x</span><span class=o>=</span>><span class=o>(</span>tr<span class=w> </span><span class=s1>'[:lower:]'</span><span class=w> </span><span class=s1>'[:upper:]'</span><span class=o>)</span><span class=w> </span>f<span class=w> </span><<span class=o>(</span><span class=nb>echo</span><span class=w> </span><span class=s1>'hi there'</span><span class=o>)</span>
|
||
|
</code></pre></div> <p>See the above section on <a href=#scope>#scope</a></p> <h2 id=bugs-and-portability-considerations>Bugs and Portability Considerations<a class=headerlink href=#bugs-and-portability-considerations title="Permanent link">¶</a></h2> <ul> <li>Process substitution is not specified by POSIX.</li> <li>Process substitution is disabled completely in Bash POSIX mode.</li> <li>Process substitution is implemented by Bash, Zsh, Ksh{88,93}, but not (yet) pdksh derivatives (mksh). Coprocesses may be used instead.</li> <li>Process substitution is supported only on systems that support either named pipes (FIFO - a <a href=../../../dict/special_file/ >special file</a>) or the <code>/dev/fd/*</code> method for accessing open files. If the system doesn't support <code>/dev/fd/*</code>, Bash falls back to creating named pipes. Note that not all shells that support process substitution have that fallback.</li> <li>Bash evaluates process substitutions within array indices, but not other arithmetic contexts. Ksh and Zsh do not. (Possible Bug)</li> </ul> <div class=highlight><pre><span></span><code><a id=__codelineno-7-1 name=__codelineno-7-1 href=#__codelineno-7-1></a><span class=c1># print "moo"</span>
|
||
|
<a id=__codelineno-7-2 name=__codelineno-7-2 href=#__codelineno-7-2></a><span class=nv>dev</span><span class=o>=</span><span class=nv>fd</span><span class=o>=</span><span class=m>1</span><span class=w> </span>_<span class=o>[</span><span class=m>1</span><<span class=o>(</span><span class=nb>echo</span><span class=w> </span>moo<span class=w> </span>><span class=p>&</span><span class=m>2</span><span class=o>)]=</span>
|
||
|
<a id=__codelineno-7-3 name=__codelineno-7-3 href=#__codelineno-7-3></a><span class=c1># fork bomb</span>
|
||
|
<a id=__codelineno-7-4 name=__codelineno-7-4 href=#__codelineno-7-4></a><span class=si>${</span><span class=nv>dev</span><span class=p>[</span><span class=si>${</span><span class=nv>dev</span><span class=p>=</span><span class=s1>'dev[1>(${dev[dev]})]'</span><span class=si>}</span><span class=p>]</span><span class=si>}</span>
|
||
|
</code></pre></div> <ul> <li>Issues with wait, race conditions, etc: <a href="https://groups.google.com/forum/?fromgroups=#!topic/comp.unix.shell/GqLNzUA4ulA">https://groups.google.com/forum/?fromgroups=#!topic/comp.unix.shell/GqLNzUA4ulA</a></li> </ul> <h2 id=see-also>See also<a class=headerlink href=#see-also title="Permanent link">¶</a></h2> <ul> <li>Internal: <a href=../intro/ >Introduction to expansion and substitution</a></li> <li>Internal: <a href=../../../scripting/processtree/ >Bash in the process tree</a> (subshells)</li> <li>Internal: <a href=../../redirection/ >Redirection</a></li> </ul> <aside class=md-source-file> <span class=md-source-file__fact> <span class=md-icon title="Last update"> <svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M21 13.1c-.1 0-.3.1-.4.2l-1 1 2.1 2.1 1-1c.2-.2.2-.6 0-.8l-1.3-1.3c-.1-.1-.2-.2-.4-.2m-1.9 1.8-6.1 6V23h2.1l6.1-6.1zM12.5 7v5.2l4 2.4-1 1L11 13V7zM11 21.9c-5.1-.5-9-4.8-9-9.9C2 6.5 6.5 2 12 2c5.3 0 9.6 4.1 10 9.3-.3-.1-.6-.2-1-.2s-.7.1-1 .2C19.6 7.2 16.2 4 12 4c-4.4 0-8 3.6-8 8 0 4.1 3.1 7.5 7.1 7.9l-.1.2z"/></svg> </span> <span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">October 12, 2024</span> </span> <span class=md-source-file__fact> <span class=md-icon title=Created> <svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M14.47 15.08 11 13V7h1.5v5.25l3.08 1.83c-.41.28-.79.62-1.11 1m-1.39 4.84c-.36.05-.71.08-1.08.08-4.42 0-8-3.58-8-8s3.58-8 8-8 8 3.58 8 8c0 .37-.03.72-.08 1.08.69.1 1.33.32 1.92.64.1-.56.16-1.13.16-1.72 0-5.5-4.5-10-10-10S2 6.5 2 12s4.47 10 10 10c.59 0 1.16-.06 1.72-.16-.32-.59-.54-1.23-.64-1.92M18 15v3h-3v2h3v3h2v-3h3v-2h-3v-3z"/></svg> </span> <span class="git-revision-date-localized-plugin git-revision-date-localized-plugin-date">October 12, 2024</span> </span> </aside> <h2 id=__comments>Comments</h2> <script src=https://giscus.app/client.js data-repo=flokoe/bash-hackers-wiki data-repo-id=R_kgDOJ3Nr6Q data-category="Giscus Page Comments" data-category-id=DIC_kwDOJ3Nr6c4CXq9t data-mapping=pathname data-strict=1 data-reactions-enabled=1 data-emit-metadata=0 data-input-position=top data-theme=preferred_color_scheme data-lang=en data-loading=lazy crossorigin=anonymous async>
|
||
|
</script> <script>
|
||
|
var giscus = document.querySelector("script[src*=giscus]")
|
||
|
|
||
|
/* Set palette on initial load */
|
||
|
var palette = __md_get("__palette")
|
||
|
if (palette && typeof palette.color === "object") {
|
||
|
var theme = palette.color.scheme === "slate" ? "dark" : "light"
|
||
|
giscus.setAttribute("data-theme", theme)
|
||
|
}
|
||
|
|
||
|
/* Register event handlers after documented loaded */
|
||
|
document.addEventListener("DOMContentLoaded", function() {
|
||
|
var ref = document.querySelector("[data-md-component=palette]")
|
||
|
ref.addEventListener("change", function() {
|
||
|
var palette = __md_get("__palette")
|
||
|
if (palette && typeof palette.color === "object") {
|
||
|
var theme = palette.color.scheme === "slate" ? "dark" : "light"
|
||
|
|
||
|
/* Instruct Giscus to change theme */
|
||
|
var frame = document.querySelector(".giscus-frame")
|
||
|
frame.contentWindow.postMessage(
|
||
|
{ giscus: { setConfig: { theme } } },
|
||
|
"https://giscus.app"
|
||
|
)
|
||
|
}
|
||
|
})
|
||
|
})
|
||
|
</script> </article> </div> <script>var target=document.getElementById(location.hash.slice(1));target&&target.name&&(target.checked=target.name.startsWith("__tabbed_"))</script> </div> <button type=button class="md-top md-icon" data-md-component=top hidden> <svg xmlns=http://www.w3.org/2000/svg viewbox="0 0 24 24"><path d="M13 20h-2V8l-5.5 5.5-1.42-1.42L12 4.16l7.92 7.92-1.42 1.42L13 8z"/></svg> Back to top </button> </main> <footer class=md-footer> <div class="md-footer-meta md-typeset"> <div class="md-footer-meta__inner md-grid"> <div class=md-copyright> Made with <a href=https://squidfunk.github.io/mkdocs-material/ target=_blank rel=noopener> Material for MkDocs </a> </div> </div> </div> </footer> </div> <div class=md-dialog data-md-component=dialog> <div class="md-dialog__inner md-typeset"></div> </div> <script id=__config type=application/json>{"base": "../../..", "features": ["navigation.instant", "navigation.tracking", "navigation.tabs", "navigation.sections", "navigation.top", "content.action.view", "content.action.edit", "search.suggest", "search.highlight", "content.code.copy"], "search": "../../../assets/javascripts/workers/search.6ce7567c.min.js", "translations": {"clipboard.copied": "Copied to clipboard", "clipboard.copy": "Copy to clipboard", "search.result.more.one": "1 more on this page", "search.result.more.other": "# more on this page", "search.result.none": "No matching documents", "search.result.one": "1 matching document", "search.result.other": "# matching documents", "search.result.placeholder": "Type to start searching", "search.result.term.missing": "Missing", "select.version": "Select version"}}</script> <script src=../../../assets/javascripts/bundle.525ec568.min.js></script> </body> </html>
|