github.com/lmorg/murex@v0.0.0-20240217211045-e081c89cd4ef/docs/user-guide/pipeline.md (about)

     1  # Pipeline
     2  
     3  > Overview of what a "pipeline" is
     4  
     5  ## Description
     6  
     7  In the Murex docs you'll often see the term "pipeline". This refers to any
     8  commands sequenced together.
     9  
    10  A pipeline can be joined via any pipe token (eg `|`, `->`, `=>`, `?`). But,
    11  for the sake of documentation, a pipeline might even be a solitary command.
    12  
    13  ## Examples
    14  
    15  Typical Murex pipeline:
    16  
    17  ```
    18  open example.json -> [[ /node/0 ]]
    19  ```
    20  
    21  Example of a single command pipeline:
    22  
    23  ```
    24  top
    25  ```
    26  
    27  Pipeline you might see in Bash / Zsh (this is also valid in Murex):
    28  
    29  ```
    30  cat names.txt | sort | uniq
    31  ```
    32  
    33  Pipeline filtering out a specific error from `example-cmd`
    34  
    35  ```
    36  example-cmd ? grep "File not found"
    37  ```
    38  
    39  ## Detail
    40  
    41  A pipeline isn't a Murex specific construct but rather something inherited
    42  from Unix. Where Murex differs is that it can support sending typed
    43  information to compatible functions (unlike standard Unix pipes which are
    44  dumb-byte streams).
    45  
    46  Wikipedia has a page on [Pipeline (Unix)](https://en.wikipedia.org/wiki/Pipeline_(Unix)):
    47  
    48  > In Unix-like computer operating systems, a pipeline is a mechanism for
    49  > inter-process communication using message passing. A pipeline is a set of
    50  > processes chained together by their standard streams, so that the output
    51  > text of each process (stdout) is passed directly as input (stdin) to the
    52  > next one. The second process is started as the first process is still
    53  > executing, and they are executed concurrently. The concept of pipelines was
    54  > championed by Douglas McIlroy at Unix's ancestral home of Bell Labs, during
    55  > the development of Unix, shaping its toolbox philosophy. It is named by
    56  > analogy to a physical pipeline. A key feature of these pipelines is their
    57  > "hiding of internals" (Ritchie & Thompson, 1974). This in turn allows for
    58  > more clarity and simplicity in the system. 
    59  
    60  ## Named Pipes
    61  
    62  The drawback with pipes is that it assumes each command runs sequentially one
    63  after another and that everything fits neatly into the concept of "output" and
    64  "errors". The moment you need to use background (`bg`) processes, do anything
    65  more specific with data streams (even if just ignore them entirely), or use
    66  more than one data stream, then this concept breaks down. This is where named
    67  pipes come to the rescue. Named pipes are out of scope for this specific
    68  document but you can read more on them in links the links below.
    69  
    70  ## See Also
    71  
    72  * [Bang Prefix](../user-guide/bang-prefix.md):
    73    Bang prefixing to reverse default actions
    74  * [Schedulers](../user-guide/schedulers.md):
    75    Overview of the different schedulers (or 'run modes') in Murex
    76  * [`->` Arrow Pipe](../parser/pipe-arrow.md):
    77    Pipes STDOUT from the left hand command to STDIN of the right hand command
    78  * [`=>` Generic Pipe](../parser/pipe-generic.md):
    79    Pipes a reformatted STDOUT stream from the left hand command to STDIN of the right hand command
    80  * [`?` STDERR Pipe](../parser/pipe-err.md):
    81    Pipes STDERR from the left hand command to STDIN of the right hand command (DEPRECATED)
    82  * [`bg`](../commands/bg.md):
    83    Run processes in the background
    84  * [`|` POSIX Pipe](../parser/pipe-posix.md):
    85    Pipes STDOUT from the left hand command to STDIN of the right hand command
    86  
    87  <hr/>
    88  
    89  This document was generated from [gen/user-guide/pipeline_doc.yaml](https://github.com/lmorg/murex/blob/master/gen/user-guide/pipeline_doc.yaml).