github.com/lmorg/murex@v0.0.0-20240217211045-e081c89cd4ef/gen/user-guide/pipeline_doc.yaml (about)

     1  - DocumentID: pipeline
     2    Title: >-
     3      Pipeline
     4    CategoryID: user-guide
     5    Summary: >-
     6      Overview of what a "pipeline" is
     7    Description: |-
     8      ## Description
     9      
    10      In the Murex docs you'll often see the term "pipeline". This refers to any
    11      commands sequenced together.
    12      
    13      A pipeline can be joined via any pipe token (eg `|`, `->`, `=>`, `?`). But,
    14      for the sake of documentation, a pipeline might even be a solitary command.
    15    
    16      ## Examples
    17    
    18      Typical Murex pipeline:
    19  
    20      ```
    21      open example.json -> [[ /node/0 ]]
    22      ```
    23  
    24      Example of a single command pipeline:
    25  
    26      ```
    27      top
    28      ```
    29  
    30      Pipeline you might see in Bash / Zsh (this is also valid in Murex):
    31  
    32      ```
    33      cat names.txt | sort | uniq
    34      ```
    35  
    36      Pipeline filtering out a specific error from `example-cmd`
    37  
    38      ```
    39      example-cmd ? grep "File not found"
    40      ```
    41  
    42      ## Detail
    43  
    44      A pipeline isn't a Murex specific construct but rather something inherited
    45      from Unix. Where Murex differs is that it can support sending typed
    46      information to compatible functions (unlike standard Unix pipes which are
    47      dumb-byte streams).
    48  
    49      Wikipedia has a page on [Pipeline (Unix)](https://en.wikipedia.org/wiki/Pipeline_(Unix)):
    50  
    51      > In Unix-like computer operating systems, a pipeline is a mechanism for
    52      > inter-process communication using message passing. A pipeline is a set of
    53      > processes chained together by their standard streams, so that the output
    54      > text of each process (stdout) is passed directly as input (stdin) to the
    55      > next one. The second process is started as the first process is still
    56      > executing, and they are executed concurrently. The concept of pipelines was
    57      > championed by Douglas McIlroy at Unix's ancestral home of Bell Labs, during
    58      > the development of Unix, shaping its toolbox philosophy. It is named by
    59      > analogy to a physical pipeline. A key feature of these pipelines is their
    60      > "hiding of internals" (Ritchie & Thompson, 1974). This in turn allows for
    61      > more clarity and simplicity in the system. 
    62  
    63      ## Named Pipes
    64  
    65      The drawback with pipes is that it assumes each command runs sequentially one
    66      after another and that everything fits neatly into the concept of "output" and
    67      "errors". The moment you need to use background (`bg`) processes, do anything
    68      more specific with data streams (even if just ignore them entirely), or use
    69      more than one data stream, then this concept breaks down. This is where named
    70      pipes come to the rescue. Named pipes are out of scope for this specific
    71      document but you can read more on them in links the links below.
    72    Related:
    73    - schedulers
    74    - pipe-arrow
    75    - pipe-posix
    76    - pipe-generic
    77    - pipe-err
    78    - bang-prefix
    79    - bg