github.com/lmorg/murex@v0.0.0-20240217211045-e081c89cd4ef/gen/user-guide/pipeline_doc.yaml (about) 1 - DocumentID: pipeline 2 Title: >- 3 Pipeline 4 CategoryID: user-guide 5 Summary: >- 6 Overview of what a "pipeline" is 7 Description: |- 8 ## Description 9 10 In the Murex docs you'll often see the term "pipeline". This refers to any 11 commands sequenced together. 12 13 A pipeline can be joined via any pipe token (eg `|`, `->`, `=>`, `?`). But, 14 for the sake of documentation, a pipeline might even be a solitary command. 15 16 ## Examples 17 18 Typical Murex pipeline: 19 20 ``` 21 open example.json -> [[ /node/0 ]] 22 ``` 23 24 Example of a single command pipeline: 25 26 ``` 27 top 28 ``` 29 30 Pipeline you might see in Bash / Zsh (this is also valid in Murex): 31 32 ``` 33 cat names.txt | sort | uniq 34 ``` 35 36 Pipeline filtering out a specific error from `example-cmd` 37 38 ``` 39 example-cmd ? grep "File not found" 40 ``` 41 42 ## Detail 43 44 A pipeline isn't a Murex specific construct but rather something inherited 45 from Unix. Where Murex differs is that it can support sending typed 46 information to compatible functions (unlike standard Unix pipes which are 47 dumb-byte streams). 48 49 Wikipedia has a page on [Pipeline (Unix)](https://en.wikipedia.org/wiki/Pipeline_(Unix)): 50 51 > In Unix-like computer operating systems, a pipeline is a mechanism for 52 > inter-process communication using message passing. A pipeline is a set of 53 > processes chained together by their standard streams, so that the output 54 > text of each process (stdout) is passed directly as input (stdin) to the 55 > next one. The second process is started as the first process is still 56 > executing, and they are executed concurrently. The concept of pipelines was 57 > championed by Douglas McIlroy at Unix's ancestral home of Bell Labs, during 58 > the development of Unix, shaping its toolbox philosophy. It is named by 59 > analogy to a physical pipeline. A key feature of these pipelines is their 60 > "hiding of internals" (Ritchie & Thompson, 1974). This in turn allows for 61 > more clarity and simplicity in the system. 62 63 ## Named Pipes 64 65 The drawback with pipes is that it assumes each command runs sequentially one 66 after another and that everything fits neatly into the concept of "output" and 67 "errors". The moment you need to use background (`bg`) processes, do anything 68 more specific with data streams (even if just ignore them entirely), or use 69 more than one data stream, then this concept breaks down. This is where named 70 pipes come to the rescue. Named pipes are out of scope for this specific 71 document but you can read more on them in links the links below. 72 Related: 73 - schedulers 74 - pipe-arrow 75 - pipe-posix 76 - pipe-generic 77 - pipe-err 78 - bang-prefix 79 - bg