github.com/lmorg/murex@v0.0.0-20240217211045-e081c89cd4ef/docs/blog/reading_lists.md (about)

     1  # Reading Lists From The Command Line
     2  
     3  > How hard can it be to read a list of data from the command line? If your list is line delimited then it should be easy. However what if your list is a JSON array? This post will explore how to work with lists in a different command line environments.
     4  
     5  <h2>Table of Contents</h2>
     6  
     7  <div id="toc">
     8  
     9  - [Preface](#preface)
    10  - [Reading lines in Bash and similar shells](#reading-lines-in-bash-and-similar-shells)
    11  - [But what if my files aren't line delimited?](#but-what-if-my-files-arent-line-delimited)
    12  - [Iteration in Bash via `jq`](#iteration-in-bash-via-jq)
    13  - [Iteration in Murex via `foreach`](#iteration-in-murex-via-foreach)
    14  - [Reading JSON arrays in PowerShell](#reading-json-arrays-in-powershell)
    15  - [Conclusion](#conclusion)
    16  
    17  </div>
    18  
    19  ## Preface
    20  
    21  A common problem we resort to shell scripting for is iterating through lists. This was easy in the days of old when most data was `\n` (new line) delimited but these days structured data is common place with formats like JSON, YAML, TOML, XML and even S-Expressions appearing commonly throughout developer and DevOps tooling.
    22  
    23  So lets explore a few techniques for iterating through lists.
    24  
    25  ## Reading lines in Bash and similar shells
    26  
    27  Bash shell is a popular command-line interface for Unix and Linux operating systems. One of its many useful features is the ability to read files line by line. This can be helpful for processing large files or performing repetitive tasks on a file's contents. The most basic way to read a file line by line is to use a while loop with the `read` command:
    28  
    29  ```
    30  while read line; do
    31      echo $line
    32  done < file.txt
    33  ```
    34  
    35  In this example, the `while` loop reads each line of the `file.txt` file and stores it in the `$line` variable. The `echo` command then prints the contents of the `$line` variable to the console. The `<` symbol tells Bash to redirect the contents of the file into the loop.
    36  
    37  The `read` command is what actually reads each line of the file. By default, it reads one line at a time and stores it in the variable specified. You can also use the `-r` option with the `read` command to disable backslash interpretation, which can be useful when dealing with files that contain backslashes.
    38  
    39  Another useful feature of Bash is the ability to perform operations on each line of a file before processing it. For example, you can use `sed` to replace text within each line of a file:
    40  
    41  ```
    42  while read line; do
    43      new_line=$(echo $line | sed 's/foo/bar/g')
    44      echo $new_line
    45  done < file.txt
    46  ```
    47  
    48  In this example, `sed` replaces all instances of "foo" with "bar" in each line of the file. The modified line is then stored in the `$new_line` variable and printed to the console.
    49  
    50  Of course you could just run
    51  
    52  ```
    53  sed 's/foo/bar/g' file.txt
    54  ```
    55  
    56  ...but the reasons for the for this contrived example will follow.
    57  
    58  ## But what if my files aren't line delimited?
    59  
    60  The problem with Bash, and all traditional Linux or UNIX shells, is that they operate on byte streams. To be fair, this isn't so much a fault of Bash _per se_ but more a result of the design of UNIX where (almost) everything is a file, including pipes. This means everything is treated as bytes. Unlike, for example, Powershell which passes .NET objects around. Byte streams make complete sense when you're working on '70s or '80s mainframes but it is a little less productive in the modern world of structured formats like JSON.
    61  
    62  So how do you read lists from objects in, for example, JSON? In Bash, this isn't so easy. You need to rely on third party tools like `jq`. However you do have the benefit of compatibility with all of the older core utilities, like `sed`, that have become muscle memory by now. This does also come with its own drawbacks as well, which I'll explore in the following section.
    63  
    64  ## Iteration in Bash via `jq`
    65  
    66  `jq` is a fantastic tool that has become a staple of many a CI/CD pipeline however it is not part of most operating systems base platform, so it would need to be installed separately. This also creates additional complications whereby you end up having a language within a language -- like running `awk` or `sed` inside Bash, you're now introducing `jq` too. Thus its syntax isn't always the easiest to grok when delving deep into nested JSON with conditionals and such like compared with shells that offer first party tools for working with objects. We can delve deeper into the power of `jq` in another article but for now we are going to keep things intentionally simple:
    67  
    68  Lets create a JSON array:
    69  
    70  ```
    71  json='["Monday","Tuesday","Wednesday","Thursday","Friday","Saturday","Sunday"]'
    72  ```
    73  
    74  Straight away you should be able to see that Bash, with its reliance on whitespace delimitations, couldn't natively parse this. So now lets run it through `jq`:
    75  
    76  ```
    77  $ echo $json | jq -r '.[]' | while read -r day do; echo "Happy $day"; done
    78  Happy Monday
    79  Happy Tuesday
    80  Happy Wednesday
    81  Happy Thursday
    82  Happy Friday
    83  Happy Saturday
    84  Happy Sunday
    85  ```
    86  
    87  What's happening here is the `jq` tool is converting our JSON array into a `\n` delimited list. And from there, we can use `while` and `read` just like we did in our first example at the start of this article.
    88  
    89  The `-r` flag tells `jq` to strip quotation marks around the values. Without `-r` you'd see `Happy "Monday"'` and so on and so forth.
    90  
    91  `.[]` is `jq` syntax for "all elements (`[]`) in the root object space (`.`).
    92  
    93  ## Iteration in Murex via `foreach`
    94  
    95  Murex doesn't just treat files as byte streams, it passes type annotations too. And it uses those annotations to dynamically alter how to read files. The following examples will also use JSON as the input format, however Murex natively supports other structured data formats too, like YAML, CSV and S-Expressions.
    96  
    97  Lets use the same JSON array as we did earlier, except use one of Murex's features to generate arrays programmatically:
    98  
    99  ```
   100  ยป %[Monday..Sunday]
   101  [
   102      "Monday",
   103      "Tuesday",
   104      "Wednesday",
   105      "Thursday",
   106      "Friday",
   107      "Saturday",
   108      "Sunday"
   109  ]
   110  ```
   111  
   112  The `jq` example rewritten in Murex would look like the following:
   113  
   114  ```
   115  %[Monday..Sunday] | foreach day { out "Happy $day" }
   116  ```
   117  
   118  What's happening here is `%[...]` creates the JSON array (as described above) and then the `foreach` builtin iterates through the array and assigns that element to a variable named `day`.
   119  
   120  > `out` in Murex is the equivalent of `echo` in Bash. In fact you can still use `echo` in Murex albeit that is just aliased to `out`.
   121  
   122  ## Reading JSON arrays in PowerShell
   123  
   124  Microsoft PowerShell is a typed shell, like Murex, which was originally built for Windows but has since been ported to macOS and Linux too. Where PowerShell differs is that rather than using byte streams with type annotations, PowerShell passes .NET objects. Thus you'll see a little more boilerplate code in PowerShell where you need to explicitly convert types -- whereas Murex can get away with implicit definitions.
   125  
   126  ```
   127  $jsonString = '["Monday","Tuesday","Wednesday","Thursday","Friday","Saturday","Sunday"]'
   128  $jsonObject = ConvertFrom-Json $jsonString
   129  foreach ($day in $jsonObject) { Write-Host "Hello $day" } 
   130  ```
   131  
   132  The first line is just creating a JSON string and allocating that to `$jsonString`. We can largely ignore that as it is the same as we've seen in Bash already. The second line is more interesting as that is where the type conversion happens. `ConvertFrom-Json` does exactly as it describes -- PowerShell is generally pretty good at having descriptive names for commands,
   133  
   134  From there on it looks relatively similar to Murex syntax: `foreach` being the statement name, followed by the variable name.
   135  
   136  ## Conclusion
   137  
   138  Iterating over a JSON array from the command line can be done in various ways using different shells. PowerShell, `jq`, and Murex offer their unique approaches and syntaxes, making it easy to work with JSON data in different environments. Whether you're a Windows user who prefers PowerShell or a Linux user who prefers Bash or Murex, there are many options available to suit your needs. Regardless of the shell you choose, mastering the art of iterating over JSON arrays can greatly enhance your command-line skills and help you work more efficiently with JSON data.
   139  
   140  <hr>
   141  
   142  Published: 22.04.2023 at 11:43
   143  
   144  ## See Also
   145  
   146  * [`%[]` Create Array](../parser/create-array.md):
   147    Quickly generate arrays
   148  * [`a` (mkarray)](../commands/a.md):
   149    A sophisticated yet simple way to build an array or list
   150  * [`cast`](../commands/cast.md):
   151    Alters the data type of the previous function without altering it's output
   152  * [`foreach`](../commands/foreach.md):
   153    Iterate through an array
   154  * [`formap`](../commands/formap.md):
   155    Iterate through a map or other collection of data
   156  * [`out`](../commands/out.md):
   157    Print a string to the STDOUT with a trailing new line character
   158  
   159  <hr/>
   160  
   161  This document was generated from [gen/blog/reading_lists_doc.yaml](https://github.com/lmorg/murex/blob/master/gen/blog/reading_lists_doc.yaml).