github.com/lmorg/murex@v0.0.0-20240217211045-e081c89cd4ef/docs/blog/split_personalities.md (about) 1 # The Split Personalities of Shell Usage 2 3 > Shell usage is split between the need to write something quickly and frequently verses the need to write something more complex but only the once. In this article is explore those opposing use cases and how different $SHELLs have chosen to address them. 4 5 ## A Very Brief History 6 7  8 9 In the very early days of UNIX you had the Thompson shell which supported 10 pipes, some basic control structures and wildcards. Thompson shell was based 11 after the Multics shell, which in turn was inspired from `RUNCOM`. In fact the 12 'rc' extension often seen in shell profiles is directly taken from `RUNCOM`. 13 14 It wasn't until a little later that variables were a feature in shells. That 15 came with the PWB shell, which was designed to be upwardly-compatible with the 16 Thompson shell, supporting Thompson syntax while bringing advancements intended 17 to make shell scripting much more practical. 18 19 While the inspiration behind modern shells, `RUNCOM`, is a program that 20 literally just ran commands from a file; it is this authors opinion that early 21 UNIX shells were originally designed to be interactive terminals for launching 22 applications first and foremost, with scripting as a feature that took a few 23 years to mature. Furthermore, the ALGOL-inspired scripting commands were 24 originally external executables and only later rewritten as shell builtins for 25 performance reasons. For example running `if` in the shell would originally 26 `fork()` the executable `/bin/if` but that quickly became call a builtin 27 function that was part of the shell itself. 28 29 I believe it is these reasons why $SHELLs based on that lineage, be it the 30 Bourne shell, Bash or Zsh, all share a scripting syntax which very much feels 31 like it is extended from REPL usage. 32 33 ## Opposing Requirements 34 35  36 37 The problem with shell usage is it falls into two contradictory categories 38 equally: 39 40 1. You need an interactive terminal that is optimized for the operators 41 productivity. Since it is a REPL environment, any instructions you do pass 42 are going to be write-many read-once. In other words, the syntax needs to be 43 quick to type because it's going to be typed often. However it doesn't have 44 to be particularly readable because you're not going to save and read back 45 whatever instructions you've keyed into the REPL. 46 47 2. You need the ability to write short scripts. The language here needs to be 48 familiar because it is aimed at non-developers (otherwise they might just as 49 well use C, FORTRAN, ALGOL or others) and succinct (again, otherwise a 50 developer might as well use a compiled language). However it also should be 51 readable because scripts are saved, recalled, reused and often extended over 52 time. So they fall into the write-once read-many category. 53 54 In an interactive program manager it makes sense to forgo quotation marks 55 around strings, commas to separate parameters and semi-colons to terminate the 56 line. Even the C shell, `csh` then later `tcsh`, doesn't follow C's syntax that 57 strictly -- instead understanding that brevity is required for interactive use. 58 59 When I first started writing my own shell, Murex, I originally started out 60 with syntax that was inspired by the C. A pipeline would look something like 61 the following: 62 63 ``` 64 cat ("./example.csv") | grep ("-n", "foobar") 65 ``` 66 67 While this came with some readability improvements, it was a _massive_ pain to 68 write over and over. So I added some syntax completion to the terminal, 69 inspired by IDE's and how they attempt to minimize the repetition of entering 70 syntax tokens. However this didn't remove the pain entirely, it just masked it 71 a little. So I removed the redundant braces. But the enforced quotation marks 72 were still annoying, so I decided to make the quotation marks optional. Then 73 the commas were removed...and before I knew it, I'd basically just reinvented 74 the same syntax for writing commands as everyone had already been using for a 75 multitude of decades prior. What started out as the example above ended up 76 looking more like the example below: 77 78 ``` 79 cat ./example.csv | grep -n foobar 80 ``` 81 82 (please excuse the useless use of `cat` in these examples -- it's purely there 83 for illustrative reasons) 84 85 ## The Traditional 86 87  88 89 As I've already hinted in the section before, Bourne, Bash, Zsh all fall nicely 90 into the first camp. The write-many read-once camp. And that makes sense to me 91 when I consider the evolution of those shells. Their heritage does stem from 92 interactive terminals firstly and scripting secondly. 93 94 The problem with traditional shells is that their grammar is lousy for anyone 95 who needs a write-once read-many language. Worse still, while a significant 96 amount of their grammar has now been included as builtins, for practical use 97 operators often find themselves inlining other languages anyway, such as awk, 98 sed, Perl and others. So it is understandable that a great many chose to do 99 away with traditional shells for scripting and instead use more other, more 100 powerful and readable languages like Python. 101 102 Unfortunately the same problems transfer the other way too, in that I have 103 already demonstrated why Python (and other programming languages) don't always 104 make good shells. While I will conceded that there is a loyal fanbase who will 105 swear by their Python REPL for terminal usage, and if they're happy with that 106 then I salute them, their usage is as niche as those who enjoy using Bash for 107 complex scripts. Perhaps the only language I've used which translates well both 108 for terse REPLs and lengthier scripts is LISP. 109 110 ## The Modern 111 112  113 114 So how are modern shells addressing these split concerns? 115 116 ### Powershell 117 118 Microsoft had the benefit of being able to start from a clean room. They didn't 119 need to inherit 50+ years of UNIX legacy when they wrote Powershell. So their 120 approach was naturally to base their shell on .NET. Passing .NET objects around 121 has a number of advantages over the POSIX specification of passing files, byte 122 streams, to applications. This allows developers to write richer command line 123 applications in their preferred .NET language rather than being tied to the 124 shell's syntax. However one could argue the same is true with POSIX shells and 125 how you can write a program in any language you like. But in Powershell those 126 other .NET programs feel more tightly integrated into Powershell than a forked 127 process does in Bash. Again, I put this down to Powershell passing .NET objects 128 along the pipeline. 129 130 Where Powershell falls down for me is in two key areas: 131 132 1. Many of the flags passed are verbose. Calling .NET objects can be verbose. 133 Take this example of base64 encoding a string: 134 ``` 135 [Convert]::ToBase64String([System.Text.Encoding]::Unicode.GetBytes("TextToEncode")) 136 ``` 137 138 2. Powershell doesn't play nicely with POSIX. Okay, I'm arguably contradicting 139 myself now because earlier I raised this as a benefit. And in many ways it 140 is. However if you wish to run Powershell on Linux, which you can do, you 141 may find that you'll want to work with CLI tools that do "think" in terms of 142 byte streams. Many of these tools have equivalent aliases written in .NET so 143 you can appear to use them without escaping the rich programming environment 144 provided by Powershell. However you may, and I often did, run into a great 145 many scenarios where my expectations didn't match the practicalities of 146 Powershell. 147 148 (I will talk more about the second point in another article where I'll discuss 149 pipelines, data types and the need for modern shells to understand rich data 150 rather than treating everything as a flat stream of bytes) 151 152 There is no question that Powershell is a more powerful REPL than Bash but it 153 definitely slides more towards the "write-once read-many" end of the spectrum. 154 155 ### Oil 156 157 [Oil](https://www.oilshell.org/) describes itself as the following: 158 159 > Oil is a new Unix shell. It's our upgrade path from bash to a better language 160 > and runtime. It's also for Python and JavaScript users who avoid shell! 161 162 The way Oil achieves this is a lot of how PWB improved upon the Thompson shell 163 in the 1970s. Oil aims to be upwards-compatible with Bash. Any command line or 164 shell script you can run in Bash should, eventually, be supported in Oil as 165 well. Oil can extend on that and support a syntax and grammar that is more 166 readable and sane to write longer lived scripts in. Thus bridging the conflict 167 between "write-many" and "read-many" languages. 168 169 This make Oil one of the most interesting alternative shells I have come 170 across. 171 172 ### Murex 173 174  175 176 The approach Murex takes sits somewhere in between the previous two shells. 177 It attempts to retain familiarity with POSIX syntax but isn't afraid to break 178 compatibility where it makes sense. The emphasis is on creating grammar that 179 is both succinct but also readable. This mission was driven from originally 180 attempting to create something more familiar to Javascript developers then 181 falling back to some old Bash-ism's when I realized that for all of it's warts, 182 Bash and its kin aren't actually bad for quick REPL usage of C-style braces 183 over ALGOL style named scopes: 184 185 **POSIX:** 186 187 ``` 188 if [ 0 -eq 1 ]; then 189 echo '0 == 1' 190 else 191 echo '0 != 1' 192 fi 193 ``` 194 195 **Murex:** 196 197 ``` 198 if { 0 == 1 } then { 199 echo '0 == 1' 200 } else { 201 echo '0 != 1' 202 } 203 ``` 204 205 But since the curly braces are tokens, grammar like `then` / `else` become 206 superfluous words that only exist for readability. So then we can make them 207 optional. And you end up with a syntax that allows for a certain amount of 208 golfing in the REPL should the operator want to save a few key strokes 209 210 ``` 211 if { 0 == 1 } { echo '0 == 1' } { echo '0 != 1' } 212 ``` 213 214 ## Conclusion 215 216 The write-many read-once tendencies of the interactive terminal and the 217 write-once read-many demands of scripting might be difficult to consolidate 218 but I do think it is achievable and I'm not convinced the current heavy weights 219 do a good job at addressing those conflicting concerns. Whereas alternative 220 shells like [Oil](https://www.oilshell.org/), [Elfish](https://elv.sh/) and 221 [Murex](https://github.com/lmorg/murex) seem to be putting a lot more thought 222 into this problem and it is really exciting seeing the different ideas that are 223 being produced. 224 225 <hr> 226 227 Published: 02.10.2021 at 22:42 228 229 ## See Also 230 231 * [Interactive Shell](../user-guide/interactive-shell.md): 232 What's different about Murex's interactive shell? 233 * [Reading Lists From The Command Line](../blog/reading_lists.md): 234 How hard can it be to read a list of data from the command line? If your list is line delimited then it should be easy. However what if your list is a JSON array? This post will explore how to work with lists in a different command line environments. 235 * [Rosetta Stone](../user-guide/rosetta-stone.md): 236 A tabulated list of Bashism's and their equivalent Murex syntax 237 * [`if`](../commands/if.md): 238 Conditional statement to execute different blocks of code depending on the result of the condition 239 240 <hr/> 241 242 This document was generated from [gen/blog/split_personalities_doc.yaml](https://github.com/lmorg/murex/blob/master/gen/blog/split_personalities_doc.yaml).