github.com/upcmd/up@v0.8.1-0.20230108151705-ad8b797bf04f/README.md (about) 1 # UPcmd [Docs](https://upcmd.netlify.app/) [Telegram Channel](https://t.me/joinchat/CM3uyxPLSPFPbrbttCQiow) 2 3 The Ultimate Provisioner: the modern configuration management, build and automation tool 4 5 * Sick of using Makefile, Ansible, Ant, Gradle, Rake and different frameworks ... ? 6 * Tired of patching Shell scripts, integrating different tools together, elegantly? 7 * Lacking of an overall simple solution of automation, nicely integrated in a Cloud environments? 8 * Feeling the pains of your DevOps, Ci/CD best practice? 9 * Need the interoperability to collaborate with popular tools: terraform, packer, vagrant, docker, kubectl, helm .... ? 10 11 No worries of replacing anything you already setup, UPcmd does not dictate and work exclusively with other tools, rather it incorporates and collaborates with others, but it is capable to be a framework in case you do need it 12 13 ## How does it look like? 14 You could use a quick terminal GUI to select the task to execute 15 ![Interactive GUI in terminal](https://raw.githubusercontent.com/upcmd/up-demo/master/demos/menu_selection.png) 16 17 All the project build, test, regression tests, the documentation site generation, publishing the tagged release and latest rolling release are using Up tasks 18 19 This is how publishing latest bleeding edge release look like when you run [UP task](https://github.com/upcmd/up/blob/master/up.yml) 20 21 ![up ngo Publish_latest](https://raw.githubusercontent.com/upcmd/up-demo/master/demos/publish_latest.gif) 22 23 ## [UPcmd - The Ultimate Provisioner](https://upcmd.netlify.app/usage/cli_usage/) 24 25 UP is designed and implemented to shine as a modern tool for below: 26 27 * Configuration management 28 * Build, continuous delivery, integration with CI/CD 29 * Comprehensive workflow orchestration: full support of almost all type of condition, loop(recursive), break, until 30 * Flexible configuration organisation 31 * No dependency hell issue 32 * Precise modeling, data is object, object is the data 33 * Design of composition, separate func type, data and implementation 34 * Use inteface(call func) for abstraction of intention, data input and implementation 35 * Many builtin features: dry run, assert, pause, user prompt, input validation, debugging/trace/inspect, developer friendly 36 * ... many more for you to discover, check out the docs 37 38 It is a build tool like Ansible, Make, Rake, Ant, Gradle, Puppet, Taskfile etc, but it is little smarter to try to make things a little easier 39 40 The goal of UP is to provide a quick (I'd say the quickest) solution to enable continuous integration and continuous deployment (CI/CD). It is simple to use and yet powerful to address many common challenges nowadays devop teams facing in the Cloud environment. 41 42 It is designed with mindful consideration of collaboration with automation in Kubernetes, helm charts, api call. 43 44 It follows best practices integrating with common CI/CD tools, such as GOCD, Jenkins, Drone, Gitlab CI and so on and is a good company of all types of CLI tools. 45 46 It brings a fun DSL programming interface, a way of modeling and engineering into CLI. It enables OO design and rapid Test Driven Development (TDD) and shorter software delivery cycle. 47 48 * Hello, world 49 50 ``` 51 tasks: 52 - 53 name: task 54 task: 55 - 56 func: shell 57 do: 58 - echo "hello, world" 59 ``` 60 61 * Appetizers 62 63 Below shows: 64 65 * It is a simple deployment of a web application, it has a prior step of database deployment 66 * The database deployment will depend on the db configuration base on per environment 67 * the non prod envs: staging and dev share the same database configurations using a shared database 68 * the non prod envs: staging and dev each individually will use different database host name though 69 * a dvar db_password using aes to manage the password 70 71 To deploy, simply specify a instanceid to be associated with an environment, eg: dev, staging or prod 72 73 ``` 74 up ngo -i staging 75 ``` 76 77 The config: 78 ``` 79 80 scopes: 81 - name: global 82 vars: 83 db_driver: postgres 84 port: 5432 85 86 - name: nonprod 87 members: 88 - dev 89 - staging 90 vars: 91 db_host: nonpord_database.test.host 92 db_user: test_db_user 93 db_password: could_be_encrypted_using_upcmd_too 94 dvars: 95 - name: db_password 96 value: '6HmsmiJIW1PfIXcF4WwOKOMDiL7PstgfKs2aRFajrwY=' 97 98 - name: prod 99 members: [prod] 100 vars: 101 host_alias: prod 102 103 - name: dev 104 vars: 105 host_alias: dev 106 107 - name: staging 108 vars: 109 host_alias: staging 110 111 - name: prod 112 vars: 113 host_alias: prod 114 db_host: pord_database.proddb.host 115 db_user: prod_db_user 116 dvars: 117 - name: db_password 118 value: 'prod_encrypte_aes' 119 120 dvars: 121 - name: db_hostname 122 value: '{{.host_alias}}.myapp.com' 123 - name: db_url 124 value: 'jdbc:{{.db_driver}}://{{.db_hostname}}:{{.db_port}}/test?user={{.db_user}}&password={{.db_password}}&ssl=true' 125 126 tasks: 127 - 128 name: Main 129 desc: deploy my web app stack 130 task: 131 - 132 func: call 133 do: 134 - deploy_database 135 - deploy_web 136 137 - 138 func: shell 139 do: 140 - systemctl start my_database 141 - systemctl start my_web_server 142 143 - 144 name: deploy_web 145 task: 146 - 147 func: shell 148 do: 149 - deploy myweb_server 150 151 - 152 name: deploy_database 153 task: 154 - 155 func: shell 156 do: 157 - deploy mydatabase 158 159 ``` 160 161 With the evolving of the up.yml file, you could externalize the configuration to individual files or make them as module to be reused or shared. Please check out the doc for more details. 162 163 ### High level design 164 165 At high level, UPcmd process flows like below: 166 167 * The process engine process the scope vars and merge them with global vars, then in the run time it will merge with local vars again 168 * For the callee task, the local vars will be overriden by the vars passed from caller task 169 170 ![high level design](https://raw.githubusercontent.com/upcmd/updocs/master/static/up_high_level.png) 171 172 ### Possible applications 173 174 UPcmd is a generic automation tool, given your automation solution being backed by Unix Shell. You do not need Shell executable though, as it has default GOSH builtin just in case you will need one to fall back to. 175 176 There could be application as below, but not limit to: 177 * Build, package, publish, test, deploy for all different types of applications in your local machine, or integrate with CI/CD tools/pipelines 178 * UPcmd could be used as tool/platform/pipeline agnostic abstraction layer, leave the most configuration to UPcmd to manage as an execution profile, expose only the profileid to be linked with the tool, eg: jenkins/gitlab ci, so that all your automation is portable. In case you need to switch from one to another, you don't need to rewrite all the automation. In this case, UPcmd's configured tasks could be regarded as pipeline as code. 179 * A collection of util like (tool box) for local machine automation, for example, 180 1. bootstrap the whole Macbook with all upgraded packages, setup all your dotfiles 181 2. bootstrap the whole Linux box/virtual VM/vagrant box/docker container 182 * Create CLI program, prompt with user input, encrypt/decrypt secrets 183 * Web service/rest api call and message transformation 184 * Cloud service provisioning, eg drive complicated workflow to manage to create full application stack in AWS or k8s cluster, utilise and integrate with other CLI commands, such as packer, aws cli, kubectl, helm, terraform 185 * Reuse/consume or share modules to deal with a particular use case. 186 * Resolve the dependencies issue by simply invoking different version of the relevant CLI/docker run 187 * The orchestration of UPcmd task itself could be seen as prototyping tool and design tool, or use the defined workflow as skeleton to guide the implementation from different part 188 189 ### Installation 190 191 There are 32 different distro for different combination of OS and Arch type, check them out: [release](https://github.com/upcmd/up/releases) 192 193 #### Generic Installation 194 195 1. Download the binary for your platform from the 196 * [latest stable tagged release](https://github.com/upcmd/up/releases/latest) or 197 * [latest bleeding edge release](https://github.com/upcmd/up/releases/latest) - Full regression tested 198 2. Rename it to up, or up.exe in windows 199 3. Move it to be under your one of your env PATH 200 201 #### Auto Latest Installation (recommended) 202 203 Always try to use the latest unless you have CI/CD pipeline to progressively to promote to production, then use tagged version 204 205 * Source this shell function 206 207 ``` 208 install_latest(){ 209 if [ "$1" == "" ];then 210 echo "syntax exaple: install_latest darwin | linux | windows" 211 else 212 os=$1 213 curl -s https://api.github.com/repos/upcmd/up/releases \ 214 |grep ${os}_amd64_latest \ 215 |grep download \ 216 |head -n 1 \ 217 |awk '{print $2}' \ 218 |xargs -I % curl -L % -o up \ 219 && chmod +x up 220 fi 221 } 222 ``` 223 224 1. install for mac: 225 226 ``` 227 install_latest darwin 228 ``` 229 230 2. install for linux: 231 232 ``` 233 install_latest linux 234 ``` 235 236 1. install for windows: 237 238 ``` 239 install_latest windows 240 ``` 241 242 * [tagged install details](https://upcmd.netlify.app/usage/installation/) 243 244 #### Install from source 245 246 Ensure you use go 1.14 (prefered) 247 248 ``` 249 go get -v github.com/upcmd/up/app/up 250 ``` 251 252 The up CLI command will be installed to: $HOME/go/bin, make sure you have this in your PATH 253 254 #### Use up cli command in docker 255 256 This will map your current working directory as /workspace directory inside of docker container: 257 258 ``` 259 docker run -it --rm -v `pwd`:/workspace docker.pkg.github.com/upcmd/up/upcli:latest /bin/sh 260 ``` 261 262 Or you can source this from the funcs.rc 263 264 ``` 265 . ./funcs.rc 266 run_upcli_docker 267 ``` 268 269 In the container: 270 271 ``` 272 cd /workspace 273 up ngo 274 ``` 275 276 ### A little taste of UPcmd 277 278 Below is a simple greeting example, and a list, inspect and execution view of the task. 279 280 * With some smarts: logic and loop etc [doc](https://upcmd.netlify.app/quick-start/c0151/) | [source](https://github.com/upcmd/up/blob/master/tests/functests/c0151.yml) 281 282 This shows: 283 * the greet task is an implementation, by default it was called with default global var greet_to value, but with supply of local var of "Grace", it changes the behaviour [see concept of interface](https://upcmd.netlify.app/call-func/c0020/) 284 * loop through 285 * if/else logic 286 * chain through tasks 287 288 ``` 289 290 Vars: 291 greet_to: Tom 292 weather: sunny 293 294 tasks: 295 - 296 name: task 297 desc: main task of hello world demo of UPcmd 298 task: 299 - 300 func: call 301 desc: greet to Tom 302 do: 303 - greet 304 305 - 306 func: call 307 desc: greet to Grace 308 vars: 309 greet_to: Grace 310 do: 311 - greet 312 313 314 - 315 func: cmd 316 desc: do you get the idea? 317 do: 318 - name: print 319 cmd: | 320 Have you got a little taste of using the UPcmd? 321 322 - 323 func: call 324 desc: greet to a team 325 vars: 326 team: 327 - Jason 328 - Connie 329 weather: stormy 330 loop: team 331 do: 332 - sayhi 333 334 - 335 name: greet 336 desc: greet to some one 337 task: 338 - 339 func: shell 340 desc: say hello 341 do: 342 - echo "Hello, {{.greet_to}}" 343 344 - 345 func: cmd 346 desc: talk about weather 347 do: 348 - name: print 349 cmd: 'It is {{.weather}}' 350 351 - 352 func: cmd 353 desc: ice break 354 do: 355 - name: print 356 cmd: 'What a great day!' 357 if: '{{eq .weather "sunny"}}' 358 else: 359 - 360 func: cmd 361 do: 362 - name: print 363 cmd: 'What a bad day!!' 364 365 - 366 name: sayhi 367 desc: say hi to some one 368 task: 369 - 370 func: cmd 371 desc: say hi to someone 372 do: 373 - name: print 374 cmd: 'Hi {{.loopitem}}, how are you?' 375 376 - 377 func: call 378 desc: greet to the team member 379 dvars: 380 - name: greet_to 381 value: '{{.loopitem}}' 382 do: 383 - greet 384 385 ``` 386 387 ![A little taste](https://raw.githubusercontent.com/upcmd/updocs/master/static/a_little_taste.png) 388 389 ### Demo 390 391 It demos: 392 * create upcmd task skeleton using init command 393 * show the intro demo code and execution 394 * use module 395 * test driven, assert and color print 396 397 Check it out yourself: [source](https://github.com/upcmd/up-demo/blob/master/demo.sh) and try to have fun to run though the examples by yourself 398 399 ![demo](https://raw.githubusercontent.com/upcmd/up-demo/master/demos/intro.gif) 400 401 ### Why yet another build tool 402 403 * Make was initially designed and used for building C program, even though it could be adopted for other purpose, the hard to learn trivial often causes problems than the benefits added to the team, and it is burning the brain. It is hard to make automation task extended to a more advanced level, readability degrades rapidly, and it is risky to implement critical logic using Make. Make is just a little old for modern business requirements. (Sorry, maybe this is just from one not good at using Makefile) 404 405 * Rake is smart and powerful. If you don't mind learning Ruby, it is a good choice of building tool. Similarly Ant and Gradle are all bind to a language specific, it is just not right when it comes to the case that you want to automate things in cloud environment. In most cases when it requires automation in a cloud environment, in a given spun up AWS EC2 instance, a shell session, a kubernete pod, you would want something just works without any dependencies. You simply do not want to maintain the consistency of chain of upgrading path for all language packages in multiple environments. In these cases, Rake, Gradle, Ant are not best options. Due to history reasons, devops teams might have adopted them and take the advantages in the early phase. When it comes to gradual improvement and upgrade consistently in long term, the effort and cost to upgrade the whole ecosystem is just too huge and often wrong solution used in order to keep it going, until it is start sinking. 406 407 * Ansible, Puppet and chef are configuration management tools. They are powerful, there are many builtin well tested modules you could use. However, Ansible might be too huge for little job. Most of the time it tends to over kill, also it suffers the same problem of python/python packages dependencies. Using it means bringing the whole hard to maintain forest of software packages, os libraries all into your execution context. 408 409 A common usage of Ansible for many teams is to use the local ssh execution with group/host vars for templating and workflow automation, which is simply not right. The way the vars managed are not fine-grained. The ansible role as a reusable module is not flexible to implement for more complicated tasks. Specifically, it does not support leaf level merge; it is hard and nearly not possible(elegantly) to do a simple validation of command line input; its controller and role one-way communication is not flexible ... 410 411 * Inspired by https://taskfile.dev/, it is tiny tool making build and automation easier and elegant, however it lacks some features in a practical cloud environment for CI/CD, devops automation 412 413 With all above considerations, UP is designed to be a generic, tiny footprint (zero depedency), effective automation tool in a cloud environment. 414 415 ### Features 416 417 1. Drop in replacement for Makefile, but way more powerful. It uses a composition model rather than dependency model for flexibility/composibility 418 2. Implemented in golang, so no dependency hell, no maintenance of runtime and ensure the version consistency across multiple/many execution contexts 419 3. Use scopes to manage group of execution context, the variables associated with the scope. Fine grained scoping model to support variable auto overriding/merging. Similar to Ansible global/group vars, host vars, but more powerful to support leaf level objects auto merging 420 4. Use dvar - dynamic var, a special design to achieve many incredible features, for example: 421 * manage security: encrypt/decrypt, like ansible-vault, builtin 422 * dynamics on dynamics: it allows you to specify how many layers of expansion you'd like to dynamically render a variable 423 * builtin templating capability 424 * use golang template, supporting all(220+) (builtin|sprig|gtf) funcs/pipeline so that your configuration could be well controlled in template using objects 425 * auto message transformation between yaml|json result to object used internally 426 * conform the hierarchical scoping model for var merging to leaf level 427 * manage setup/read env vars in the same scoping model so that you could have seamless integration with minimal exposed demanding ENV vars from CD/CI tool 428 * auto validation for mandatory vars 429 5. Color print and adjustable verbose level 430 6. Flexible programming model to allow you to separate implementation with interface so that common code could be reused via call func 431 Allow empty skeleton to be laid for testing driving process or guide as seudo code, but fill in the details and implementation gradually 432 7. Flow control: 433 * ignoreError 434 * dry run 435 * if condition support 436 * loop support to iterate through a list of items 437 * mult-layered loop, break and until condition 438 * block and embedded block of code for execution 439 * finally/rescue to ensure cleanup 440 8. Flexible configuration style to load dvar, scope, flow from external yaml so that the programming code will be a little cleaner and organised. Your code could evolve starting from simple, then externalize detailed implementation to files. 441 9. Support the unlimited yml object, so yml config in var is text and it is also object.It could be merged in scopes automatically, it could be processed using go template 442 10. Battery included for common builtin commands: print, reg, deReg, template, readFile, writeFile 443 11. Builtin yml liter and object query, modification 444 12. Call func is really shining powerful design to be used: 445 * Compose the sequential execution of block of code 446 * Use the stack design, so it segregates all its local vars so that the vars used in its implementation will not pollute the caller's vars 447 * It serves like a interface to separates the goal and implementation and makes the code is reusable 448 13. The shell execution binary is configurable, builtin support for GOSH (mvdan.cc/sh). This means that you do not need native shell/bash/zsh installed in order for task execution, you can run task from windows machine. 449 14. It provides a module mechanism to encourage community to share modular code so that you do not need to reinvent the wheel to develop the same function again 450 15. Use execution profile to simplify your Ci/CD pipeline integration with zero arguments in your command line but all managed in a nice configurable way. 451 16. Use virtualEnv cmd to snapshot an env context, unset all env vars to create a pure clean execution env context, or restore to a point of time of the execution 452 453 ### Real Examples 454 455 Both UPcmd project build and the docs entire site build use the UPcmd itself 456 457 ##### Project release for UPcmd [source](https://github.com/upcmd/up/blob/master/up.yml) 458 ``` 459 up ngo publish 460 ``` 461 462 ##### Documentation [doc site](https://upcmd.netlify.app/) 463 464 build of the entire doc site using one build task: 465 * [source](https://github.com/upcmd/updocs/blob/master/up.yml) 466 * [details](https://upcmd.netlify.app/advanced-cases/upcmd-doc-gen/) 467 468 ##### A web scripting example [how?](https://upcmd.netlify.app/advanced-cases/web-scraping/) 469 470 ### Testing 471 472 There are over 230+ test cases, every release come with a full passed regression test of all cases defined, [source](https://github.com/upcmd/up/tree/master/tests) 473 474 * [common examples](https://github.com/upcmd/up/tree/master/tests/functests) 475 * [module usage examples](https://github.com/upcmd/up/tree/master/tests/modtests) 476 477 These test cases are not only about the tests, they are the usage examples with documentation self explaned 478 479 ### License 480 481 This project is under [MPL-2.0 License](https://github.com/upcmd/up/blob/master/LICENSE)