github.com/yunabe/lgo@v0.0.0-20190709125917-42c42d410fdf/README.md (about) 1 # lgo [![Binder](https://mybinder.org/badge.svg)](https://mybinder.org/v2/gh/yunabe/lgo-binder/master?filepath=basics.ipynb) [![Go Report Card](https://goreportcard.com/badge/github.com/yunabe/lgo)](https://goreportcard.com/report/github.com/yunabe/lgo) 2 3 Go (golang) Jupyter Notebook kernel and an interactive REPL 4 5 ## Disclaimer 6 7 Since go1.10, this Go kernel has performance issue due to [a performance regression in Go tool chain](https://github.com/golang/go/issues/24034). 8 9 Also, this Go kernel can not be built with go1.12 due to [another regression in Go tool chain](https://github.com/golang/go/issues/30768). 10 11 Now, the compiler options this kernel relies on are completely broken and I'm not sure when they will fix the regressions. 12 Unfortunately, they don't plan to fix this in go1.13 as of July 8th 2019. If you are interested in using this kernel, please upvote the bugs. For a while, please use other Go kernels if you want to use the later version of Go with Jupyter notebook. 13 14 ## Medium Post 15 - [Go (golang) Jupyter Notebook kernel and an interactive REPL - April 16, 2018](https://medium.com/@yunabe/interactive-go-programming-with-jupyter-93fbf089aff1) 16 17 # Features 18 - Write and execute Go (golang) interactively like Python. 19 - Jupyter Notebook integration 20 - Full Go (golang) language spec support. 100% gc (go compiler) compatible. 21 - Code completion and inspection in Jupyter Notebooks 22 - Display images, HTML, JavaScript, SVG, etc... 23 - Currently, lgo is only supported on Linux. But you can use lgo on Mac and Windows with virtual machines or Docker. 24 25 <img src="https://drive.google.com/uc?export=view&id=12_7fHfKfdSy8SNXi0nsWznbsRgix9tGJ" width="400" height="366"> 26 27 # Jupyter notebook examples 28 You can view example notebooks of lgo from 29 [Example notebooks on Jupyter nbviewer](https://nbviewer.jupyter.org/github/yunabe/lgo/blob/master/examples/basics.ipynb) 30 31 If you want to execute these notebooks, you can try these notebooks on your browser without installation from [![Binder](https://mybinder.org/badge.svg)](https://mybinder.org/v2/gh/yunabe/lgo-binder/master?filepath=basics.ipynb) 32 33 # Try lgo from your browser without installation 34 [![Binder](https://mybinder.org/badge.svg)](https://mybinder.org/v2/gh/yunabe/lgo-binder/master?filepath=basics.ipynb) 35 36 Thanks to [binder (mybinder.org)](https://mybinder.org/), you can try lgo on your browsers with temporary docker containers on binder. 37 Open your temporary Jupyter Notebook from the button above and enjoy lgo. 38 39 # Quick Start with Docker 40 1. Install [Docker](https://docs.docker.com/engine/installation/) and [Docker Compose](https://docs.docker.com/compose/). 41 2. Clone the respository and run the docker container with docker-compose. 42 ``` 43 > git clone https://github.com/yunabe/lgo.git 44 > cd lgo/docker/jupyter 45 > docker-compose up -d 46 ``` 47 48 If you want to use a port other than `8888` on host, update ports config in `lgo/docker/jupyter/docker-compose.yaml` before running `docker-compose up`. 49 50 3. Get the URL to open the Jupyter Notebook 51 ``` 52 > docker-compose exec jupyter jupyter notebook list 53 Currently running servers: 54 http://0.0.0.0:8888/?token=50dfee7e328bf86e70c234a2f06021e1df63a19641c86676 :: /examples 55 ``` 56 4. Open the Jupyter Notebook server with the authentication token above. 57 58 ## Linux/Mac OS 59 If you are using Linux or Mac OS, you can use start/stop scripts instead. Web browser will open the URL automatically. 60 61 ```bash 62 # start server 63 > ./up.sh 64 # stop server 65 > ./down.sh 66 ``` 67 68 # Install 69 ## Prerequisites 70 - lgo is supported only on Linux at this moment. On Windows or Mac OS, use virtual machines or dockers. 71 - [Install Go 1.9 or Go 1.8](https://golang.org/doc/install). 72 - [Note for Go 1.10](#go110) 73 - Install [Jupyter Notebook](http://jupyter.readthedocs.io/en/latest/install.html) 74 - [Install ZMQ](http://zeromq.org/distro:debian) 75 - e.g. `sudo apt-get install libzmq3-dev` 76 - [Install pkg-config](https://www.freedesktop.org/wiki/Software/pkg-config/) 77 - e.g. `sudo apt-get install pkg-config` 78 79 ## Install 80 - `go get github.com/yunabe/lgo/cmd/lgo && go get -d github.com/yunabe/lgo/cmd/lgo-internal` 81 - This installs `lgo` command into your `$(go env GOPATH)/bin` 82 - Set `LGOPATH` environment variable 83 - `lgo install` will install binaries into the directory specified with `LGOPATH`. 84 - You can use any empty directory with write permission as `LGOPATH`. 85 - Run `lgo install` 86 - This installs std libraries and the internal lgo tool into `LGOPATH` with specific compiler flags. 87 - If `lgo install` fails, please check install log stored in `$LGOPATH/install.log` 88 - (Optional) Run `lgo installpkg [packages]` to install third-party packages to `LGOPATH` 89 - You can preinstall third-party packages into `LGOPATH`. 90 - This step is optional. If packages are not preinstalled, lgo installs the packages on the fly. 91 - But, installing packages is a heavy and slow process. I recommend you to preinstall packages 92 which you will use in the future with high probability. 93 - If `lgo installpkg` fails, please check the log stored in `$LGOPATH/installpkg.log`. 94 - See [go's manual](https://golang.org/cmd/go/#hdr-Package_lists) about the format of `[packages]` args. 95 - Install the kernel configuration to Jupyter Notebook 96 - `python $(go env GOPATH)/src/github.com/yunabe/lgo/bin/install_kernel` 97 - Make sure to use the same version of `python` as you used to install `jupyter`. For example, use `python3` instead of `python` if you install `jupyter` with `pip3`. 98 - (Optional) If you want to use `lgo` with JupyterLab, install a jupyterlab extension for `lgo` 99 - `jupyter labextension install @yunabe/lgo_extension` 100 - This extension adds "Go Format" button to the toolbar in JupyterLab. 101 102 ## Usage: Jupyter Notebook 103 - Run `jupyter notebook` command to start Juyputer Notebook and select "Go (lgo)" from New Notebook menu. 104 - To show documents of packages, functions and variables in your code, move the cursor to the identifier you want to inspect and press `Shift-Tab`. 105 - Press `Tab` to complete code 106 - Click `Format Go` button in the toolbar to format code. 107 - lgo works with [JupyterLab](https://github.com/jupyterlab/jupyterlab). To use lgo from JupyterLab, install JupyterLab and run `jupyter lab`. 108 109 <img width="400" height="225" src="doc/inspect.jpg"> 110 <img width="400" height="225" src="doc/complete.jpg"> 111 112 ## Usage: REPL console 113 You can use lgo from command line with [Jupyter Console](https://github.com/jupyter/jupyter_console) or build-in REPL mode of lgo 114 115 ### Jupyter Console (Recommended) 116 Run `jupyter console --kernel lgo` 117 118 ```go 119 In [1]: a, b := 3, 4 120 121 In [2]: func sum(x, y int) int { 122 : return x + y 123 : } 124 125 In [3]: import "fmt" 126 127 In [4]: fmt.Sprintf("sum(%d, %d) = %d", a, b, sum(a, b)) 128 sum(3, 4) = 7 129 ``` 130 131 ### built-in REPL mode 132 Run `lgo run` 133 134 ```go 135 $ lgo run 136 >>> a, b := 3, 4 137 >>> func sum(x, y int) int { 138 ... return x + y 139 ... } 140 >>> import "fmt" 141 >>> fmt.Sprintf("sum(%d, %d) = %d", a, b, sum(a, b)) 142 sum(3, 4) = 7 143 ``` 144 145 # Tips 146 ## go get and lgo 147 The packages you want to use in lgo must be prebuilt and installed into `$LGOPATH` by `lgo install` command. 148 Please make sure to run `lgo install` after you fetch a new package with `go get` command. 149 150 ## Update go version 151 Please run `lgo install --clean` after you update `go` version. 152 153 `lgo install` installs prebuilt packages into `$LGOPATH`. 154 When you update `go` version, you need to reinstall these prebuilt packages with the newer `go` 155 because binary formats of prebuilt packages may change in the newer version of go. 156 157 ## Display HTML and images 158 To display HTML and images in lgo, use [`_ctx.Display`](https://godoc.org/github.com/yunabe/lgo/core#LgoContext). 159 See [the example of `_ctx.Display`](http://nbviewer.jupyter.org/github/yunabe/lgo/blob/master/examples/basics.ipynb#Display) in an example notebook 160 161 ## Cancellation 162 In lgo, you can interrupt execution by pressing "Stop" button (or pressing `I, I`) in Jupyter Notebook and pressing `Ctrl-C` in the interactive shell. 163 164 However, as you may know, Go does not allow you to cancel running goroutines with `Ctrl-C`. Go does not provide any API to cancel specific goroutines. The standard way to handle cancellation in Go today is to use [`context.Context`](https://golang.org/pkg/context/#Context) (Read [Go Concurrency Patterns: Context](https://blog.golang.org/context) if you are not familiar with context.Context in Go). 165 166 lgo creates a special context `_ctx` on every execution and `_ctx` is cancelled when the execution is cancelled. Please pass `_ctx` as a context.Context param of Go libraries you want to cancel. Here is [an example notebook of cancellation in lgo](http://nbviewer.jupyter.org/github/yunabe/lgo/blob/master/examples/interrupt.ipynb). 167 168 ## Memory Management 169 In lgo, memory is managed by the garbage collector of Go. Memory not referenced from any variables or goroutines is collected and released automatically. 170 171 One caveat of memory management in lgo is that memory referenced from global variables are not released automatically when the global variables are shadowed by other global variables with the same names. For example, if you run the following code blocks, the 32MB RAM reserved in `[1]` is not released after executing `[2]` and `[3]` because 172 173 - `[2]` does not reset the value of `b` in `[1]`. It just defines another global variable `b` with the same name and shadows the reference to the first `b`. 174 - `[3]` resets `b` defined in `[2]`. The memory reserved in `[2]` will be released after `[3]`. But the memory reserved in `[1]` will not be released. 175 176 ``` 177 [1] 178 // Assign 32MB ram to b. 179 b := make([]byte, 1 << 25) 180 [2] 181 // This shadows the first b. 182 b := make([]byte, 1 << 24) 183 [3] 184 // This sets nil to the second b. 185 b = nil 186 ``` 187 188 ## go1.10 189 lgo works with go1.10. But the overhead of code execution is 4-5x larger in go1.10 than go1.9. 190 It is due to [a regression of the cache mechnism of `go install` in go1.10](https://github.com/golang/go/issues/24034). 191 I recommend you to use lgo with go1.9 until the bug is fixed in go1.10. 192 193 # Comparisons with similar projects 194 ## gore 195 [gore](https://github.com/motemen/gore), which was released in Feb 2015, is the most famous REPL implementation for Go as of Dec 2017. gore is a great tool to try out very short code snippets in REPL style. 196 197 But gore does not fit to data science or heavy data processing at all. 198 gore executes your inputs by concatinating all of your inputs, 199 wrapping it with `main` function and running it with `go run` command. 200 This means every time you input your code, gore executes all your inputs from the begining. 201 For example, if you are writing something like 202 203 1. Loads a very large CSV file as an input. It takes 1 min to load. 204 2. Analyzes the loaded data. For example, calculates max, min, avg, etc.. 205 206 gore always runs the first step when you calculate something and you need to wait for 1 min every time. 207 This behavior is not acceptable for real data science works. Also, gore is not good at tyring code with side effects (even fmt.Println) because code snippets with side effects are executed repeatedly and repeatedly. 208 lgo chose a totally different approach to execute Go code interactively and does not have the same shortcoming. 209 210 gore is a CLI tool and it does not support Jupyter Notebook. 211 212 ## gophernotes 213 ||lgo|gophernotes| 214 |:---|:---|:---| 215 |Backend|gc (go compiler)|An unofficial interpreter| 216 |Full Go Language Specs|:heavy_check_mark:|| 217 |100% gc compatible|:heavy_check_mark:|| 218 |Static typing|:heavy_check_mark:|to some extent| 219 |Performance|Fast|Slow| 220 |Overhead|500ms|1ms| 221 |[Cancellation](https://github.com/yunabe/lgo/blob/master/README.md#cancellation)|:heavy_check_mark:|| 222 |Code completion|:heavy_check_mark:|| 223 |Code inspection|:heavy_check_mark:|| 224 |Code formatting|:heavy_check_mark:|| 225 |[Display HTML and images](https://github.com/yunabe/lgo/blob/master/README.md#display-html-and-images)|:heavy_check_mark:|| 226 |Windows, Mac|Use Docker or VM|Partial| 227 |License|BSD|LGPL| 228 229 gophernotes was the first Jupyter kernel for Go, released in Jan 2016. 230 Before [Sep 2017](https://github.com/gopherdata/gophernotes/commit/69792d8af799d6905e2c576164d1a189ac021784#diff-04c6e90faac2675aa89e2176d2eec7d8), it used the same technology gore uses to evaluate Go code. This means it did not fit to heavy data processing or data analysis at all. 231 From Sep 2017, gophernotes switched from `go run` approach to [gomacro](https://github.com/cosmos72/gomacro), one of unofficial golang interpreters by [cosmos72](https://github.com/cosmos72). This solved the problem gore has. Now, the code execution mechnism of gophernotes also fits to heavy data analysis. 232 233 The shortcomings of using an unofficial interpreter are 234 - It does not support all Go language features. Especially, it does not support one of the most important Go feature, `interface`. 235 As of go1.10, it is hard to support `interface` in an interpreter written in Go because of the lack of API in `reflect` package. 236 - Interpreters are generally slow. 237 - Unofficial interpreters are not well-tested compared to the official gc (go compiler) tools. 238 239 The advantages of this approach are 240 - The overhead of code execution is small because it does not compile and link code. 241 - Windows/Mac partial support. lgo works only on Linux and you need to use VMs or Docker to run it on Windows/Mac. 242 gophernotes (gomacro) works on Windows/Mac natively [if you do not need third-party packages](https://github.com/gopherdata/gophernotes#limitations). 243 244 These disadvantage and advantages are not something inevitable in interperters. But they are not easy to solve under the limited development resource. 245 246 Also, lgo kernel supports more rich features in Jupyter Notebook as of Dec 2017, including code completion, code inspection and images/HTML/JavaScript output supports. 247 248 # Troubleshooting 249 250 ## Dead kernel 251 ### Symptom 252 Got an error message like: 253 254 ``` 255 Kernel Restarting 256 The kernel appears to have died. It will restart automatically. 257 ``` 258 259 ### Solutions 260 First, please confirm your code does not call [`os.Exit`](https://golang.org/pkg/os/#Exit) directly or indirectly. 261 In lgo, your code is executed in the processs of lgo kernel. If you evaluate `os.Exit` in lgo, it terminates the lgo kernel process and jupyter notebook server loses the connection with the kernel. 262 Thus, you must not evaluate `os.Exit` or functions that call it internally (e.g. `log.Fatal`) in lgo. 263 264 If `os.Exit` is not the reason of "Dead kernel", please check crash logs of the kernel. 265 If you run your notebook with `jupyter notebook` command in a terminal, the crash log should be there. 266 If you run your notebook in docker, attach the container's terminal with [`docker attach`](https://docs.docker.com/engine/reference/commandline/attach/) to view the logs. 267 If you can see the logs of `jupyter notebook`, you should see logs like 268 269 ``` 270 2018/03/01 20:30:45 lgo-internal failed: exit status 1 271 [I 22:34:00.500 NotebookApp] KernelRestarter: restarting kernel (1/5) 272 kernel abcd1234-5678-efghi-xxxx-777eeffcccbb restarted 273 ``` 274 275 and you can probably see helpful information before `lgo-internal failed` message. 276 277 ## multiple roots 278 ### Sympton 279 Got an error message like: 280 281 ``` 282 multiple roots $LGOPATH/pkg & 283 Failed to build a shared library of github.com/yunabe/lgo/sess7b..7d/exec1: exit status 1 284 ``` 285 286 ### Solutions 287 This error occurs when the `go` command you are currently using is different from the `go` command you used to run `lgo install`. 288 For example, this happens if you update `go` from 1.9 to 1.10 but did not run `lgo install --clean` with the new `go` after the update. 289 290 If you encouter this issue, please double-check that you are using `go` which you used to run `lgo install` to install packages into `$LGOPATH`. 291 292 ## old export format no longer supported 293 ### Symptom 294 Got error messages like: 295 296 ``` 297 could not import github.com/yunabe/mylib (/home/yunabe/local/gocode/pkg/linux_amd64/github.com/yunabe/mylib.a: import "github.com/yunabe/mylib": old export format no longer supported (recompile library)) 298 ``` 299 300 ### Reason and Solution 301 Some libraries installed in your `$GOPATH` are in the old format, which are built go1.6 or before. 302 Make sure all libraries under your `$GOPATH` are recompiled with your current go compiler. 303 304 ``` 305 cd $GOPATH/src; go install ./... 306 ```