github.com/akaros/go-akaros@v0.0.0-20181004170632-85005d477eab/README.akaros (about) 1 Go on Akaros 2 Brian Wheatman 3 2018-08-31 4 5 ################################################# 6 Getting started with Go on Akaros 7 ################################################# 8 This guide assumes you have set up Akaros following the directions here 9 https://github.com/brho/akaros/blob/master/GETTING_STARTED.md 10 as well as having ssh working by using https://github.com/akaros/dropbear-akaros. 11 Roughly you should be able to start your instance and ssh into by using “ssh <akaros_server>”. 12 13 14 The default server is currently qemu, but if you want to use a different server then set the value of AKAROS_SERVER to the name of the server. 15 16 First clone the go-akaros repo and make a directory outside of it for your Go workspace. 17 You will need a variable for each of these locations, mine are as followed. 18 19 export AKAROS_GOROOT="$HOME/go-akaros1.4/go-akaros/" 20 export AKAROS_GOPATH="$HOME/go-akaros1.4/go-workspace/" 21 22 You will then need to build the Go source for Akaros. 23 24 cd to $AKAROS_GOROOT/src and run “./akaros.bash make” 25 26 27 Start your ufs server with 28 ufs -root="$AKAROS_GOROOT" -addr="127.0.0.1:1025" & 29 30 On the Akaros side you will need to run ./mountroot to set up the ufs server on the Akaros side. 31 32 Then when you want to run a command with Akaros's version of Go just use 33 "GOOS=akaros GOARCH=amd64 GOROOT=$AKAROS_GOROOT GOPATH=$AKAROS_GOPATH PATH=$AKAROS_GOROOT/bin:$AKAROS_GOROOT/misc/akaros/bin:$AKAROS_GOPATH/bin:$PATH CGO_ENABLED=1 $AKAROS_GOROOT/bin/go" 34 35 Personally I set this up to an alias as 36 alias akaros_go="GOOS=akaros GOARCH=amd64 GOROOT=$AKAROS_GOROOT GOPATH=$AKAROS_GOPATH PATH=$AKAROS_GOROOT/bin:$AKAROS_GOROOT/misc/akaros/bin:$AKAROS_GOPATH/bin:$PATH CGO_ENABLED=1 $AKAROS_GOROOT/bin/go" 37 38 But you could also map it to the go command itself. 39 40 This command is designed to be self contained so that it does not mess with your environment at all. 41 42 ################################################# 43 The differences between Akaros and other OS ports 44 ################################################# 45 46 Most of the major difference in the Akaros port of Go vs that of other OS’s is that Akaros does much more in user space. 47 In many ways Akaros’s second level scheduler (2LS) is like part of the OS of other operating systems. 48 Because these operations are dealt with in user space we are able to deal with them directly instead of trapping into the kernel. 49 50 We sometimes want the ability to call functions from Akaros’s standard libraries, for example things like syscall, futex, yield, or enable_profalarm. 51 52 There are a few issues to deal with when calling into the 2LS. 53 54 The first is calling convention. 55 As of this writing (Go 1.4) Go passes all of the arguments on the stack. 56 While gcc (which the 2LS is compiled with) passes the first 6 arguments in registers then the rest on the stack. 57 58 The next issue is linking. 59 Go code is compiled separately from the 2LS code where many of the functions we want to call are; these must be fixed at link time or run time. 60 61 We also have to worry about what stack we will run the function on. 62 Each Go routine has its own stack which grows as needed; these events are called stack splits. 63 When these happen not only is the data in the stack moved to a new location, 64 but also all pointers into the stack are updated to point to their new location. 65 This can cause issues because only Go pointers are updated and not C pointers. 66 A more detailed description of this and how it can cause issues is found in the syscall section below. 67 The C code we are calling has no knowledge of how this all works. 68 Thus, before it is called we need to switch over to a stack which is safe for arbitrary code execution. 69 70 71 We have two separate methods for calling GCC C code, one from Go the other from Ken C. 72 73 ################################################# 74 Calling standard library code from Ken C 75 ################################################# 76 77 The C side roughly lives in src/runtime/sys_akaros.c, src/runtime/parlib/gcc_akaros.c, and src/runtime/parlib/gcc_akaros.h. 78 The set of functions that this has been set up for are: 79 ros_syscall_sync 80 futex, pthread_yield 81 sigaction 82 sigaltstack 83 pthread_sigmask 84 enable_profalarm 85 disable_profalarm 86 To call one of these follow the example in src/runtime/sys_akaros.c. 87 You need to use the wrappers defined in src/runtime/sys_akaros.c and use runtime·asmcgocall. 88 Also you are limited to a single argument, so the wrappers take a pointer to a struct. 89 To make a new one you need to define the struct of arguments in src/runtime/parlib/gcc_akaros.h. 90 Define the wrapper and const gcc_call_t type in src/runtime/parlib/gcc_akaros.c, you can then call it with runtime·asmcgocall. 91 92 The gcc prefix to the file names controls which compilier is used to compile the file 93 and runtime·asmcgocall allows calling of a C function and deals with changing stacks and calling convention. 94 One of the issues is its limit to a single argument. 95 The linking issue is dealt with by special cgo imports and externing the functions. 96 97 ################################################# 98 Calling standard library code from Go 99 ################################################# 100 101 The Go side lives in the src/usys package. 102 This is supposed to model a system call as much as possible. 103 The first argument is what function to run followed by the arguments. 104 The difference is that it does not need to change rings so we can call directly whatever function we want. 105 The calling convention conversion and stack switching is done in the usys package. 106 107 The set of functions that are already set up are: 108 abort_syscall_at_abs_unix 109 unset_alarm 110 go_syscall 111 which wraps ros_syscall_sync and syscall_retval_is_error which 112 ensures that errno and errstr are zero when there is no error 113 go_usys_tester 114 which is to test the usys package 115 futex 116 serialize_argv_envp 117 free 118 To call one of these functions just import the usys package and use the usys Call or Call1 methods. 119 These take one of the constants defined at the top of src/usys/usys.go as the first argument 120 which tells which function to run then just takes the arguments to the function. 121 Call takes arbitary arguments, while Call1 takes exactly one argument to the function. 122 The advantage of Call1 is that in Go variable arguments are passed in as a slice which causes an extra alloc, so Call1 is preferable. 123 124 The way usys works is that when you import usys the init function is run before any of your code, 125 this causes a general protection fault with known high 16 bits. 126 The lower 48 bits are used to pass in the address of a table. 127 When the 2LS fault handler gets a fault with the correct high order bits in fills in the table with function addresses. 128 Then when a Call or Call1 is made it deals with stack switching and calls the correct function pointer with the arguments directly. 129 130 To add a new function you need to add it to both sides: both the fault handler and the usys package. 131 In the usys package, just add to the constants at the top insuring to increment num_functions. 132 In $AKAROS_ROOT/user/pthread/pthread.c in set_up_go_table, add the function you want to call. 133 134 The stack issue and calling convention are dealt with by the assembly in src/usys/usys.s. 135 This changes the stack and calling convention. 136 We don’t have any linking issues since the code is not linked together and instead the function pointer table is set up at runtime. 137 This does have the extra concern that the two sides must be kept in sync. 138 There is a sentinel at the end of the table, but full checking cannot be done. 139 140 Ways to improve: 141 Usys could also be used to get constants from Akaros. 142 This is currently done using cgo, 143 but if we wanted to remove cgo we could get the constants in the same was as the function pointer table is set up. 144 You could set up a second table to hold the constants and pass it in with another general protection fault 145 then set up the constants in the table in the fault handler. 146 Ideally we wouldn’t have two separate paths 147 We could combine the gcc_ path and the usys path into a single package that could be called from both C and Go code. 148 It is also possible the C path with no longer be necessary when Go removes C from its source 149 150 151 ################################################# 152 Syscall 153 ################################################# 154 155 One of the big users of the usys package is the syscall package. 156 Syscalls now work very similar to how they work on other OS’s. 157 158 There is a perl script, src/syscall/mksyscall.pl, 159 which generates a function for each system call (zsyscall_akaros_amd64.go), 160 some system calls need extra wrappers, others can be made entirely by the perl script. 161 The ones that are wrapped use lowercase for their generated function so that function is not exported, 162 it also leaves the basic name for use by the exported wrapper. 163 These autogenerated functions basically just take the arguments and pass them through to whatever actually makes the system call. 164 The autogenerated functions also do error checking. 165 166 On most OS’s this is a small assembly function which sets up arguments and makes the syscall instruction. 167 Akaros instead uses usys instead of the syscall instruction. 168 This generated function also deals with the fact that Akaros syscall errors contain both an error number and a string. 169 The requires an extra alloc for the object which contains both of these. 170 We use usys.Call1 to limit the number of extra mallocs since Call passes the arguments in a slice and we don’t want this extra creation. 171 172 This next section will walk through the old method and other solutions to the problem as an illustration in the ways that things can go wrong. 173 174 We used to have the following two functions: 175 176 In package syscall: 177 func Syscall6(trap, a1, a2, a3, a4, a5, a6 uintptr) (r1, r2 uintptr, err error) { 178 // I have syscall numbers >=300 stubbed out since they are not yet 179 // implemented. If we are trying to call one of those, print out a warning 180 // and return an error. 181 if trap >= 300 { 182 parlib.SyscallError(trap); 183 return r1, r2, EINVAL 184 } 185 186 // Otherwise, run the syscall! 187 __r1, __err, __errstr := parlib.Syscall(uint32(trap), int(a1), int(a2), 188 int(a3), int(a4), int(a5), int(a6)) 189 190 var akaerror error = nil 191 if __r1 == -1 { 192 akaerror = NewAkaError(Errno(__err), string(__errstr)) 193 } 194 return uintptr(__r1), r2, akaerror 195 } 196 197 And in package parlib 198 func Syscall(_num uint32, _a0, _a1, _a2, _a3, _a4, _a5 int) (ret int, err int32, errstr string) { 199 var syscall SyscallType 200 syscall.num = C.uint(_num) 201 syscall.ev_q = (*C.struct_event_queue)(unsafe.Pointer(nil)) 202 syscall.arg0 = C.long(_a0) 203 syscall.arg1 = C.long(_a1) 204 syscall.arg2 = C.long(_a2) 205 syscall.arg3 = C.long(_a3) 206 syscall.arg4 = C.long(_a4) 207 syscall.arg5 = C.long(_a5) 208 C.ros_syscall_sync((*C.struct_syscall)(unsafe.Pointer(&syscall))) 209 return int(syscall.retval), int32(syscall.err), C.GoString(&syscall.errstr[0]) 210 } 211 212 This allowed the generated code to be almost identical to all other OS’s 213 since we could just call the Syscall6 function just like all of them called their 214 own Syscall6 function (the wrapper around the syscall instruction). 215 216 ################################################# 217 NOSPLIT 218 ################################################# 219 220 The main issue with this is dealing with the possibility of stack splits. 221 The Go compiler inserts a preamble into every function which checks how close to the end of the stack we currently are 222 and if we will get too close to the end in the current function it allocates a new stack and copies over all the data. 223 In this process it also fixes all pointers so that any pointer that pointed into the old stack 224 now points to the corresponding object in the new stack. 225 The issue with this is that the Syscall6 function took in uintptr types. 226 These are to the Go compiler an int type which happens to be the same size as a ptr type. 227 This means that in the event of a stack split they are not fixed. 228 The way to stop these stack checks are to mark the function as nosplit. 229 This stops the stack split in that function, but it only works if every function that might be called, 230 and everything they might call are also nosplit. 231 Also there is a total size the nosplit stack is allowed to be and it is quite small. 232 233 An easy solution to the problem seems to be to mark the functions as nosplit. 234 The issue with this is that nosplit functions are not able to create objects or use cgo, 235 since both of these have functions that could possibly split. 236 So syscall cannot be marked as nosplit since it both uses cgo and creates objects. 237 238 The next solution might be to bubble up the cgo call into the individual generated functions. 239 Since it is in these functions that we convert objects from their real Go types into uintptrs. 240 If we place this conversion in the same place as the call into cgo we don’t have any stack splits in between. 241 The issue with this is that we cannot use cgo in the syscall package. 242 This is because cgo implicitly depends on syscall and using cgo in syscall would create a dependency loop. 243 A solution to this is to move the individual generated functions over into Go's runtime·parlib so that they can make their direct calls into cgo. 244 To be honest, this does work, but it creates a giant mess. 245 Not only is the syscall logic broken into two packages, 246 but also we need to duplicate struct definitions since we need these objects in both places and we need to be careful about dependency loops. 247 We also need to be considerate that standard Go code will call these functions 248 and not just Akaros specific code meaning changing types and signatures is difficult. 249 250 This is why we use usys, everything below the usys call is nosplit and it has no dependencies. 251 As a note the usys call itself cannot go in the nosplit function since it makes a slice for the variable number of arguments, 252 Call1 fixes this issue, other constant argument lengths could be made in the same way if needed. 253 The reason we do not have a single syscall function and which handles error checking and calling out to usys like other OS’s 254 is that our errors involve a struct that has both a number and a string. 255 This requires a new object, so we cannot have a general function which takes in the syscall argument and returns the result and error, 256 since those arguments would have to be uintptrs, meaning we couldn’t have any splits and the creation of the struct object could cause a split. 257 258 259 ################################################# 260 Signals 261 ################################################# 262 The other major difference is in how we handle signals. 263 The relevant code is in src/runtime/parlib/signal.go and src/runtime/sys_akaros_amd64.s. 264 265 This has a lot of things that, at first, seem like a mess, but each are necessary. 266 I will walk through how we process signals and try and explain how each work and why it is done the way it is. 267 268 It starts with the init function in parlib.go. 269 An init function runs at the start of the program whenever that package is imported. 270 And if package foo imports package bar then foo’s init will run before bar’s init. 271 This function installs the signal handlers using Signal, sets up the wtf variable (which will be explained later), 272 and starts the process process_signals (also explained later). 273 Installing a signal handler involves setting up the function in the Go table of signal handlers and installs it in the system using sigaction. 274 The sigaction installed is always sig_hand declared in cgo at the top of signal.go. 275 When we update the signal handler we do not update the sigaction. 276 277 When a signal is triggered sig_hand is called, it first checks whether or not we are in vcore context. 278 If we are in vcore context we are not able to do much of anything. 279 So we kick a futex to wake up a thread so that we can process the signal from outside of vcore context. 280 If we are not in vcore context we can directly call the signal handler through wtf. 281 The process that is waiting on the futex is process_signals (started in init). 282 This determines which signal to call (its possible multiple were triggered at once), then for the first signal calls the appropriate signal handler. 283 If somebody has changed it we assume they have done so properly and we directly call whatever function they set up. 284 If nobody has changed it then we convert the signal into an internal signal. 285 This is done by calling a pthread_kill with the appropriate signal number and immediately yielding. 286 This is done so we only need to deal with one type of signal: process-wide (run in vcore-context) signals are converted to a per-uthread signal. 287 This will loop back around and trigger sig_hand once again. 288 Eventually (most likely immediately) we will be in sig_hand while not in vcore context. 289 290 Now the wtf variable. 291 This is equal to the address of the default signal handler. 292 There is some extra complexity that we need to access this both from the runtime package and the parlib package and we don’t want to have imports. 293 Also we need it in both cgo code and in Go’s C code. 294 Added to this is the fact that Go does not let cgo code see functions that have not been defined. 295 This means that cgo code cannot directly call functions which have been defined in assembly in Go. 296 We get around this by passing in the address of the function at runtime into cgo. 297 The get_value function deals all the casting to get around Go’s issues with passing pointers. 298 299 Moving on, once we are in sig_hand while not in vcore context we call defaultSighandler (wtf in cgo). 300 This is defined in src/runtime/sys_akaros_amd64.s. 301 This just does some calling convention conversion and calls sigtramp_real. 302 Sigtramp_real has two cases, the first is that we are not on a g, if we are not on a g we call sig_hand and loop back around. 303 This can happen if an odd thread handles the signal. 304 If we are on a g we switch the that g’s signal stack and run runtime.sighandler. 305 306 ################################################# 307 Process of updating 1.3 to 1.4 308 ################################################# 309 Here I will explain my process for going from Go 1.3 to Go 1.4. 310 I tried to be fairly systematic in my approach. 311 My goal was that once I finished it would look like the Akaros port had been in development along with the main Go branch. 312 313 I started with a git tree of the akros port of Go 1.3 which worked. 314 I then found the split point of Go 1.3 and Go 1.4. 315 I then rebased all Akaros specific commits from after the split point onto this split point. 316 After this I rebased all the commits from Go 1.4 from after the split point on top of those Akaros specific commits. 317 318 From this point I used a multi pass approach. 319 At first I moved forward through the commit history and made sure it could compile at every point. 320 This mostly involved adding function definitions for new ideas that were added. 321 The second pass involved making sure all of the tests could compile. 322 Lastly, the third pass involved actually running the tests. 323 The goal was that at each point in the commit history I wanted the branch to work to some degree. 324 325 The advantage of this multi pass approach is it was easy to see what commits broke since you could easily test the commit before and after. 326 One example of this being particularly helpful is that we had lots of issues that all seemed unrelated, 327 but were actually the result of changing the default size of the stack. 328 By determining the commit that broke those tests just changed the size of the stack it was much easier to find and fix. 329 330