github.com/xhghs/rclone@v1.51.1-0.20200430155106-e186a28cced8/CONTRIBUTING.md (about) 1 # Contributing to rclone # 2 3 This is a short guide on how to contribute things to rclone. 4 5 ## Reporting a bug ## 6 7 If you've just got a question or aren't sure if you've found a bug 8 then please use the [rclone forum](https://forum.rclone.org/) instead 9 of filing an issue. 10 11 When filing an issue, please include the following information if 12 possible as well as a description of the problem. Make sure you test 13 with the [latest beta of rclone](https://beta.rclone.org/): 14 15 * Rclone version (eg output from `rclone -V`) 16 * Which OS you are using and how many bits (eg Windows 7, 64 bit) 17 * The command you were trying to run (eg `rclone copy /tmp remote:tmp`) 18 * A log of the command with the `-vv` flag (eg output from `rclone -vv copy /tmp remote:tmp`) 19 * if the log contains secrets then edit the file with a text editor first to obscure them 20 21 ## Submitting a pull request ## 22 23 If you find a bug that you'd like to fix, or a new feature that you'd 24 like to implement then please submit a pull request via GitHub. 25 26 If it is a big feature then make an issue first so it can be discussed. 27 28 You'll need a Go environment set up with GOPATH set. See [the Go 29 getting started docs](https://golang.org/doc/install) for more info. 30 31 First in your web browser press the fork button on [rclone's GitHub 32 page](https://github.com/rclone/rclone). 33 34 Now in your terminal 35 36 go get -u github.com/rclone/rclone 37 cd $GOPATH/src/github.com/rclone/rclone 38 git remote rename origin upstream 39 git remote add origin git@github.com:YOURUSER/rclone.git 40 41 Make a branch to add your new feature 42 43 git checkout -b my-new-feature 44 45 And get hacking. 46 47 When ready - run the unit tests for the code you changed 48 49 go test -v 50 51 Note that you may need to make a test remote, eg `TestSwift` for some 52 of the unit tests. 53 54 Note the top level Makefile targets 55 56 * make check 57 * make test 58 59 Both of these will be run by Travis when you make a pull request but 60 you can do this yourself locally too. These require some extra go 61 packages which you can install with 62 63 * make build_dep 64 65 Make sure you 66 67 * Add [documentation](#writing-documentation) for a new feature. 68 * Follow the [commit message guidelines](#commit-messages). 69 * Add [unit tests](#testing) for a new feature 70 * squash commits down to one per feature 71 * rebase to master with `git rebase master` 72 73 When you are done with that 74 75 git push origin my-new-feature 76 77 Go to the GitHub website and click [Create pull 78 request](https://help.github.com/articles/creating-a-pull-request/). 79 80 You patch will get reviewed and you might get asked to fix some stuff. 81 82 If so, then make the changes in the same branch, squash the commits, 83 rebase it to master then push it to GitHub with `--force`. 84 85 ## CI for your fork ## 86 87 rclone currently uses [GitHub Actions](https://github.com/rclone/rclone/actions) to build and test the project, which should be automatically available for your fork too from the `Actions` tab in your repository. 88 89 ## Testing ## 90 91 rclone's tests are run from the go testing framework, so at the top 92 level you can run this to run all the tests. 93 94 go test -v ./... 95 96 rclone contains a mixture of unit tests and integration tests. 97 Because it is difficult (and in some respects pointless) to test cloud 98 storage systems by mocking all their interfaces, rclone unit tests can 99 run against any of the backends. This is done by making specially 100 named remotes in the default config file. 101 102 If you wanted to test changes in the `drive` backend, then you would 103 need to make a remote called `TestDrive`. 104 105 You can then run the unit tests in the drive directory. These tests 106 are skipped if `TestDrive:` isn't defined. 107 108 cd backend/drive 109 go test -v 110 111 You can then run the integration tests which tests all of rclone's 112 operations. Normally these get run against the local filing system, 113 but they can be run against any of the remotes. 114 115 cd fs/sync 116 go test -v -remote TestDrive: 117 go test -v -remote TestDrive: -fast-list 118 119 cd fs/operations 120 go test -v -remote TestDrive: 121 122 If you want to use the integration test framework to run these tests 123 all together with an HTML report and test retries then from the 124 project root: 125 126 go install github.com/rclone/rclone/fstest/test_all 127 test_all -backend drive 128 129 If you want to run all the integration tests against all the remotes, 130 then change into the project root and run 131 132 make test 133 134 This command is run daily on the integration test server. You can 135 find the results at https://pub.rclone.org/integration-tests/ 136 137 ## Code Organisation ## 138 139 Rclone code is organised into a small number of top level directories 140 with modules beneath. 141 142 * backend - the rclone backends for interfacing to cloud providers - 143 * all - import this to load all the cloud providers 144 * ...providers 145 * bin - scripts for use while building or maintaining rclone 146 * cmd - the rclone commands 147 * all - import this to load all the commands 148 * ...commands 149 * docs - the documentation and website 150 * content - adjust these docs only - everything else is autogenerated 151 * fs - main rclone definitions - minimal amount of code 152 * accounting - bandwidth limiting and statistics 153 * asyncreader - an io.Reader which reads ahead 154 * config - manage the config file and flags 155 * driveletter - detect if a name is a drive letter 156 * filter - implements include/exclude filtering 157 * fserrors - rclone specific error handling 158 * fshttp - http handling for rclone 159 * fspath - path handling for rclone 160 * hash - defines rclones hash types and functions 161 * list - list a remote 162 * log - logging facilities 163 * march - iterates directories in lock step 164 * object - in memory Fs objects 165 * operations - primitives for sync, eg Copy, Move 166 * sync - sync directories 167 * walk - walk a directory 168 * fstest - provides integration test framework 169 * fstests - integration tests for the backends 170 * mockdir - mocks an fs.Directory 171 * mockobject - mocks an fs.Object 172 * test_all - Runs integration tests for everything 173 * graphics - the images used in the website etc 174 * lib - libraries used by the backend 175 * atexit - register functions to run when rclone exits 176 * dircache - directory ID to name caching 177 * oauthutil - helpers for using oauth 178 * pacer - retries with backoff and paces operations 179 * readers - a selection of useful io.Readers 180 * rest - a thin abstraction over net/http for REST 181 * vendor - 3rd party code managed by `go mod` 182 * vfs - Virtual FileSystem layer for implementing rclone mount and similar 183 184 ## Writing Documentation ## 185 186 If you are adding a new feature then please update the documentation. 187 188 If you add a new general flag (not for a backend), then document it in 189 `docs/content/docs.md` - the flags there are supposed to be in 190 alphabetical order. 191 192 If you add a new backend option/flag, then it should be documented in 193 the source file in the `Help:` field. The first line of this is used 194 for the flag help, the remainder is shown to the user in `rclone 195 config` and is added to the docs with `make backenddocs`. 196 197 The only documentation you need to edit are the `docs/content/*.md` 198 files. The MANUAL.*, rclone.1, web site etc are all auto generated 199 from those during the release process. See the `make doc` and `make 200 website` targets in the Makefile if you are interested in how. You 201 don't need to run these when adding a feature. 202 203 Documentation for rclone sub commands is with their code, eg 204 `cmd/ls/ls.go`. 205 206 ## Making a release ## 207 208 There are separate instructions for making a release in the RELEASE.md 209 file. 210 211 ## Commit messages ## 212 213 Please make the first line of your commit message a summary of the 214 change that a user (not a developer) of rclone would like to read, and 215 prefix it with the directory of the change followed by a colon. The 216 changelog gets made by looking at just these first lines so make it 217 good! 218 219 If you have more to say about the commit, then enter a blank line and 220 carry on the description. Remember to say why the change was needed - 221 the commit itself shows what was changed. 222 223 Writing more is better than less. Comparing the behaviour before the 224 change to that after the change is very useful. Imagine you are 225 writing to yourself in 12 months time when you've forgotten everything 226 about what you just did and you need to get up to speed quickly. 227 228 If the change fixes an issue then write `Fixes #1234` in the commit 229 message. This can be on the subject line if it will fit. If you 230 don't want to close the associated issue just put `#1234` and the 231 change will get linked into the issue. 232 233 Here is an example of a short commit message: 234 235 ``` 236 drive: add team drive support - fixes #885 237 ``` 238 239 And here is an example of a longer one: 240 241 ``` 242 mount: fix hang on errored upload 243 244 In certain circumstances if an upload failed then the mount could hang 245 indefinitely. This was fixed by closing the read pipe after the Put 246 completed. This will cause the write side to return a pipe closed 247 error fixing the hang. 248 249 Fixes #1498 250 ``` 251 252 ## Adding a dependency ## 253 254 rclone uses the [go 255 modules](https://tip.golang.org/cmd/go/#hdr-Modules__module_versions__and_more) 256 support in go1.11 and later to manage its dependencies. 257 258 **NB** you must be using go1.11 or above to add a dependency to 259 rclone. Rclone will still build with older versions of go, but we use 260 the `go mod` command for dependencies which is only in go1.11 and 261 above. 262 263 rclone can be built with modules outside of the GOPATH, but for 264 backwards compatibility with older go versions, rclone also maintains 265 a `vendor` directory with all the external code rclone needs for 266 building. 267 268 The `vendor` directory is entirely managed by the `go mod` tool, do 269 not add things manually. 270 271 To add a dependency `github.com/ncw/new_dependency` see the 272 instructions below. These will fetch the dependency, add it to 273 `go.mod` and `go.sum` and vendor it for older go versions. 274 275 GO111MODULE=on go get github.com/ncw/new_dependency 276 GO111MODULE=on go mod vendor 277 278 You can add constraints on that package when doing `go get` (see the 279 go docs linked above), but don't unless you really need to. 280 281 Please check in the changes generated by `go mod` including the 282 `vendor` directory and `go.mod` and `go.sum` in a single commit 283 separate from any other code changes with the title "vendor: add 284 github.com/ncw/new_dependency". Remember to `git add` any new files 285 in `vendor`. 286 287 ## Updating a dependency ## 288 289 If you need to update a dependency then run 290 291 GO111MODULE=on go get -u github.com/pkg/errors 292 GO111MODULE=on go mod vendor 293 294 Check in in a single commit as above. 295 296 ## Updating all the dependencies ## 297 298 In order to update all the dependencies then run `make update`. This 299 just uses the go modules to update all the modules to their latest 300 stable release. Check in the changes in a single commit as above. 301 302 This should be done early in the release cycle to pick up new versions 303 of packages in time for them to get some testing. 304 305 ## Updating a backend ## 306 307 If you update a backend then please run the unit tests and the 308 integration tests for that backend. 309 310 Assuming the backend is called `remote`, make create a config entry 311 called `TestRemote` for the tests to use. 312 313 Now `cd remote` and run `go test -v` to run the unit tests. 314 315 Then `cd fs` and run `go test -v -remote TestRemote:` to run the 316 integration tests. 317 318 The next section goes into more detail about the tests. 319 320 ## Writing a new backend ## 321 322 Choose a name. The docs here will use `remote` as an example. 323 324 Note that in rclone terminology a file system backend is called a 325 remote or an fs. 326 327 Research 328 329 * Look at the interfaces defined in `fs/fs.go` 330 * Study one or more of the existing remotes 331 332 Getting going 333 334 * Create `backend/remote/remote.go` (copy this from a similar remote) 335 * box is a good one to start from if you have a directory based remote 336 * b2 is a good one to start from if you have a bucket based remote 337 * Add your remote to the imports in `backend/all/all.go` 338 * HTTP based remotes are easiest to maintain if they use rclone's rest module, but if there is a really good go SDK then use that instead. 339 * Try to implement as many optional methods as possible as it makes the remote more usable. 340 * Use lib/encoder to make sure we can encode any path name and `rclone info` to help determine the encodings needed 341 * `rclone purge -v TestRemote:rclone-info` 342 * `rclone info --remote-encoding None -vv --write-json remote.json TestRemote:rclone-info` 343 * `go run cmd/info/internal/build_csv/main.go -o remote.csv remote.json` 344 * open `remote.csv` in a spreadsheet and examine 345 346 Unit tests 347 348 * Create a config entry called `TestRemote` for the unit tests to use 349 * Create a `backend/remote/remote_test.go` - copy and adjust your example remote 350 * Make sure all tests pass with `go test -v` 351 352 Integration tests 353 354 * Add your backend to `fstest/test_all/config.yaml` 355 * Once you've done that then you can use the integration test framework from the project root: 356 * go install ./... 357 * test_all -backends remote 358 359 Or if you want to run the integration tests manually: 360 361 * Make sure integration tests pass with 362 * `cd fs/operations` 363 * `go test -v -remote TestRemote:` 364 * `cd fs/sync` 365 * `go test -v -remote TestRemote:` 366 * If your remote defines `ListR` check with this also 367 * `go test -v -remote TestRemote: -fast-list` 368 369 See the [testing](#testing) section for more information on integration tests. 370 371 Add your fs to the docs - you'll need to pick an icon for it from 372 [fontawesome](http://fontawesome.io/icons/). Keep lists of remotes in 373 alphabetical order of full name of remote (eg `drive` is ordered as 374 `Google Drive`) but with the local file system last. 375 376 * `README.md` - main GitHub page 377 * `docs/content/remote.md` - main docs page (note the backend options are automatically added to this file with `make backenddocs`) 378 * make sure this has the `autogenerated options` comments in (see your reference backend docs) 379 * update them with `make backenddocs` - revert any changes in other backends 380 * `docs/content/overview.md` - overview docs 381 * `docs/content/docs.md` - list of remotes in config section 382 * `docs/content/about.md` - front page of rclone.org 383 * `docs/layouts/chrome/navbar.html` - add it to the website navigation 384 * `bin/make_manual.py` - add the page to the `docs` constant 385 386 Once you've written the docs, run `make serve` and check they look OK 387 in the web browser and the links (internal and external) all work. 388 389 ## Writing a plugin ## 390 391 New features (backends, commands) can also be added "out-of-tree", through Go plugins. 392 Changes will be kept in a dynamically loaded file instead of being compiled into the main binary. 393 This is useful if you can't merge your changes upstream or don't want to maintain a fork of rclone. 394 395 Usage 396 397 - Naming 398 - Plugins names must have the pattern `librcloneplugin_KIND_NAME.so`. 399 - `KIND` should be one of `backend`, `command` or `bundle`. 400 - Example: A plugin with backend support for PiFS would be called 401 `librcloneplugin_backend_pifs.so`. 402 - Loading 403 - Supported on macOS & Linux as of now. ([Go issue for Windows support](https://github.com/golang/go/issues/19282)) 404 - Supported on rclone v1.50 or greater. 405 - All plugins in the folder specified by variable `$RCLONE_PLUGIN_PATH` are loaded. 406 - If this variable doesn't exist, plugin support is disabled. 407 - Plugins must be compiled against the exact version of rclone to work. 408 (The rclone used during building the plugin must be the same as the source of rclone) 409 410 Building 411 412 To turn your existing additions into a Go plugin, move them to an external repository 413 and change the top-level package name to `main`. 414 415 Check `rclone --version` and make sure that the plugin's rclone dependency and host Go version match. 416 417 Then, run `go build -buildmode=plugin -o PLUGIN_NAME.so .` to build the plugin. 418 419 [Go reference](https://godoc.org/github.com/rclone/rclone/lib/plugin) 420 421 [Minimal example](https://gist.github.com/terorie/21b517ee347828e899e1913efc1d684f)