github.com/ncw/rclone@v1.48.1-0.20190724201158-a35aa1360e3e/CONTRIBUTING.md (about) 1 # Contributing to rclone # 2 3 This is a short guide on how to contribute things to rclone. 4 5 ## Reporting a bug ## 6 7 If you've just got a question or aren't sure if you've found a bug 8 then please use the [rclone forum](https://forum.rclone.org/) instead 9 of filing an issue. 10 11 When filing an issue, please include the following information if 12 possible as well as a description of the problem. Make sure you test 13 with the [latest beta of rclone](https://beta.rclone.org/): 14 15 * Rclone version (eg output from `rclone -V`) 16 * Which OS you are using and how many bits (eg Windows 7, 64 bit) 17 * The command you were trying to run (eg `rclone copy /tmp remote:tmp`) 18 * A log of the command with the `-vv` flag (eg output from `rclone -vv copy /tmp remote:tmp`) 19 * if the log contains secrets then edit the file with a text editor first to obscure them 20 21 ## Submitting a pull request ## 22 23 If you find a bug that you'd like to fix, or a new feature that you'd 24 like to implement then please submit a pull request via GitHub. 25 26 If it is a big feature then make an issue first so it can be discussed. 27 28 You'll need a Go environment set up with GOPATH set. See [the Go 29 getting started docs](https://golang.org/doc/install) for more info. 30 31 First in your web browser press the fork button on [rclone's GitHub 32 page](https://github.com/ncw/rclone). 33 34 Now in your terminal 35 36 go get -u github.com/ncw/rclone 37 cd $GOPATH/src/github.com/ncw/rclone 38 git remote rename origin upstream 39 git remote add origin git@github.com:YOURUSER/rclone.git 40 41 Make a branch to add your new feature 42 43 git checkout -b my-new-feature 44 45 And get hacking. 46 47 When ready - run the unit tests for the code you changed 48 49 go test -v 50 51 Note that you may need to make a test remote, eg `TestSwift` for some 52 of the unit tests. 53 54 Note the top level Makefile targets 55 56 * make check 57 * make test 58 59 Both of these will be run by Travis when you make a pull request but 60 you can do this yourself locally too. These require some extra go 61 packages which you can install with 62 63 * make build_dep 64 65 Make sure you 66 67 * Add [documentation](#writing-documentation) for a new feature. 68 * Follow the [commit message guidelines](#commit-messages). 69 * Add [unit tests](#testing) for a new feature 70 * squash commits down to one per feature 71 * rebase to master with `git rebase master` 72 73 When you are done with that 74 75 git push origin my-new-feature 76 77 Go to the GitHub website and click [Create pull 78 request](https://help.github.com/articles/creating-a-pull-request/). 79 80 You patch will get reviewed and you might get asked to fix some stuff. 81 82 If so, then make the changes in the same branch, squash the commits, 83 rebase it to master then push it to GitHub with `--force`. 84 85 ## Enabling CI for your fork ## 86 87 The CI config files for rclone have taken care of forks of the project, so you can enable CI for your fork repo easily. 88 89 rclone currently uses [Travis CI](https://travis-ci.org/), [AppVeyor](https://ci.appveyor.com/), and 90 [Circle CI](https://circleci.com/) to build the project. To enable them for your fork, simply go into their 91 websites, find your fork of rclone, and enable building there. 92 93 ## Testing ## 94 95 rclone's tests are run from the go testing framework, so at the top 96 level you can run this to run all the tests. 97 98 go test -v ./... 99 100 rclone contains a mixture of unit tests and integration tests. 101 Because it is difficult (and in some respects pointless) to test cloud 102 storage systems by mocking all their interfaces, rclone unit tests can 103 run against any of the backends. This is done by making specially 104 named remotes in the default config file. 105 106 If you wanted to test changes in the `drive` backend, then you would 107 need to make a remote called `TestDrive`. 108 109 You can then run the unit tests in the drive directory. These tests 110 are skipped if `TestDrive:` isn't defined. 111 112 cd backend/drive 113 go test -v 114 115 You can then run the integration tests which tests all of rclone's 116 operations. Normally these get run against the local filing system, 117 but they can be run against any of the remotes. 118 119 cd fs/sync 120 go test -v -remote TestDrive: 121 go test -v -remote TestDrive: -subdir 122 123 cd fs/operations 124 go test -v -remote TestDrive: 125 126 If you want to use the integration test framework to run these tests 127 all together with an HTML report and test retries then from the 128 project root: 129 130 go install github.com/ncw/rclone/fstest/test_all 131 test_all -backend drive 132 133 If you want to run all the integration tests against all the remotes, 134 then change into the project root and run 135 136 make test 137 138 This command is run daily on the integration test server. You can 139 find the results at https://pub.rclone.org/integration-tests/ 140 141 ## Code Organisation ## 142 143 Rclone code is organised into a small number of top level directories 144 with modules beneath. 145 146 * backend - the rclone backends for interfacing to cloud providers - 147 * all - import this to load all the cloud providers 148 * ...providers 149 * bin - scripts for use while building or maintaining rclone 150 * cmd - the rclone commands 151 * all - import this to load all the commands 152 * ...commands 153 * docs - the documentation and website 154 * content - adjust these docs only - everything else is autogenerated 155 * fs - main rclone definitions - minimal amount of code 156 * accounting - bandwidth limiting and statistics 157 * asyncreader - an io.Reader which reads ahead 158 * config - manage the config file and flags 159 * driveletter - detect if a name is a drive letter 160 * filter - implements include/exclude filtering 161 * fserrors - rclone specific error handling 162 * fshttp - http handling for rclone 163 * fspath - path handling for rclone 164 * hash - defines rclones hash types and functions 165 * list - list a remote 166 * log - logging facilities 167 * march - iterates directories in lock step 168 * object - in memory Fs objects 169 * operations - primitives for sync, eg Copy, Move 170 * sync - sync directories 171 * walk - walk a directory 172 * fstest - provides integration test framework 173 * fstests - integration tests for the backends 174 * mockdir - mocks an fs.Directory 175 * mockobject - mocks an fs.Object 176 * test_all - Runs integration tests for everything 177 * graphics - the images used in the website etc 178 * lib - libraries used by the backend 179 * atexit - register functions to run when rclone exits 180 * dircache - directory ID to name caching 181 * oauthutil - helpers for using oauth 182 * pacer - retries with backoff and paces operations 183 * readers - a selection of useful io.Readers 184 * rest - a thin abstraction over net/http for REST 185 * vendor - 3rd party code managed by `go mod` 186 * vfs - Virtual FileSystem layer for implementing rclone mount and similar 187 188 ## Writing Documentation ## 189 190 If you are adding a new feature then please update the documentation. 191 192 If you add a new general flag (not for a backend), then document it in 193 `docs/content/docs.md` - the flags there are supposed to be in 194 alphabetical order. 195 196 If you add a new backend option/flag, then it should be documented in 197 the source file in the `Help:` field. The first line of this is used 198 for the flag help, the remainder is shown to the user in `rclone 199 config` and is added to the docs with `make backenddocs`. 200 201 The only documentation you need to edit are the `docs/content/*.md` 202 files. The MANUAL.*, rclone.1, web site etc are all auto generated 203 from those during the release process. See the `make doc` and `make 204 website` targets in the Makefile if you are interested in how. You 205 don't need to run these when adding a feature. 206 207 Documentation for rclone sub commands is with their code, eg 208 `cmd/ls/ls.go`. 209 210 ## Making a release ## 211 212 There are separate instructions for making a release in the RELEASE.md 213 file. 214 215 ## Commit messages ## 216 217 Please make the first line of your commit message a summary of the 218 change that a user (not a developer) of rclone would like to read, and 219 prefix it with the directory of the change followed by a colon. The 220 changelog gets made by looking at just these first lines so make it 221 good! 222 223 If you have more to say about the commit, then enter a blank line and 224 carry on the description. Remember to say why the change was needed - 225 the commit itself shows what was changed. 226 227 Writing more is better than less. Comparing the behaviour before the 228 change to that after the change is very useful. Imagine you are 229 writing to yourself in 12 months time when you've forgotten everything 230 about what you just did and you need to get up to speed quickly. 231 232 If the change fixes an issue then write `Fixes #1234` in the commit 233 message. This can be on the subject line if it will fit. If you 234 don't want to close the associated issue just put `#1234` and the 235 change will get linked into the issue. 236 237 Here is an example of a short commit message: 238 239 ``` 240 drive: add team drive support - fixes #885 241 ``` 242 243 And here is an example of a longer one: 244 245 ``` 246 mount: fix hang on errored upload 247 248 In certain circumstances if an upload failed then the mount could hang 249 indefinitely. This was fixed by closing the read pipe after the Put 250 completed. This will cause the write side to return a pipe closed 251 error fixing the hang. 252 253 Fixes #1498 254 ``` 255 256 ## Adding a dependency ## 257 258 rclone uses the [go 259 modules](https://tip.golang.org/cmd/go/#hdr-Modules__module_versions__and_more) 260 support in go1.11 and later to manage its dependencies. 261 262 **NB** you must be using go1.11 or above to add a dependency to 263 rclone. Rclone will still build with older versions of go, but we use 264 the `go mod` command for dependencies which is only in go1.11 and 265 above. 266 267 rclone can be built with modules outside of the GOPATH, but for 268 backwards compatibility with older go versions, rclone also maintains 269 a `vendor` directory with all the external code rclone needs for 270 building. 271 272 The `vendor` directory is entirely managed by the `go mod` tool, do 273 not add things manually. 274 275 To add a dependency `github.com/ncw/new_dependency` see the 276 instructions below. These will fetch the dependency, add it to 277 `go.mod` and `go.sum` and vendor it for older go versions. 278 279 GO111MODULE=on go get github.com/ncw/new_dependency 280 GO111MODULE=on go mod vendor 281 282 You can add constraints on that package when doing `go get` (see the 283 go docs linked above), but don't unless you really need to. 284 285 Please check in the changes generated by `go mod` including the 286 `vendor` directory and `go.mod` and `go.sum` in a single commit 287 separate from any other code changes with the title "vendor: add 288 github.com/ncw/new_dependency". Remember to `git add` any new files 289 in `vendor`. 290 291 ## Updating a dependency ## 292 293 If you need to update a dependency then run 294 295 GO111MODULE=on go get -u github.com/pkg/errors 296 GO111MODULE=on go mod vendor 297 298 Check in in a single commit as above. 299 300 ## Updating all the dependencies ## 301 302 In order to update all the dependencies then run `make update`. This 303 just uses the go modules to update all the modules to their latest 304 stable release. Check in the changes in a single commit as above. 305 306 This should be done early in the release cycle to pick up new versions 307 of packages in time for them to get some testing. 308 309 ## Updating a backend ## 310 311 If you update a backend then please run the unit tests and the 312 integration tests for that backend. 313 314 Assuming the backend is called `remote`, make create a config entry 315 called `TestRemote` for the tests to use. 316 317 Now `cd remote` and run `go test -v` to run the unit tests. 318 319 Then `cd fs` and run `go test -v -remote TestRemote:` to run the 320 integration tests. 321 322 The next section goes into more detail about the tests. 323 324 ## Writing a new backend ## 325 326 Choose a name. The docs here will use `remote` as an example. 327 328 Note that in rclone terminology a file system backend is called a 329 remote or an fs. 330 331 Research 332 333 * Look at the interfaces defined in `fs/fs.go` 334 * Study one or more of the existing remotes 335 336 Getting going 337 338 * Create `backend/remote/remote.go` (copy this from a similar remote) 339 * box is a good one to start from if you have a directory based remote 340 * b2 is a good one to start from if you have a bucket based remote 341 * Add your remote to the imports in `backend/all/all.go` 342 * HTTP based remotes are easiest to maintain if they use rclone's rest module, but if there is a really good go SDK then use that instead. 343 * Try to implement as many optional methods as possible as it makes the remote more usable. 344 345 Unit tests 346 347 * Create a config entry called `TestRemote` for the unit tests to use 348 * Create a `backend/remote/remote_test.go` - copy and adjust your example remote 349 * Make sure all tests pass with `go test -v` 350 351 Integration tests 352 353 * Add your backend to `fstest/test_all/config.yaml` 354 * Once you've done that then you can use the integration test framework from the project root: 355 * go install ./... 356 * test_all -backend remote 357 358 Or if you want to run the integration tests manually: 359 360 * Make sure integration tests pass with 361 * `cd fs/operations` 362 * `go test -v -remote TestRemote:` 363 * `cd fs/sync` 364 * `go test -v -remote TestRemote:` 365 * If you are making a bucket based remote, then check with this also 366 * `go test -v -remote TestRemote: -subdir` 367 * And if your remote defines `ListR` this also 368 * `go test -v -remote TestRemote: -fast-list` 369 370 See the [testing](#testing) section for more information on integration tests. 371 372 Add your fs to the docs - you'll need to pick an icon for it from [fontawesome](http://fontawesome.io/icons/). Keep lists of remotes in alphabetical order but with the local file system last. 373 374 * `README.md` - main GitHub page 375 * `docs/content/remote.md` - main docs page (note the backend options are automatically added to this file with `make backenddocs`) 376 * `docs/content/overview.md` - overview docs 377 * `docs/content/docs.md` - list of remotes in config section 378 * `docs/content/about.md` - front page of rclone.org 379 * `docs/layouts/chrome/navbar.html` - add it to the website navigation 380 * `bin/make_manual.py` - add the page to the `docs` constant