github.com/artpar/rclone@v1.67.3/CONTRIBUTING.md (about) 1 # Contributing to rclone 2 3 This is a short guide on how to contribute things to rclone. 4 5 ## Reporting a bug 6 7 If you've just got a question or aren't sure if you've found a bug 8 then please use the [rclone forum](https://forum.rclone.org/) instead 9 of filing an issue. 10 11 When filing an issue, please include the following information if 12 possible as well as a description of the problem. Make sure you test 13 with the [latest beta of rclone](https://beta.rclone.org/): 14 15 - Rclone version (e.g. output from `rclone version`) 16 - Which OS you are using and how many bits (e.g. Windows 10, 64 bit) 17 - The command you were trying to run (e.g. `rclone copy /tmp remote:tmp`) 18 - A log of the command with the `-vv` flag (e.g. output from `rclone -vv copy /tmp remote:tmp`) 19 - if the log contains secrets then edit the file with a text editor first to obscure them 20 21 ## Submitting a new feature or bug fix 22 23 If you find a bug that you'd like to fix, or a new feature that you'd 24 like to implement then please submit a pull request via GitHub. 25 26 If it is a big feature, then [make an issue](https://github.com/artpar/artpar/issues) first so it can be discussed. 27 28 To prepare your pull request first press the fork button on [rclone's GitHub 29 page](https://github.com/artpar/artpar). 30 31 Then [install Git](https://git-scm.com/downloads) and set your public contribution [name](https://docs.github.com/en/github/getting-started-with-github/setting-your-username-in-git) and [email](https://docs.github.com/en/github/setting-up-and-managing-your-github-user-account/setting-your-commit-email-address#setting-your-commit-email-address-in-git). 32 33 Next open your terminal, change directory to your preferred folder and initialise your local rclone project: 34 35 git clone https://github.com/artpar/artpar.git 36 cd rclone 37 git remote rename origin upstream 38 # if you have SSH keys setup in your GitHub account: 39 git remote add origin git@github.com:YOURUSER/rclone.git 40 # otherwise: 41 git remote add origin https://github.com/YOURUSER/rclone.git 42 43 Note that most of the terminal commands in the rest of this guide must be executed from the rclone folder created above. 44 45 Now [install Go](https://golang.org/doc/install) and verify your installation: 46 47 go version 48 49 Great, you can now compile and execute your own version of rclone: 50 51 go build 52 ./rclone version 53 54 (Note that you can also replace `go build` with `make`, which will include a 55 more accurate version number in the executable as well as enable you to specify 56 more build options.) Finally make a branch to add your new feature 57 58 git checkout -b my-new-feature 59 60 And get hacking. 61 62 You may like one of the [popular editors/IDE's for Go](https://github.com/golang/go/wiki/IDEsAndTextEditorPlugins) and a quick view on the rclone [code organisation](#code-organisation). 63 64 When ready - test the affected functionality and run the unit tests for the code you changed 65 66 cd folder/with/changed/files 67 go test -v 68 69 Note that you may need to make a test remote, e.g. `TestSwift` for some 70 of the unit tests. 71 72 This is typically enough if you made a simple bug fix, otherwise please read the rclone [testing](#testing) section too. 73 74 Make sure you 75 76 - Add [unit tests](#testing) for a new feature. 77 - Add [documentation](#writing-documentation) for a new feature. 78 - [Commit your changes](#committing-your-changes) using the [commit message guidelines](#commit-messages). 79 80 When you are done with that push your changes to GitHub: 81 82 git push -u origin my-new-feature 83 84 and open the GitHub website to [create your pull 85 request](https://help.github.com/articles/creating-a-pull-request/). 86 87 Your changes will then get reviewed and you might get asked to fix some stuff. If so, then make the changes in the same branch, commit and push your updates to GitHub. 88 89 You may sometimes be asked to [base your changes on the latest master](#basing-your-changes-on-the-latest-master) or [squash your commits](#squashing-your-commits). 90 91 ## Using Git and GitHub 92 93 ### Committing your changes 94 95 Follow the guideline for [commit messages](#commit-messages) and then: 96 97 git checkout my-new-feature # To switch to your branch 98 git status # To see the new and changed files 99 git add FILENAME # To select FILENAME for the commit 100 git status # To verify the changes to be committed 101 git commit # To do the commit 102 git log # To verify the commit. Use q to quit the log 103 104 You can modify the message or changes in the latest commit using: 105 106 git commit --amend 107 108 If you amend to commits that have been pushed to GitHub, then you will have to [replace your previously pushed commits](#replacing-your-previously-pushed-commits). 109 110 ### Replacing your previously pushed commits 111 112 Note that you are about to rewrite the GitHub history of your branch. It is good practice to involve your collaborators before modifying commits that have been pushed to GitHub. 113 114 Your previously pushed commits are replaced by: 115 116 git push --force origin my-new-feature 117 118 ### Basing your changes on the latest master 119 120 To base your changes on the latest version of the [rclone master](https://github.com/artpar/artpar/tree/master) (upstream): 121 122 git checkout master 123 git fetch upstream 124 git merge --ff-only 125 git push origin --follow-tags # optional update of your fork in GitHub 126 git checkout my-new-feature 127 git rebase master 128 129 If you rebase commits that have been pushed to GitHub, then you will have to [replace your previously pushed commits](#replacing-your-previously-pushed-commits). 130 131 ### Squashing your commits ### 132 133 To combine your commits into one commit: 134 135 git log # To count the commits to squash, e.g. the last 2 136 git reset --soft HEAD~2 # To undo the 2 latest commits 137 git status # To check everything is as expected 138 139 If everything is fine, then make the new combined commit: 140 141 git commit # To commit the undone commits as one 142 143 otherwise, you may roll back using: 144 145 git reflog # To check that HEAD{1} is your previous state 146 git reset --soft 'HEAD@{1}' # To roll back to your previous state 147 148 If you squash commits that have been pushed to GitHub, then you will have to [replace your previously pushed commits](#replacing-your-previously-pushed-commits). 149 150 Tip: You may like to use `git rebase -i master` if you are experienced or have a more complex situation. 151 152 ### GitHub Continuous Integration 153 154 rclone currently uses [GitHub Actions](https://github.com/artpar/artpar/actions) to build and test the project, which should be automatically available for your fork too from the `Actions` tab in your repository. 155 156 ## Testing 157 158 ### Code quality tests 159 160 If you install [golangci-lint](https://github.com/golangci/golangci-lint) then you can run the same tests as get run in the CI which can be very helpful. 161 162 You can run them with `make check` or with `golangci-lint run ./...`. 163 164 Using these tests ensures that the rclone codebase all uses the same coding standards. These tests also check for easy mistakes to make (like forgetting to check an error return). 165 166 ### Quick testing 167 168 rclone's tests are run from the go testing framework, so at the top 169 level you can run this to run all the tests. 170 171 go test -v ./... 172 173 You can also use `make`, if supported by your platform 174 175 make quicktest 176 177 The quicktest is [automatically run by GitHub](#github-continuous-integration) when you push your branch to GitHub. 178 179 ### Backend testing 180 181 rclone contains a mixture of unit tests and integration tests. 182 Because it is difficult (and in some respects pointless) to test cloud 183 storage systems by mocking all their interfaces, rclone unit tests can 184 run against any of the backends. This is done by making specially 185 named remotes in the default config file. 186 187 If you wanted to test changes in the `drive` backend, then you would 188 need to make a remote called `TestDrive`. 189 190 You can then run the unit tests in the drive directory. These tests 191 are skipped if `TestDrive:` isn't defined. 192 193 cd backend/drive 194 go test -v 195 196 You can then run the integration tests which test all of rclone's 197 operations. Normally these get run against the local file system, 198 but they can be run against any of the remotes. 199 200 cd fs/sync 201 go test -v -remote TestDrive: 202 go test -v -remote TestDrive: -fast-list 203 204 cd fs/operations 205 go test -v -remote TestDrive: 206 207 If you want to use the integration test framework to run these tests 208 altogether with an HTML report and test retries then from the 209 project root: 210 211 go install github.com/artpar/artpar/fstest/test_all 212 test_all -backend drive 213 214 ### Full integration testing 215 216 If you want to run all the integration tests against all the remotes, 217 then change into the project root and run 218 219 make check 220 make test 221 222 The commands may require some extra go packages which you can install with 223 224 make build_dep 225 226 The full integration tests are run daily on the integration test server. You can 227 find the results at https://pub.rclone.org/integration-tests/ 228 229 ## Code Organisation 230 231 Rclone code is organised into a small number of top level directories 232 with modules beneath. 233 234 - backend - the rclone backends for interfacing to cloud providers - 235 - all - import this to load all the cloud providers 236 - ...providers 237 - bin - scripts for use while building or maintaining rclone 238 - cmd - the rclone commands 239 - all - import this to load all the commands 240 - ...commands 241 - cmdtest - end-to-end tests of commands, flags, environment variables,... 242 - docs - the documentation and website 243 - content - adjust these docs only - everything else is autogenerated 244 - command - these are auto-generated - edit the corresponding .go file 245 - fs - main rclone definitions - minimal amount of code 246 - accounting - bandwidth limiting and statistics 247 - asyncreader - an io.Reader which reads ahead 248 - config - manage the config file and flags 249 - driveletter - detect if a name is a drive letter 250 - filter - implements include/exclude filtering 251 - fserrors - rclone specific error handling 252 - fshttp - http handling for rclone 253 - fspath - path handling for rclone 254 - hash - defines rclone's hash types and functions 255 - list - list a remote 256 - log - logging facilities 257 - march - iterates directories in lock step 258 - object - in memory Fs objects 259 - operations - primitives for sync, e.g. Copy, Move 260 - sync - sync directories 261 - walk - walk a directory 262 - fstest - provides integration test framework 263 - fstests - integration tests for the backends 264 - mockdir - mocks an fs.Directory 265 - mockobject - mocks an fs.Object 266 - test_all - Runs integration tests for everything 267 - graphics - the images used in the website, etc. 268 - lib - libraries used by the backend 269 - atexit - register functions to run when rclone exits 270 - dircache - directory ID to name caching 271 - oauthutil - helpers for using oauth 272 - pacer - retries with backoff and paces operations 273 - readers - a selection of useful io.Readers 274 - rest - a thin abstraction over net/http for REST 275 - librclone - in memory interface to rclone's API for embedding rclone 276 - vfs - Virtual FileSystem layer for implementing rclone mount and similar 277 278 ## Writing Documentation 279 280 If you are adding a new feature then please update the documentation. 281 282 If you add a new general flag (not for a backend), then document it in 283 `docs/content/docs.md` - the flags there are supposed to be in 284 alphabetical order. 285 286 If you add a new backend option/flag, then it should be documented in 287 the source file in the `Help:` field. 288 289 - Start with the most important information about the option, 290 as a single sentence on a single line. 291 - This text will be used for the command-line flag help. 292 - It will be combined with other information, such as any default value, 293 and the result will look odd if not written as a single sentence. 294 - It should end with a period/full stop character, which will be shown 295 in docs but automatically removed when producing the flag help. 296 - Try to keep it below 80 characters, to reduce text wrapping in the terminal. 297 - More details can be added in a new paragraph, after an empty line (`"\n\n"`). 298 - Like with docs generated from Markdown, a single line break is ignored 299 and two line breaks creates a new paragraph. 300 - This text will be shown to the user in `rclone config` 301 and in the docs (where it will be added by `make backenddocs`, 302 normally run some time before next release). 303 - To create options of enumeration type use the `Examples:` field. 304 - Each example value have their own `Help:` field, but they are treated 305 a bit different than the main option help text. They will be shown 306 as an unordered list, therefore a single line break is enough to 307 create a new list item. Also, for enumeration texts like name of 308 countries, it looks better without an ending period/full stop character. 309 310 The only documentation you need to edit are the `docs/content/*.md` 311 files. The `MANUAL.*`, `rclone.1`, website, etc. are all auto-generated 312 from those during the release process. See the `make doc` and `make 313 website` targets in the Makefile if you are interested in how. You 314 don't need to run these when adding a feature. 315 316 Documentation for rclone sub commands is with their code, e.g. 317 `cmd/ls/ls.go`. Write flag help strings as a single sentence on a single 318 line, without a period/full stop character at the end, as it will be 319 combined unmodified with other information (such as any default value). 320 321 Note that you can use [GitHub's online editor](https://help.github.com/en/github/managing-files-in-a-repository/editing-files-in-another-users-repository) 322 for small changes in the docs which makes it very easy. 323 324 ## Making a release 325 326 There are separate instructions for making a release in the RELEASE.md 327 file. 328 329 ## Commit messages 330 331 Please make the first line of your commit message a summary of the 332 change that a user (not a developer) of rclone would like to read, and 333 prefix it with the directory of the change followed by a colon. The 334 changelog gets made by looking at just these first lines so make it 335 good! 336 337 If you have more to say about the commit, then enter a blank line and 338 carry on the description. Remember to say why the change was needed - 339 the commit itself shows what was changed. 340 341 Writing more is better than less. Comparing the behaviour before the 342 change to that after the change is very useful. Imagine you are 343 writing to yourself in 12 months time when you've forgotten everything 344 about what you just did and you need to get up to speed quickly. 345 346 If the change fixes an issue then write `Fixes #1234` in the commit 347 message. This can be on the subject line if it will fit. If you 348 don't want to close the associated issue just put `#1234` and the 349 change will get linked into the issue. 350 351 Here is an example of a short commit message: 352 353 ``` 354 drive: add team drive support - fixes #885 355 ``` 356 357 And here is an example of a longer one: 358 359 ``` 360 mount: fix hang on errored upload 361 362 In certain circumstances, if an upload failed then the mount could hang 363 indefinitely. This was fixed by closing the read pipe after the Put 364 completed. This will cause the write side to return a pipe closed 365 error fixing the hang. 366 367 Fixes #1498 368 ``` 369 370 ## Adding a dependency 371 372 rclone uses the [go 373 modules](https://tip.golang.org/cmd/go/#hdr-Modules__module_versions__and_more) 374 support in go1.11 and later to manage its dependencies. 375 376 rclone can be built with modules outside of the `GOPATH`. 377 378 To add a dependency `github.com/ncw/new_dependency` see the 379 instructions below. These will fetch the dependency and add it to 380 `go.mod` and `go.sum`. 381 382 go get github.com/ncw/new_dependency 383 384 You can add constraints on that package when doing `go get` (see the 385 go docs linked above), but don't unless you really need to. 386 387 Please check in the changes generated by `go mod` including `go.mod` 388 and `go.sum` in the same commit as your other changes. 389 390 ## Updating a dependency 391 392 If you need to update a dependency then run 393 394 go get golang.org/x/crypto 395 396 Check in a single commit as above. 397 398 ## Updating all the dependencies 399 400 In order to update all the dependencies then run `make update`. This 401 just uses the go modules to update all the modules to their latest 402 stable release. Check in the changes in a single commit as above. 403 404 This should be done early in the release cycle to pick up new versions 405 of packages in time for them to get some testing. 406 407 ## Updating a backend 408 409 If you update a backend then please run the unit tests and the 410 integration tests for that backend. 411 412 Assuming the backend is called `remote`, make create a config entry 413 called `TestRemote` for the tests to use. 414 415 Now `cd remote` and run `go test -v` to run the unit tests. 416 417 Then `cd fs` and run `go test -v -remote TestRemote:` to run the 418 integration tests. 419 420 The next section goes into more detail about the tests. 421 422 ## Writing a new backend 423 424 Choose a name. The docs here will use `remote` as an example. 425 426 Note that in rclone terminology a file system backend is called a 427 remote or an fs. 428 429 ### Research 430 431 - Look at the interfaces defined in `fs/types.go` 432 - Study one or more of the existing remotes 433 434 ### Getting going 435 436 - Create `backend/remote/remote.go` (copy this from a similar remote) 437 - box is a good one to start from if you have a directory-based remote (and shows how to use the directory cache) 438 - b2 is a good one to start from if you have a bucket-based remote 439 - Add your remote to the imports in `backend/all/all.go` 440 - HTTP based remotes are easiest to maintain if they use rclone's [lib/rest](https://pkg.go.dev/github.com/artpar/artpar/lib/rest) module, but if there is a really good Go SDK from the provider then use that instead. 441 - Try to implement as many optional methods as possible as it makes the remote more usable. 442 - Use [lib/encoder](https://pkg.go.dev/github.com/artpar/artpar/lib/encoder) to make sure we can encode any path name and `rclone info` to help determine the encodings needed 443 - `rclone purge -v TestRemote:rclone-info` 444 - `rclone test info --all --remote-encoding None -vv --write-json remote.json TestRemote:rclone-info` 445 - `go run cmd/test/info/internal/build_csv/main.go -o remote.csv remote.json` 446 - open `remote.csv` in a spreadsheet and examine 447 448 ### Guidelines for a speedy merge 449 450 - **Do** use [lib/rest](https://pkg.go.dev/github.com/artpar/artpar/lib/rest) if you are implementing a REST like backend and parsing XML/JSON in the backend. 451 - **Do** use rclone's Client or Transport from [fs/fshttp](https://pkg.go.dev/github.com/artpar/artpar/fs/fshttp) if your backend is HTTP based - this adds features like `--dump bodies`, `--tpslimit`, `--user-agent` without you having to code anything! 452 - **Do** follow your example backend exactly - use the same code order, function names, layout, structure. **Don't** move stuff around and **Don't** delete the comments. 453 - **Do not** split your backend up into `fs.go` and `object.go` (there are a few backends like that - don't follow them!) 454 - **Do** put your API type definitions in a separate file - by preference `api/types.go` 455 - **Remember** we have >50 backends to maintain so keeping them as similar as possible to each other is a high priority! 456 457 ### Unit tests 458 459 - Create a config entry called `TestRemote` for the unit tests to use 460 - Create a `backend/remote/remote_test.go` - copy and adjust your example remote 461 - Make sure all tests pass with `go test -v` 462 463 ### Integration tests 464 465 - Add your backend to `fstest/test_all/config.yaml` 466 - Once you've done that then you can use the integration test framework from the project root: 467 - go install ./... 468 - test_all -backends remote 469 470 Or if you want to run the integration tests manually: 471 472 - Make sure integration tests pass with 473 - `cd fs/operations` 474 - `go test -v -remote TestRemote:` 475 - `cd fs/sync` 476 - `go test -v -remote TestRemote:` 477 - If your remote defines `ListR` check with this also 478 - `go test -v -remote TestRemote: -fast-list` 479 480 See the [testing](#testing) section for more information on integration tests. 481 482 ### Backend documentation 483 484 Add your backend to the docs - you'll need to pick an icon for it from 485 [fontawesome](http://fontawesome.io/icons/). Keep lists of remotes in 486 alphabetical order of full name of remote (e.g. `drive` is ordered as 487 `Google Drive`) but with the local file system last. 488 489 - `README.md` - main GitHub page 490 - `docs/content/remote.md` - main docs page (note the backend options are automatically added to this file with `make backenddocs`) 491 - make sure this has the `autogenerated options` comments in (see your reference backend docs) 492 - update them in your backend with `bin/make_backend_docs.py remote` 493 - `docs/content/overview.md` - overview docs 494 - `docs/content/docs.md` - list of remotes in config section 495 - `docs/content/_index.md` - front page of rclone.org 496 - `docs/layouts/chrome/navbar.html` - add it to the website navigation 497 - `bin/make_manual.py` - add the page to the `docs` constant 498 499 Once you've written the docs, run `make serve` and check they look OK 500 in the web browser and the links (internal and external) all work. 501 502 ## Adding a new s3 provider 503 504 It is quite easy to add a new S3 provider to rclone. 505 506 You'll need to modify the following files 507 508 - `backend/s3/s3.go` 509 - Add the provider to `providerOption` at the top of the file 510 - Add endpoints and other config for your provider gated on the provider in `fs.RegInfo`. 511 - Exclude your provider from genric config questions (eg `region` and `endpoint). 512 - Add the provider to the `setQuirks` function - see the documentation there. 513 - `docs/content/s3.md` 514 - Add the provider at the top of the page. 515 - Add a section about the provider linked from there. 516 - Add a transcript of a trial `rclone config` session 517 - Edit the transcript to remove things which might change in subsequent versions 518 - **Do not** alter or add to the autogenerated parts of `s3.md` 519 - **Do not** run `make backenddocs` or `bin/make_backend_docs.py s3` 520 - `README.md` - this is the home page in github 521 - Add the provider and a link to the section you wrote in `docs/contents/s3.md` 522 - `docs/content/_index.md` - this is the home page of rclone.org 523 - Add the provider and a link to the section you wrote in `docs/contents/s3.md` 524 525 When adding the provider, endpoints, quirks, docs etc keep them in 526 alphabetical order by `Provider` name, but with `AWS` first and 527 `Other` last. 528 529 Once you've written the docs, run `make serve` and check they look OK 530 in the web browser and the links (internal and external) all work. 531 532 Once you've written the code, test `rclone config` works to your 533 satisfaction, and check the integration tests work `go test -v -remote 534 NewS3Provider:`. You may need to adjust the quirks to get them to 535 pass. Some providers just can't pass the tests with control characters 536 in the names so if these fail and the provider doesn't support 537 `urlEncodeListings` in the quirks then ignore them. Note that the 538 `SetTier` test may also fail on non AWS providers. 539 540 For an example of adding an s3 provider see [eb3082a1](https://github.com/artpar/artpar/commit/eb3082a1ebdb76d5625f14cedec3f5154a5e7b10). 541 542 ## Writing a plugin 543 544 New features (backends, commands) can also be added "out-of-tree", through Go plugins. 545 Changes will be kept in a dynamically loaded file instead of being compiled into the main binary. 546 This is useful if you can't merge your changes upstream or don't want to maintain a fork of rclone. 547 548 ### Usage 549 550 - Naming 551 - Plugins names must have the pattern `librcloneplugin_KIND_NAME.so`. 552 - `KIND` should be one of `backend`, `command` or `bundle`. 553 - Example: A plugin with backend support for PiFS would be called 554 `librcloneplugin_backend_pifs.so`. 555 - Loading 556 - Supported on macOS & Linux as of now. ([Go issue for Windows support](https://github.com/golang/go/issues/19282)) 557 - Supported on rclone v1.50 or greater. 558 - All plugins in the folder specified by variable `$RCLONE_PLUGIN_PATH` are loaded. 559 - If this variable doesn't exist, plugin support is disabled. 560 - Plugins must be compiled against the exact version of rclone to work. 561 (The rclone used during building the plugin must be the same as the source of rclone) 562 563 ### Building 564 565 To turn your existing additions into a Go plugin, move them to an external repository 566 and change the top-level package name to `main`. 567 568 Check `rclone --version` and make sure that the plugin's rclone dependency and host Go version match. 569 570 Then, run `go build -buildmode=plugin -o PLUGIN_NAME.so .` to build the plugin. 571 572 [Go reference](https://godoc.org/github.com/artpar/artpar/lib/plugin) 573 574 [Minimal example](https://gist.github.com/terorie/21b517ee347828e899e1913efc1d684f)