github.com/artpar/rclone@v1.67.3/docs/content/commands/rclone_dedupe.md (about)

     1  ---
     2  title: "rclone dedupe"
     3  description: "Interactively find duplicate filenames and delete/rename them."
     4  slug: rclone_dedupe
     5  url: /commands/rclone_dedupe/
     6  groups: Important
     7  versionIntroduced: v1.27
     8  # autogenerated - DO NOT EDIT, instead edit the source code in cmd/dedupe/ and as part of making a release run "make commanddocs"
     9  ---
    10  # rclone dedupe
    11  
    12  Interactively find duplicate filenames and delete/rename them.
    13  
    14  ## Synopsis
    15  
    16  
    17  
    18  By default `dedupe` interactively finds files with duplicate
    19  names and offers to delete all but one or rename them to be
    20  different. This is known as deduping by name.
    21  
    22  Deduping by name is only useful with a small group of backends (e.g. Google Drive,
    23  Opendrive) that can have duplicate file names. It can be run on wrapping backends
    24  (e.g. crypt) if they wrap a backend which supports duplicate file
    25  names.
    26  
    27  However if `--by-hash` is passed in then dedupe will find files with
    28  duplicate hashes instead which will work on any backend which supports
    29  at least one hash. This can be used to find files with duplicate
    30  content. This is known as deduping by hash.
    31  
    32  If deduping by name, first rclone will merge directories with the same
    33  name.  It will do this iteratively until all the identically named
    34  directories have been merged.
    35  
    36  Next, if deduping by name, for every group of duplicate file names /
    37  hashes, it will delete all but one identical file it finds without
    38  confirmation.  This means that for most duplicated files the `dedupe` command will not be interactive.
    39  
    40  `dedupe` considers files to be identical if they have the
    41  same file path and the same hash. If the backend does not support hashes (e.g. crypt wrapping
    42  Google Drive) then they will never be found to be identical. If you
    43  use the `--size-only` flag then files will be considered
    44  identical if they have the same size (any hash will be ignored). This
    45  can be useful on crypt backends which do not support hashes.
    46  
    47  Next rclone will resolve the remaining duplicates. Exactly which
    48  action is taken depends on the dedupe mode. By default, rclone will
    49  interactively query the user for each one.
    50  
    51  **Important**: Since this can cause data loss, test first with the
    52  `--dry-run` or the `--interactive`/`-i` flag.
    53  
    54  Here is an example run.
    55  
    56  Before - with duplicates
    57  
    58      $ rclone lsl drive:dupes
    59        6048320 2016-03-05 16:23:16.798000000 one.txt
    60        6048320 2016-03-05 16:23:11.775000000 one.txt
    61         564374 2016-03-05 16:23:06.731000000 one.txt
    62        6048320 2016-03-05 16:18:26.092000000 one.txt
    63        6048320 2016-03-05 16:22:46.185000000 two.txt
    64        1744073 2016-03-05 16:22:38.104000000 two.txt
    65         564374 2016-03-05 16:22:52.118000000 two.txt
    66  
    67  Now the `dedupe` session
    68  
    69      $ rclone dedupe drive:dupes
    70      2016/03/05 16:24:37 Google drive root 'dupes': Looking for duplicates using interactive mode.
    71      one.txt: Found 4 files with duplicate names
    72      one.txt: Deleting 2/3 identical duplicates (MD5 "1eedaa9fe86fd4b8632e2ac549403b36")
    73      one.txt: 2 duplicates remain
    74        1:      6048320 bytes, 2016-03-05 16:23:16.798000000, MD5 1eedaa9fe86fd4b8632e2ac549403b36
    75        2:       564374 bytes, 2016-03-05 16:23:06.731000000, MD5 7594e7dc9fc28f727c42ee3e0749de81
    76      s) Skip and do nothing
    77      k) Keep just one (choose which in next step)
    78      r) Rename all to be different (by changing file.jpg to file-1.jpg)
    79      s/k/r> k
    80      Enter the number of the file to keep> 1
    81      one.txt: Deleted 1 extra copies
    82      two.txt: Found 3 files with duplicate names
    83      two.txt: 3 duplicates remain
    84        1:       564374 bytes, 2016-03-05 16:22:52.118000000, MD5 7594e7dc9fc28f727c42ee3e0749de81
    85        2:      6048320 bytes, 2016-03-05 16:22:46.185000000, MD5 1eedaa9fe86fd4b8632e2ac549403b36
    86        3:      1744073 bytes, 2016-03-05 16:22:38.104000000, MD5 851957f7fb6f0bc4ce76be966d336802
    87      s) Skip and do nothing
    88      k) Keep just one (choose which in next step)
    89      r) Rename all to be different (by changing file.jpg to file-1.jpg)
    90      s/k/r> r
    91      two-1.txt: renamed from: two.txt
    92      two-2.txt: renamed from: two.txt
    93      two-3.txt: renamed from: two.txt
    94  
    95  The result being
    96  
    97      $ rclone lsl drive:dupes
    98        6048320 2016-03-05 16:23:16.798000000 one.txt
    99         564374 2016-03-05 16:22:52.118000000 two-1.txt
   100        6048320 2016-03-05 16:22:46.185000000 two-2.txt
   101        1744073 2016-03-05 16:22:38.104000000 two-3.txt
   102  
   103  Dedupe can be run non interactively using the `--dedupe-mode` flag or by using an extra parameter with the same value
   104  
   105    * `--dedupe-mode interactive` - interactive as above.
   106    * `--dedupe-mode skip` - removes identical files then skips anything left.
   107    * `--dedupe-mode first` - removes identical files then keeps the first one.
   108    * `--dedupe-mode newest` - removes identical files then keeps the newest one.
   109    * `--dedupe-mode oldest` - removes identical files then keeps the oldest one.
   110    * `--dedupe-mode largest` - removes identical files then keeps the largest one.
   111    * `--dedupe-mode smallest` - removes identical files then keeps the smallest one.
   112    * `--dedupe-mode rename` - removes identical files then renames the rest to be different.
   113    * `--dedupe-mode list` - lists duplicate dirs and files only and changes nothing.
   114  
   115  For example, to rename all the identically named photos in your Google Photos directory, do
   116  
   117      rclone dedupe --dedupe-mode rename "drive:Google Photos"
   118  
   119  Or
   120  
   121      rclone dedupe rename "drive:Google Photos"
   122  
   123  
   124  ```
   125  rclone dedupe [mode] remote:path [flags]
   126  ```
   127  
   128  ## Options
   129  
   130  ```
   131        --by-hash              Find identical hashes rather than names
   132        --dedupe-mode string   Dedupe mode interactive|skip|first|newest|oldest|largest|smallest|rename (default "interactive")
   133    -h, --help                 help for dedupe
   134  ```
   135  
   136  
   137  ## Important Options
   138  
   139  Important flags useful for most commands.
   140  
   141  ```
   142    -n, --dry-run         Do a trial run with no permanent changes
   143    -i, --interactive     Enable interactive mode
   144    -v, --verbose count   Print lots more stuff (repeat for more)
   145  ```
   146  
   147  See the [global flags page](/flags/) for global options not listed here.
   148  
   149  # SEE ALSO
   150  
   151  * [rclone](/commands/rclone/)	 - Show help for rclone commands, flags and backends.
   152