github.com/josephvusich/fdf@v0.0.0-20230522095411-9326dd32e33f/README.md (about) 1 # File Duplicate Finder (fdf) 2 3  4 5 A cross-platform duplicate file finder supporting deduplication via copy-on-write clones and hard links. Inspired by [Olof Laderkvist's Windows-only fdf utility](http://www.ltr-data.se/opencode.html/). 6 7 ## Installation 8 9 `go install github.com/josephvusich/fdf@latest` 10 11 ### System Requirements 12 13 * Go (with CGO support enabled on non-Windows platforms) 14 * One of the following platforms: 15 16 | Platform | Minimum version | Reason | 17 |---|---|---| 18 | Linux | 2.6.33+ | File-to-file `sendfile()` support | 19 | Mac OS X | Sierra 10.12+ | `clonefile()` and APFS support | 20 | Windows | Windows 10 or Windows Server 2016+ | `FSCTL_DUPLICATE_EXTENTS_TO_FILE` support | 21 22 ## Usage 23 ``` 24 usage: fdf [--clone | --copy | --delete | --link] [-hqrtv] 25 [-m FIELDS] [-z BYTES] [-n LENGTH] 26 [--protect PATTERN] [--unprotect PATTERN] [directory ...] 27 28 -a, --clone (verb) create copy-on-write clones instead of hardlinks (not supported on all filesystems) 29 -c, --copy (verb) split existing hardlinks via copy 30 mutually exclusive with --ignore-hardlinks 31 -d, --delete (verb) delete duplicate files 32 -t, --dry-run don't actually do anything, just show what would be done 33 --exclude GLOB exclude files matching GLOB from scanning 34 --exclude-dir DIR exclude DIR from scanning, throws error if DIR does not exist 35 --help show this help screen and exit 36 --if-kept GLOB only remove files if the 'kept' file matches the provided GLOB 37 --if-kept-dir DIR only remove files if the 'kept' file is a descendant of DIR 38 --ignore-content allow --match without 'content' 39 -h, --ignore-hardlinks ignore existing hardlinks 40 mutually exclusive with --copy 41 --include GLOB include GLOB, opposite of --exclude 42 --include-dir DIR include DIR, throws error if DIR does not exist 43 --json-report FILE on completion, dump JSON match data to FILE 44 -l, --link (verb) hardlink duplicate files 45 -m, --match FIELDS Evaluate FIELDS to determine file equality, where valid fields are: 46 name (case insensitive) 47 range notation supported: name[offset:len,offset:len,...] 48 name[0:-1] whole string 49 name[0:-2] all except last character 50 name[1:2] second and third characters 51 name[-1:1] last character 52 name[-3:3] last 3 characters 53 copyname (case insensitive) 54 'foo.bar' == 'foo (1).bar' == 'Copy of foo.bar', also requires +size or +content 55 namesuffix (case insensitive) 56 one filename must end with the other, e.g.: 'foo-1.bar' and '1.bar' 57 nameprefix (case insensitive) 58 one filename must begin with the other, e.g., 'foo-1.bar' and 'foo.bar' 59 parent (case insensitive name of immediate parent directory) 60 range notation supported: see 'name' for examples 61 path 62 match parent directory path 63 relpath 64 match parent directory path relative to input dir(s) 65 size 66 content (default, also implies size) 67 specify multiple fields using '+', e.g.: name+content 68 -z, --minimum-size BYTES skip files smaller than BYTES, must be greater than the sum of --skip-header and --skip-footer (default 1) 69 --preserve PATTERN (deprecated) alias for --protect PATTERN 70 -p, --protect PATTERN prevent files matching glob PATTERN from being modified or deleted 71 may appear more than once to support multiple patterns 72 rules are applied in the order specified 73 --protect-dir DIR similar to --protect 'DIR/**/*', but throws error if DIR does not exist 74 -q, --quiet don't display current filename during scanning 75 -r, --recursive traverse subdirectories 76 --skip-footer LENGTH skip LENGTH bytes at the end of each file when comparing 77 -n, --skip-header LENGTH skip LENGTH bytes at the beginning of each file when comparing 78 --unprotect value remove files added by --protect 79 may appear more than once 80 rules are applied in the order specified 81 --unprotect-dir DIR similar to --unprotect 'DIR/**/*', but throws error if DIR does not exist 82 -v, --verbose display additional details regarding protected paths 83 ``` 84 85 ## Copy-on-write Cloning 86 87 The `--clone` flag enables copy-on-write clones on compatible filesystems. Common filesystems with support include APFS, ReFS, and Btrfs. See [Comparison of file systems](https://en.wikipedia.org/wiki/Comparison_of_file_systems) on Wikipedia for more. Note that `--copy` may also create clones when using Mac OS X with an APFS filesystem. 88 89 ## License 90 91 Licensed under the [Apache 2.0 license](LICENSE). 92 93 * [clonefile_windows.go](clonefile_windows.go) is adapted from [git-lfs](https://github.com/git-lfs/git-lfs/blob/285eebdddf3a47e83d3cc457397b2bcc798cf935/tools/util_windows.go), licensed under the [MIT license](LICENSE-git-lfs.md). 94 * [path_windows.go](path_windows.go) is adapted from the [Go standard library](https://github.com/golang/go/blob/b86e76681366447798c94abb959bb60875bcc856/src/os/path_windows.go), licensed under a [BSD-style license](LICENSE-golang).