github.com/artpar/rclone@v1.67.3/docs/content/internetarchive.md (about) 1 --- 2 title: "Internet Archive" 3 description: "Rclone docs for Internet Archive" 4 versionIntroduced: "v1.59" 5 --- 6 7 # {{< icon "fa fa-archive" >}} Internet Archive 8 9 The Internet Archive backend utilizes Items on [archive.org](https://archive.org/) 10 11 Refer to [IAS3 API documentation](https://archive.org/services/docs/api/ias3.html) for the API this backend uses. 12 13 Paths are specified as `remote:bucket` (or `remote:` for the `lsd` 14 command.) You may put subdirectories in too, e.g. `remote:item/path/to/dir`. 15 16 Unlike S3, listing up all items uploaded by you isn't supported. 17 18 Once you have made a remote, you can use it like this: 19 20 Make a new item 21 22 rclone mkdir remote:item 23 24 List the contents of a item 25 26 rclone ls remote:item 27 28 Sync `/home/local/directory` to the remote item, deleting any excess 29 files in the item. 30 31 rclone sync --interactive /home/local/directory remote:item 32 33 ## Notes 34 Because of Internet Archive's architecture, it enqueues write operations (and extra post-processings) in a per-item queue. You can check item's queue at https://catalogd.archive.org/history/item-name-here . Because of that, all uploads/deletes will not show up immediately and takes some time to be available. 35 The per-item queue is enqueued to an another queue, Item Deriver Queue. [You can check the status of Item Deriver Queue here.](https://catalogd.archive.org/catalog.php?whereami=1) This queue has a limit, and it may block you from uploading, or even deleting. You should avoid uploading a lot of small files for better behavior. 36 37 You can optionally wait for the server's processing to finish, by setting non-zero value to `wait_archive` key. 38 By making it wait, rclone can do normal file comparison. 39 Make sure to set a large enough value (e.g. `30m0s` for smaller files) as it can take a long time depending on server's queue. 40 41 ## About metadata 42 This backend supports setting, updating and reading metadata of each file. 43 The metadata will appear as file metadata on Internet Archive. 44 However, some fields are reserved by both Internet Archive and rclone. 45 46 The following are reserved by Internet Archive: 47 - `name` 48 - `source` 49 - `size` 50 - `md5` 51 - `crc32` 52 - `sha1` 53 - `format` 54 - `old_version` 55 - `viruscheck` 56 - `summation` 57 58 Trying to set values to these keys is ignored with a warning. 59 Only setting `mtime` is an exception. Doing so make it the identical behavior as setting ModTime. 60 61 rclone reserves all the keys starting with `rclone-`. Setting value for these keys will give you warnings, but values are set according to request. 62 63 If there are multiple values for a key, only the first one is returned. 64 This is a limitation of rclone, that supports one value per one key. 65 It can be triggered when you did a server-side copy. 66 67 Reading metadata will also provide custom (non-standard nor reserved) ones. 68 69 ## Filtering auto generated files 70 71 The Internet Archive automatically creates metadata files after 72 upload. These can cause problems when doing an `rclone sync` as rclone 73 will try, and fail, to delete them. These metadata files are not 74 changeable, as they are created by the Internet Archive automatically. 75 76 These auto-created files can be excluded from the sync using [metadata 77 filtering](/filtering/#metadata). 78 79 rclone sync ... --metadata-exclude "source=metadata" --metadata-exclude "format=Metadata" 80 81 Which excludes from the sync any files which have the 82 `source=metadata` or `format=Metadata` flags which are added to 83 Internet Archive auto-created files. 84 85 ## Configuration 86 87 Here is an example of making an internetarchive configuration. 88 Most applies to the other providers as well, any differences are described [below](#providers). 89 90 First run 91 92 rclone config 93 94 This will guide you through an interactive setup process. 95 96 ``` 97 No remotes found, make a new one? 98 n) New remote 99 s) Set configuration password 100 q) Quit config 101 n/s/q> n 102 name> remote 103 Option Storage. 104 Type of storage to configure. 105 Choose a number from below, or type in your own value. 106 XX / InternetArchive Items 107 \ (internetarchive) 108 Storage> internetarchive 109 Option access_key_id. 110 IAS3 Access Key. 111 Leave blank for anonymous access. 112 You can find one here: https://archive.org/account/s3.php 113 Enter a value. Press Enter to leave empty. 114 access_key_id> XXXX 115 Option secret_access_key. 116 IAS3 Secret Key (password). 117 Leave blank for anonymous access. 118 Enter a value. Press Enter to leave empty. 119 secret_access_key> XXXX 120 Edit advanced config? 121 y) Yes 122 n) No (default) 123 y/n> y 124 Option endpoint. 125 IAS3 Endpoint. 126 Leave blank for default value. 127 Enter a string value. Press Enter for the default (https://s3.us.archive.org). 128 endpoint> 129 Option front_endpoint. 130 Host of InternetArchive Frontend. 131 Leave blank for default value. 132 Enter a string value. Press Enter for the default (https://archive.org). 133 front_endpoint> 134 Option disable_checksum. 135 Don't store MD5 checksum with object metadata. 136 Normally rclone will calculate the MD5 checksum of the input before 137 uploading it so it can ask the server to check the object against checksum. 138 This is great for data integrity checking but can cause long delays for 139 large files to start uploading. 140 Enter a boolean value (true or false). Press Enter for the default (true). 141 disable_checksum> true 142 Option encoding. 143 The encoding for the backend. 144 See the [encoding section in the overview](/overview/#encoding) for more info. 145 Enter a encoder.MultiEncoder value. Press Enter for the default (Slash,Question,Hash,Percent,Del,Ctl,InvalidUtf8,Dot). 146 encoding> 147 Edit advanced config? 148 y) Yes 149 n) No (default) 150 y/n> n 151 -------------------- 152 [remote] 153 type = internetarchive 154 access_key_id = XXXX 155 secret_access_key = XXXX 156 -------------------- 157 y) Yes this is OK (default) 158 e) Edit this remote 159 d) Delete this remote 160 y/e/d> y 161 ``` 162 163 {{< rem autogenerated options start" - DO NOT EDIT - instead edit fs.RegInfo in backend/internetarchive/internetarchive.go then run make backenddocs" >}} 164 ### Standard options 165 166 Here are the Standard options specific to internetarchive (Internet Archive). 167 168 #### --internetarchive-access-key-id 169 170 IAS3 Access Key. 171 172 Leave blank for anonymous access. 173 You can find one here: https://archive.org/account/s3.php 174 175 Properties: 176 177 - Config: access_key_id 178 - Env Var: RCLONE_INTERNETARCHIVE_ACCESS_KEY_ID 179 - Type: string 180 - Required: false 181 182 #### --internetarchive-secret-access-key 183 184 IAS3 Secret Key (password). 185 186 Leave blank for anonymous access. 187 188 Properties: 189 190 - Config: secret_access_key 191 - Env Var: RCLONE_INTERNETARCHIVE_SECRET_ACCESS_KEY 192 - Type: string 193 - Required: false 194 195 ### Advanced options 196 197 Here are the Advanced options specific to internetarchive (Internet Archive). 198 199 #### --internetarchive-endpoint 200 201 IAS3 Endpoint. 202 203 Leave blank for default value. 204 205 Properties: 206 207 - Config: endpoint 208 - Env Var: RCLONE_INTERNETARCHIVE_ENDPOINT 209 - Type: string 210 - Default: "https://s3.us.archive.org" 211 212 #### --internetarchive-front-endpoint 213 214 Host of InternetArchive Frontend. 215 216 Leave blank for default value. 217 218 Properties: 219 220 - Config: front_endpoint 221 - Env Var: RCLONE_INTERNETARCHIVE_FRONT_ENDPOINT 222 - Type: string 223 - Default: "https://archive.org" 224 225 #### --internetarchive-disable-checksum 226 227 Don't ask the server to test against MD5 checksum calculated by rclone. 228 Normally rclone will calculate the MD5 checksum of the input before 229 uploading it so it can ask the server to check the object against checksum. 230 This is great for data integrity checking but can cause long delays for 231 large files to start uploading. 232 233 Properties: 234 235 - Config: disable_checksum 236 - Env Var: RCLONE_INTERNETARCHIVE_DISABLE_CHECKSUM 237 - Type: bool 238 - Default: true 239 240 #### --internetarchive-wait-archive 241 242 Timeout for waiting the server's processing tasks (specifically archive and book_op) to finish. 243 Only enable if you need to be guaranteed to be reflected after write operations. 244 0 to disable waiting. No errors to be thrown in case of timeout. 245 246 Properties: 247 248 - Config: wait_archive 249 - Env Var: RCLONE_INTERNETARCHIVE_WAIT_ARCHIVE 250 - Type: Duration 251 - Default: 0s 252 253 #### --internetarchive-encoding 254 255 The encoding for the backend. 256 257 See the [encoding section in the overview](/overview/#encoding) for more info. 258 259 Properties: 260 261 - Config: encoding 262 - Env Var: RCLONE_INTERNETARCHIVE_ENCODING 263 - Type: Encoding 264 - Default: Slash,LtGt,CrLf,Del,Ctl,InvalidUtf8,Dot 265 266 #### --internetarchive-description 267 268 Description of the remote 269 270 Properties: 271 272 - Config: description 273 - Env Var: RCLONE_INTERNETARCHIVE_DESCRIPTION 274 - Type: string 275 - Required: false 276 277 ### Metadata 278 279 Metadata fields provided by Internet Archive. 280 If there are multiple values for a key, only the first one is returned. 281 This is a limitation of Rclone, that supports one value per one key. 282 283 Owner is able to add custom keys. Metadata feature grabs all the keys including them. 284 285 Here are the possible system metadata items for the internetarchive backend. 286 287 | Name | Help | Type | Example | Read Only | 288 |------|------|------|---------|-----------| 289 | crc32 | CRC32 calculated by Internet Archive | string | 01234567 | **Y** | 290 | format | Name of format identified by Internet Archive | string | Comma-Separated Values | **Y** | 291 | md5 | MD5 hash calculated by Internet Archive | string | 01234567012345670123456701234567 | **Y** | 292 | mtime | Time of last modification, managed by Rclone | RFC 3339 | 2006-01-02T15:04:05.999999999Z | **Y** | 293 | name | Full file path, without the bucket part | filename | backend/internetarchive/internetarchive.go | **Y** | 294 | old_version | Whether the file was replaced and moved by keep-old-version flag | boolean | true | **Y** | 295 | rclone-ia-mtime | Time of last modification, managed by Internet Archive | RFC 3339 | 2006-01-02T15:04:05.999999999Z | N | 296 | rclone-mtime | Time of last modification, managed by Rclone | RFC 3339 | 2006-01-02T15:04:05.999999999Z | N | 297 | rclone-update-track | Random value used by Rclone for tracking changes inside Internet Archive | string | aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa | N | 298 | sha1 | SHA1 hash calculated by Internet Archive | string | 0123456701234567012345670123456701234567 | **Y** | 299 | size | File size in bytes | decimal number | 123456 | **Y** | 300 | source | The source of the file | string | original | **Y** | 301 | summation | Check https://forum.rclone.org/t/31922 for how it is used | string | md5 | **Y** | 302 | viruscheck | The last time viruscheck process was run for the file (?) | unixtime | 1654191352 | **Y** | 303 304 See the [metadata](/docs/#metadata) docs for more info. 305 306 {{< rem autogenerated options stop >}}