github.com/artpar/rclone@v1.67.3/docs/content/netstorage.md (about) 1 --- 2 title: "Akamai Netstorage" 3 description: "Rclone docs for Akamai NetStorage" 4 versionIntroduced: "v1.58" 5 --- 6 7 # {{< icon "fas fa-database" >}} Akamai NetStorage 8 9 Paths are specified as `remote:` 10 You may put subdirectories in too, e.g. `remote:/path/to/dir`. 11 If you have a CP code you can use that as the folder after the domain such as \<domain>\/\<cpcode>\/\<internal directories within cpcode>. 12 13 For example, this is commonly configured with or without a CP code: 14 * **With a CP code**. `[your-domain-prefix]-nsu.akamaihd.net/123456/subdirectory/` 15 * **Without a CP code**. `[your-domain-prefix]-nsu.akamaihd.net` 16 17 18 See all buckets 19 rclone lsd remote: 20 The initial setup for Netstorage involves getting an account and secret. Use `rclone config` to walk you through the setup process. 21 22 ## Configuration 23 24 Here's an example of how to make a remote called `ns1`. 25 26 1. To begin the interactive configuration process, enter this command: 27 28 ``` 29 rclone config 30 ``` 31 32 2. Type `n` to create a new remote. 33 34 ``` 35 n) New remote 36 d) Delete remote 37 q) Quit config 38 e/n/d/q> n 39 ``` 40 41 3. For this example, enter `ns1` when you reach the name> prompt. 42 43 ``` 44 name> ns1 45 ``` 46 47 4. Enter `netstorage` as the type of storage to configure. 48 49 ``` 50 Type of storage to configure. 51 Enter a string value. Press Enter for the default (""). 52 Choose a number from below, or type in your own value 53 XX / NetStorage 54 \ "netstorage" 55 Storage> netstorage 56 ``` 57 58 5. Select between the HTTP or HTTPS protocol. Most users should choose HTTPS, which is the default. HTTP is provided primarily for debugging purposes. 59 60 61 ``` 62 Enter a string value. Press Enter for the default (""). 63 Choose a number from below, or type in your own value 64 1 / HTTP protocol 65 \ "http" 66 2 / HTTPS protocol 67 \ "https" 68 protocol> 1 69 ``` 70 71 6. Specify your NetStorage host, CP code, and any necessary content paths using this format: `<domain>/<cpcode>/<content>/` 72 73 ``` 74 Enter a string value. Press Enter for the default (""). 75 host> baseball-nsu.akamaihd.net/123456/content/ 76 ``` 77 78 7. Set the netstorage account name 79 ``` 80 Enter a string value. Press Enter for the default (""). 81 account> username 82 ``` 83 84 8. Set the Netstorage account secret/G2O key which will be used for authentication purposes. Select the `y` option to set your own password then enter your secret. 85 Note: The secret is stored in the `rclone.conf` file with hex-encoded encryption. 86 87 ``` 88 y) Yes type in my own password 89 g) Generate random password 90 y/g> y 91 Enter the password: 92 password: 93 Confirm the password: 94 password: 95 ``` 96 97 9. View the summary and confirm your remote configuration. 98 99 ``` 100 [ns1] 101 type = netstorage 102 protocol = http 103 host = baseball-nsu.akamaihd.net/123456/content/ 104 account = username 105 secret = *** ENCRYPTED *** 106 -------------------- 107 y) Yes this is OK (default) 108 e) Edit this remote 109 d) Delete this remote 110 y/e/d> y 111 ``` 112 113 This remote is called `ns1` and can now be used. 114 115 ## Example operations 116 117 Get started with rclone and NetStorage with these examples. For additional rclone commands, visit https://rclone.org/commands/. 118 119 ### See contents of a directory in your project 120 121 rclone lsd ns1:/974012/testing/ 122 123 ### Sync the contents local with remote 124 125 rclone sync . ns1:/974012/testing/ 126 127 ### Upload local content to remote 128 rclone copy notes.txt ns1:/974012/testing/ 129 130 ### Delete content on remote 131 rclone delete ns1:/974012/testing/notes.txt 132 133 ### Move or copy content between CP codes. 134 135 Your credentials must have access to two CP codes on the same remote. You can't perform operations between different remotes. 136 137 rclone move ns1:/974012/testing/notes.txt ns1:/974450/testing2/ 138 139 ## Features 140 141 ### Symlink Support 142 143 The Netstorage backend changes the rclone `--links, -l` behavior. When uploading, instead of creating the .rclonelink file, use the "symlink" API in order to create the corresponding symlink on the remote. The .rclonelink file will not be created, the upload will be intercepted and only the symlink file that matches the source file name with no suffix will be created on the remote. 144 145 This will effectively allow commands like copy/copyto, move/moveto and sync to upload from local to remote and download from remote to local directories with symlinks. Due to internal rclone limitations, it is not possible to upload an individual symlink file to any remote backend. You can always use the "backend symlink" command to create a symlink on the NetStorage server, refer to "symlink" section below. 146 147 Individual symlink files on the remote can be used with the commands like "cat" to print the destination name, or "delete" to delete symlink, or copy, copy/to and move/moveto to download from the remote to local. Note: individual symlink files on the remote should be specified including the suffix .rclonelink. 148 149 **Note**: No file with the suffix .rclonelink should ever exist on the server since it is not possible to actually upload/create a file with .rclonelink suffix with rclone, it can only exist if it is manually created through a non-rclone method on the remote. 150 151 ### Implicit vs. Explicit Directories 152 153 With NetStorage, directories can exist in one of two forms: 154 155 1. **Explicit Directory**. This is an actual, physical directory that you have created in a storage group. 156 2. **Implicit Directory**. This refers to a directory within a path that has not been physically created. For example, during upload of a file, nonexistent subdirectories can be specified in the target path. NetStorage creates these as "implicit." While the directories aren't physically created, they exist implicitly and the noted path is connected with the uploaded file. 157 158 Rclone will intercept all file uploads and mkdir commands for the NetStorage remote and will explicitly issue the mkdir command for each directory in the uploading path. This will help with the interoperability with the other Akamai services such as SFTP and the Content Management Shell (CMShell). Rclone will not guarantee correctness of operations with implicit directories which might have been created as a result of using an upload API directly. 159 160 ### `--fast-list` / ListR support 161 162 NetStorage remote supports the ListR feature by using the "list" NetStorage API action to return a lexicographical list of all objects within the specified CP code, recursing into subdirectories as they're encountered. 163 164 * **Rclone will use the ListR method for some commands by default**. Commands such as `lsf -R` will use ListR by default. To disable this, include the `--disable listR` option to use the non-recursive method of listing objects. 165 166 * **Rclone will not use the ListR method for some commands**. Commands such as `sync` don't use ListR by default. To force using the ListR method, include the `--fast-list` option. 167 168 There are pros and cons of using the ListR method, refer to [rclone documentation](https://rclone.org/docs/#fast-list). In general, the sync command over an existing deep tree on the remote will run faster with the "--fast-list" flag but with extra memory usage as a side effect. It might also result in higher CPU utilization but the whole task can be completed faster. 169 170 **Note**: There is a known limitation that "lsf -R" will display number of files in the directory and directory size as -1 when ListR method is used. The workaround is to pass "--disable listR" flag if these numbers are important in the output. 171 172 ### Purge 173 174 NetStorage remote supports the purge feature by using the "quick-delete" NetStorage API action. The quick-delete action is disabled by default for security reasons and can be enabled for the account through the Akamai portal. Rclone will first try to use quick-delete action for the purge command and if this functionality is disabled then will fall back to a standard delete method. 175 176 **Note**: Read the [NetStorage Usage API](https://learn.akamai.com/en-us/webhelp/netstorage/netstorage-http-api-developer-guide/GUID-15836617-9F50-405A-833C-EA2556756A30.html) for considerations when using "quick-delete". In general, using quick-delete method will not delete the tree immediately and objects targeted for quick-delete may still be accessible. 177 178 {{< rem autogenerated options start" - DO NOT EDIT - instead edit fs.RegInfo in backend/netstorage/netstorage.go then run make backenddocs" >}} 179 ### Standard options 180 181 Here are the Standard options specific to netstorage (Akamai NetStorage). 182 183 #### --netstorage-host 184 185 Domain+path of NetStorage host to connect to. 186 187 Format should be `<domain>/<internal folders>` 188 189 Properties: 190 191 - Config: host 192 - Env Var: RCLONE_NETSTORAGE_HOST 193 - Type: string 194 - Required: true 195 196 #### --netstorage-account 197 198 Set the NetStorage account name 199 200 Properties: 201 202 - Config: account 203 - Env Var: RCLONE_NETSTORAGE_ACCOUNT 204 - Type: string 205 - Required: true 206 207 #### --netstorage-secret 208 209 Set the NetStorage account secret/G2O key for authentication. 210 211 Please choose the 'y' option to set your own password then enter your secret. 212 213 **NB** Input to this must be obscured - see [rclone obscure](/commands/rclone_obscure/). 214 215 Properties: 216 217 - Config: secret 218 - Env Var: RCLONE_NETSTORAGE_SECRET 219 - Type: string 220 - Required: true 221 222 ### Advanced options 223 224 Here are the Advanced options specific to netstorage (Akamai NetStorage). 225 226 #### --netstorage-protocol 227 228 Select between HTTP or HTTPS protocol. 229 230 Most users should choose HTTPS, which is the default. 231 HTTP is provided primarily for debugging purposes. 232 233 Properties: 234 235 - Config: protocol 236 - Env Var: RCLONE_NETSTORAGE_PROTOCOL 237 - Type: string 238 - Default: "https" 239 - Examples: 240 - "http" 241 - HTTP protocol 242 - "https" 243 - HTTPS protocol 244 245 #### --netstorage-description 246 247 Description of the remote 248 249 Properties: 250 251 - Config: description 252 - Env Var: RCLONE_NETSTORAGE_DESCRIPTION 253 - Type: string 254 - Required: false 255 256 ## Backend commands 257 258 Here are the commands specific to the netstorage backend. 259 260 Run them with 261 262 rclone backend COMMAND remote: 263 264 The help below will explain what arguments each command takes. 265 266 See the [backend](/commands/rclone_backend/) command for more 267 info on how to pass options and arguments. 268 269 These can be run on a running backend using the rc command 270 [backend/command](/rc/#backend-command). 271 272 ### du 273 274 Return disk usage information for a specified directory 275 276 rclone backend du remote: [options] [<arguments>+] 277 278 The usage information returned, includes the targeted directory as well as all 279 files stored in any sub-directories that may exist. 280 281 ### symlink 282 283 You can create a symbolic link in ObjectStore with the symlink action. 284 285 rclone backend symlink remote: [options] [<arguments>+] 286 287 The desired path location (including applicable sub-directories) ending in 288 the object that will be the target of the symlink (for example, /links/mylink). 289 Include the file extension for the object, if applicable. 290 `rclone backend symlink <src> <path>` 291 292 {{< rem autogenerated options stop >}}