Sep 11, 2023
+Nov 26, 2023
NOTE: This version of rclone will not support mount any more (see #5373). If mounting is wanted on macOS, either install a precompiled binary or enable the relevant option when installing from source.
Note that this is a third party installer not controlled by the rclone developers so it may be out of date. Its current version is as below.
On macOS, rclone can also be installed via MacPorts:
+sudo port install rcloneNote that this is a third party installer not controlled by the rclone developers so it may be out of date. Its current version is as below.
+
More information here.
To avoid problems with macOS gatekeeper enforcing the binary to be signed and notarized it is enough to download with curl.
Download the latest version of rclone.
@@ -392,8 +401,8 @@
Make sure you have Snapd installed
-$ sudo snap install rcloneDue to the strict confinement of Snap, rclone snap cannot acess real /home/$USER/.config/rclone directory, default config path is as below.
+$ sudo snap install rcloneDue to the strict confinement of Snap, rclone snap cannot access real /home/$USER/.config/rclone directory, default config path is as below.
Note that this is controlled by community maintainer not the rclone developers so it may be out of date. Its current version is as below.
Make sure you have git and Go installed. Go version 1.17 or newer is required, latest release is recommended. You can get it from your package manager, or download it from golang.org/dl. Then you can run the following:
+Make sure you have git and Go installed. Go version 1.18 or newer is required, the latest release is recommended. You can get it from your package manager, or download it from golang.org/dl. Then you can run the following:
git clone https://github.com/rclone/rclone.git
cd rclone
go buildNote that on macOS and Windows the mount command will not be available unless you specify an additional build tag cmount.
go build -tags cmountThis assumes you have a GCC compatible C compiler (GCC or Clang) in your PATH, as it uses cgo. But on Windows, the cgofuse library that the cmount implementation is based on, also supports building without cgo, i.e. by setting environment variable CGO_ENABLED to value 0 (static linking). This is how the official Windows release of rclone is being built, starting with version 1.59. It is still possible to build with cgo on Windows as well, by using the MinGW port of GCC, e.g. by installing it in a MSYS2 distribution (make sure you install it in the classic mingw64 subsystem, the ucrt64 version is not compatible).
-Additionally, on Windows, you must install the third party utility WinFsp, with the "Developer" feature selected. If building with cgo, you must also set environment variable CPATH pointing to the fuse include directory within the WinFsp installation (normally C:\Program Files (x86)\WinFsp\inc\fuse).
You may also add arguments -ldflags -s (with or without -tags cmount), to omit symbol table and debug information, making the executable file smaller, and -trimpath to remove references to local file system paths. This is how the official rclone releases are built.
Additionally, to build with mount on Windows, you must install the third party utility WinFsp, with the "Developer" feature selected. If building with cgo, you must also set environment variable CPATH pointing to the fuse include directory within the WinFsp installation (normally C:\Program Files (x86)\WinFsp\inc\fuse).
You may add arguments -ldflags -s to omit symbol table and debug information, making the executable file smaller, and -trimpath to remove references to local file system paths. The official rclone releases are built with both of these.
go build -trimpath -ldflags -s -tags cmountInstead of executing the go build command directly, you can run it via the Makefile. It changes the version number suffix from "-DEV" to "-beta" and appends commit details. It also copies the resulting rclone executable into your GOPATH bin folder ($(go env GOPATH)/bin, which corresponds to ~/go/bin/rclone by default).
If you want to customize the version string, as reported by the rclone version command, you can set one of the variables fs.Version, fs.VersionTag (to keep default suffix but customize the number), or fs.VersionSuffix (to keep default number but customize the suffix). This can be done from the build command, by adding to the -ldflags argument value as shown below.
go build -trimpath -ldflags "-s -X github.com/rclone/rclone/fs.Version=v9.9.9-test" -tags cmountOn Windows, the official executables also have the version information, as well as a file icon, embedded as binary resources. To get that with your own build you need to run the following command before the build command. It generates a Windows resource system object file, with extension .syso, e.g. resource_windows_amd64.syso, that will be automatically picked up by future build commands.
go run bin/resource_windows.goThe above command will generate a resource file containing version information based on the fs.Version variable in source at the time you run the command, which means if the value of this variable changes you need to re-run the command for it to be reflected in the version information. Also, if you override this version variable in the build command as described above, you need to do that also when generating the resource file, or else it will still use the value from the source.
+go run bin/resource_windows.go -version v9.9.9-testInstead of executing the go build command directly, you can run it via the Makefile. The default target changes the version suffix from "-DEV" to "-beta" followed by additional commit details, embeds version information binary resources on Windows, and copies the resulting rclone executable into your GOPATH bin folder ($(go env GOPATH)/bin, which corresponds to ~/go/bin/rclone by default).
makeTo include mount command on macOS and Windows with Makefile build:
make GOTAGS=cmountThere are other make targets that can be used for more advanced builds, such as cross-compiling for all supported os/architectures, embedding icon and version info resources into windows executable, and packaging results into release artifacts. See Makefile and cross-compile.go for details.
-Another alternative is to download the source, build and install rclone in one operation, as a regular Go package. The source will be stored it in the Go module cache, and the resulting executable will be in your GOPATH bin folder ($(go env GOPATH)/bin, which corresponds to ~/go/bin/rclone by default).
With Go version 1.17 or newer:
+There are other make targets that can be used for more advanced builds, such as cross-compiling for all supported os/architectures, and packaging results into release artifacts. See Makefile and cross-compile.go for details.
+Another alternative method for source installation is to download the source, build and install rclone - all in one operation, as a regular Go package. The source will be stored it in the Go module cache, and the resulting executable will be in your GOPATH bin folder ($(go env GOPATH)/bin, which corresponds to ~/go/bin/rclone by default).
go install github.com/rclone/rclone@latestWith Go versions older than 1.17 (do not use the -u flag, it causes Go to try to update the dependencies that rclone uses and sometimes these don't work with the current version):
go get github.com/rclone/rcloneThis can be done with Stefan Weichinger's ansible role.
Instructions
@@ -466,7 +478,7 @@The WinFsp service infrastructure supports incorporating services for file system implementations, such as rclone, into its own launcher service, as kind of "child services". This has the additional advantage that it also implements a network provider that integrates into Windows standard methods for managing network drives. This is currently not officially supported by Rclone, but with WinFsp version 2019.3 B2 / v1.5B2 or later it should be possible through path rewriting as described here.
To Windows service running any rclone command, the excellent third-party utility NSSM, the "Non-Sucking Service Manager", can be used. It includes some advanced features such as adjusting process priority, defining process environment variables, redirect to file anything written to stdout, and customized response to different exit codes, with a GUI to configure everything from (although it can also be used from command line ).
-There are also several other alternatives. To mention one more, WinSW, "Windows Service Wrapper", is worth checking out. It requires .NET Framework, but it is preinstalled on newer versions of Windows, and it also provides alternative standalone distributions which includes necessary runtime (.NET 5). WinSW is a command-line only utility, where you have to manually create an XML file with service configuration. This may be a drawback for some, but it can also be an advantage as it is easy to back up and re-use the configuration settings, without having go through manual steps in a GUI. One thing to note is that by default it does not restart the service on error, one have to explicit enable this in the configuration file (via the "onfailure" parameter).
+There are also several other alternatives. To mention one more, WinSW, "Windows Service Wrapper", is worth checking out. It requires .NET Framework, but it is preinstalled on newer versions of Windows, and it also provides alternative standalone distributions which includes necessary runtime (.NET 5). WinSW is a command-line only utility, where you have to manually create an XML file with service configuration. This may be a drawback for some, but it can also be an advantage as it is easy to back up and reuse the configuration settings, without having go through manual steps in a GUI. One thing to note is that by default it does not restart the service on error, one have to explicit enable this in the configuration file (via the "onfailure" parameter).
To always run rclone in background, relevant for mount commands etc, you can use systemd to set up rclone as a system or user service. Running as a system service ensures that it is run at startup even if the user it is running as has no active session. Running rclone as a user service ensures that it only starts after the configured user has logged into the system.
@@ -507,10 +519,12 @@Checks the files in the source against a SUM file.
+Checks the files in the destination against a SUM file.
Checks that hashsums of source files match the SUM file. It compares hashes (MD5, SHA1, etc) and logs a report of files which don't match. It doesn't alter the file system.
-If you supply the --download flag, it will download the data from remote and calculate the contents hash on the fly. This can be useful for remotes that don't support hashes or if you really want to check all the data.
Checks that hashsums of destination files match the SUM file. It compares hashes (MD5, SHA1, etc) and logs a report of files which don't match. It doesn't alter the file system.
+The sumfile is treated as the source and the dst:path is treated as the destination for the purposes of the output.
+If you supply the --download flag, it will download the data from the remote and calculate the content hash on the fly. This can be useful for remotes that don't support hashes or if you really want to check all the data.
Note that hash values in the SUM file are treated as case insensitive.
If you supply the --one-way flag, it will only check that files in the source match the files in the destination, not the other way around. This means that extra files in the destination that are not in the source will not be detected.
The --differ, --missing-on-dst, --missing-on-src, --match and --error flags write paths, one per line, to the file name (or stdout if it is -) supplied. What they write is described in the help below. For example --differ will write all paths which are present on both the source and destination but different.
! path means there was an error reading or hashing the source or dest.The default number of parallel checks is 8. See the --checkers=N option for more information.
-rclone checksum <hash> sumfile src:path [flags]rclone checksum <hash> sumfile dst:path [flags] --combined string Make a combined report of changes to this file
--differ string Report all non-matching files to this file
@@ -2194,11 +2213,11 @@ Copy Options
-c, --checksum Check for changes with size & checksum (if available, or fallback to size only).
--compare-dest stringArray Include additional comma separated server-side paths during comparison
--copy-dest stringArray Implies --compare-dest but also copies files from paths into destination
- --cutoff-mode string Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default "HARD")
+ --cutoff-mode HARD|SOFT|CAUTIOUS Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default HARD)
--ignore-case-sync Ignore case when synchronizing
--ignore-checksum Skip post copy check of checksums
--ignore-existing Skip all files that exist on destination
- --ignore-size Ignore size when skipping use mod-time or checksum
+ --ignore-size Ignore size when skipping use modtime or checksum
-I, --ignore-times Don't skip files that match size and time - transfer all files
--immutable Do not modify files, fail if existing files have been modified
--inplace Download directly to destination file instead of atomic download to temp/rename
@@ -2213,11 +2232,12 @@ Copy Options
--multi-thread-write-buffer-size SizeSuffix In memory buffer size for writing when in multi-thread mode (default 128Ki)
--no-check-dest Don't check the destination, copy regardless
--no-traverse Don't traverse destination file system on copy
- --no-update-modtime Don't update destination mod-time if files identical
+ --no-update-modtime Don't update destination modtime if files identical
--order-by string Instructions on how to order the transfers, e.g. 'size,descending'
+ --partial-suffix string Add partial-suffix to temporary file name when --inplace is not used (default ".partial")
--refresh-times Refresh the modtime of remote files
--server-side-across-configs Allow server-side operations (e.g. copy) to work across different configs
- --size-only Skip based on size only, not mod-time or checksum
+ --size-only Skip based on size only, not modtime or checksum
--streaming-upload-cutoff SizeSuffix Cutoff for switching to chunked upload if file size is unknown, upload starts after reaching cutoff or when file ends (default 100Ki)
-u, --update Skip files that are newer on the destinationThen
$ rclone hashsum MD5 remote:pathNote that hash names are case insensitive and values are output in lower case.
-rclone hashsum <hash> remote:path [flags]rclone hashsum [<hash> remote:path] [flags] --base64 Output base64 encoded hashsum
-C, --checkfile string Validate hashes against a given SUM file instead of printing them
@@ -2840,7 +2856,9 @@ Windows caveats
It is also possible to make a drive mount available to everyone on the system, by running the process creating it as the built-in SYSTEM account. There are several ways to do this: One is to use the command-line utility PsExec, from Microsoft's Sysinternals suite, which has option -s to start processes as the SYSTEM account. Another alternative is to run the mount command from a Windows Scheduled Task, or a Windows Service, configured to run as the SYSTEM account. A third alternative is to use the WinFsp.Launcher infrastructure). Read more in the install documentation. Note that when running rclone as another user, it will not use the configuration file from your profile unless you tell it to with the --config option. Note also that it is now the SYSTEM account that will have the owner permissions, and other accounts will have permissions according to the group or others scopes. As mentioned above, these will then not get the "write extended attributes" permission, and this may prevent writing to files. You can work around this with the FileSecurity option, see example above.
Note that mapping to a directory path, instead of a drive letter, does not suffer from the same limitations.
Mounting on macOS
-Mounting on macOS can be done either via macFUSE (also known as osxfuse) or FUSE-T. macFUSE is a traditional FUSE driver utilizing a macOS kernel extension (kext). FUSE-T is an alternative FUSE system which "mounts" via an NFSv4 local server.
+Mounting on macOS can be done either via built-in NFS server, macFUSE (also known as osxfuse) or FUSE-T. macFUSE is a traditional FUSE driver utilizing a macOS kernel extension (kext). FUSE-T is an alternative FUSE system which "mounts" via an NFSv4 local server.
+NFS mount
+This method spins up an NFS server using serve nfs command and mounts it to the specified mountpoint. If you run this in background mode using |--daemon|, you will need to send SIGTERM signal to the rclone process using |kill| command to stop the mount.
macFUSE Notes
If installing macFUSE using dmg packages from the website, rclone will locate the macFUSE libraries without any further intervention. If however, macFUSE is installed using the macports package manager, the following addition steps are required.
sudo mkdir /usr/local/lib
@@ -2860,7 +2878,7 @@ Unicode Normalization
Read Only mounts
When mounting with --read-only, attempts to write to files will fail silently as opposed to with a clear warning as in macFUSE.
Limitations
-Without the use of --vfs-cache-mode this can only write files sequentially, it can only seek when reading. This means that many applications won't work with their files on an rclone mount without --vfs-cache-mode writes or --vfs-cache-mode full. See the VFS File Caching section for more info.
+Without the use of --vfs-cache-mode this can only write files sequentially, it can only seek when reading. This means that many applications won't work with their files on an rclone mount without --vfs-cache-mode writes or --vfs-cache-mode full. See the VFS File Caching section for more info. When using NFS mount on macOS, if you don't specify |--vfs-cache-mode| the mount point will be read-only.
The bucket-based remotes (e.g. Swift, S3, Google Compute Storage, B2) do not support the concept of empty directories, so empty directories will have a tendency to disappear once they fall out of the directory cache.
When rclone mount is invoked on Unix with --daemon flag, the main rclone program will wait for the background mount to become ready or until the timeout specified by the --daemon-wait flag. On Linux it can check mount status using ProcFS so the flag in fact sets maximum time to wait, while the real wait can be less. On macOS / BSD the time to wait is constant and the check is performed only at the end. We advise you to set wait time on macOS reasonably.
Only supported on Linux, FreeBSD, OS X and Windows at the moment.
@@ -2912,9 +2930,8 @@ Rclone as Unix mount helper
command=cmount can be used to run cmount or any other rclone command rather than the default mount.args2env will pass mount options to the mount helper running in background via environment variables instead of command line arguments. This allows to hide secrets from such commands as ps or pgrep.vv... will be transformed into appropriate --verbose=Nstandard mount options like x-systemd.automount, _netdev, nosuid and alike are intended only for Automountd and ignored by rclone.
+standard mount options like x-systemd.automount, _netdev, nosuid and alike are intended only for Automountd and ignored by rclone. ## VFS - Virtual File System
-VFS - Virtual File System
This command uses the VFS layer. This adapts the cloud storage objects that rclone uses into something which looks much more like a disk filing system.
Cloud storage objects have lots of properties which aren't like disk files - you can't extend them or write to the middle of them, so the VFS layer has to deal with that. Because there is no one right way of doing this there are various options explained below.
The VFS layer also implements a directory cache - this caches info about files and directories (but not the data) in memory.
@@ -3075,6 +3092,7 @@ Options
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi)
--vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off)
--vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms)
+ --vfs-refresh Refreshes the directory cache recursively on start
--vfs-used-is-size rclone size Use the rclone size algorithm for Used size
--vfs-write-back Duration Time to writeback files after last use when using cache (default 5s)
--vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s)
@@ -3135,11 +3153,11 @@ Copy Options
-c, --checksum Check for changes with size & checksum (if available, or fallback to size only).
--compare-dest stringArray Include additional comma separated server-side paths during comparison
--copy-dest stringArray Implies --compare-dest but also copies files from paths into destination
- --cutoff-mode string Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default "HARD")
+ --cutoff-mode HARD|SOFT|CAUTIOUS Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default HARD)
--ignore-case-sync Ignore case when synchronizing
--ignore-checksum Skip post copy check of checksums
--ignore-existing Skip all files that exist on destination
- --ignore-size Ignore size when skipping use mod-time or checksum
+ --ignore-size Ignore size when skipping use modtime or checksum
-I, --ignore-times Don't skip files that match size and time - transfer all files
--immutable Do not modify files, fail if existing files have been modified
--inplace Download directly to destination file instead of atomic download to temp/rename
@@ -3154,11 +3172,12 @@ Copy Options
--multi-thread-write-buffer-size SizeSuffix In memory buffer size for writing when in multi-thread mode (default 128Ki)
--no-check-dest Don't check the destination, copy regardless
--no-traverse Don't traverse destination file system on copy
- --no-update-modtime Don't update destination mod-time if files identical
+ --no-update-modtime Don't update destination modtime if files identical
--order-by string Instructions on how to order the transfers, e.g. 'size,descending'
+ --partial-suffix string Add partial-suffix to temporary file name when --inplace is not used (default ".partial")
--refresh-times Refresh the modtime of remote files
--server-side-across-configs Allow server-side operations (e.g. copy) to work across different configs
- --size-only Skip based on size only, not mod-time or checksum
+ --size-only Skip based on size only, not modtime or checksum
--streaming-upload-cutoff SizeSuffix Cutoff for switching to chunked upload if file size is unknown, upload starts after reaching cutoff or when file ends (default 100Ki)
-u, --update Skip files that are newer on the destination
Important Options
@@ -3451,6 +3470,37 @@ Template
+The server also makes the following functions available so that they can be used within the template. These functions help extend the options for dynamic rendering of HTML. They can be used to render HTML based on specific conditions.
+
+
+
+
+
+Function
+Description
+
+
+
+
+afterEpoch
+Returns the time since the epoch for the given time.
+
+
+contains
+Checks whether a given substring is present or not in a given string.
+
+
+hasPrefix
+Checks whether the given string begins with the specified prefix.
+
+
+hasSuffix
+Checks whether the given string end with the specified suffix.
+
+
+
Authentication
By default this will serve files without needing a login.
You can either use an htpasswd file which can take lots of users, or set a single username and password with the --rc-user and --rc-pass flags.
@@ -3565,7 +3615,9 @@ SEE ALSO
rclone serve docker - Serve any remote on docker's volume plugin API.
rclone serve ftp - Serve remote:path over FTP.
rclone serve http - Serve the remote over HTTP.
+rclone serve nfs - Serve the remote as an NFS mount
rclone serve restic - Serve the remote for restic's REST API.
+rclone serve s3 - Serve remote:path over s3.
rclone serve sftp - Serve the remote over SFTP.
rclone serve webdav - Serve remote:path over WebDAV.
@@ -3577,8 +3629,7 @@ Synopsis
Server options
Use --addr to specify which IP address and port the server should listen on, e.g. --addr 1.2.3.4:8000 or --addr :8080 to listen to all IPs.
Use --name to choose the friendly server name, which is by default "rclone (hostname)".
-Use --log-trace in conjunction with -vv to enable additional debug logging of all UPNP traffic.
-VFS - Virtual File System
+Use --log-trace in conjunction with -vv to enable additional debug logging of all UPNP traffic. ## VFS - Virtual File System
This command uses the VFS layer. This adapts the cloud storage objects that rclone uses into something which looks much more like a disk filing system.
Cloud storage objects have lots of properties which aren't like disk files - you can't extend them or write to the middle of them, so the VFS layer has to deal with that. Because there is no one right way of doing this there are various options explained below.
The VFS layer also implements a directory cache - this caches info about files and directories (but not the data) in memory.
@@ -3726,6 +3777,7 @@ Options
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi)
--vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off)
--vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms)
+ --vfs-refresh Refreshes the directory cache recursively on start
--vfs-used-is-size rclone size Use the rclone size algorithm for Used size
--vfs-write-back Duration Time to writeback files after last use when using cache (default 5s)
--vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s)Running rclone serve docker will create the said socket, listening for commands from Docker to create the necessary Volumes. Normally you need not give the --socket-addr flag. The API will listen on the unix domain socket at /run/docker/plugins/rclone.sock. In the example above rclone will create a TCP socket and a small file /etc/docker/plugins/rclone.spec containing the socket address. We use sudo because both paths are writeable only by the root user.
If you later decide to change listening socket, the docker daemon must be restarted to reconnect to /run/docker/plugins/rclone.sock or parse new /etc/docker/plugins/rclone.spec. Until you restart, any volume related docker commands will timeout trying to access the old socket. Running directly is supported on Linux only, not on Windows or MacOS. This is not a problem with managed plugin mode described in details in the full documentation.
The command will create volume mounts under the path given by --base-dir (by default /var/lib/docker-volumes/rclone available only to root) and maintain the JSON formatted file docker-plugin.state in the rclone cache directory with book-keeping records of created and mounted volumes.
All mount and VFS options are submitted by the docker daemon via API, but you can also provide defaults on the command line as well as set path to the config file and cache directory or adjust logging verbosity.
-All mount and VFS options are submitted by the docker daemon via API, but you can also provide defaults on the command line as well as set path to the config file and cache directory or adjust logging verbosity. ## VFS - Virtual File System
This command uses the VFS layer. This adapts the cloud storage objects that rclone uses into something which looks much more like a disk filing system.
Cloud storage objects have lots of properties which aren't like disk files - you can't extend them or write to the middle of them, so the VFS layer has to deal with that. Because there is no one right way of doing this there are various options explained below.
The VFS layer also implements a directory cache - this caches info about files and directories (but not the data) in memory.
@@ -3934,6 +3985,7 @@If you set --addr to listen on a public or LAN accessible IP address then using Authentication is advised - see the next section for info.
By default this will serve files without needing a login.
-You can set a single username and password with the --user and --pass flags.
-You can set a single username and password with the --user and --pass flags. ## VFS - Virtual File System
This command uses the VFS layer. This adapts the cloud storage objects that rclone uses into something which looks much more like a disk filing system.
Cloud storage objects have lots of properties which aren't like disk files - you can't extend them or write to the middle of them, so the VFS layer has to deal with that. Because there is no one right way of doing this there are various options explained below.
The VFS layer also implements a directory cache - this caches info about files and directories (but not the data) in memory.
@@ -4159,6 +4210,7 @@The server also makes the following functions available so that they can be used within the template. These functions help extend the options for dynamic rendering of HTML. They can be used to render HTML based on specific conditions.
+| Function | +Description | +
|---|---|
| afterEpoch | +Returns the time since the epoch for the given time. | +
| contains | +Checks whether a given substring is present or not in a given string. | +
| hasPrefix | +Checks whether the given string begins with the specified prefix. | +
| hasSuffix | +Checks whether the given string end with the specified suffix. | +
By default this will serve files without needing a login.
You can either use an htpasswd file which can take lots of users, or set a single username and password with the --user and --pass flags.
The password file can be updated while rclone is running.
Use --realm to set the authentication realm.
Use --salt to change the password hashing salt from the default.
Use --salt to change the password hashing salt from the default. ## VFS - Virtual File System
This command uses the VFS layer. This adapts the cloud storage objects that rclone uses into something which looks much more like a disk filing system.
Cloud storage objects have lots of properties which aren't like disk files - you can't extend them or write to the middle of them, so the VFS layer has to deal with that. Because there is no one right way of doing this there are various options explained below.
The VFS layer also implements a directory cache - this caches info about files and directories (but not the data) in memory.
@@ -4492,6 +4574,7 @@Serve the remote as an NFS mount
+Create an NFS server that serves the given remote over the network.
+The primary purpose for this command is to enable mount command on recent macOS versions where installing FUSE is very cumbersome.
+Since this is running on NFSv3, no authentication method is available. Any client will be able to access the data. To limit access, you can use serve NFS on loopback address and rely on secure tunnels (such as SSH). For this reason, by default, a random TCP port is chosen and loopback interface is used for the listening address; meaning that it is only available to the local machine. If you want other machines to access the NFS mount over local network, you need to specify the listening address and port using --addr flag.
Modifying files through NFS protocol requires VFS caching. Usually you will need to specify --vfs-cache-mode in order to be able to write to the mountpoint (full is recommended). If you don't specify VFS cache mode, the mount will be read-only.
To serve NFS over the network use following command:
+rclone serve nfs remote: --addr 0.0.0.0:$PORT --vfs-cache-mode=fullWe specify a specific port that we can use in the mount command:
+To mount the server under Linux/macOS, use the following command:
+mount -oport=$PORT,mountport=$PORT $HOSTNAME: path/to/mountpointWhere $PORT is the same port number we used in the serve nfs command.
This feature is only available on Unix platforms.
+This command uses the VFS layer. This adapts the cloud storage objects that rclone uses into something which looks much more like a disk filing system.
+Cloud storage objects have lots of properties which aren't like disk files - you can't extend them or write to the middle of them, so the VFS layer has to deal with that. Because there is no one right way of doing this there are various options explained below.
+The VFS layer also implements a directory cache - this caches info about files and directories (but not the data) in memory.
+Using the --dir-cache-time flag, you can control how long a directory should be considered up to date and not refreshed from the backend. Changes made through the VFS will appear immediately or invalidate the cache.
--dir-cache-time duration Time to cache directory entries for (default 5m0s)
+--poll-interval duration Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable (default 1m0s)However, changes made directly on the cloud storage by the web interface or a different copy of rclone will only be picked up once the directory cache expires if the backend configured does not support polling for changes. If the backend supports polling, changes will be picked up within the polling interval.
+You can send a SIGHUP signal to rclone for it to flush all directory caches, regardless of how old they are. Assuming only one rclone instance is running, you can reset the cache like this:
kill -SIGHUP $(pidof rclone)If you configure rclone with a remote control then you can use rclone rc to flush the whole directory cache:
+rclone rc vfs/forgetOr individual files or directories:
+rclone rc vfs/forget file=path/to/file dir=path/to/dirThe --buffer-size flag determines the amount of memory, that will be used to buffer data in advance.
Each open file will try to keep the specified amount of data in memory at all times. The buffered data is bound to one open file and won't be shared.
+This flag is a upper limit for the used memory per open file. The buffer will only use memory for data that is downloaded but not not yet read. If the buffer is empty, only a small amount of memory will be used.
+The maximum memory used by rclone for buffering can be up to --buffer-size * open files.
These flags control the VFS file caching options. File caching is necessary to make the VFS layer appear compatible with a normal file system. It can be disabled at the cost of some compatibility.
+For example you'll need to enable VFS caching if you want to read and write simultaneously to a file. See below for more details.
+Note that the VFS cache is separate from the cache backend and you may find that you need one or the other or both.
+--cache-dir string Directory rclone will use for caching.
+--vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off)
+--vfs-cache-max-age duration Max time since last access of objects in the cache (default 1h0m0s)
+--vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off)
+--vfs-cache-min-free-space SizeSuffix Target minimum free space on the disk containing the cache (default off)
+--vfs-cache-poll-interval duration Interval to poll the cache for stale objects (default 1m0s)
+--vfs-write-back duration Time to writeback files after last use when using cache (default 5s)If run with -vv rclone will print the location of the file cache. The files are stored in the user cache file area which is OS dependent but can be controlled with --cache-dir or setting the appropriate environment variable.
The cache has 4 different modes selected by --vfs-cache-mode. The higher the cache mode the more compatible rclone becomes at the cost of using disk space.
Note that files are written back to the remote only when they are closed and if they haven't been accessed for --vfs-write-back seconds. If rclone is quit or dies with files that haven't been uploaded, these will be uploaded next time rclone is run with the same flags.
If using --vfs-cache-max-size or --vfs-cache-min-free-size note that the cache may exceed these quotas for two reasons. Firstly because it is only checked every --vfs-cache-poll-interval. Secondly because open files cannot be evicted from the cache. When --vfs-cache-max-size or --vfs-cache-min-free-size is exceeded, rclone will attempt to evict the least accessed files from the cache first. rclone will start with files that haven't been accessed for the longest. This cache flushing strategy is efficient and more relevant files are likely to remain cached.
The --vfs-cache-max-age will evict files from the cache after the set time since last access has passed. The default value of 1 hour will start evicting files from cache that haven't been accessed for 1 hour. When a cached file is accessed the 1 hour timer is reset to 0 and will wait for 1 more hour before evicting. Specify the time with standard notation, s, m, h, d, w .
You should not run two copies of rclone using the same VFS cache with the same or overlapping remotes if using --vfs-cache-mode > off. This can potentially cause data corruption if you do. You can work around this by giving each rclone its own cache hierarchy with --cache-dir. You don't need to worry about this if the remotes in use don't overlap.
In this mode (the default) the cache will read directly from the remote and write directly to the remote without caching anything on disk.
+This will mean some operations are not possible
+This is very similar to "off" except that files opened for read AND write will be buffered to disk. This means that files opened for write will be a lot more compatible, but uses the minimal disk space.
+These operations are not possible
+In this mode files opened for read only are still read directly from the remote, write only and read/write files are buffered to disk first.
+This mode should support all normal file system operations.
+If an upload fails it will be retried at exponentially increasing intervals up to 1 minute.
+In this mode all reads and writes are buffered to and from disk. When data is read from the remote this is buffered to disk as well.
+In this mode the files in the cache will be sparse files and rclone will keep track of which bits of the files it has downloaded.
+So if an application only reads the starts of each file, then rclone will only buffer the start of the file. These files will appear to be their full size in the cache, but they will be sparse files with only the data that has been downloaded present in them.
+This mode should support all normal file system operations and is otherwise identical to --vfs-cache-mode writes.
When reading a file rclone will read --buffer-size plus --vfs-read-ahead bytes ahead. The --buffer-size is buffered in memory whereas the --vfs-read-ahead is buffered on disk.
When using this mode it is recommended that --buffer-size is not set too large and --vfs-read-ahead is set large if required.
IMPORTANT not all file systems support sparse files. In particular FAT/exFAT do not. Rclone will perform very badly if the cache directory is on a filesystem which doesn't support sparse files and it will log an ERROR message if one is detected.
+Various parts of the VFS use fingerprinting to see if a local file copy has changed relative to a remote file. Fingerprints are made from:
+where available on an object.
+On some backends some of these attributes are slow to read (they take an extra API call per object, or extra work per object).
+For example hash is slow with the local and sftp backends as they have to read the entire file and hash it, and modtime is slow with the s3, swift, ftp and qinqstor backends because they need to do an extra API call to fetch it.
If you use the --vfs-fast-fingerprint flag then rclone will not include the slow operations in the fingerprint. This makes the fingerprinting less accurate but much faster and will improve the opening time of cached files.
If you are running a vfs cache over local, s3 or swift backends then using this flag is recommended.
Note that if you change the value of this flag, the fingerprints of the files in the cache may be invalidated and the files will need to be downloaded again.
+When rclone reads files from a remote it reads them in chunks. This means that rather than requesting the whole file rclone reads the chunk specified. This can reduce the used download quota for some remotes by requesting only chunks from the remote that are actually read, at the cost of an increased number of requests.
+These flags control the chunking:
+--vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M)
+--vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off)Rclone will start reading a chunk of size --vfs-read-chunk-size, and then double the size for each read. When --vfs-read-chunk-size-limit is specified, and greater than --vfs-read-chunk-size, the chunk size for each open file will get doubled only until the specified value is reached. If the value is "off", which is the default, the limit is disabled and the chunk size will grow indefinitely.
With --vfs-read-chunk-size 100M and --vfs-read-chunk-size-limit 0 the following parts will be downloaded: 0-100M, 100M-200M, 200M-300M, 300M-400M and so on. When --vfs-read-chunk-size-limit 500M is specified, the result would be 0-100M, 100M-300M, 300M-700M, 700M-1200M, 1200M-1700M and so on.
Setting --vfs-read-chunk-size to 0 or "off" disables chunked reading.
These flags may be used to enable/disable features of the VFS for performance or other reasons. See also the chunked reading feature.
+In particular S3 and Swift benefit hugely from the --no-modtime flag (or use --use-server-modtime for a slightly different effect) as each read of the modification time takes a transaction.
--no-checksum Don't compare checksums on up/download.
+--no-modtime Don't read/write the modification time (can speed things up).
+--no-seek Don't allow seeking in files.
+--read-only Only allow read-only access.Sometimes rclone is delivered reads or writes out of order. Rather than seeking rclone will wait a short time for the in sequence read or write to come in. These flags only come into effect when not using an on disk cache file.
+--vfs-read-wait duration Time to wait for in-sequence read before seeking (default 20ms)
+--vfs-write-wait duration Time to wait for in-sequence write before giving error (default 1s)When using VFS write caching (--vfs-cache-mode with value writes or full), the global flag --transfers can be set to adjust the number of parallel uploads of modified files from the cache (the related global flag --checkers has no effect on the VFS).
--transfers int Number of file transfers to run in parallel (default 4)Linux file systems are case-sensitive: two files can differ only by case, and the exact case must be used when opening a file.
+File systems in modern Windows are case-insensitive but case-preserving: although existing files can be opened using any case, the exact case used to create the file is preserved and available for programs to query. It is not allowed for two files in the same directory to differ only by case.
+Usually file systems on macOS are case-insensitive. It is possible to make macOS file systems case-sensitive but that is not the default.
+The --vfs-case-insensitive VFS flag controls how rclone handles these two cases. If its value is "false", rclone passes file names to the remote as-is. If the flag is "true" (or appears without a value on the command line), rclone may perform a "fixup" as explained below.
The user may specify a file name to open/delete/rename/etc with a case different than what is stored on the remote. If an argument refers to an existing file with exactly the same name, then the case of the existing file on the disk will be used. However, if a file name with exactly the same name is not found but a name differing only by case exists, rclone will transparently fixup the name. This fixup happens only when an existing file is requested. Case sensitivity of file names created anew by rclone is controlled by the underlying remote.
+Note that case sensitivity of the operating system running rclone (the target) may differ from case sensitivity of a file system presented by rclone (the source). The flag controls whether "fixup" is performed to satisfy the target.
+If the flag is not provided on the command line, then its default value depends on the operating system where rclone runs: "true" on Windows and macOS, "false" otherwise. If the flag is provided without a value, then it is "true".
+This flag allows you to manually set the statistics about the filing system. It can be useful when those statistics cannot be read correctly automatically.
+--vfs-disk-space-total-size Manually set the total disk space size (example: 256G, default: -1)Some backends, most notably S3, do not report the amount of bytes used. If you need this information to be available when running df on the filesystem, then pass the flag --vfs-used-is-size to rclone. With this flag set, instead of relying on the backend to report this information, rclone will scan the whole remote similar to rclone size and compute the total used space itself.
WARNING. Contrary to rclone size, this flag ignores filters so that the result is accurate. However, this is very inefficient and may cost lots of API calls resulting in extra charges. Use it as a last resort and only with caching.
rclone serve nfs remote:path [flags] --addr string IPaddress:Port or :Port to bind server to
+ --dir-cache-time Duration Time to cache directory entries for (default 5m0s)
+ --dir-perms FileMode Directory permissions (default 0777)
+ --file-perms FileMode File permissions (default 0666)
+ --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000)
+ -h, --help help for nfs
+ --no-checksum Don't compare checksums on up/download
+ --no-modtime Don't read/write the modification time (can speed things up)
+ --no-seek Don't allow seeking in files
+ --poll-interval Duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s)
+ --read-only Only allow read-only access
+ --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000)
+ --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2)
+ --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s)
+ --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off)
+ --vfs-cache-min-free-space SizeSuffix Target minimum free space on the disk containing the cache (default off)
+ --vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off)
+ --vfs-cache-poll-interval Duration Interval to poll the cache for stale objects (default 1m0s)
+ --vfs-case-insensitive If a file name not found, find a case insensitive match
+ --vfs-disk-space-total-size SizeSuffix Specify the total space of disk (default off)
+ --vfs-fast-fingerprint Use fast (less accurate) fingerprints for change detection
+ --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full
+ --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi)
+ --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off)
+ --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms)
+ --vfs-refresh Refreshes the directory cache recursively on start
+ --vfs-used-is-size rclone size Use the rclone size algorithm for Used size
+ --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s)
+ --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s)Flags for filtering directory listings.
+ --delete-excluded Delete files on dest excluded from sync
+ --exclude stringArray Exclude files matching pattern
+ --exclude-from stringArray Read file exclude patterns from file (use - to read from stdin)
+ --exclude-if-present stringArray Exclude directories if filename is present
+ --files-from stringArray Read list of source-file names from file (use - to read from stdin)
+ --files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin)
+ -f, --filter stringArray Add a file filtering rule
+ --filter-from stringArray Read file filtering patterns from a file (use - to read from stdin)
+ --ignore-case Ignore case in filters (case insensitive)
+ --include stringArray Include files matching pattern
+ --include-from stringArray Read file include patterns from file (use - to read from stdin)
+ --max-age Duration Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off)
+ --max-depth int If set limits the recursion depth to this (default -1)
+ --max-size SizeSuffix Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off)
+ --metadata-exclude stringArray Exclude metadatas matching pattern
+ --metadata-exclude-from stringArray Read metadata exclude patterns from file (use - to read from stdin)
+ --metadata-filter stringArray Add a metadata filtering rule
+ --metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin)
+ --metadata-include stringArray Include metadatas matching pattern
+ --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin)
+ --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off)
+ --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off)See the global flags page for global options not listed here.
+Serve the remote for restic's REST API.
-Run a basic web server to serve a remote over restic's REST backend API over HTTP. This allows restic to use rclone as a data storage mechanism for cloud providers that restic does not support directly.
Restic is a command-line program for doing backups.
The server will log errors. Use -v to see access logs.
@@ -4592,7 +4866,7 @@Use --realm to set the authentication realm.
Use --salt to change the password hashing salt from the default.
rclone serve restic remote:path [flags] --addr stringArray IPaddress:Port or :Port to bind server to (default [127.0.0.1:8080])
--allow-origin string Origin which cross-domain request (CORS) can be executed from
--append-only Disallow deletion of repository data
@@ -4614,32 +4888,84 @@ Options
--stdio Run an HTTP2 server on stdin/stdout
--user string User name for authenticationSee the global flags page for global options not listed here.
-Serve the remote over SFTP.
-Run an SFTP server to serve a remote over SFTP. This can be used with an SFTP client or you can make a remote of type sftp to use with it.
-You can use the filter flags (e.g. --include, --exclude) to control what is served.
The server will respond to a small number of shell commands, mainly md5sum, sha1sum and df, which enable it to provide support for checksums and the about feature when accessed from an sftp remote.
-Note that this server uses standard 32 KiB packet payload size, which means you must not configure the client to expect anything else, e.g. with the chunk_size option on an sftp remote.
-The server will log errors. Use -v to see access logs.
--bwlimit will be respected for file transfers. Use --stats to control the stats printing.
You must provide some means of authentication, either with --user/--pass, an authorized keys file (specify location with --authorized-keys - the default is the same as ssh), an --auth-proxy, or set the --no-auth flag for no authentication when logging in.
If you don't supply a host --key then rclone will generate rsa, ecdsa and ed25519 variants, and cache them for later use in rclone's cache directory (see rclone help flags cache-dir) in the "serve-sftp" directory.
By default the server binds to localhost:2022 - if you want it to be reachable externally then supply --addr :2022 for example.
Note that the default of --vfs-cache-mode off is fine for the rclone sftp backend, but it may not be with other SFTP clients.
If --stdio is specified, rclone will serve SFTP over stdio, which can be used with sshd via ~/.ssh/authorized_keys, for example:
restrict,command="rclone serve sftp --stdio ./photos" ssh-rsa ...On the client you need to set --transfers 1 when using --stdio. Otherwise multiple instances of the rclone server are started by OpenSSH which can lead to "corrupted on transfer" errors. This is the case because the client chooses indiscriminately which server to send commands to while the servers all have different views of the state of the filing system.
The "restrict" in authorized_keys prevents SHA1SUMs and MD5SUMs from being used. Omitting "restrict" and using --sftp-path-override to enable checksumming is possible but less secure and you could use the SFTP server provided by OpenSSH in this case.
Serve remote:path over s3.
+serve s3 implements a basic s3 server that serves a remote via s3. This can be viewed with an s3 client, or you can make an s3 type remote to read and write to it with rclone.
serve s3 is considered Experimental so use with care.
S3 server supports Signature Version 4 authentication. Just use --auth-key accessKey,secretKey and set the Authorization header correctly in the request. (See the AWS docs).
--auth-key can be repeated for multiple auth pairs. If --auth-key is not provided then serve s3 will allow anonymous access.
Please note that some clients may require HTTPS endpoints. See the SSL docs for more information.
+This command uses the VFS directory cache. All the functionality will work with --vfs-cache-mode off. Using --vfs-cache-mode full (or writes) can be used to cache objects locally to improve performance.
Use --force-path-style=false if you want to use the bucket name as a part of the hostname (such as mybucket.local)
Use --etag-hash if you want to change the hash uses for the ETag. Note that using anything other than MD5 (the default) is likely to cause problems for S3 clients which rely on the Etag being the MD5.
For a simple set up, to serve remote:path over s3, run the server like this:
rclone serve s3 --auth-key ACCESS_KEY_ID,SECRET_ACCESS_KEY remote:pathThis will be compatible with an rclone remote which is defined like this:
+[serves3]
+type = s3
+provider = Rclone
+endpoint = http://127.0.0.1:8080/
+access_key_id = ACCESS_KEY_ID
+secret_access_key = SECRET_ACCESS_KEY
+use_multipart_uploads = falseNote that setting disable_multipart_uploads = true is to work around a bug which will be fixed in due course.
When uploading multipart files serve s3 holds all the parts in memory (see #7453). This is a limitaton of the library rclone uses for serving S3 and will hopefully be fixed at some point.
Multipart server side copies do not work (see #7454). These take a very long time and eventually fail. The default threshold for multipart server side copies is 5G which is the maximum it can be, so files above this side will fail to be server side copied.
+For a current list of serve s3 bugs see the serve s3 bug category on GitHub.
serve s3 will treat all directories in the root as buckets and ignore all files in the root. You can use CreateBucket to create folders under the root, but you can't create empty folders under other folders not in the root.
When using PutObject or DeleteObject, rclone will automatically create or clean up empty folders. If you don't want to clean up empty folders automatically, use --no-cleanup.
When using ListObjects, rclone will use / when the delimiter is empty. This reduces backend requests with no effect on most operations, but if the delimiter is something other than / and empty, rclone will do a full recursive search of the backend, which can take some time.
Versioning is not currently supported.
+Metadata will only be saved in memory other than the rclone mtime metadata which will be set as the modification time of the file.
serve s3 currently supports the following operations.
ListBucketsCreateBucketDeleteBucketHeadObjectListObjectsGetObjectPutObjectDeleteObjectDeleteObjectsCreateMultipartUploadCompleteMultipartUploadAbortMultipartUploadCopyObjectUploadPartOther operations will return error Unimplemented.
Use --addr to specify which IP address and port the server should listen on, eg --addr 1.2.3.4:8000 or --addr :8080 to listen to all IPs. By default it only listens on localhost. You can use port :0 to let the OS choose an available port.
If you set --addr to listen on a public or LAN accessible IP address then using Authentication is advised - see the next section for info.
You can use a unix socket by setting the url to unix:///path/to/socket or just by using an absolute path name. Note that unix sockets bypass the authentication - this is expected to be done with file system permissions.
--addr may be repeated to listen on multiple IPs/ports/sockets.
--server-read-timeout and --server-write-timeout can be used to control the timeouts on the server. Note that this is the total time for a transfer.
--max-header-bytes controls the maximum number of bytes the server will accept in the HTTP header.
--baseurl controls the URL prefix that rclone serves from. By default rclone will serve from the root. If you used --baseurl "/rclone" then rclone would serve from a URL starting with "/rclone/". This is useful if you wish to proxy rclone serve. Rclone automatically inserts leading and trailing "/" on --baseurl, so --baseurl "rclone", --baseurl "/rclone" and --baseurl "/rclone/" are all treated identically.
By default this will serve over http. If you want you can serve over https. You will need to supply the --cert and --key flags. If you wish to do client side certificate validation then you will need to supply --client-ca also.
--cert should be a either a PEM encoded certificate or a concatenation of that with the CA certificate. --key should be the PEM encoded private key and --client-ca should be the PEM encoded client certificate authority certificate.
--min-tls-version is minimum TLS version that is acceptable. Valid values are "tls1.0", "tls1.1", "tls1.2" and "tls1.3" (default "tls1.0"). ## VFS - Virtual File System
This command uses the VFS layer. This adapts the cloud storage objects that rclone uses into something which looks much more like a disk filing system.
Cloud storage objects have lots of properties which aren't like disk files - you can't extend them or write to the middle of them, so the VFS layer has to deal with that. Because there is no one right way of doing this there are various options explained below.
The VFS layer also implements a directory cache - this caches info about files and directories (but not the data) in memory.
-Using the --dir-cache-time flag, you can control how long a directory should be considered up to date and not refreshed from the backend. Changes made through the VFS will appear immediately or invalidate the cache.
--dir-cache-time duration Time to cache directory entries for (default 5m0s)
--poll-interval duration Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable (default 1m0s)rclone rc vfs/forgetOr individual files or directories:
rclone rc vfs/forget file=path/to/file dir=path/to/dirThe --buffer-size flag determines the amount of memory, that will be used to buffer data in advance.
Each open file will try to keep the specified amount of data in memory at all times. The buffered data is bound to one open file and won't be shared.
This flag is a upper limit for the used memory per open file. The buffer will only use memory for data that is downloaded but not not yet read. If the buffer is empty, only a small amount of memory will be used.
The maximum memory used by rclone for buffering can be up to --buffer-size * open files.
These flags control the VFS file caching options. File caching is necessary to make the VFS layer appear compatible with a normal file system. It can be disabled at the cost of some compatibility.
For example you'll need to enable VFS caching if you want to read and write simultaneously to a file. See below for more details.
Note that the VFS cache is separate from the cache backend and you may find that you need one or the other or both.
@@ -4672,7 +4998,7 @@If using --vfs-cache-max-size or --vfs-cache-min-free-size note that the cache may exceed these quotas for two reasons. Firstly because it is only checked every --vfs-cache-poll-interval. Secondly because open files cannot be evicted from the cache. When --vfs-cache-max-size or --vfs-cache-min-free-size is exceeded, rclone will attempt to evict the least accessed files from the cache first. rclone will start with files that haven't been accessed for the longest. This cache flushing strategy is efficient and more relevant files are likely to remain cached.
The --vfs-cache-max-age will evict files from the cache after the set time since last access has passed. The default value of 1 hour will start evicting files from cache that haven't been accessed for 1 hour. When a cached file is accessed the 1 hour timer is reset to 0 and will wait for 1 more hour before evicting. Specify the time with standard notation, s, m, h, d, w .
You should not run two copies of rclone using the same VFS cache with the same or overlapping remotes if using --vfs-cache-mode > off. This can potentially cause data corruption if you do. You can work around this by giving each rclone its own cache hierarchy with --cache-dir. You don't need to worry about this if the remotes in use don't overlap.
In this mode (the default) the cache will read directly from the remote and write directly to the remote without caching anything on disk.
This will mean some operations are not possible
This is very similar to "off" except that files opened for read AND write will be buffered to disk. This means that files opened for write will be a lot more compatible, but uses the minimal disk space.
These operations are not possible
In this mode files opened for read only are still read directly from the remote, write only and read/write files are buffered to disk first.
This mode should support all normal file system operations.
If an upload fails it will be retried at exponentially increasing intervals up to 1 minute.
-In this mode all reads and writes are buffered to and from disk. When data is read from the remote this is buffered to disk as well.
In this mode the files in the cache will be sparse files and rclone will keep track of which bits of the files it has downloaded.
So if an application only reads the starts of each file, then rclone will only buffer the start of the file. These files will appear to be their full size in the cache, but they will be sparse files with only the data that has been downloaded present in them.
@@ -4705,7 +5031,7 @@When reading a file rclone will read --buffer-size plus --vfs-read-ahead bytes ahead. The --buffer-size is buffered in memory whereas the --vfs-read-ahead is buffered on disk.
When using this mode it is recommended that --buffer-size is not set too large and --vfs-read-ahead is set large if required.
IMPORTANT not all file systems support sparse files. In particular FAT/exFAT do not. Rclone will perform very badly if the cache directory is on a filesystem which doesn't support sparse files and it will log an ERROR message if one is detected.
-Various parts of the VFS use fingerprinting to see if a local file copy has changed relative to a remote file. Fingerprints are made from:
If you use the --vfs-fast-fingerprint flag then rclone will not include the slow operations in the fingerprint. This makes the fingerprinting less accurate but much faster and will improve the opening time of cached files.
If you are running a vfs cache over local, s3 or swift backends then using this flag is recommended.
Note that if you change the value of this flag, the fingerprints of the files in the cache may be invalidated and the files will need to be downloaded again.
-When rclone reads files from a remote it reads them in chunks. This means that rather than requesting the whole file rclone reads the chunk specified. This can reduce the used download quota for some remotes by requesting only chunks from the remote that are actually read, at the cost of an increased number of requests.
These flags control the chunking:
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M)
@@ -4726,7 +5052,7 @@ VFS Chunked Reading
Rclone will start reading a chunk of size --vfs-read-chunk-size, and then double the size for each read. When --vfs-read-chunk-size-limit is specified, and greater than --vfs-read-chunk-size, the chunk size for each open file will get doubled only until the specified value is reached. If the value is "off", which is the default, the limit is disabled and the chunk size will grow indefinitely.
With --vfs-read-chunk-size 100M and --vfs-read-chunk-size-limit 0 the following parts will be downloaded: 0-100M, 100M-200M, 200M-300M, 300M-400M and so on. When --vfs-read-chunk-size-limit 500M is specified, the result would be 0-100M, 100M-300M, 300M-700M, 700M-1200M, 1200M-1700M and so on.
Setting --vfs-read-chunk-size to 0 or "off" disables chunked reading.
-VFS Performance
+VFS Performance
These flags may be used to enable/disable features of the VFS for performance or other reasons. See also the chunked reading feature.
In particular S3 and Swift benefit hugely from the --no-modtime flag (or use --use-server-modtime for a slightly different effect) as each read of the modification time takes a transaction.
--no-checksum Don't compare checksums on up/download.
@@ -4738,7 +5064,7 @@ VFS Performance
--vfs-write-wait duration Time to wait for in-sequence write before giving error (default 1s)
When using VFS write caching (--vfs-cache-mode with value writes or full), the global flag --transfers can be set to adjust the number of parallel uploads of modified files from the cache (the related global flag --checkers has no effect on the VFS).
--transfers int Number of file transfers to run in parallel (default 4)
-VFS Case Sensitivity
+VFS Case Sensitivity
Linux file systems are case-sensitive: two files can differ only by case, and the exact case must be used when opening a file.
File systems in modern Windows are case-insensitive but case-preserving: although existing files can be opened using any case, the exact case used to create the file is preserved and available for programs to query. It is not allowed for two files in the same directory to differ only by case.
Usually file systems on macOS are case-insensitive. It is possible to make macOS file systems case-sensitive but that is not the default.
@@ -4746,64 +5072,40 @@ VFS Case Sensitivity
The user may specify a file name to open/delete/rename/etc with a case different than what is stored on the remote. If an argument refers to an existing file with exactly the same name, then the case of the existing file on the disk will be used. However, if a file name with exactly the same name is not found but a name differing only by case exists, rclone will transparently fixup the name. This fixup happens only when an existing file is requested. Case sensitivity of file names created anew by rclone is controlled by the underlying remote.
Note that case sensitivity of the operating system running rclone (the target) may differ from case sensitivity of a file system presented by rclone (the source). The flag controls whether "fixup" is performed to satisfy the target.
If the flag is not provided on the command line, then its default value depends on the operating system where rclone runs: "true" on Windows and macOS, "false" otherwise. If the flag is provided without a value, then it is "true".
-VFS Disk Options
+VFS Disk Options
This flag allows you to manually set the statistics about the filing system. It can be useful when those statistics cannot be read correctly automatically.
--vfs-disk-space-total-size Manually set the total disk space size (example: 256G, default: -1)
-Alternate report of used bytes
+Alternate report of used bytes
Some backends, most notably S3, do not report the amount of bytes used. If you need this information to be available when running df on the filesystem, then pass the flag --vfs-used-is-size to rclone. With this flag set, instead of relying on the backend to report this information, rclone will scan the whole remote similar to rclone size and compute the total used space itself.
WARNING. Contrary to rclone size, this flag ignores filters so that the result is accurate. However, this is very inefficient and may cost lots of API calls resulting in extra charges. Use it as a last resort and only with caching.
-Auth Proxy
-If you supply the parameter --auth-proxy /path/to/program then rclone will use that program to generate backends on the fly which then are used to authenticate incoming requests. This uses a simple JSON based protocol with input on STDIN and output on STDOUT.
-PLEASE NOTE: --auth-proxy and --authorized-keys cannot be used together, if --auth-proxy is set the authorized keys option will be ignored.
-There is an example program bin/test_proxy.py in the rclone source code.
-The program's job is to take a user and pass on the input and turn those into the config for a backend on STDOUT in JSON format. This config will have any default parameters for the backend added, but it won't use configuration from environment variables or command line options - it is the job of the proxy program to make a complete config.
-This config generated must have this extra parameter - _root - root to use for the backend
-And it may have this parameter - _obscure - comma separated strings for parameters to obscure
-If password authentication was used by the client, input to the proxy process (on STDIN) would look similar to this:
-{
- "user": "me",
- "pass": "mypassword"
-}
-If public-key authentication was used by the client, input to the proxy process (on STDIN) would look similar to this:
-{
- "user": "me",
- "public_key": "AAAAB3NzaC1yc2EAAAADAQABAAABAQDuwESFdAe14hVS6omeyX7edc...JQdf"
-}
-And as an example return this on STDOUT
-{
- "type": "sftp",
- "_root": "",
- "_obscure": "pass",
- "user": "me",
- "pass": "mypassword",
- "host": "sftp.example.com"
-}
-This would mean that an SFTP backend would be created on the fly for the user and pass/public_key returned in the output to the host given. Note that since _obscure is set to pass, rclone will obscure the pass parameter before creating the backend (which is required for sftp backends).
-The program can manipulate the supplied user in any way, for example to make proxy to many different sftp backends, you could make the user be user@example.com and then set the host to example.com in the output and the user to user. For security you'd probably want to restrict the host to a limited list.
-Note that an internal cache is keyed on user so only use that for configuration, don't use pass or public_key. This also means that if a user's password or public-key is changed the cache will need to expire (which takes 5 mins) before it takes effect.
-This can be used to build general purpose proxies to any kind of backend that rclone supports.
-rclone serve sftp remote:path [flags]
-Options
- --addr string IPaddress:Port or :Port to bind server to (default "localhost:2022")
- --auth-proxy string A program to use to create the backend from the auth
- --authorized-keys string Authorized keys file (default "~/.ssh/authorized_keys")
+rclone serve s3 remote:path [flags]
+Options
+ --addr stringArray IPaddress:Port or :Port to bind server to (default [127.0.0.1:8080])
+ --allow-origin string Origin which cross-domain request (CORS) can be executed from
+ --auth-key stringArray Set key pair for v4 authorization: access_key_id,secret_access_key
+ --baseurl string Prefix for URLs - leave blank for root
+ --cert string TLS PEM key (concatenation of certificate and CA certificate)
+ --client-ca string Client certificate authority to verify clients with
--dir-cache-time Duration Time to cache directory entries for (default 5m0s)
--dir-perms FileMode Directory permissions (default 0777)
+ --etag-hash string Which hash to use for the ETag, or auto or blank for off (default "MD5")
--file-perms FileMode File permissions (default 0666)
+ --force-path-style If true use path style access if false use virtual hosted style (default true) (default true)
--gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000)
- -h, --help help for sftp
- --key stringArray SSH private host key file (Can be multi-valued, leave blank to auto generate)
- --no-auth Allow connections with no authentication if set
+ -h, --help help for s3
+ --key string TLS PEM Private key
+ --max-header-bytes int Maximum size of request header (default 4096)
+ --min-tls-version string Minimum TLS version that is acceptable (default "tls1.0")
--no-checksum Don't compare checksums on up/download
+ --no-cleanup Not to cleanup empty folder after object is deleted
--no-modtime Don't read/write the modification time (can speed things up)
--no-seek Don't allow seeking in files
- --pass string Password for authentication
--poll-interval Duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s)
--read-only Only allow read-only access
- --stdio Run an sftp server on stdin/stdout
+ --server-read-timeout Duration Timeout for server reading data (default 1h0m0s)
+ --server-write-timeout Duration Timeout for server writing data (default 1h0m0s)
--uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000)
--umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2)
- --user string User name for authentication
--vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s)
--vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off)
--vfs-cache-min-free-space SizeSuffix Target minimum free space on the disk containing the cache (default off)
@@ -4816,10 +5118,11 @@ Options
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi)
--vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off)
--vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms)
+ --vfs-refresh Refreshes the directory cache recursively on start
--vfs-used-is-size rclone size Use the rclone size algorithm for Used size
--vfs-write-back Duration Time to writeback files after last use when using cache (default 5s)
--vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s)
-Filter Options
+Filter Options
Flags for filtering directory listings.
--delete-excluded Delete files on dest excluded from sync
--exclude stringArray Exclude files matching pattern
@@ -4844,13 +5147,244 @@ Filter Options
--min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off)
--min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off)
See the global flags page for global options not listed here.
-SEE ALSO
+SEE ALSO
+
+- rclone serve - Serve a remote over a protocol.
+
+rclone serve sftp
+Serve the remote over SFTP.
+Synopsis
+Run an SFTP server to serve a remote over SFTP. This can be used with an SFTP client or you can make a remote of type sftp to use with it.
+You can use the filter flags (e.g. --include, --exclude) to control what is served.
+The server will respond to a small number of shell commands, mainly md5sum, sha1sum and df, which enable it to provide support for checksums and the about feature when accessed from an sftp remote.
+Note that this server uses standard 32 KiB packet payload size, which means you must not configure the client to expect anything else, e.g. with the chunk_size option on an sftp remote.
+The server will log errors. Use -v to see access logs.
+--bwlimit will be respected for file transfers. Use --stats to control the stats printing.
+You must provide some means of authentication, either with --user/--pass, an authorized keys file (specify location with --authorized-keys - the default is the same as ssh), an --auth-proxy, or set the --no-auth flag for no authentication when logging in.
+If you don't supply a host --key then rclone will generate rsa, ecdsa and ed25519 variants, and cache them for later use in rclone's cache directory (see rclone help flags cache-dir) in the "serve-sftp" directory.
+By default the server binds to localhost:2022 - if you want it to be reachable externally then supply --addr :2022 for example.
+Note that the default of --vfs-cache-mode off is fine for the rclone sftp backend, but it may not be with other SFTP clients.
+If --stdio is specified, rclone will serve SFTP over stdio, which can be used with sshd via ~/.ssh/authorized_keys, for example:
+restrict,command="rclone serve sftp --stdio ./photos" ssh-rsa ...
+On the client you need to set --transfers 1 when using --stdio. Otherwise multiple instances of the rclone server are started by OpenSSH which can lead to "corrupted on transfer" errors. This is the case because the client chooses indiscriminately which server to send commands to while the servers all have different views of the state of the filing system.
+The "restrict" in authorized_keys prevents SHA1SUMs and MD5SUMs from being used. Omitting "restrict" and using --sftp-path-override to enable checksumming is possible but less secure and you could use the SFTP server provided by OpenSSH in this case.
+VFS - Virtual File System
+This command uses the VFS layer. This adapts the cloud storage objects that rclone uses into something which looks much more like a disk filing system.
+Cloud storage objects have lots of properties which aren't like disk files - you can't extend them or write to the middle of them, so the VFS layer has to deal with that. Because there is no one right way of doing this there are various options explained below.
+The VFS layer also implements a directory cache - this caches info about files and directories (but not the data) in memory.
+VFS Directory Cache
+Using the --dir-cache-time flag, you can control how long a directory should be considered up to date and not refreshed from the backend. Changes made through the VFS will appear immediately or invalidate the cache.
+--dir-cache-time duration Time to cache directory entries for (default 5m0s)
+--poll-interval duration Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable (default 1m0s)
+However, changes made directly on the cloud storage by the web interface or a different copy of rclone will only be picked up once the directory cache expires if the backend configured does not support polling for changes. If the backend supports polling, changes will be picked up within the polling interval.
+You can send a SIGHUP signal to rclone for it to flush all directory caches, regardless of how old they are. Assuming only one rclone instance is running, you can reset the cache like this:
+kill -SIGHUP $(pidof rclone)
+If you configure rclone with a remote control then you can use rclone rc to flush the whole directory cache:
+rclone rc vfs/forget
+Or individual files or directories:
+rclone rc vfs/forget file=path/to/file dir=path/to/dir
+VFS File Buffering
+The --buffer-size flag determines the amount of memory, that will be used to buffer data in advance.
+Each open file will try to keep the specified amount of data in memory at all times. The buffered data is bound to one open file and won't be shared.
+This flag is a upper limit for the used memory per open file. The buffer will only use memory for data that is downloaded but not not yet read. If the buffer is empty, only a small amount of memory will be used.
+The maximum memory used by rclone for buffering can be up to --buffer-size * open files.
+VFS File Caching
+These flags control the VFS file caching options. File caching is necessary to make the VFS layer appear compatible with a normal file system. It can be disabled at the cost of some compatibility.
+For example you'll need to enable VFS caching if you want to read and write simultaneously to a file. See below for more details.
+Note that the VFS cache is separate from the cache backend and you may find that you need one or the other or both.
+--cache-dir string Directory rclone will use for caching.
+--vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off)
+--vfs-cache-max-age duration Max time since last access of objects in the cache (default 1h0m0s)
+--vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off)
+--vfs-cache-min-free-space SizeSuffix Target minimum free space on the disk containing the cache (default off)
+--vfs-cache-poll-interval duration Interval to poll the cache for stale objects (default 1m0s)
+--vfs-write-back duration Time to writeback files after last use when using cache (default 5s)
+If run with -vv rclone will print the location of the file cache. The files are stored in the user cache file area which is OS dependent but can be controlled with --cache-dir or setting the appropriate environment variable.
+The cache has 4 different modes selected by --vfs-cache-mode. The higher the cache mode the more compatible rclone becomes at the cost of using disk space.
+Note that files are written back to the remote only when they are closed and if they haven't been accessed for --vfs-write-back seconds. If rclone is quit or dies with files that haven't been uploaded, these will be uploaded next time rclone is run with the same flags.
+If using --vfs-cache-max-size or --vfs-cache-min-free-size note that the cache may exceed these quotas for two reasons. Firstly because it is only checked every --vfs-cache-poll-interval. Secondly because open files cannot be evicted from the cache. When --vfs-cache-max-size or --vfs-cache-min-free-size is exceeded, rclone will attempt to evict the least accessed files from the cache first. rclone will start with files that haven't been accessed for the longest. This cache flushing strategy is efficient and more relevant files are likely to remain cached.
+The --vfs-cache-max-age will evict files from the cache after the set time since last access has passed. The default value of 1 hour will start evicting files from cache that haven't been accessed for 1 hour. When a cached file is accessed the 1 hour timer is reset to 0 and will wait for 1 more hour before evicting. Specify the time with standard notation, s, m, h, d, w .
+You should not run two copies of rclone using the same VFS cache with the same or overlapping remotes if using --vfs-cache-mode > off. This can potentially cause data corruption if you do. You can work around this by giving each rclone its own cache hierarchy with --cache-dir. You don't need to worry about this if the remotes in use don't overlap.
+--vfs-cache-mode off
+In this mode (the default) the cache will read directly from the remote and write directly to the remote without caching anything on disk.
+This will mean some operations are not possible
+
+- Files can't be opened for both read AND write
+- Files opened for write can't be seeked
+- Existing files opened for write must have O_TRUNC set
+- Files open for read with O_TRUNC will be opened write only
+- Files open for write only will behave as if O_TRUNC was supplied
+- Open modes O_APPEND, O_TRUNC are ignored
+- If an upload fails it can't be retried
+
+--vfs-cache-mode minimal
+This is very similar to "off" except that files opened for read AND write will be buffered to disk. This means that files opened for write will be a lot more compatible, but uses the minimal disk space.
+These operations are not possible
+
+- Files opened for write only can't be seeked
+- Existing files opened for write must have O_TRUNC set
+- Files opened for write only will ignore O_APPEND, O_TRUNC
+- If an upload fails it can't be retried
+
+--vfs-cache-mode writes
+In this mode files opened for read only are still read directly from the remote, write only and read/write files are buffered to disk first.
+This mode should support all normal file system operations.
+If an upload fails it will be retried at exponentially increasing intervals up to 1 minute.
+--vfs-cache-mode full
+In this mode all reads and writes are buffered to and from disk. When data is read from the remote this is buffered to disk as well.
+In this mode the files in the cache will be sparse files and rclone will keep track of which bits of the files it has downloaded.
+So if an application only reads the starts of each file, then rclone will only buffer the start of the file. These files will appear to be their full size in the cache, but they will be sparse files with only the data that has been downloaded present in them.
+This mode should support all normal file system operations and is otherwise identical to --vfs-cache-mode writes.
+When reading a file rclone will read --buffer-size plus --vfs-read-ahead bytes ahead. The --buffer-size is buffered in memory whereas the --vfs-read-ahead is buffered on disk.
+When using this mode it is recommended that --buffer-size is not set too large and --vfs-read-ahead is set large if required.
+IMPORTANT not all file systems support sparse files. In particular FAT/exFAT do not. Rclone will perform very badly if the cache directory is on a filesystem which doesn't support sparse files and it will log an ERROR message if one is detected.
+Fingerprinting
+Various parts of the VFS use fingerprinting to see if a local file copy has changed relative to a remote file. Fingerprints are made from:
+
+- size
+- modification time
+- hash
+
+where available on an object.
+On some backends some of these attributes are slow to read (they take an extra API call per object, or extra work per object).
+For example hash is slow with the local and sftp backends as they have to read the entire file and hash it, and modtime is slow with the s3, swift, ftp and qinqstor backends because they need to do an extra API call to fetch it.
+If you use the --vfs-fast-fingerprint flag then rclone will not include the slow operations in the fingerprint. This makes the fingerprinting less accurate but much faster and will improve the opening time of cached files.
+If you are running a vfs cache over local, s3 or swift backends then using this flag is recommended.
+Note that if you change the value of this flag, the fingerprints of the files in the cache may be invalidated and the files will need to be downloaded again.
+VFS Chunked Reading
+When rclone reads files from a remote it reads them in chunks. This means that rather than requesting the whole file rclone reads the chunk specified. This can reduce the used download quota for some remotes by requesting only chunks from the remote that are actually read, at the cost of an increased number of requests.
+These flags control the chunking:
+--vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M)
+--vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off)
+Rclone will start reading a chunk of size --vfs-read-chunk-size, and then double the size for each read. When --vfs-read-chunk-size-limit is specified, and greater than --vfs-read-chunk-size, the chunk size for each open file will get doubled only until the specified value is reached. If the value is "off", which is the default, the limit is disabled and the chunk size will grow indefinitely.
+With --vfs-read-chunk-size 100M and --vfs-read-chunk-size-limit 0 the following parts will be downloaded: 0-100M, 100M-200M, 200M-300M, 300M-400M and so on. When --vfs-read-chunk-size-limit 500M is specified, the result would be 0-100M, 100M-300M, 300M-700M, 700M-1200M, 1200M-1700M and so on.
+Setting --vfs-read-chunk-size to 0 or "off" disables chunked reading.
+VFS Performance
+These flags may be used to enable/disable features of the VFS for performance or other reasons. See also the chunked reading feature.
+In particular S3 and Swift benefit hugely from the --no-modtime flag (or use --use-server-modtime for a slightly different effect) as each read of the modification time takes a transaction.
+--no-checksum Don't compare checksums on up/download.
+--no-modtime Don't read/write the modification time (can speed things up).
+--no-seek Don't allow seeking in files.
+--read-only Only allow read-only access.
+Sometimes rclone is delivered reads or writes out of order. Rather than seeking rclone will wait a short time for the in sequence read or write to come in. These flags only come into effect when not using an on disk cache file.
+--vfs-read-wait duration Time to wait for in-sequence read before seeking (default 20ms)
+--vfs-write-wait duration Time to wait for in-sequence write before giving error (default 1s)
+When using VFS write caching (--vfs-cache-mode with value writes or full), the global flag --transfers can be set to adjust the number of parallel uploads of modified files from the cache (the related global flag --checkers has no effect on the VFS).
+--transfers int Number of file transfers to run in parallel (default 4)
+VFS Case Sensitivity
+Linux file systems are case-sensitive: two files can differ only by case, and the exact case must be used when opening a file.
+File systems in modern Windows are case-insensitive but case-preserving: although existing files can be opened using any case, the exact case used to create the file is preserved and available for programs to query. It is not allowed for two files in the same directory to differ only by case.
+Usually file systems on macOS are case-insensitive. It is possible to make macOS file systems case-sensitive but that is not the default.
+The --vfs-case-insensitive VFS flag controls how rclone handles these two cases. If its value is "false", rclone passes file names to the remote as-is. If the flag is "true" (or appears without a value on the command line), rclone may perform a "fixup" as explained below.
+The user may specify a file name to open/delete/rename/etc with a case different than what is stored on the remote. If an argument refers to an existing file with exactly the same name, then the case of the existing file on the disk will be used. However, if a file name with exactly the same name is not found but a name differing only by case exists, rclone will transparently fixup the name. This fixup happens only when an existing file is requested. Case sensitivity of file names created anew by rclone is controlled by the underlying remote.
+Note that case sensitivity of the operating system running rclone (the target) may differ from case sensitivity of a file system presented by rclone (the source). The flag controls whether "fixup" is performed to satisfy the target.
+If the flag is not provided on the command line, then its default value depends on the operating system where rclone runs: "true" on Windows and macOS, "false" otherwise. If the flag is provided without a value, then it is "true".
+VFS Disk Options
+This flag allows you to manually set the statistics about the filing system. It can be useful when those statistics cannot be read correctly automatically.
+--vfs-disk-space-total-size Manually set the total disk space size (example: 256G, default: -1)
+Alternate report of used bytes
+Some backends, most notably S3, do not report the amount of bytes used. If you need this information to be available when running df on the filesystem, then pass the flag --vfs-used-is-size to rclone. With this flag set, instead of relying on the backend to report this information, rclone will scan the whole remote similar to rclone size and compute the total used space itself.
+WARNING. Contrary to rclone size, this flag ignores filters so that the result is accurate. However, this is very inefficient and may cost lots of API calls resulting in extra charges. Use it as a last resort and only with caching.
+Auth Proxy
+If you supply the parameter --auth-proxy /path/to/program then rclone will use that program to generate backends on the fly which then are used to authenticate incoming requests. This uses a simple JSON based protocol with input on STDIN and output on STDOUT.
+PLEASE NOTE: --auth-proxy and --authorized-keys cannot be used together, if --auth-proxy is set the authorized keys option will be ignored.
+There is an example program bin/test_proxy.py in the rclone source code.
+The program's job is to take a user and pass on the input and turn those into the config for a backend on STDOUT in JSON format. This config will have any default parameters for the backend added, but it won't use configuration from environment variables or command line options - it is the job of the proxy program to make a complete config.
+This config generated must have this extra parameter - _root - root to use for the backend
+And it may have this parameter - _obscure - comma separated strings for parameters to obscure
+If password authentication was used by the client, input to the proxy process (on STDIN) would look similar to this:
+{
+ "user": "me",
+ "pass": "mypassword"
+}
+If public-key authentication was used by the client, input to the proxy process (on STDIN) would look similar to this:
+{
+ "user": "me",
+ "public_key": "AAAAB3NzaC1yc2EAAAADAQABAAABAQDuwESFdAe14hVS6omeyX7edc...JQdf"
+}
+And as an example return this on STDOUT
+{
+ "type": "sftp",
+ "_root": "",
+ "_obscure": "pass",
+ "user": "me",
+ "pass": "mypassword",
+ "host": "sftp.example.com"
+}
+This would mean that an SFTP backend would be created on the fly for the user and pass/public_key returned in the output to the host given. Note that since _obscure is set to pass, rclone will obscure the pass parameter before creating the backend (which is required for sftp backends).
+The program can manipulate the supplied user in any way, for example to make proxy to many different sftp backends, you could make the user be user@example.com and then set the host to example.com in the output and the user to user. For security you'd probably want to restrict the host to a limited list.
+Note that an internal cache is keyed on user so only use that for configuration, don't use pass or public_key. This also means that if a user's password or public-key is changed the cache will need to expire (which takes 5 mins) before it takes effect.
+This can be used to build general purpose proxies to any kind of backend that rclone supports.
+rclone serve sftp remote:path [flags]
+Options
+ --addr string IPaddress:Port or :Port to bind server to (default "localhost:2022")
+ --auth-proxy string A program to use to create the backend from the auth
+ --authorized-keys string Authorized keys file (default "~/.ssh/authorized_keys")
+ --dir-cache-time Duration Time to cache directory entries for (default 5m0s)
+ --dir-perms FileMode Directory permissions (default 0777)
+ --file-perms FileMode File permissions (default 0666)
+ --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000)
+ -h, --help help for sftp
+ --key stringArray SSH private host key file (Can be multi-valued, leave blank to auto generate)
+ --no-auth Allow connections with no authentication if set
+ --no-checksum Don't compare checksums on up/download
+ --no-modtime Don't read/write the modification time (can speed things up)
+ --no-seek Don't allow seeking in files
+ --pass string Password for authentication
+ --poll-interval Duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s)
+ --read-only Only allow read-only access
+ --stdio Run an sftp server on stdin/stdout
+ --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000)
+ --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2)
+ --user string User name for authentication
+ --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s)
+ --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off)
+ --vfs-cache-min-free-space SizeSuffix Target minimum free space on the disk containing the cache (default off)
+ --vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off)
+ --vfs-cache-poll-interval Duration Interval to poll the cache for stale objects (default 1m0s)
+ --vfs-case-insensitive If a file name not found, find a case insensitive match
+ --vfs-disk-space-total-size SizeSuffix Specify the total space of disk (default off)
+ --vfs-fast-fingerprint Use fast (less accurate) fingerprints for change detection
+ --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full
+ --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi)
+ --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off)
+ --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms)
+ --vfs-refresh Refreshes the directory cache recursively on start
+ --vfs-used-is-size rclone size Use the rclone size algorithm for Used size
+ --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s)
+ --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s)
+Filter Options
+Flags for filtering directory listings.
+ --delete-excluded Delete files on dest excluded from sync
+ --exclude stringArray Exclude files matching pattern
+ --exclude-from stringArray Read file exclude patterns from file (use - to read from stdin)
+ --exclude-if-present stringArray Exclude directories if filename is present
+ --files-from stringArray Read list of source-file names from file (use - to read from stdin)
+ --files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin)
+ -f, --filter stringArray Add a file filtering rule
+ --filter-from stringArray Read file filtering patterns from a file (use - to read from stdin)
+ --ignore-case Ignore case in filters (case insensitive)
+ --include stringArray Include files matching pattern
+ --include-from stringArray Read file include patterns from file (use - to read from stdin)
+ --max-age Duration Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off)
+ --max-depth int If set limits the recursion depth to this (default -1)
+ --max-size SizeSuffix Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off)
+ --metadata-exclude stringArray Exclude metadatas matching pattern
+ --metadata-exclude-from stringArray Read metadata exclude patterns from file (use - to read from stdin)
+ --metadata-filter stringArray Add a metadata filtering rule
+ --metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin)
+ --metadata-include stringArray Include metadatas matching pattern
+ --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin)
+ --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off)
+ --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off)
+See the global flags page for global options not listed here.
+SEE ALSO
- rclone serve - Serve a remote over a protocol.
rclone serve webdav
Serve remote:path over WebDAV.
-Synopsis
+Synopsis
Run a basic WebDAV server to serve a remote over HTTP via the WebDAV protocol. This can be viewed with a WebDAV client, through a web browser, or you can make a remote of type WebDAV to read and write it.
WebDAV options
--etag-hash
@@ -4861,7 +5395,7 @@ Access WebDAV on Windows
Access Office applications on WebDAV
Navigate to following registry HKEY_CURRENT_USER[14.0/15.0/16.0] Create a new DWORD BasicAuthLevel with value 2. 0 - Basic authentication disabled 1 - Basic authentication enabled for SSL connections only 2 - Basic authentication enabled for SSL and for non-SSL connections
https://learn.microsoft.com/en-us/office/troubleshoot/powerpoint/office-opens-blank-from-sharepoint
-Server options
+Server options
Use --addr to specify which IP address and port the server should listen on, eg --addr 1.2.3.4:8000 or --addr :8080 to listen to all IPs. By default it only listens on localhost. You can use port :0 to let the OS choose an available port.
If you set --addr to listen on a public or LAN accessible IP address then using Authentication is advised - see the next section for info.
You can use a unix socket by setting the url to unix:///path/to/socket or just by using an absolute path name. Note that unix sockets bypass the authentication - this is expected to be done with file system permissions.
@@ -4869,7 +5403,7 @@ Server options
--server-read-timeout and --server-write-timeout can be used to control the timeouts on the server. Note that this is the total time for a transfer.
--max-header-bytes controls the maximum number of bytes the server will accept in the HTTP header.
--baseurl controls the URL prefix that rclone serves from. By default rclone will serve from the root. If you used --baseurl "/rclone" then rclone would serve from a URL starting with "/rclone/". This is useful if you wish to proxy rclone serve. Rclone automatically inserts leading and trailing "/" on --baseurl, so --baseurl "rclone", --baseurl "/rclone" and --baseurl "/rclone/" are all treated identically.
-TLS (SSL)
+TLS (SSL)
By default this will serve over http. If you want you can serve over https. You will need to supply the --cert and --key flags. If you wish to do client side certificate validation then you will need to supply --client-ca also.
--cert should be a either a PEM encoded certificate or a concatenation of that with the CA certificate. --key should be the PEM encoded private key and --client-ca should be the PEM encoded client certificate authority certificate.
--min-tls-version is minimum TLS version that is acceptable. Valid values are "tls1.0", "tls1.1", "tls1.2" and "tls1.3" (default "tls1.0").
@@ -4953,6 +5487,37 @@ Template
+The server also makes the following functions available so that they can be used within the template. These functions help extend the options for dynamic rendering of HTML. They can be used to render HTML based on specific conditions.
+
+
+
+
+
+Function
+Description
+
+
+
+
+afterEpoch
+Returns the time since the epoch for the given time.
+
+
+contains
+Checks whether a given substring is present or not in a given string.
+
+
+hasPrefix
+Checks whether the given string begins with the specified prefix.
+
+
+hasSuffix
+Checks whether the given string end with the specified suffix.
+
+
+
Authentication
By default this will serve files without needing a login.
You can either use an htpasswd file which can take lots of users, or set a single username and password with the --user and --pass flags.
@@ -4964,12 +5529,11 @@ Authentication
htpasswd -B htpasswd anotherUser
The password file can be updated while rclone is running.
Use --realm to set the authentication realm.
-Use --salt to change the password hashing salt from the default.
-VFS - Virtual File System
+Use --salt to change the password hashing salt from the default. ## VFS - Virtual File System
This command uses the VFS layer. This adapts the cloud storage objects that rclone uses into something which looks much more like a disk filing system.
Cloud storage objects have lots of properties which aren't like disk files - you can't extend them or write to the middle of them, so the VFS layer has to deal with that. Because there is no one right way of doing this there are various options explained below.
The VFS layer also implements a directory cache - this caches info about files and directories (but not the data) in memory.
-VFS Directory Cache
+VFS Directory Cache
Using the --dir-cache-time flag, you can control how long a directory should be considered up to date and not refreshed from the backend. Changes made through the VFS will appear immediately or invalidate the cache.
--dir-cache-time duration Time to cache directory entries for (default 5m0s)
--poll-interval duration Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable (default 1m0s)
@@ -4980,12 +5544,12 @@ VFS Directory Cache
rclone rc vfs/forget
Or individual files or directories:
rclone rc vfs/forget file=path/to/file dir=path/to/dir
-VFS File Buffering
+VFS File Buffering
The --buffer-size flag determines the amount of memory, that will be used to buffer data in advance.
Each open file will try to keep the specified amount of data in memory at all times. The buffered data is bound to one open file and won't be shared.
This flag is a upper limit for the used memory per open file. The buffer will only use memory for data that is downloaded but not not yet read. If the buffer is empty, only a small amount of memory will be used.
The maximum memory used by rclone for buffering can be up to --buffer-size * open files.
-VFS File Caching
+VFS File Caching
These flags control the VFS file caching options. File caching is necessary to make the VFS layer appear compatible with a normal file system. It can be disabled at the cost of some compatibility.
For example you'll need to enable VFS caching if you want to read and write simultaneously to a file. See below for more details.
Note that the VFS cache is separate from the cache backend and you may find that you need one or the other or both.
@@ -5002,7 +5566,7 @@ VFS File Caching
If using --vfs-cache-max-size or --vfs-cache-min-free-size note that the cache may exceed these quotas for two reasons. Firstly because it is only checked every --vfs-cache-poll-interval. Secondly because open files cannot be evicted from the cache. When --vfs-cache-max-size or --vfs-cache-min-free-size is exceeded, rclone will attempt to evict the least accessed files from the cache first. rclone will start with files that haven't been accessed for the longest. This cache flushing strategy is efficient and more relevant files are likely to remain cached.
The --vfs-cache-max-age will evict files from the cache after the set time since last access has passed. The default value of 1 hour will start evicting files from cache that haven't been accessed for 1 hour. When a cached file is accessed the 1 hour timer is reset to 0 and will wait for 1 more hour before evicting. Specify the time with standard notation, s, m, h, d, w .
You should not run two copies of rclone using the same VFS cache with the same or overlapping remotes if using --vfs-cache-mode > off. This can potentially cause data corruption if you do. You can work around this by giving each rclone its own cache hierarchy with --cache-dir. You don't need to worry about this if the remotes in use don't overlap.
---vfs-cache-mode off
+--vfs-cache-mode off
In this mode (the default) the cache will read directly from the remote and write directly to the remote without caching anything on disk.
This will mean some operations are not possible
@@ -5014,7 +5578,7 @@ --vfs-cache-mode off
- Open modes O_APPEND, O_TRUNC are ignored
- If an upload fails it can't be retried
---vfs-cache-mode minimal
+--vfs-cache-mode minimal
This is very similar to "off" except that files opened for read AND write will be buffered to disk. This means that files opened for write will be a lot more compatible, but uses the minimal disk space.
These operations are not possible
@@ -5023,11 +5587,11 @@ --vfs-cache-mode minimal
- Files opened for write only will ignore O_APPEND, O_TRUNC
- If an upload fails it can't be retried
---vfs-cache-mode writes
+--vfs-cache-mode writes
In this mode files opened for read only are still read directly from the remote, write only and read/write files are buffered to disk first.
This mode should support all normal file system operations.
If an upload fails it will be retried at exponentially increasing intervals up to 1 minute.
---vfs-cache-mode full
+--vfs-cache-mode full
In this mode all reads and writes are buffered to and from disk. When data is read from the remote this is buffered to disk as well.
In this mode the files in the cache will be sparse files and rclone will keep track of which bits of the files it has downloaded.
So if an application only reads the starts of each file, then rclone will only buffer the start of the file. These files will appear to be their full size in the cache, but they will be sparse files with only the data that has been downloaded present in them.
@@ -5035,7 +5599,7 @@ --vfs-cache-mode full
When reading a file rclone will read --buffer-size plus --vfs-read-ahead bytes ahead. The --buffer-size is buffered in memory whereas the --vfs-read-ahead is buffered on disk.
When using this mode it is recommended that --buffer-size is not set too large and --vfs-read-ahead is set large if required.
IMPORTANT not all file systems support sparse files. In particular FAT/exFAT do not. Rclone will perform very badly if the cache directory is on a filesystem which doesn't support sparse files and it will log an ERROR message if one is detected.
-Fingerprinting
+Fingerprinting
Various parts of the VFS use fingerprinting to see if a local file copy has changed relative to a remote file. Fingerprints are made from:
- size
@@ -5048,7 +5612,7 @@ Fingerprinting
If you use the --vfs-fast-fingerprint flag then rclone will not include the slow operations in the fingerprint. This makes the fingerprinting less accurate but much faster and will improve the opening time of cached files.
If you are running a vfs cache over local, s3 or swift backends then using this flag is recommended.
Note that if you change the value of this flag, the fingerprints of the files in the cache may be invalidated and the files will need to be downloaded again.
-VFS Chunked Reading
+VFS Chunked Reading
When rclone reads files from a remote it reads them in chunks. This means that rather than requesting the whole file rclone reads the chunk specified. This can reduce the used download quota for some remotes by requesting only chunks from the remote that are actually read, at the cost of an increased number of requests.
These flags control the chunking:
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M)
@@ -5056,7 +5620,7 @@ VFS Chunked Reading
Rclone will start reading a chunk of size --vfs-read-chunk-size, and then double the size for each read. When --vfs-read-chunk-size-limit is specified, and greater than --vfs-read-chunk-size, the chunk size for each open file will get doubled only until the specified value is reached. If the value is "off", which is the default, the limit is disabled and the chunk size will grow indefinitely.
With --vfs-read-chunk-size 100M and --vfs-read-chunk-size-limit 0 the following parts will be downloaded: 0-100M, 100M-200M, 200M-300M, 300M-400M and so on. When --vfs-read-chunk-size-limit 500M is specified, the result would be 0-100M, 100M-300M, 300M-700M, 700M-1200M, 1200M-1700M and so on.
Setting --vfs-read-chunk-size to 0 or "off" disables chunked reading.
-VFS Performance
+VFS Performance
These flags may be used to enable/disable features of the VFS for performance or other reasons. See also the chunked reading feature.
In particular S3 and Swift benefit hugely from the --no-modtime flag (or use --use-server-modtime for a slightly different effect) as each read of the modification time takes a transaction.
--no-checksum Don't compare checksums on up/download.
@@ -5068,7 +5632,7 @@ VFS Performance
--vfs-write-wait duration Time to wait for in-sequence write before giving error (default 1s)
When using VFS write caching (--vfs-cache-mode with value writes or full), the global flag --transfers can be set to adjust the number of parallel uploads of modified files from the cache (the related global flag --checkers has no effect on the VFS).
--transfers int Number of file transfers to run in parallel (default 4)
-VFS Case Sensitivity
+VFS Case Sensitivity
Linux file systems are case-sensitive: two files can differ only by case, and the exact case must be used when opening a file.
File systems in modern Windows are case-insensitive but case-preserving: although existing files can be opened using any case, the exact case used to create the file is preserved and available for programs to query. It is not allowed for two files in the same directory to differ only by case.
Usually file systems on macOS are case-insensitive. It is possible to make macOS file systems case-sensitive but that is not the default.
@@ -5076,10 +5640,10 @@ VFS Case Sensitivity
The user may specify a file name to open/delete/rename/etc with a case different than what is stored on the remote. If an argument refers to an existing file with exactly the same name, then the case of the existing file on the disk will be used. However, if a file name with exactly the same name is not found but a name differing only by case exists, rclone will transparently fixup the name. This fixup happens only when an existing file is requested. Case sensitivity of file names created anew by rclone is controlled by the underlying remote.
Note that case sensitivity of the operating system running rclone (the target) may differ from case sensitivity of a file system presented by rclone (the source). The flag controls whether "fixup" is performed to satisfy the target.
If the flag is not provided on the command line, then its default value depends on the operating system where rclone runs: "true" on Windows and macOS, "false" otherwise. If the flag is provided without a value, then it is "true".
-VFS Disk Options
+VFS Disk Options
This flag allows you to manually set the statistics about the filing system. It can be useful when those statistics cannot be read correctly automatically.
--vfs-disk-space-total-size Manually set the total disk space size (example: 256G, default: -1)
-Alternate report of used bytes
+Alternate report of used bytes
Some backends, most notably S3, do not report the amount of bytes used. If you need this information to be available when running df on the filesystem, then pass the flag --vfs-used-is-size to rclone. With this flag set, instead of relying on the backend to report this information, rclone will scan the whole remote similar to rclone size and compute the total used space itself.
WARNING. Contrary to rclone size, this flag ignores filters so that the result is accurate. However, this is very inefficient and may cost lots of API calls resulting in extra charges. Use it as a last resort and only with caching.
Auth Proxy
@@ -5113,7 +5677,7 @@ Auth Proxy
Note that an internal cache is keyed on user so only use that for configuration, don't use pass or public_key. This also means that if a user's password or public-key is changed the cache will need to expire (which takes 5 mins) before it takes effect.
This can be used to build general purpose proxies to any kind of backend that rclone supports.
rclone serve webdav remote:path [flags]
-Options
+Options
--addr stringArray IPaddress:Port or :Port to bind server to (default [127.0.0.1:8080])
--allow-origin string Origin which cross-domain request (CORS) can be executed from
--auth-proxy string A program to use to create the backend from the auth
@@ -5157,10 +5721,11 @@ Options
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi)
--vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off)
--vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms)
+ --vfs-refresh Refreshes the directory cache recursively on start
--vfs-used-is-size rclone size Use the rclone size algorithm for Used size
--vfs-write-back Duration Time to writeback files after last use when using cache (default 5s)
--vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s)
-Filter Options
+Filter Options
Flags for filtering directory listings.
--delete-excluded Delete files on dest excluded from sync
--exclude stringArray Exclude files matching pattern
@@ -5185,13 +5750,13 @@ Filter Options
--min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off)
--min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off)
See the global flags page for global options not listed here.
-SEE ALSO
+SEE ALSO
- rclone serve - Serve a remote over a protocol.
rclone settier
Changes storage class/tier of objects in remote.
-Synopsis
+Synopsis
rclone settier changes storage tier or class at remote if supported. Few cloud storage services provides different storage classes on objects, for example AWS S3 and Glacier, Azure Blob storage - Hot, Cool and Archive, Google Cloud Storage, Regional Storage, Nearline, Coldline etc.
Note that, certain tier changes make objects not available to access immediately. For example tiering to archive in azure blob storage makes objects in frozen state, user can restore by setting tier to Hot/Cool, similarly S3 to Glacier makes object inaccessible.true
You can use it to tier single object
@@ -5201,25 +5766,25 @@ Synopsis
Or just provide remote directory and all files in directory will be tiered
rclone settier tier remote:path/dir
rclone settier tier remote:path [flags]
-Options
+Options
-h, --help help for settier
See the global flags page for global options not listed here.
-SEE ALSO
+SEE ALSO
- rclone - Show help for rclone commands, flags and backends.
rclone test
Run a test command
-Synopsis
+Synopsis
Rclone test is used to run test commands.
Select which test command you want with the subcommand, eg
rclone test memory remote:
Each subcommand has its own options which you can see in their help.
NB Be careful running these commands, they may do strange things so reading their documentation first is recommended.
-Options
+Options
-h, --help help for test
See the global flags page for global options not listed here.
-SEE ALSO
+SEE ALSO
- rclone - Show help for rclone commands, flags and backends.
- rclone test changenotify - Log any change notify requests for the remote passed in.
@@ -5232,34 +5797,34 @@ SEE ALSO
rclone test changenotify
Log any change notify requests for the remote passed in.
rclone test changenotify remote: [flags]
-Options
+Options
-h, --help help for changenotify
--poll-interval Duration Time to wait between polling for changes (default 10s)
See the global flags page for global options not listed here.
-SEE ALSO
+SEE ALSO
- rclone test - Run a test command
rclone test histogram
Makes a histogram of file name characters.
-Synopsis
+Synopsis
This command outputs JSON which shows the histogram of characters used in filenames in the remote:path specified.
The data doesn't contain any identifying information but is useful for the rclone developers when developing filename compression.
rclone test histogram [remote:path] [flags]
-Options
+Options
-h, --help help for histogram
See the global flags page for global options not listed here.
-SEE ALSO
+SEE ALSO
- rclone test - Run a test command
rclone test info
Discovers file name or other limitations for paths.
-Synopsis
+Synopsis
rclone info discovers what filenames and upload methods are possible to write to the paths passed in and how long they can be. It can take some time. It will write test files into the remote:path passed in. It outputs a bit of go code for each one.
NB this can create undeletable files and other hazards - use with care
rclone test info [remote:path]+ [flags]
-Options
+Options
--all Run all tests
--check-base32768 Check can store all possible base32768 characters
--check-control Check control characters
@@ -5270,14 +5835,14 @@ Options
--upload-wait Duration Wait after writing a file (default 0s)
--write-json string Write results to file
See the global flags page for global options not listed here.
-SEE ALSO
+SEE ALSO
- rclone test - Run a test command
rclone test makefile
Make files with random contents of the size given
rclone test makefile <size> [<file>]+ [flags]
-Options
+Options
--ascii Fill files with random ASCII printable bytes only
--chargen Fill files with a ASCII chargen pattern
-h, --help help for makefile
@@ -5286,14 +5851,14 @@ Options
--sparse Make the files sparse (appear to be filled with ASCII 0x00)
--zero Fill files with ASCII 0x00
See the global flags page for global options not listed here.
-SEE ALSO
+SEE ALSO
- rclone test - Run a test command
rclone test makefiles
Make a random file hierarchy in a directory
rclone test makefiles <dir> [flags]
-Options
+Options
--ascii Fill files with random ASCII printable bytes only
--chargen Fill files with a ASCII chargen pattern
--files int Number of files to create (default 1000)
@@ -5309,23 +5874,23 @@ Options
--sparse Make the files sparse (appear to be filled with ASCII 0x00)
--zero Fill files with ASCII 0x00
See the global flags page for global options not listed here.
-SEE ALSO
+SEE ALSO
- rclone test - Run a test command
rclone test memory
Load all the objects at remote:path into memory and report memory stats.
rclone test memory remote:path [flags]
-Options
+Options
-h, --help help for memory
See the global flags page for global options not listed here.
-SEE ALSO
+SEE ALSO
- rclone test - Run a test command
rclone touch
Create new file or change file modification time.
-Synopsis
+Synopsis
Set the modification time on file(s) as specified by remote:path to have the current time.
If remote:path does not exist then a zero sized file will be created, unless --no-create or --recursive is provided.
If --recursive is used then recursively sets the modification time on all existing files that is found under the path. Filters are supported, and you can test with the --dry-run or the --interactive/-i flag.
@@ -5337,7 +5902,7 @@ Synopsis
Note that value of --timestamp is in UTC. If you want local time then add the --localtime flag.
rclone touch remote:path [flags]
-Options
+Options
-h, --help help for touch
--localtime Use localtime for timestamp, not UTC
-C, --no-create Do not create the file if it does not exist (implied with --recursive)
@@ -5348,7 +5913,7 @@ Important Options
-n, --dry-run Do a trial run with no permanent changes
-i, --interactive Enable interactive mode
-v, --verbose count Print lots more stuff (repeat for more)
-Filter Options
+Filter Options
Flags for filtering directory listings.
--delete-excluded Delete files on dest excluded from sync
--exclude stringArray Exclude files matching pattern
@@ -5377,13 +5942,13 @@ Listing Options
--default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z)
--fast-list Use recursive list if available; uses more memory but fewer transactions
See the global flags page for global options not listed here.
-SEE ALSO
+SEE ALSO
- rclone - Show help for rclone commands, flags and backends.
rclone tree
List the contents of the remote in a tree like fashion.
-Synopsis
+Synopsis
rclone tree lists the contents of a remote in a similar way to the unix tree command.
For example
$ rclone tree remote:path
@@ -5400,7 +5965,7 @@ Synopsis
The tree command has many options for controlling the listing which are compatible with the tree command, for example you can include file sizes with --size. Note that not all of them have short options as they conflict with rclone's short options.
For a more interactive navigation of the remote see the ncdu command.
rclone tree remote:path [flags]
-Options
+Options
-a, --all All files are listed (list . files too)
-d, --dirs-only List directories only
--dirsfirst List directories before files (-U disables)
@@ -5420,7 +5985,7 @@ Options
-r, --sort-reverse Reverse the order of the sort
-U, --unsorted Leave files unsorted
--version Sort files alphanumerically by version
-Filter Options
+Filter Options
Flags for filtering directory listings.
--delete-excluded Delete files on dest excluded from sync
--exclude stringArray Exclude files matching pattern
@@ -5449,7 +6014,7 @@ Listing Options
--default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z)
--fast-list Use recursive list if available; uses more memory but fewer transactions
See the global flags page for global options not listed here.
-SEE ALSO
+SEE ALSO
- rclone - Show help for rclone commands, flags and backends.
@@ -5570,6 +6135,7 @@ Metadata support
Rclone only supports a one-time sync of metadata. This means that metadata will be synced from the source object to the destination object only when the source object has changed and needs to be re-uploaded. If the metadata subsequently changes on the source object without changing the object itself then it won't be synced to the destination object. This is in line with the way rclone syncs Content-Type without the --metadata flag.
Using --metadata when syncing from local to local will preserve file attributes such as file mode, owner, extended attributes (not Windows).
Note that arbitrary metadata may be added to objects using the --metadata-set key=value flag when the object is first uploaded. This flag can be repeated as many times as necessary.
+The --metadata-mapper flag can be used to pass the name of a program in which can transform metadata when it is being copied from source to destination.
Types of metadata
Metadata is divided into two type. System metadata and User metadata.
Metadata which the backend uses itself is called system metadata. For example on the local backend the system metadata uid will store the user ID of the file when used on a unix based platform.
@@ -5647,26 +6213,31 @@ Standard system metadata
2006-01-02T15:04:05.999999999Z07:00
+utime
+Time of file upload: RFC 3339
+2006-01-02T15:04:05.999999999Z07:00
+
+
cache-control
Cache-Control header
no-cache
-
+
content-disposition
Content-Disposition header
inline
-
+
content-encoding
Content-Encoding header
gzip
-
+
content-language
Content-Language header
en-US
-
+
content-type
Content-Type header
text/plain
@@ -5675,7 +6246,7 @@ Standard system metadata
The metadata keys mtime and content-type will take precedence if supplied in the metadata over reading the Content-Type or modification time of the source object.
Hashes are not included in system metadata as there is a well defined way of reading those already.
-Options
+Options
Rclone has a number of options to control its behaviour.
Options that take parameters can have the values passed in two ways, --option=value or --option value. However boolean (true/false) options behave slightly differently to the other options in that --boolean sets the option to true and the absence of the flag sets it to false. It is also possible to specify --boolean=false or --boolean=true. Note that --boolean false is not valid - this is parsed as --boolean and the false is parsed as an extra command line argument for rclone.
Time or duration options
@@ -5925,7 +6496,7 @@ --inplace
ftp
sftp
-Without --inplace (the default) rclone will first upload to a temporary file with an extension like this where XXXXXX represents a random string.
+Without --inplace (the default) rclone will first upload to a temporary file with an extension like this, where XXXXXX represents a random string and .partial is --partial-suffix value (.partial by default).
original-file-name.XXXXXX.partial
(rclone will make sure the final name is no longer than 100 characters by truncating the original-file-name part if necessary).
When the upload is complete, rclone will rename the .partial file to the correct name, overwriting any existing file at that point. If the upload fails then the .partial file will be deleted.
@@ -6006,9 +6577,81 @@ --cutoff-mode=hard|soft|cautious
Specifying --cutoff-mode=soft will stop starting new transfers when Rclone reaches the limit.
Specifying --cutoff-mode=cautious will try to prevent Rclone from reaching the limit. Only applicable for --max-transfer
-M, --metadata
-Setting this flag enables rclone to copy the metadata from the source to the destination. For local backends this is ownership, permissions, xattr etc. See the #metadata for more info.
+Setting this flag enables rclone to copy the metadata from the source to the destination. For local backends this is ownership, permissions, xattr etc. See the metadata section for more info.
+--metadata-mapper SpaceSepList
+If you supply the parameter --metadata-mapper /path/to/program then rclone will use that program to map metadata from source object to destination object.
+The argument to this flag should be a command with an optional space separated list of arguments. If one of the arguments has a space in then enclose it in ", if you want a literal " in an argument then enclose the argument in " and double the ". See CSV encoding for more info.
+--metadata-mapper "python bin/test_metadata_mapper.py"
+--metadata-mapper 'python bin/test_metadata_mapper.py "argument with a space"'
+--metadata-mapper 'python bin/test_metadata_mapper.py "argument with ""two"" quotes"'
+This uses a simple JSON based protocol with input on STDIN and output on STDOUT. This will be called for every file and directory copied and may be called concurrently.
+The program's job is to take a metadata blob on the input and turn it into a metadata blob on the output suitable for the destination backend.
+Input to the program (via STDIN) might look like this. This provides some context for the Metadata which may be important.
+
+SrcFs is the config string for the remote that the object is currently on.SrcFsType is the name of the source backend.DstFs is the config string for the remote that the object is being copied toDstFsType is the name of the destination backend.Remote is the path of the file relative to the root.Size, MimeType, ModTime are attributes of the file.IsDir is true if this is a directory (not yet implemented).ID is the source ID of the file if known.Metadata is the backend specific metadata as described in the backend docs.
+{
+ "SrcFs": "gdrive:",
+ "SrcFsType": "drive",
+ "DstFs": "newdrive:user",
+ "DstFsType": "onedrive",
+ "Remote": "test.txt",
+ "Size": 6,
+ "MimeType": "text/plain; charset=utf-8",
+ "ModTime": "2022-10-11T17:53:10.286745272+01:00",
+ "IsDir": false,
+ "ID": "xyz",
+ "Metadata": {
+ "btime": "2022-10-11T16:53:11Z",
+ "content-type": "text/plain; charset=utf-8",
+ "mtime": "2022-10-11T17:53:10.286745272+01:00",
+ "owner": "user1@domain1.com",
+ "permissions": "...",
+ "description": "my nice file",
+ "starred": "false"
+ }
+}
+The program should then modify the input as desired and send it to STDOUT. The returned Metadata field will be used in its entirety for the destination object. Any other fields will be ignored. Note in this example we translate user names and permissions and add something to the description:
+{
+ "Metadata": {
+ "btime": "2022-10-11T16:53:11Z",
+ "content-type": "text/plain; charset=utf-8",
+ "mtime": "2022-10-11T17:53:10.286745272+01:00",
+ "owner": "user1@domain2.com",
+ "permissions": "...",
+ "description": "my nice file [migrated from domain1]",
+ "starred": "false"
+ }
+}
+Metadata can be removed here too.
+An example python program might look something like this to implement the above transformations.
+import sys, json
+
+i = json.load(sys.stdin)
+metadata = i["Metadata"]
+# Add tag to description
+if "description" in metadata:
+ metadata["description"] += " [migrated from domain1]"
+else:
+ metadata["description"] = "[migrated from domain1]"
+# Modify owner
+if "owner" in metadata:
+ metadata["owner"] = metadata["owner"].replace("domain1.com", "domain2.com")
+o = { "Metadata": metadata }
+json.dump(o, sys.stdout, indent="\t")
+You can find this example (slightly expanded) in the rclone source code at bin/test_metadata_mapper.py.
+If you want to see the input to the metadata mapper and the output returned from it in the log you can use -vv --dump mapper.
+See the metadata section for more info.
--metadata-set key=value
-Add metadata key = value when uploading. This can be repeated as many times as required. See the #metadata for more info.
+Add metadata key = value when uploading. This can be repeated as many times as required. See the metadata section for more info.
--modify-window=TIME
When checking whether a file has been modified, this is the maximum allowed time difference that a file can have and still be considered equivalent.
The default is 1ns unless this is overridden by a remote. For example OS X only stores modification times to the nearest second so if you are reading and writing to an OS X filing system this will be 1s by default.
@@ -6083,7 +6726,7 @@ --order-by string
--order-by name - send the files with alphabetically by path firstIf the --order-by flag is not supplied or it is supplied with an empty string then the default ordering will be used which is as scanned. With --checkers 1 this is mostly alphabetical, however with the default --checkers 8 it is somewhat random.
-Limitations
+Limitations
The --order-by flag does not do a separate pass over the data. This means that it may transfer some files out of the order specified if
- there are no files in the backlog or the source has not been fully scanned yet
@@ -6091,13 +6734,17 @@ Limitations
Rclone will do its best to transfer the best file it has so in practice this should not cause a problem. Think of --order-by as being more of a best efforts flag rather than a perfect ordering.
If you want perfect ordering then you will need to specify --check-first which will find all the files which need transferring first before transferring any.
+--partial-suffix
+When --inplace is not used, it causes rclone to use the --partial-suffix as suffix for temporary files.
+Suffix length limit is 16 characters.
+The default is .partial.
--password-command SpaceSepList
This flag supplies a program which should supply the config password when run. This is an alternative to rclone prompting for the password or setting the RCLONE_CONFIG_PASS variable.
The argument to this should be a command with a space separated list of arguments. If one of the arguments has a space in then enclose it in ", if you want a literal " in an argument then enclose the argument in " and double the ". See CSV encoding for more info.
Eg
---password-command echo hello
---password-command echo "hello with space"
---password-command echo "hello with ""quotes"" and space"
+--password-command "echo hello"
+--password-command 'echo "hello with space"'
+--password-command 'echo "hello with ""quotes"" and space"'
See the Configuration Encryption for more info.
See a Windows PowerShell example on the Wiki.
-P, --progress
@@ -6220,18 +6867,12 @@ --delete-(before,during,after)
Specifying --delete-during will delete files while checking and uploading files. This is the fastest option and uses the least memory.
Specifying --delete-after (the default value) will delay deletion of files until all new/updated files have been successfully transferred. The files to be deleted are collected in the copy pass then deleted after the copy pass has completed successfully. The files to be deleted are held in memory so this mode may use more memory. This is the safest mode as it will only delete files if there have been no errors subsequent to that. If there have been errors before the deletions start then you will get the message not deleting files as there were IO errors.
--fast-list
-When doing anything which involves a directory listing (e.g. sync, copy, ls - in fact nearly every command), rclone normally lists a directory and processes it before using more directory lists to process any subdirectories. This can be parallelised and works very quickly using the least amount of memory.
-However, some remotes have a way of listing all files beneath a directory in one (or a small number) of transactions. These tend to be the bucket-based remotes (e.g. S3, B2, GCS, Swift).
-If you use the --fast-list flag then rclone will use this method for listing directories. This will have the following consequences for the listing:
-
-- It will use fewer transactions (important if you pay for them)
-- It will use more memory. Rclone has to load the whole listing into memory.
-- It may be faster because it uses fewer transactions
-- It may be slower because it can't be parallelized
-
-rclone should always give identical results with and without --fast-list.
-If you pay for transactions and can fit your entire sync listing into memory then --fast-list is recommended. If you have a very big sync to do then don't use --fast-list otherwise you will run out of memory.
-If you use --fast-list on a remote which doesn't support it, then rclone will just ignore it.
+When doing anything which involves a directory listing (e.g. sync, copy, ls - in fact nearly every command), rclone has different strategies to choose from.
+The basic strategy is to list one directory and processes it before using more directory lists to process any subdirectories. This is a mandatory backend feature, called List, which means it is supported by all backends. This strategy uses small amount of memory, and because it can be parallelised it is fast for operations involving processing of the list results.
+Some backends provide the support for an alternative strategy, where all files beneath a directory can be listed in one (or a small number) of transactions. Rclone supports this alternative strategy through an optional backend feature called ListR. You can see in the storage system overview documentation's optional features section which backends it is enabled for (these tend to be the bucket-based ones, e.g. S3, B2, GCS, Swift). This strategy requires fewer transactions for highly recursive operations, which is important on backends where this is charged or heavily rate limited. It may be faster (due to fewer transactions) or slower (because it can't be parallelized) depending on different parameters, and may require more memory if rclone has to keep the whole listing in memory.
+Which listing strategy rclone picks for a given operation is complicated, but in general it tries to choose the best possible. It will prefer ListR in situations where it doesn't need to store the listed files in memory, e.g. for unlimited recursive ls command variants. In other situations it will prefer List, e.g. for sync and copy, where it needs to keep the listed files in memory, and is performing operations on them where parallelization may be a huge advantage.
+Rclone is not able to take all relevant parameters into account for deciding the best strategy, and therefore allows you to influence the choice in two ways: You can stop rclone from using ListR by disabling the feature, using the --disable option (--disable ListR), or you can allow rclone to use ListR where it would normally choose not to do so due to higher memory usage, using the --fast-list option. Rclone should always produce identical results either way. Using --disable ListR or --fast-list on a remote which doesn't support ListR does nothing, rclone will just ignore it.
+A rule of thumb is that if you pay for transactions and can fit your entire sync listing into memory, then --fast-list is recommended. If you have a very big sync to do, then don't use --fast-list, otherwise you will run out of memory. Run some tests and compare before you decide, and if in doubt then just leave the default, let rclone decide, i.e. not use --fast-list.
--timeout=TIME
This sets the IO idle timeout. If a transfer has started but then becomes idle for this long it is considered broken and disconnected.
The default is 5m. Set to 0 to disable.
@@ -6348,6 +6989,8 @@ --dump goroutines
This dumps a list of the running go-routines at the end of the command to standard output.
--dump openfiles
This dumps a list of the open files at the end of the command. It uses the lsof command to do that so you'll need that installed to use it.
+--dump mapper
+This shows the JSON blobs being sent to the program supplied with --metadata-mapper and received from it. It can be useful for debugging the metadata mapper interface.
--memprofile=FILE
Write memory profile to file. This can be analysed with go tool pprof.
Filtering
@@ -6414,7 +7057,7 @@ List of exit codes
Environment Variables
Rclone can be configured entirely using environment variables. These can be used to set defaults for options or config file entries.
-Options
+Options
Every option in rclone can have its default set by environment variable.
To find the name of the environment variable, first, take the long option name, strip the leading --, change - to _, make upper case and prepend RCLONE_.
For example, to always set --stats 5s, set the environment variable RCLONE_STATS=5s. If you set stats on the command line this will override the environment variable setting.
@@ -7777,6 +8420,40 @@ operations/about: Return the space used on the remote<
The result is as returned from rclone about --json
See the about command for more information on the above.
Authentication is required for this call.
+operations/check: check the source and destination are the same
+Checks the files in the source and destination match. It compares sizes and hashes and logs a report of files that don't match. It doesn't alter the source or destination.
+This takes the following parameters:
+
+- srcFs - a remote name string e.g. "drive:" for the source, "/" for local filesystem
+- dstFs - a remote name string e.g. "drive2:" for the destination, "/" for local filesystem
+- download - check by downloading rather than with hash
+- checkFileHash - treat checkFileFs:checkFileRemote as a SUM file with hashes of given type
+- checkFileFs - treat checkFileFs:checkFileRemote as a SUM file with hashes of given type
+- checkFileRemote - treat checkFileFs:checkFileRemote as a SUM file with hashes of given type
+- oneWay - check one way only, source files must exist on remote
+- combined - make a combined report of changes (default false)
+- missingOnSrc - report all files missing from the source (default true)
+- missingOnDst - report all files missing from the destination (default true)
+- match - report all matching files (default false)
+- differ - report all non-matching files (default true)
+- error - report all files with errors (hashing or reading) (default true)
+
+If you supply the download flag, it will download the data from both remotes and check them against each other on the fly. This can be useful for remotes that don't support hashes or if you really want to check all the data.
+If you supply the size-only global flag, it will only compare the sizes not the hashes as well. Use this for a quick check.
+If you supply the checkFileHash option with a valid hash name, the checkFileFs:checkFileRemote must point to a text file in the SUM format. This treats the checksum file as the source and dstFs as the destination. Note that srcFs is not used and should not be supplied in this case.
+Returns:
+
+- success - true if no error, false otherwise
+- status - textual summary of check, OK or text string
+- hashType - hash used in check, may be missing
+- combined - array of strings of combined report of changes
+- missingOnSrc - array of strings of all files missing from the source
+- missingOnDst - array of strings of all files missing from the destination
+- match - array of strings of all matching files
+- differ - array of strings of all non-matching files
+- error - array of strings of all files with errors (hashing or reading)
+
+Authentication is required for this call.
operations/cleanup: Remove trashed files in the remote or path
This takes the following parameters:
@@ -8505,7 +9182,7 @@ Features
Google Drive
-MD5
+MD5, SHA1, SHA256
R/W
No
Yes
@@ -8564,7 +9241,7 @@ Features
Yes
No
R
--
+RW
Koofr
@@ -8576,6 +9253,15 @@ Features
-
+Linkbox
+-
+R
+No
+No
+-
+-
+
+
Mail.ru Cloud
Mailru â¶
R/W
@@ -8584,7 +9270,7 @@ Features
-
-
-
+
Mega
-
-
@@ -8593,7 +9279,7 @@ Features
-
-
-
+
Memory
MD5
R/W
@@ -8602,7 +9288,7 @@ Features
-
-
-
+
Microsoft Azure Blob Storage
MD5
R/W
@@ -8611,6 +9297,15 @@ Features
R/W
-
+
+Microsoft Azure Files Storage
+MD5
+R/W
+Yes
+No
+R/W
+-
+
Microsoft OneDrive
QuickXorHash âµ
@@ -8740,7 +9435,7 @@ Features
SMB
-
--
+R/W
Yes
No
-
@@ -8811,7 +9506,6 @@ Features
-Notes
¹ Dropbox supports its own custom hash. This is an SHA256 sum of all the 4 MiB block SHA256s.
² SFTP supports checksums if the same login has shell access and md5sum or sha1sum as well as echo are in the remote's PATH.
³ WebDAV supports hashes when used with Fastmail Files, Owncloud and Nextcloud only.
@@ -8821,7 +9515,7 @@ Notes
â· pCloud only supports SHA1 (not MD5) in its EU region
⸠Opendrive does not support creation of duplicate files using their web client interface or other stock clients, but the underlying storage platform has been determined to allow duplicate files, and it is possible to create them with rclone. It may be that this is a mistake or an unsupported feature.
â¹ QingStor does not support SetModTime for objects bigger than 5 GiB.
-¹ⰠFTP supports modtimes for the major FTP servers, and also others if they advertised required protocol extensions. See this for more details.
+¹ⰠFTP supports modtimes for the major FTP servers, and also others if they advertised required protocol extensions. See this for more details.
¹¹ Internet Archive requires option wait_archive to be set to a non-zero value for full modtime support.
¹² HiDrive supports its own custom hash. It combines SHA1 sums for each 4 KiB block hierarchically to a single top-level sum.
Hash
@@ -9385,7 +10079,7 @@ Optional Features
Yes
Yes
Yes
-Yes ‡‡
+Yes
No
Yes
No
@@ -9632,20 +10326,34 @@ Optional Features
No
+Microsoft Azure Files Storage
+No
+Yes
+Yes
+Yes
+No
+No
+Yes
+Yes
+No
+Yes
+Yes
+
+
Microsoft OneDrive
Yes
Yes
Yes
Yes
Yes
-No
+Yes âµ
No
No
Yes
Yes
Yes
-
+
OpenDrive
Yes
Yes
@@ -9659,9 +10367,9 @@ Optional Features
No
Yes
-
+
OpenStack Swift
-Yes â€
+Yes ¹
Yes
No
No
@@ -9673,7 +10381,7 @@ Optional Features
Yes
No
-
+
Oracle Object Storage
No
Yes
@@ -9682,12 +10390,12 @@ Optional Features
Yes
Yes
Yes
-No
+Yes
No
No
No
-
+
pCloud
Yes
Yes
@@ -9701,7 +10409,7 @@ Optional Features
Yes
Yes
-
+
PikPak
Yes
Yes
@@ -9715,7 +10423,7 @@ Optional Features
Yes
Yes
-
+
premiumize.me
Yes
No
@@ -9729,7 +10437,7 @@ Optional Features
Yes
Yes
-
+
put.io
Yes
No
@@ -9743,7 +10451,7 @@ Optional Features
Yes
Yes
-
+
Proton Drive
Yes
No
@@ -9757,7 +10465,7 @@ Optional Features
Yes
Yes
-
+
QingStor
No
Yes
@@ -9771,7 +10479,7 @@ Optional Features
No
No
-
+
Quatrix by Maytech
Yes
Yes
@@ -9785,7 +10493,7 @@ Optional Features
Yes
Yes
-
+
Seafile
Yes
Yes
@@ -9799,10 +10507,10 @@ Optional Features
Yes
Yes
-
+
SFTP
No
-No
+Yes â´
Yes
Yes
No
@@ -9813,7 +10521,7 @@ Optional Features
Yes
Yes
-
+
Sia
No
No
@@ -9827,7 +10535,7 @@ Optional Features
No
Yes
-
+
SMB
No
No
@@ -9841,7 +10549,7 @@ Optional Features
No
Yes
-
+
SugarSync
Yes
Yes
@@ -9855,9 +10563,9 @@ Optional Features
No
Yes
-
+
Storj
-Yes ☨
+Yes ²
Yes
Yes
No
@@ -9869,7 +10577,7 @@ Optional Features
No
No
-
+
Uptobox
No
Yes
@@ -9883,7 +10591,7 @@ Optional Features
No
No
-
+
WebDAV
Yes
Yes
@@ -9891,13 +10599,13 @@ Optional Features
Yes
No
No
-Yes ‡
+Yes ³
No
No
Yes
Yes
-
+
Yandex Disk
Yes
Yes
@@ -9911,7 +10619,7 @@ Optional Features
Yes
Yes
-
+
Zoho WorkDrive
Yes
Yes
@@ -9925,7 +10633,7 @@ Optional Features
Yes
Yes
-
+
The local filesystem
Yes
No
@@ -9941,11 +10649,13 @@ Optional Features
+¹ Note Swift implements this in order to delete directory markers but it doesn't actually have a quicker way of deleting files other than deleting them individually.
+² Storj implements this efficiently only for entire buckets. If purging a directory inside a bucket, files are deleted individually.
+³ StreamUpload is not supported with Nextcloud
+â´ Use the --sftp-copy-is-hardlink flag to enable.
+âµ Use the --onedrive-delta flag to enable.
Purge
This deletes a directory quicker than just deleting all the files in the directory.
-†Note Swift implements this in order to delete directory markers but they don't actually have a quicker way of deleting files other than deleting them individually.
-☨ Storj implements this efficiently only for entire buckets. If purging a directory inside a bucket, files are deleted individually.
-‡ StreamUpload is not supported with Nextcloud
Copy
Used when copying an object to and from the same remote. This known as a server-side copy so you can copy a file without downloading it and uploading it again. It is used if you use rclone copy or rclone move if the remote doesn't support Move directly.
If the server doesn't support Copy directly then for copy operations the file is downloaded then re-uploaded.
@@ -9981,11 +10691,11 @@ Copy
-c, --checksum Check for changes with size & checksum (if available, or fallback to size only).
--compare-dest stringArray Include additional comma separated server-side paths during comparison
--copy-dest stringArray Implies --compare-dest but also copies files from paths into destination
- --cutoff-mode string Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default "HARD")
+ --cutoff-mode HARD|SOFT|CAUTIOUS Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default HARD)
--ignore-case-sync Ignore case when synchronizing
--ignore-checksum Skip post copy check of checksums
--ignore-existing Skip all files that exist on destination
- --ignore-size Ignore size when skipping use mod-time or checksum
+ --ignore-size Ignore size when skipping use modtime or checksum
-I, --ignore-times Don't skip files that match size and time - transfer all files
--immutable Do not modify files, fail if existing files have been modified
--inplace Download directly to destination file instead of atomic download to temp/rename
@@ -10000,11 +10710,12 @@ Copy
--multi-thread-write-buffer-size SizeSuffix In memory buffer size for writing when in multi-thread mode (default 128Ki)
--no-check-dest Don't check the destination, copy regardless
--no-traverse Don't traverse destination file system on copy
- --no-update-modtime Don't update destination mod-time if files identical
+ --no-update-modtime Don't update destination modtime if files identical
--order-by string Instructions on how to order the transfers, e.g. 'size,descending'
+ --partial-suffix string Add partial-suffix to temporary file name when --inplace is not used (default ".partial")
--refresh-times Refresh the modtime of remote files
--server-side-across-configs Allow server-side operations (e.g. copy) to work across different configs
- --size-only Skip based on size only, not mod-time or checksum
+ --size-only Skip based on size only, not modtime or checksum
--streaming-upload-cutoff SizeSuffix Cutoff for switching to chunked upload if file size is unknown, upload starts after reaching cutoff or when file ends (default 100Ki)
-u, --update Skip files that are newer on the destination
Sync
@@ -10050,7 +10761,7 @@ Networking
--tpslimit float Limit HTTP transactions per second to this
--tpslimit-burst int Max burst of transactions for --tpslimit (default 1)
--use-cookies Enable session cookiejar
- --user-agent string Set the user-agent to a specified string (default "rclone/v1.64.0")
+ --user-agent string Set the user-agent to a specified string (default "rclone/v1.65.0")
Performance
Flags helpful for increasing performance.
--buffer-size SizeSuffix In memory buffer size when reading files for each --transfer (default 16Mi)
@@ -10061,7 +10772,7 @@ Config
--ask-password Allow prompt for password for encrypted configuration (default true)
--auto-confirm If enabled, do not request console confirmation
--cache-dir string Directory rclone will use for caching (default "$HOME/.cache/rclone")
- --color string When to show colors (and other ANSI codes) AUTO|NEVER|ALWAYS (default "AUTO")
+ --color AUTO|NEVER|ALWAYS When to show colors (and other ANSI codes) AUTO|NEVER|ALWAYS (default AUTO)
--config string Config file (default "$HOME/.config/rclone/rclone.conf")
--default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z)
--disable string Disable a comma separated list of features (use --disable help to see a list)
@@ -10084,7 +10795,7 @@ Config
Debugging
Flags for developers.
--cpuprofile string Write cpu profile to file
- --dump DumpFlags List of items to dump from: headers,bodies,requests,responses,auth,filters,goroutines,openfiles
+ --dump DumpFlags List of items to dump from: headers, bodies, requests, responses, auth, filters, goroutines, openfiles, mapper
--dump-bodies Dump HTTP headers and bodies - may contain sensitive info
--dump-headers Dump HTTP headers - may contain sensitive info
--memprofile string Write memory profile to file
@@ -10120,7 +10831,7 @@ Logging
Logging and statistics.
--log-file string Log everything to this file
--log-format string Comma separated list of log format options (default "date,time")
- --log-level string Log level DEBUG|INFO|NOTICE|ERROR (default "NOTICE")
+ --log-level LogLevel Log level DEBUG|INFO|NOTICE|ERROR (default NOTICE)
--log-systemd Activate systemd integration for the logger
--max-stats-groups int Maximum number of stats groups to keep in memory, on max oldest is discarded (default 1000)
-P, --progress Show progress during transfer
@@ -10128,7 +10839,7 @@ Logging
-q, --quiet Print as little stuff as possible
--stats Duration Interval between printing stats, e.g. 500ms, 60s, 5m (0 to disable) (default 1m0s)
--stats-file-name-length int Max file name length in stats (0 for no limit) (default 45)
- --stats-log-level string Log level to show --stats output DEBUG|INFO|NOTICE|ERROR (default "INFO")
+ --stats-log-level LogLevel Log level to show --stats output DEBUG|INFO|NOTICE|ERROR (default INFO)
--stats-one-line Make the stats fit on one line
--stats-one-line-date Enable --stats-one-line and add current date/time prefix
--stats-one-line-date-format string Enable --stats-one-line-date and use custom formatted date: Enclose date string in double quotes ("), see https://golang.org/pkg/time/#Time.Format
@@ -10146,6 +10857,7 @@ Metadata
--metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin)
--metadata-include stringArray Include metadatas matching pattern
--metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin)
+ --metadata-mapper SpaceSepList Program to run to transforming metadata before upload
--metadata-set stringArray Add metadata key=value when uploading
RC
Flags to control the Remote Control API.
@@ -10182,13 +10894,13 @@ Backend
--acd-auth-url string Auth server URL
--acd-client-id string OAuth Client Id
--acd-client-secret string OAuth Client Secret
- --acd-encoding MultiEncoder The encoding for the backend (default Slash,InvalidUtf8,Dot)
+ --acd-encoding Encoding The encoding for the backend (default Slash,InvalidUtf8,Dot)
--acd-templink-threshold SizeSuffix Files >= this size will be downloaded via their tempLink (default 9Gi)
--acd-token string OAuth Access Token as a JSON blob
--acd-token-url string Token server url
--acd-upload-wait-per-gb Duration Additional time per GiB to wait after a failed complete upload to see if it appears (default 3m0s)
--alias-remote string Remote or path to alias
- --azureblob-access-tier string Access tier of blob: hot, cool or archive
+ --azureblob-access-tier string Access tier of blob: hot, cool, cold or archive
--azureblob-account string Azure Storage Account Name
--azureblob-archive-tier-delete Delete archive tier blobs before overwriting
--azureblob-chunk-size SizeSuffix Upload chunk size (default 4Mi)
@@ -10199,7 +10911,7 @@ Backend
--azureblob-client-send-certificate-chain Send the certificate chain when using certificate auth
--azureblob-directory-markers Upload an empty object with a trailing slash when a new directory is created
--azureblob-disable-checksum Don't store MD5 checksum with object metadata
- --azureblob-encoding MultiEncoder The encoding for the backend (default Slash,BackSlash,Del,Ctl,RightPeriod,InvalidUtf8)
+ --azureblob-encoding Encoding The encoding for the backend (default Slash,BackSlash,Del,Ctl,RightPeriod,InvalidUtf8)
--azureblob-endpoint string Endpoint for the service
--azureblob-env-auth Read credentials from runtime (environment variables, CLI or MSI)
--azureblob-key string Storage Account Shared Key
@@ -10219,18 +10931,43 @@ Backend
--azureblob-use-emulator Uses local storage emulator if provided as 'true'
--azureblob-use-msi Use a managed service identity to authenticate (only works in Azure)
--azureblob-username string User name (usually an email address)
+ --azurefiles-account string Azure Storage Account Name
+ --azurefiles-chunk-size SizeSuffix Upload chunk size (default 4Mi)
+ --azurefiles-client-certificate-password string Password for the certificate file (optional) (obscured)
+ --azurefiles-client-certificate-path string Path to a PEM or PKCS12 certificate file including the private key
+ --azurefiles-client-id string The ID of the client in use
+ --azurefiles-client-secret string One of the service principal's client secrets
+ --azurefiles-client-send-certificate-chain Send the certificate chain when using certificate auth
+ --azurefiles-connection-string string Azure Files Connection String
+ --azurefiles-encoding Encoding The encoding for the backend (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Del,Ctl,RightPeriod,InvalidUtf8,Dot)
+ --azurefiles-endpoint string Endpoint for the service
+ --azurefiles-env-auth Read credentials from runtime (environment variables, CLI or MSI)
+ --azurefiles-key string Storage Account Shared Key
+ --azurefiles-max-stream-size SizeSuffix Max size for streamed files (default 10Gi)
+ --azurefiles-msi-client-id string Object ID of the user-assigned MSI to use, if any
+ --azurefiles-msi-mi-res-id string Azure resource ID of the user-assigned MSI to use, if any
+ --azurefiles-msi-object-id string Object ID of the user-assigned MSI to use, if any
+ --azurefiles-password string The user's password (obscured)
+ --azurefiles-sas-url string SAS URL
+ --azurefiles-service-principal-file string Path to file containing credentials for use with a service principal
+ --azurefiles-share-name string Azure Files Share Name
+ --azurefiles-tenant string ID of the service principal's tenant. Also called its directory ID
+ --azurefiles-upload-concurrency int Concurrency for multipart uploads (default 16)
+ --azurefiles-use-msi Use a managed service identity to authenticate (only works in Azure)
+ --azurefiles-username string User name (usually an email address)
--b2-account string Account ID or Application Key ID
--b2-chunk-size SizeSuffix Upload chunk size (default 96Mi)
--b2-copy-cutoff SizeSuffix Cutoff for switching to multipart copy (default 4Gi)
--b2-disable-checksum Disable checksums for large (> upload cutoff) files
--b2-download-auth-duration Duration Time before the authorization token will expire in s or suffix ms|s|m|h|d (default 1w)
--b2-download-url string Custom endpoint for downloads
- --b2-encoding MultiEncoder The encoding for the backend (default Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot)
+ --b2-encoding Encoding The encoding for the backend (default Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot)
--b2-endpoint string Endpoint for the service
--b2-hard-delete Permanently delete files on remote removal, otherwise hide files
--b2-key string Application Key
+ --b2-lifecycle int Set the number of days deleted files should be kept when creating a bucket
--b2-test-mode string A flag string for X-Bz-Test-Mode header for debugging
- --b2-upload-concurrency int Concurrency for multipart uploads (default 16)
+ --b2-upload-concurrency int Concurrency for multipart uploads (default 4)
--b2-upload-cutoff SizeSuffix Cutoff for switching to chunked upload (default 200Mi)
--b2-version-at Time Show file versions as they were at the specified time (default off)
--b2-versions Include old versions in directory listings
@@ -10241,7 +10978,7 @@ Backend
--box-client-id string OAuth Client Id
--box-client-secret string OAuth Client Secret
--box-commit-retries int Max number of times to try committing a multipart file (default 100)
- --box-encoding MultiEncoder The encoding for the backend (default Slash,BackSlash,Del,Ctl,RightSpace,InvalidUtf8,Dot)
+ --box-encoding Encoding The encoding for the backend (default Slash,BackSlash,Del,Ctl,RightSpace,InvalidUtf8,Dot)
--box-impersonate string Impersonate this user ID when using a service account
--box-list-chunk int Size of listing chunk 1-1000 (default 1000)
--box-owned-by string Only show items owned by the login (email address) passed in
@@ -10299,7 +11036,7 @@ Backend
--drive-client-secret string OAuth Client Secret
--drive-copy-shortcut-content Server side copy contents of shortcuts instead of the shortcut
--drive-disable-http2 Disable drive using http2 (default true)
- --drive-encoding MultiEncoder The encoding for the backend (default InvalidUtf8)
+ --drive-encoding Encoding The encoding for the backend (default InvalidUtf8)
--drive-env-auth Get IAM credentials from runtime (environment variables or instance meta data if no env vars)
--drive-export-formats string Comma separated list of preferred formats for downloading Google docs (default "docx,xlsx,pptx,svg")
--drive-fast-list-bug-fix Work around a bug in Google Drive listing (default true)
@@ -10308,17 +11045,21 @@ Backend
--drive-import-formats string Comma separated list of preferred formats for uploading Google docs
--drive-keep-revision-forever Keep new head revision of each file forever
--drive-list-chunk int Size of listing chunk 100-1000, 0 to disable (default 1000)
+ --drive-metadata-labels Bits Control whether labels should be read or written in metadata (default off)
+ --drive-metadata-owner Bits Control whether owner should be read or written in metadata (default read)
+ --drive-metadata-permissions Bits Control whether permissions should be read or written in metadata (default off)
--drive-pacer-burst int Number of API calls to allow without sleeping (default 100)
--drive-pacer-min-sleep Duration Minimum time to sleep between API calls (default 100ms)
--drive-resource-key string Resource key for accessing a link-shared file
--drive-root-folder-id string ID of the root folder
- --drive-scope string Scope that rclone should use when requesting access from drive
+ --drive-scope string Comma separated list of scopes that rclone should use when requesting access from drive
--drive-server-side-across-configs Deprecated: use --server-side-across-configs instead
--drive-service-account-credentials string Service Account Credentials JSON blob
--drive-service-account-file string Service Account Credentials JSON file path
--drive-shared-with-me Only show files that are shared with me
+ --drive-show-all-gdocs Show all Google Docs including non-exportable ones in listings
--drive-size-as-quota Show sizes as storage quota usage, not actual size
- --drive-skip-checksum-gphotos Skip MD5 checksum on Google photos and videos only
+ --drive-skip-checksum-gphotos Skip checksums on Google photos and videos only
--drive-skip-dangling-shortcuts If set skip dangling shortcut files
--drive-skip-gdocs Skip google documents in all listings
--drive-skip-shortcuts If set skip shortcut files
@@ -10342,7 +11083,7 @@ Backend
--dropbox-chunk-size SizeSuffix Upload chunk size (< 150Mi) (default 48Mi)
--dropbox-client-id string OAuth Client Id
--dropbox-client-secret string OAuth Client Secret
- --dropbox-encoding MultiEncoder The encoding for the backend (default Slash,BackSlash,Del,RightSpace,InvalidUtf8,Dot)
+ --dropbox-encoding Encoding The encoding for the backend (default Slash,BackSlash,Del,RightSpace,InvalidUtf8,Dot)
--dropbox-impersonate string Impersonate this user when using a business account
--dropbox-pacer-min-sleep Duration Minimum time to sleep between API calls (default 10ms)
--dropbox-shared-files Instructs rclone to work on individual shared files
@@ -10351,11 +11092,11 @@ Backend
--dropbox-token-url string Token server url
--fichier-api-key string Your API Key, get it from https://1fichier.com/console/params.pl
--fichier-cdn Set if you wish to use CDN download links
- --fichier-encoding MultiEncoder The encoding for the backend (default Slash,LtGt,DoubleQuote,SingleQuote,BackQuote,Dollar,BackSlash,Del,Ctl,LeftSpace,RightSpace,InvalidUtf8,Dot)
+ --fichier-encoding Encoding The encoding for the backend (default Slash,LtGt,DoubleQuote,SingleQuote,BackQuote,Dollar,BackSlash,Del,Ctl,LeftSpace,RightSpace,InvalidUtf8,Dot)
--fichier-file-password string If you want to download a shared file that is password protected, add this parameter (obscured)
--fichier-folder-password string If you want to list the files in a shared folder that is password protected, add this parameter (obscured)
--fichier-shared-folder string If you want to download a shared folder, add this parameter
- --filefabric-encoding MultiEncoder The encoding for the backend (default Slash,Del,Ctl,InvalidUtf8,Dot)
+ --filefabric-encoding Encoding The encoding for the backend (default Slash,Del,Ctl,InvalidUtf8,Dot)
--filefabric-permanent-token string Permanent Authentication Token
--filefabric-root-folder-id string ID of the root folder
--filefabric-token string Session Token
@@ -10369,7 +11110,7 @@ Backend
--ftp-disable-mlsd Disable using MLSD even if server advertises support
--ftp-disable-tls13 Disable TLS 1.3 (workaround for FTP servers with buggy TLS)
--ftp-disable-utf8 Disable using UTF-8 even if server advertises support
- --ftp-encoding MultiEncoder The encoding for the backend (default Slash,Del,Ctl,RightSpace,Dot)
+ --ftp-encoding Encoding The encoding for the backend (default Slash,Del,Ctl,RightSpace,Dot)
--ftp-explicit-tls Use Explicit FTPS (FTP over TLS)
--ftp-force-list-hidden Use LIST -a to force listing of hidden files and folders. This will disable the use of MLSD
--ftp-host string FTP host to connect to
@@ -10391,7 +11132,7 @@ Backend
--gcs-client-secret string OAuth Client Secret
--gcs-decompress If set this will decompress gzip encoded objects
--gcs-directory-markers Upload an empty object with a trailing slash when a new directory is created
- --gcs-encoding MultiEncoder The encoding for the backend (default Slash,CrLf,InvalidUtf8,Dot)
+ --gcs-encoding Encoding The encoding for the backend (default Slash,CrLf,InvalidUtf8,Dot)
--gcs-endpoint string Endpoint for the service
--gcs-env-auth Get GCP IAM credentials from runtime (environment variables or instance meta data if no env vars)
--gcs-location string Location for the newly created buckets
@@ -10404,9 +11145,13 @@ Backend
--gcs-token-url string Token server url
--gcs-user-project string User project
--gphotos-auth-url string Auth server URL
+ --gphotos-batch-commit-timeout Duration Max time to wait for a batch to finish committing (default 10m0s)
+ --gphotos-batch-mode string Upload file batching sync|async|off (default "sync")
+ --gphotos-batch-size int Max number of files in upload batch
+ --gphotos-batch-timeout Duration Max time to allow an idle upload batch before uploading (default 0s)
--gphotos-client-id string OAuth Client Id
--gphotos-client-secret string OAuth Client Secret
- --gphotos-encoding MultiEncoder The encoding for the backend (default Slash,CrLf,InvalidUtf8,Dot)
+ --gphotos-encoding Encoding The encoding for the backend (default Slash,CrLf,InvalidUtf8,Dot)
--gphotos-include-archived Also view and download archived media
--gphotos-read-only Set to make the Google Photos backend read only
--gphotos-read-size Set to read the size of media items
@@ -10418,8 +11163,8 @@ Backend
--hasher-max-age Duration Maximum time to keep checksums in cache (0 = no cache, off = cache forever) (default off)
--hasher-remote string Remote to cache checksums for (e.g. myRemote:path)
--hdfs-data-transfer-protection string Kerberos data transfer protection: authentication|integrity|privacy
- --hdfs-encoding MultiEncoder The encoding for the backend (default Slash,Colon,Del,Ctl,InvalidUtf8,Dot)
- --hdfs-namenode string Hadoop name node and port
+ --hdfs-encoding Encoding The encoding for the backend (default Slash,Colon,Del,Ctl,InvalidUtf8,Dot)
+ --hdfs-namenode CommaSepList Hadoop name nodes and ports
--hdfs-service-principal-name string Kerberos service principal name for the namenode
--hdfs-username string Hadoop user name
--hidrive-auth-url string Auth server URL
@@ -10427,7 +11172,7 @@ Backend
--hidrive-client-id string OAuth Client Id
--hidrive-client-secret string OAuth Client Secret
--hidrive-disable-fetching-member-count Do not fetch number of objects in directories unless it is absolutely necessary
- --hidrive-encoding MultiEncoder The encoding for the backend (default Slash,Dot)
+ --hidrive-encoding Encoding The encoding for the backend (default Slash,Dot)
--hidrive-endpoint string Endpoint for the service (default "https://api.hidrive.strato.com/2.1")
--hidrive-root-prefix string The root/parent folder for all paths (default "/")
--hidrive-scope-access string Access permissions that rclone should use when requesting access from HiDrive (default "rw")
@@ -10440,9 +11185,16 @@ Backend
--http-no-head Don't use HEAD requests
--http-no-slash Set this if the site doesn't end directories with /
--http-url string URL of HTTP host to connect to
+ --imagekit-encoding Encoding The encoding for the backend (default Slash,LtGt,DoubleQuote,Dollar,Question,Hash,Percent,BackSlash,Del,Ctl,InvalidUtf8,Dot,SquareBracket)
+ --imagekit-endpoint string You can find your ImageKit.io URL endpoint in your [dashboard](https://imagekit.io/dashboard/developer/api-keys)
+ --imagekit-only-signed Restrict unsigned image URLs If you have configured Restrict unsigned image URLs in your dashboard settings, set this to true
+ --imagekit-private-key string You can find your ImageKit.io private key in your [dashboard](https://imagekit.io/dashboard/developer/api-keys)
+ --imagekit-public-key string You can find your ImageKit.io public key in your [dashboard](https://imagekit.io/dashboard/developer/api-keys)
+ --imagekit-upload-tags string Tags to add to the uploaded files, e.g. "tag1,tag2"
+ --imagekit-versions Include old versions in directory listings
--internetarchive-access-key-id string IAS3 Access Key
--internetarchive-disable-checksum Don't ask the server to test against MD5 checksum calculated by rclone (default true)
- --internetarchive-encoding MultiEncoder The encoding for the backend (default Slash,LtGt,CrLf,Del,Ctl,InvalidUtf8,Dot)
+ --internetarchive-encoding Encoding The encoding for the backend (default Slash,LtGt,CrLf,Del,Ctl,InvalidUtf8,Dot)
--internetarchive-endpoint string IAS3 Endpoint (default "https://s3.us.archive.org")
--internetarchive-front-endpoint string Host of InternetArchive Frontend (default "https://archive.org")
--internetarchive-secret-access-key string IAS3 Secret Key (password)
@@ -10450,7 +11202,7 @@ Backend
--jottacloud-auth-url string Auth server URL
--jottacloud-client-id string OAuth Client Id
--jottacloud-client-secret string OAuth Client Secret
- --jottacloud-encoding MultiEncoder The encoding for the backend (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,Del,Ctl,InvalidUtf8,Dot)
+ --jottacloud-encoding Encoding The encoding for the backend (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,Del,Ctl,InvalidUtf8,Dot)
--jottacloud-hard-delete Delete files permanently rather than putting them into the trash
--jottacloud-md5-memory-limit SizeSuffix Files bigger than this will be cached on disk to calculate the MD5 if required (default 10Mi)
--jottacloud-no-versions Avoid server side versioning by deleting files and recreating files instead of overwriting them
@@ -10458,17 +11210,18 @@ Backend
--jottacloud-token-url string Token server url
--jottacloud-trashed-only Only show files that are in the trash
--jottacloud-upload-resume-limit SizeSuffix Files bigger than this can be resumed if the upload fail's (default 10Mi)
- --koofr-encoding MultiEncoder The encoding for the backend (default Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot)
+ --koofr-encoding Encoding The encoding for the backend (default Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot)
--koofr-endpoint string The Koofr API endpoint to use
--koofr-mountid string Mount ID of the mount to use
--koofr-password string Your password for rclone (generate one at https://app.koofr.net/app/admin/preferences/password) (obscured)
--koofr-provider string Choose your storage provider
--koofr-setmtime Does the backend support setting modification time (default true)
--koofr-user string Your user name
+ --linkbox-token string Token from https://www.linkbox.to/admin/account
-l, --links Translate symlinks to/from regular files with a '.rclonelink' extension
--local-case-insensitive Force the filesystem to report itself as case insensitive
--local-case-sensitive Force the filesystem to report itself as case sensitive
- --local-encoding MultiEncoder The encoding for the backend (default Slash,Dot)
+ --local-encoding Encoding The encoding for the backend (default Slash,Dot)
--local-no-check-updated Don't check to see if the files change during upload
--local-no-preallocate Disable preallocation of disk space for transferred files
--local-no-set-modtime Disable setting modtime
@@ -10480,7 +11233,7 @@ Backend
--mailru-check-hash What should copy do if file checksum is mismatched or invalid (default true)
--mailru-client-id string OAuth Client Id
--mailru-client-secret string OAuth Client Secret
- --mailru-encoding MultiEncoder The encoding for the backend (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Del,Ctl,InvalidUtf8,Dot)
+ --mailru-encoding Encoding The encoding for the backend (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Del,Ctl,InvalidUtf8,Dot)
--mailru-pass string Password (obscured)
--mailru-speedup-enable Skip full upload if there is another file with same data hash (default true)
--mailru-speedup-file-patterns string Comma separated list of file name patterns eligible for speedup (put by hash) (default "*.mkv,*.avi,*.mp4,*.mp3,*.zip,*.gz,*.rar,*.pdf")
@@ -10490,7 +11243,7 @@ Backend
--mailru-token-url string Token server url
--mailru-user string User name (usually email)
--mega-debug Output more debug from Mega
- --mega-encoding MultiEncoder The encoding for the backend (default Slash,InvalidUtf8,Dot)
+ --mega-encoding Encoding The encoding for the backend (default Slash,InvalidUtf8,Dot)
--mega-hard-delete Delete files permanently rather than putting them into the trash
--mega-pass string Password (obscured)
--mega-use-https Use HTTPS for transfers
@@ -10506,9 +11259,10 @@ Backend
--onedrive-chunk-size SizeSuffix Chunk size to upload files with - must be multiple of 320k (327,680 bytes) (default 10Mi)
--onedrive-client-id string OAuth Client Id
--onedrive-client-secret string OAuth Client Secret
+ --onedrive-delta If set rclone will use delta listing to implement recursive listings
--onedrive-drive-id string The ID of the drive to use
--onedrive-drive-type string The type of the drive (personal | business | documentLibrary)
- --onedrive-encoding MultiEncoder The encoding for the backend (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Del,Ctl,LeftSpace,LeftTilde,RightSpace,RightPeriod,InvalidUtf8,Dot)
+ --onedrive-encoding Encoding The encoding for the backend (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Del,Ctl,LeftSpace,LeftTilde,RightSpace,RightPeriod,InvalidUtf8,Dot)
--onedrive-expose-onenote-files Set to make OneNote files show up in directory listings
--onedrive-hash-type string Specify the hash in use for the backend (default "auto")
--onedrive-link-password string Set the password for links created by the link command
@@ -10529,7 +11283,7 @@ Backend
--oos-copy-cutoff SizeSuffix Cutoff for switching to multipart copy (default 4.656Gi)
--oos-copy-timeout Duration Timeout for copy (default 1m0s)
--oos-disable-checksum Don't store MD5 checksum with object metadata
- --oos-encoding MultiEncoder The encoding for the backend (default Slash,InvalidUtf8,Dot)
+ --oos-encoding Encoding The encoding for the backend (default Slash,InvalidUtf8,Dot)
--oos-endpoint string Endpoint for Object storage API
--oos-leave-parts-on-error If true avoid calling abort upload on a failure, leaving all successfully uploaded parts for manual recovery
--oos-max-upload-parts int Maximum number of parts in a multipart upload (default 10000)
@@ -10546,13 +11300,13 @@ Backend
--oos-upload-concurrency int Concurrency for multipart uploads (default 10)
--oos-upload-cutoff SizeSuffix Cutoff for switching to chunked upload (default 200Mi)
--opendrive-chunk-size SizeSuffix Files will be uploaded in chunks this size (default 10Mi)
- --opendrive-encoding MultiEncoder The encoding for the backend (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,LeftSpace,LeftCrLfHtVt,RightSpace,RightCrLfHtVt,InvalidUtf8,Dot)
+ --opendrive-encoding Encoding The encoding for the backend (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,LeftSpace,LeftCrLfHtVt,RightSpace,RightCrLfHtVt,InvalidUtf8,Dot)
--opendrive-password string Password (obscured)
--opendrive-username string Username
--pcloud-auth-url string Auth server URL
--pcloud-client-id string OAuth Client Id
--pcloud-client-secret string OAuth Client Secret
- --pcloud-encoding MultiEncoder The encoding for the backend (default Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot)
+ --pcloud-encoding Encoding The encoding for the backend (default Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot)
--pcloud-hostname string Hostname to connect to (default "api.pcloud.com")
--pcloud-password string Your pcloud password (obscured)
--pcloud-root-folder-id string Fill in for rclone to use a non root folder as its starting point (default "d0")
@@ -10562,7 +11316,7 @@ Backend
--pikpak-auth-url string Auth server URL
--pikpak-client-id string OAuth Client Id
--pikpak-client-secret string OAuth Client Secret
- --pikpak-encoding MultiEncoder The encoding for the backend (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,LeftSpace,RightSpace,RightPeriod,InvalidUtf8,Dot)
+ --pikpak-encoding Encoding The encoding for the backend (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,LeftSpace,RightSpace,RightPeriod,InvalidUtf8,Dot)
--pikpak-hash-memory-limit SizeSuffix Files bigger than this will be cached on disk to calculate hash if required (default 10Mi)
--pikpak-pass string Pikpak password (obscured)
--pikpak-root-folder-id string ID of the root folder
@@ -10574,13 +11328,13 @@ Backend
--premiumizeme-auth-url string Auth server URL
--premiumizeme-client-id string OAuth Client Id
--premiumizeme-client-secret string OAuth Client Secret
- --premiumizeme-encoding MultiEncoder The encoding for the backend (default Slash,DoubleQuote,BackSlash,Del,Ctl,InvalidUtf8,Dot)
+ --premiumizeme-encoding Encoding The encoding for the backend (default Slash,DoubleQuote,BackSlash,Del,Ctl,InvalidUtf8,Dot)
--premiumizeme-token string OAuth Access Token as a JSON blob
--premiumizeme-token-url string Token server url
--protondrive-2fa string The 2FA code
--protondrive-app-version string The app version string (default "macos-drive@1.0.0-alpha.1+rclone")
--protondrive-enable-caching Caches the files and folders metadata to reduce API calls (default true)
- --protondrive-encoding MultiEncoder The encoding for the backend (default Slash,LeftSpace,RightSpace,InvalidUtf8,Dot)
+ --protondrive-encoding Encoding The encoding for the backend (default Slash,LeftSpace,RightSpace,InvalidUtf8,Dot)
--protondrive-mailbox-password string The mailbox password of your two-password proton account (obscured)
--protondrive-original-file-size Return the file size before encryption (default true)
--protondrive-password string The password of your proton account (obscured)
@@ -10589,13 +11343,13 @@ Backend
--putio-auth-url string Auth server URL
--putio-client-id string OAuth Client Id
--putio-client-secret string OAuth Client Secret
- --putio-encoding MultiEncoder The encoding for the backend (default Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot)
+ --putio-encoding Encoding The encoding for the backend (default Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot)
--putio-token string OAuth Access Token as a JSON blob
--putio-token-url string Token server url
--qingstor-access-key-id string QingStor Access Key ID
--qingstor-chunk-size SizeSuffix Chunk size to use for uploading (default 4Mi)
--qingstor-connection-retries int Number of connection retries (default 3)
- --qingstor-encoding MultiEncoder The encoding for the backend (default Slash,Ctl,InvalidUtf8)
+ --qingstor-encoding Encoding The encoding for the backend (default Slash,Ctl,InvalidUtf8)
--qingstor-endpoint string Enter an endpoint URL to connection QingStor API
--qingstor-env-auth Get QingStor credentials from runtime
--qingstor-secret-access-key string QingStor Secret Access Key (password)
@@ -10604,7 +11358,7 @@ Backend
--qingstor-zone string Zone to connect to
--quatrix-api-key string API key for accessing Quatrix account
--quatrix-effective-upload-time string Wanted upload time for one chunk (default "4s")
- --quatrix-encoding MultiEncoder The encoding for the backend (default Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot)
+ --quatrix-encoding Encoding The encoding for the backend (default Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot)
--quatrix-hard-delete Delete files permanently rather than putting them into the trash
--quatrix-host string Host name of Quatrix account
--quatrix-maximal-summary-chunk-size SizeSuffix The maximal summary for all chunks. It should not be less than 'transfers'*'minimal_chunk_size' (default 95.367Mi)
@@ -10619,7 +11373,7 @@ Backend
--s3-disable-checksum Don't store MD5 checksum with object metadata
--s3-disable-http2 Disable usage of http2 for S3 backends
--s3-download-url string Custom endpoint for downloads
- --s3-encoding MultiEncoder The encoding for the backend (default Slash,InvalidUtf8,Dot)
+ --s3-encoding Encoding The encoding for the backend (default Slash,InvalidUtf8,Dot)
--s3-endpoint string Endpoint for S3 API
--s3-env-auth Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars)
--s3-force-path-style If true use path style access if false use virtual hosted style (default true)
@@ -10653,14 +11407,16 @@ Backend
--s3-upload-cutoff SizeSuffix Cutoff for switching to chunked upload (default 200Mi)
--s3-use-accelerate-endpoint If true use the AWS S3 accelerated endpoint
--s3-use-accept-encoding-gzip Accept-Encoding: gzip Whether to send Accept-Encoding: gzip header (default unset)
+ --s3-use-already-exists Tristate Set if rclone should report BucketAlreadyExists errors on bucket creation (default unset)
--s3-use-multipart-etag Tristate Whether to use ETag in multipart uploads for verification (default unset)
+ --s3-use-multipart-uploads Tristate Set if rclone should use multipart uploads (default unset)
--s3-use-presigned-request Whether to use a presigned request or PutObject for single part uploads
--s3-v2-auth If true use v2 authentication
--s3-version-at Time Show file versions as they were at the specified time (default off)
--s3-versions Include old versions in directory listings
--seafile-2fa Two-factor authentication ('true' if the account has 2FA enabled)
--seafile-create-library Should rclone create a library if it doesn't exist
- --seafile-encoding MultiEncoder The encoding for the backend (default Slash,DoubleQuote,BackSlash,Ctl,InvalidUtf8)
+ --seafile-encoding Encoding The encoding for the backend (default Slash,DoubleQuote,BackSlash,Ctl,InvalidUtf8)
--seafile-library string Name of the library
--seafile-library-key string Library password (for encrypted libraries only) (obscured)
--seafile-pass string Password (obscured)
@@ -10670,6 +11426,7 @@ Backend
--sftp-chunk-size SizeSuffix Upload and download chunk size (default 32Ki)
--sftp-ciphers SpaceSepList Space separated list of ciphers to be used for session encryption, ordered by preference
--sftp-concurrency int The maximum number of outstanding requests for one file (default 64)
+ --sftp-copy-is-hardlink Set to enable server side copies using hardlinks
--sftp-disable-concurrent-reads If set don't use concurrent reads
--sftp-disable-concurrent-writes If set don't use concurrent writes
--sftp-disable-hashcheck Disable the execution of SSH commands to determine if remote file hashing is available
@@ -10704,7 +11461,7 @@ Backend
--sharefile-chunk-size SizeSuffix Upload chunk size (default 64Mi)
--sharefile-client-id string OAuth Client Id
--sharefile-client-secret string OAuth Client Secret
- --sharefile-encoding MultiEncoder The encoding for the backend (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,LeftSpace,LeftPeriod,RightSpace,RightPeriod,InvalidUtf8,Dot)
+ --sharefile-encoding Encoding The encoding for the backend (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,LeftSpace,LeftPeriod,RightSpace,RightPeriod,InvalidUtf8,Dot)
--sharefile-endpoint string Endpoint for API calls
--sharefile-root-folder-id string ID of the root folder
--sharefile-token string OAuth Access Token as a JSON blob
@@ -10712,12 +11469,12 @@ Backend
--sharefile-upload-cutoff SizeSuffix Cutoff for switching to multipart upload (default 128Mi)
--sia-api-password string Sia Daemon API Password (obscured)
--sia-api-url string Sia daemon API URL, like http://sia.daemon.host:9980 (default "http://127.0.0.1:9980")
- --sia-encoding MultiEncoder The encoding for the backend (default Slash,Question,Hash,Percent,Del,Ctl,InvalidUtf8,Dot)
+ --sia-encoding Encoding The encoding for the backend (default Slash,Question,Hash,Percent,Del,Ctl,InvalidUtf8,Dot)
--sia-user-agent string Siad User Agent (default "Sia-Agent")
--skip-links Don't warn about skipped symlinks
--smb-case-insensitive Whether the server is configured to be case-insensitive (default true)
--smb-domain string Domain name for NTLM authentication (default "WORKGROUP")
- --smb-encoding MultiEncoder The encoding for the backend (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot)
+ --smb-encoding Encoding The encoding for the backend (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot)
--smb-hide-special-share Hide special shares (e.g. print$) which users aren't supposed to access (default true)
--smb-host string SMB server hostname to connect to
--smb-idle-timeout Duration Max time before closing idle connections (default 1m0s)
@@ -10735,7 +11492,7 @@ Backend
--sugarsync-authorization string Sugarsync authorization
--sugarsync-authorization-expiry string Sugarsync authorization expiry
--sugarsync-deleted-id string Sugarsync deleted folder id
- --sugarsync-encoding MultiEncoder The encoding for the backend (default Slash,Ctl,InvalidUtf8,Dot)
+ --sugarsync-encoding Encoding The encoding for the backend (default Slash,Ctl,InvalidUtf8,Dot)
--sugarsync-hard-delete Permanently delete files if true
--sugarsync-private-access-key string Sugarsync Private Access Key
--sugarsync-refresh-token string Sugarsync refresh token
@@ -10749,7 +11506,7 @@ Backend
--swift-auth-version int AuthVersion - optional - set to (1,2,3) if your auth URL has no version (ST_AUTH_VERSION)
--swift-chunk-size SizeSuffix Above this size files will be chunked into a _segments container (default 5Gi)
--swift-domain string User domain - optional (v3 auth) (OS_USER_DOMAIN_NAME)
- --swift-encoding MultiEncoder The encoding for the backend (default Slash,InvalidUtf8)
+ --swift-encoding Encoding The encoding for the backend (default Slash,InvalidUtf8)
--swift-endpoint-type string Endpoint type to choose from the service catalogue (OS_ENDPOINT_TYPE) (default "public")
--swift-env-auth Get swift credentials from environment variables in standard OpenStack form
--swift-key string API key or password (OS_PASSWORD)
@@ -10771,7 +11528,7 @@ Backend
--union-search-policy string Policy to choose upstream on SEARCH category (default "ff")
--union-upstreams string List of space separated upstreams
--uptobox-access-token string Your access token
- --uptobox-encoding MultiEncoder The encoding for the backend (default Slash,LtGt,DoubleQuote,BackQuote,Del,Ctl,LeftSpace,InvalidUtf8,Dot)
+ --uptobox-encoding Encoding The encoding for the backend (default Slash,LtGt,DoubleQuote,BackQuote,Del,Ctl,LeftSpace,InvalidUtf8,Dot)
--uptobox-private Set to make uploaded files private
--webdav-bearer-token string Bearer token instead of user/pass (e.g. a Macaroon)
--webdav-bearer-token-command string Command to run to get a bearer token
@@ -10786,14 +11543,14 @@ Backend
--yandex-auth-url string Auth server URL
--yandex-client-id string OAuth Client Id
--yandex-client-secret string OAuth Client Secret
- --yandex-encoding MultiEncoder The encoding for the backend (default Slash,Del,Ctl,InvalidUtf8,Dot)
+ --yandex-encoding Encoding The encoding for the backend (default Slash,Del,Ctl,InvalidUtf8,Dot)
--yandex-hard-delete Delete files permanently rather than putting them into the trash
--yandex-token string OAuth Access Token as a JSON blob
--yandex-token-url string Token server url
--zoho-auth-url string Auth server URL
--zoho-client-id string OAuth Client Id
--zoho-client-secret string OAuth Client Secret
- --zoho-encoding MultiEncoder The encoding for the backend (default Del,Ctl,InvalidUtf8)
+ --zoho-encoding Encoding The encoding for the backend (default Del,Ctl,InvalidUtf8)
--zoho-region string Zoho region to connect to
--zoho-token string OAuth Access Token as a JSON blob
--zoho-token-url string Token server url
@@ -11113,7 +11870,7 @@ --ignore-listing-checksum
- While checksums are (by default) generated and stored in the listing files, they are NOT currently used for determining diffs (deltas). It is anticipated that full checksum support will be added in a future version.
--ignore-listing-checksum is NOT the same as --ignore-checksum, and you may wish to use one or the other, or both. In a nutshell: --ignore-listing-checksum controls whether checksums are considered when scanning for diffs, while --ignore-checksum controls whether checksums are considered during the copy/sync operations that follow, if there ARE diffs.- Unless
--ignore-listing-checksum is passed, bisync currently computes hashes for one path even when there's no common hash with the other path (for example, a crypt remote.)
+- Unless
--ignore-listing-checksum is passed, bisync currently computes hashes for one path even when there's no common hash with the other path (for example, a crypt remote.)
- If both paths support checksums and have a common hash, AND
--ignore-listing-checksum was not specified when creating the listings, --check-sync=only can be used to compare Path1 vs. Path2 checksums (as of the time the previous listings were created.) However, --check-sync=only will NOT include checksums if the previous listings were generated on a run using --ignore-listing-checksum. For a more robust integrity check of the current state, consider using check (or cryptcheck, if at least one path is a crypt remote.)
--resilient
@@ -11261,7 +12018,7 @@ Unusual sync checks
As of rclone v1.64, bisync is now better at detecting false positive sync conflicts, which would previously have resulted in unnecessary renames and duplicates. Now, when bisync comes to a file that it wants to rename (because it is new/changed on both sides), it first checks whether the Path1 and Path2 versions are currently identical (using the same underlying function as check.) If bisync concludes that the files are identical, it will skip them and move on. Otherwise, it will create renamed ..Path1 and ..Path2 duplicates, as before. This behavior also improves the experience of renaming directories, as a --resync is no longer required, so long as the same change has been made on both sides.
All files changed check
If all prior existing files on either of the filesystems have changed (e.g. timestamps have changed due to changing the system's timezone) then bisync will abort without making any changes. Any new files are not considered for this check. You could use --force to force the sync (whichever side has the changed timestamp files wins). Alternately, a --resync may be used (Path1 versions will be pushed to Path2). Consider the situation carefully and perhaps use --dry-run before you commit to the changes.
-Modification time
+Modification times
Bisync relies on file timestamps to identify changed files and will refuse to operate if backend lacks the modification time support.
If you or your application should change the content of a file without changing the modification time then bisync will not notice the change, and thus will not copy it to the other side.
Note that on some cloud storage systems it is not possible to have file timestamps that match precisely between the local and other filesystems.
@@ -11277,7 +12034,7 @@ Lock file
Note that while concurrent bisync runs are allowed, be very cautious that there is no overlap in the trees being synched between concurrent runs, lest there be replicated files, deleted files and general mayhem.
Return codes
rclone bisync returns the following codes to calling program: - 0 on a successful run, - 1 for a non-critical failing run (a rerun may be successful), - 2 for a critically aborted run (requires a --resync to recover).
-Limitations
+Limitations
Supported backends
Bisync is considered BETA and has been tested with the following backends: - Local filesystem - Google Drive - Dropbox - OneDrive - S3 - SFTP - Yandex Disk
It has not been fully tested with other services yet. If it works, or sorta works, please let us know and we'll update the list. Run the test suite to check for proper operation as described below.
@@ -11835,7 +12592,7 @@ Configuration
rclone ls remote:
To copy a local directory to a 1Fichier directory called backup
rclone copy /home/source remote:backup
-Modified time and hashes
+Modification times and hashes
1Fichier does not support modification times. It supports the Whirlpool hash algorithm.
Duplicated files
1Fichier can have two files with exactly the same name and path (unlike a normal file system).
@@ -11964,10 +12721,10 @@ --fichier-encoding
- Config: encoding
- Env Var: RCLONE_FICHIER_ENCODING
-- Type: MultiEncoder
+- Type: Encoding
- Default: Slash,LtGt,DoubleQuote,SingleQuote,BackQuote,Dollar,BackSlash,Del,Ctl,LeftSpace,RightSpace,InvalidUtf8,Dot
-Limitations
+Limitations
rclone about is not supported by the 1Fichier backend. Backends without this capability cannot determine free space for an rclone mount or use policy mfs (most free space) as a member of an rclone union remote.
See List of backends that do not support rclone about and rclone about
Alias
@@ -12110,9 +12867,9 @@ Configuration
rclone ls remote:
To copy a local directory to an Amazon Drive directory called backup
rclone copy /home/source remote:backup
-Modified time and MD5SUMs
+Modification times and hashes
Amazon Drive doesn't allow modification times to be changed via the API so these won't be accurate or used for syncing.
-It does store MD5SUMs so for a more accurate sync, you can use the --checksum flag.
+It does support the MD5 hash algorithm, so for a more accurate sync, you can use the --checksum flag.
Restricted filename characters
@@ -12234,10 +12991,10 @@ --acd-encoding
- Config: encoding
- Env Var: RCLONE_ACD_ENCODING
-- Type: MultiEncoder
+- Type: Encoding
- Default: Slash,InvalidUtf8,Dot
-Limitations
+Limitations
Note that Amazon Drive is case insensitive so you can't have a file called "Hello.doc" and one called "hello.doc".
Amazon Drive has rate limiting so you may notice errors in the sync (429 errors). rclone will automatically retry the sync up to 3 times by default (see --retries flag) which should hopefully work around this problem.
Amazon Drive has an internal limit of file sizes that can be uploaded to the service. This limit is not officially published, but all files larger than this will fail.
@@ -12263,10 +13020,12 @@ Amazon S3 Storage Providers
IONOS Cloud
Leviia Object Storage
Liara Object Storage
+Linode Object Storage
Minio
Petabox
Qiniu Cloud Object Storage (Kodo)
RackCorp Object Storage
+Rclone Serve S3
Scaleway
Seagate Lyve Cloud
SeaweedFS
@@ -12485,10 +13244,18 @@ Configuration
e) Edit this remote
d) Delete this remote
y/e/d>
-Modified time
+Modification times and hashes
+Modification times
The modified time is stored as metadata on the object as X-Amz-Meta-Mtime as floating point since the epoch, accurate to 1 ns.
If the modification time needs to be updated rclone will attempt to perform a server side copy to update the modification if the object can be copied in a single part. In the case the object is larger than 5Gb or is in Glacier or Glacier Deep Archive storage the object will be uploaded rather than copied.
Note that reading this from the object takes an additional HEAD request as the metadata isn't returned in object listings.
+Hashes
+For small objects which weren't uploaded as multipart uploads (objects sized below --s3-upload-cutoff if uploaded with rclone) rclone uses the ETag: header as an MD5 checksum.
+However for objects which were uploaded as multipart uploads or with server side encryption (SSE-AWS or SSE-C) the ETag header is no longer the MD5 sum of the data, so rclone adds an additional piece of metadata X-Amz-Meta-Md5chksum which is a base64 encoded MD5 hash (in the same format as is required for Content-MD5). You can use base64 -d and hexdump to check this value manually:
+echo 'VWTGdNx3LyXQDfA0e2Edxw==' | base64 -d | hexdump
+or you can use rclone check to verify the hashes are OK.
+For large objects, calculating this hash can take some time so the addition of this hash can be disabled with --s3-disable-checksum. This will mean that these objects do not have an MD5 checksum.
+Note that reading this from the object takes an additional HEAD request as the metadata isn't returned in object listings.
Reducing costs
Avoiding HEAD requests to read the modification time
By default, rclone will use the modification time of objects stored in S3 for syncing. This is stored in object metadata which unfortunately takes an extra HEAD request to read which can be expensive (in time and money).
@@ -12535,13 +13302,6 @@ Avoiding HEAD requests after PUT
By default, rclone will HEAD every object it uploads. It does this to check the object got uploaded correctly.
You can disable this with the --s3-no-head option - see there for more details.
Setting this flag increases the chance for undetected upload failures.
-Hashes
-For small objects which weren't uploaded as multipart uploads (objects sized below --s3-upload-cutoff if uploaded with rclone) rclone uses the ETag: header as an MD5 checksum.
-However for objects which were uploaded as multipart uploads or with server side encryption (SSE-AWS or SSE-C) the ETag header is no longer the MD5 sum of the data, so rclone adds an additional piece of metadata X-Amz-Meta-Md5chksum which is a base64 encoded MD5 hash (in the same format as is required for Content-MD5). You can use base64 -d and hexdump to check this value manually:
-echo 'VWTGdNx3LyXQDfA0e2Edxw==' | base64 -d | hexdump
-or you can use rclone check to verify the hashes are OK.
-For large objects, calculating this hash can take some time so the addition of this hash can be disabled with --s3-disable-checksum. This will mean that these objects do not have an MD5 checksum.
-Note that reading this from the object takes an additional HEAD request as the metadata isn't returned in object listings.
Versions
When bucket versioning is enabled (this can be done with rclone with the rclone backend versioning command) when rclone uploads a new version of a file it creates a new version of it Likewise when you delete a file, the old version will be marked hidden and still be available.
Old versions of files, where available, are visible using the --s3-versions flag.
@@ -12722,9 +13482,9 @@ Object-lock enabled S3 bucket
If you configure a default retention period on a bucket, requests to upload objects in such a bucket must include the Content-MD5 header.
-As mentioned in the Hashes section, small files that are not uploaded as multipart, use a different tag, causing the upload to fail. A simple solution is to set the --s3-upload-cutoff 0 and force all the files to be uploaded as multipart.
+As mentioned in the Modification times and hashes section, small files that are not uploaded as multipart, use a different tag, causing the upload to fail. A simple solution is to set the --s3-upload-cutoff 0 and force all the files to be uploaded as multipart.
Standard options
-Here are the Standard options specific to s3 (Amazon S3 Compliant Storage Providers including AWS, Alibaba, ArvanCloud, Ceph, China Mobile, Cloudflare, GCS, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, IDrive e2, IONOS Cloud, Leviia, Liara, Lyve Cloud, Minio, Netease, Petabox, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Synology, Tencent COS, Qiniu and Wasabi).
+Here are the Standard options specific to s3 (Amazon S3 Compliant Storage Providers including AWS, Alibaba, ArvanCloud, Ceph, ChinaMobile, Cloudflare, DigitalOcean, Dreamhost, GCS, HuaweiOBS, IBMCOS, IDrive, IONOS, LyveCloud, Leviia, Liara, Linode, Minio, Netease, Petabox, RackCorp, Rclone, Scaleway, SeaweedFS, StackPath, Storj, Synology, TencentCOS, Wasabi, Qiniu and others).
--s3-provider
Choose your S3 provider.
Properties:
@@ -12799,6 +13559,10 @@ --s3-provider
- Liara Object Storage
+"Linode"
+
+- Linode Object Storage
+
"Minio"
- Minio Object Storage
@@ -12815,6 +13579,10 @@ --s3-provider
- RackCorp Object Storage
+"Rclone"
+
+- Rclone S3 Server
+
"Scaleway"
- Scaleway Object Storage
@@ -13033,3272 +13801,3162 @@ --s3-region
---s3-region
-region - the location where your bucket will be created and your data stored.
+--s3-endpoint
+Endpoint for S3 API.
+Leave blank if using AWS to use the default endpoint for the region.
+Properties:
+
+- Config: endpoint
+- Env Var: RCLONE_S3_ENDPOINT
+- Provider: AWS
+- Type: string
+- Required: false
+
+--s3-location-constraint
+Location constraint - must be set to match the Region.
+Used when creating buckets only.
Properties:
-- Config: region
-- Env Var: RCLONE_S3_REGION
-- Provider: RackCorp
+- Config: location_constraint
+- Env Var: RCLONE_S3_LOCATION_CONSTRAINT
+- Provider: AWS
- Type: string
- Required: false
- Examples:
-- "global"
+
- ""
-- Global CDN (All locations) Region
+- Empty for US Region, Northern Virginia, or Pacific Northwest
-- "au"
+
- "us-east-2"
-- Australia (All states)
+- US East (Ohio) Region
-- "au-nsw"
+
- "us-west-1"
-- NSW (Australia) Region
+- US West (Northern California) Region
-- "au-qld"
+
- "us-west-2"
-- QLD (Australia) Region
+- US West (Oregon) Region
-- "au-vic"
+
- "ca-central-1"
-- VIC (Australia) Region
+- Canada (Central) Region
-- "au-wa"
+
- "eu-west-1"
-- Perth (Australia) Region
+- EU (Ireland) Region
-- "ph"
+
- "eu-west-2"
-- Manila (Philippines) Region
+- EU (London) Region
-- "th"
+
- "eu-west-3"
-- Bangkok (Thailand) Region
+- EU (Paris) Region
-- "hk"
+
- "eu-north-1"
-- HK (Hong Kong) Region
+- EU (Stockholm) Region
-- "mn"
+
- "eu-south-1"
-- Ulaanbaatar (Mongolia) Region
+- EU (Milan) Region
-- "kg"
+
- "EU"
-- Bishkek (Kyrgyzstan) Region
+- EU Region
-- "id"
+
- "ap-southeast-1"
-- Jakarta (Indonesia) Region
+- Asia Pacific (Singapore) Region
-- "jp"
+
- "ap-southeast-2"
-- Tokyo (Japan) Region
+- Asia Pacific (Sydney) Region
-- "sg"
+
- "ap-northeast-1"
-- SG (Singapore) Region
+- Asia Pacific (Tokyo) Region
-- "de"
+
- "ap-northeast-2"
-- Frankfurt (Germany) Region
+- Asia Pacific (Seoul) Region
-- "us"
+
- "ap-northeast-3"
-- USA (AnyCast) Region
+- Asia Pacific (Osaka-Local) Region
-- "us-east-1"
+
- "ap-south-1"
-- New York (USA) Region
+- Asia Pacific (Mumbai) Region
-- "us-west-1"
+
- "ap-east-1"
-- Freemont (USA) Region
+- Asia Pacific (Hong Kong) Region
-- "nz"
+
- "sa-east-1"
-- Auckland (New Zealand) Region
+- South America (Sao Paulo) Region
+- "me-south-1"
+
+- Middle East (Bahrain) Region
-
---s3-region
-Region to connect to.
-Properties:
+ - "af-south-1"
-- Config: region
-- Env Var: RCLONE_S3_REGION
-- Provider: Scaleway
-- Type: string
-- Required: false
-- Examples:
+
- Africa (Cape Town) Region
+
+- "cn-north-1"
-- "nl-ams"
+
- China (Beijing) Region
+
+- "cn-northwest-1"
-- Amsterdam, The Netherlands
+- China (Ningxia) Region
-- "fr-par"
+
- "us-gov-east-1"
-- Paris, France
+- AWS GovCloud (US-East) Region
-- "pl-waw"
+
- "us-gov-west-1"
-- Warsaw, Poland
+- AWS GovCloud (US) Region
---s3-region
-Region to connect to. - the location where your bucket will be created and your data stored. Need bo be same with your endpoint.
+--s3-acl
+Canned ACL used when creating buckets and storing or copying objects.
+This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.
+For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
+Note that this ACL is applied when server-side copying objects as S3 doesn't copy the ACL from the source but rather writes a fresh one.
+If the acl is an empty string then no X-Amz-Acl: header is added and the default (private) will be used.
Properties:
-- Config: region
-- Env Var: RCLONE_S3_REGION
-- Provider: HuaweiOBS
+- Config: acl
+- Env Var: RCLONE_S3_ACL
+- Provider: !Storj,Synology,Cloudflare
- Type: string
- Required: false
- Examples:
-- "af-south-1"
-
-- AF-Johannesburg
-
-- "ap-southeast-2"
-
-- AP-Bangkok
-
-- "ap-southeast-3"
-
-- AP-Singapore
-
-- "cn-east-3"
-
-- CN East-Shanghai1
-
-- "cn-east-2"
+
- "default"
-- CN East-Shanghai2
+- Owner gets Full_CONTROL.
+- No one else has access rights (default).
-- "cn-north-1"
+
- "private"
-- CN North-Beijing1
+- Owner gets FULL_CONTROL.
+- No one else has access rights (default).
-- "cn-north-4"
+
- "public-read"
-- CN North-Beijing4
+- Owner gets FULL_CONTROL.
+- The AllUsers group gets READ access.
-- "cn-south-1"
+
- "public-read-write"
-- CN South-Guangzhou
+- Owner gets FULL_CONTROL.
+- The AllUsers group gets READ and WRITE access.
+- Granting this on a bucket is generally not recommended.
-- "ap-southeast-1"
+
- "authenticated-read"
-- CN-Hong Kong
+- Owner gets FULL_CONTROL.
+- The AuthenticatedUsers group gets READ access.
-- "sa-argentina-1"
+
- "bucket-owner-read"
-- LA-Buenos Aires1
+- Object owner gets FULL_CONTROL.
+- Bucket owner gets READ access.
+- If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
-- "sa-peru-1"
+
- "bucket-owner-full-control"
-- LA-Lima1
+- Both the object owner and the bucket owner get FULL_CONTROL over the object.
+- If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
-- "na-mexico-1"
+
- "private"
-- LA-Mexico City1
+- Owner gets FULL_CONTROL.
+- No one else has access rights (default).
+- This acl is available on IBM Cloud (Infra), IBM Cloud (Storage), On-Premise COS.
-- "sa-chile-1"
+
- "public-read"
-- LA-Santiago2
+- Owner gets FULL_CONTROL.
+- The AllUsers group gets READ access.
+- This acl is available on IBM Cloud (Infra), IBM Cloud (Storage), On-Premise IBM COS.
-- "sa-brazil-1"
+
- "public-read-write"
-- LA-Sao Paulo1
+- Owner gets FULL_CONTROL.
+- The AllUsers group gets READ and WRITE access.
+- This acl is available on IBM Cloud (Infra), On-Premise IBM COS.
-- "ru-northwest-2"
+
- "authenticated-read"
-- RU-Moscow2
+- Owner gets FULL_CONTROL.
+- The AuthenticatedUsers group gets READ access.
+- Not supported on Buckets.
+- This acl is available on IBM Cloud (Infra) and On-Premise IBM COS.
---s3-region
-Region to connect to.
+--s3-server-side-encryption
+The server-side encryption algorithm used when storing this object in S3.
Properties:
-- Config: region
-- Env Var: RCLONE_S3_REGION
-- Provider: Cloudflare
+- Config: server_side_encryption
+- Env Var: RCLONE_S3_SERVER_SIDE_ENCRYPTION
+- Provider: AWS,Ceph,ChinaMobile,Minio
- Type: string
- Required: false
- Examples:
-- "auto"
+
- ""
+
+- None
+
+- "AES256"
+
+- AES256
+
+- "aws:kms"
-- R2 buckets are automatically distributed across Cloudflare's data centers for low latency.
+- aws:kms
---s3-region
-Region to connect to.
+--s3-sse-kms-key-id
+If using KMS ID you must provide the ARN of Key.
Properties:
-- Config: region
-- Env Var: RCLONE_S3_REGION
-- Provider: Qiniu
+- Config: sse_kms_key_id
+- Env Var: RCLONE_S3_SSE_KMS_KEY_ID
+- Provider: AWS,Ceph,Minio
- Type: string
- Required: false
- Examples:
-- "cn-east-1"
-
-- The default endpoint - a good choice if you are unsure.
-- East China Region 1.
-- Needs location constraint cn-east-1.
-
-- "cn-east-2"
-
-- East China Region 2.
-- Needs location constraint cn-east-2.
-
-- "cn-north-1"
-
-- North China Region 1.
-- Needs location constraint cn-north-1.
-
-- "cn-south-1"
-
-- South China Region 1.
-- Needs location constraint cn-south-1.
-
-- "us-north-1"
-
-- North America Region.
-- Needs location constraint us-north-1.
-
-- "ap-southeast-1"
+
- ""
-- Southeast Asia Region 1.
-- Needs location constraint ap-southeast-1.
+- None
-- "ap-northeast-1"
+
- "arn:aws:kms:us-east-1:*"
-- Northeast Asia Region 1.
-- Needs location constraint ap-northeast-1.
+- arn:aws:kms:*
---s3-region
-Region where your bucket will be created and your data stored.
+--s3-storage-class
+The storage class to use when storing new objects in S3.
Properties:
-- Config: region
-- Env Var: RCLONE_S3_REGION
-- Provider: IONOS
+- Config: storage_class
+- Env Var: RCLONE_S3_STORAGE_CLASS
+- Provider: AWS
- Type: string
- Required: false
- Examples:
-- "de"
+
- ""
-- Frankfurt, Germany
+- Default
-- "eu-central-2"
+
- "STANDARD"
-- Berlin, Germany
+- Standard storage class
-- "eu-south-2"
+
- "REDUCED_REDUNDANCY"
-- Logrono, Spain
-
+- Reduced redundancy storage class
-
---s3-region
-Region where your bucket will be created and your data stored.
-Properties:
-
-- Config: region
-- Env Var: RCLONE_S3_REGION
-- Provider: Petabox
-- Type: string
-- Required: false
-- Examples:
+
- "STANDARD_IA"
-- "us-east-1"
+
- Standard Infrequent Access storage class
+
+- "ONEZONE_IA"
-- US East (N. Virginia)
+- One Zone Infrequent Access storage class
-- "eu-central-1"
+
- "GLACIER"
-- Europe (Frankfurt)
+- Glacier storage class
-- "ap-southeast-1"
+
- "DEEP_ARCHIVE"
-- Asia Pacific (Singapore)
+- Glacier Deep Archive storage class
-- "me-south-1"
+
- "INTELLIGENT_TIERING"
-- Middle East (Bahrain)
+- Intelligent-Tiering storage class
-- "sa-east-1"
+
- "GLACIER_IR"
-- South America (São Paulo)
+- Glacier Instant Retrieval storage class
---s3-region
-Region where your data stored.
+Advanced options
+Here are the Advanced options specific to s3 (Amazon S3 Compliant Storage Providers including AWS, Alibaba, ArvanCloud, Ceph, ChinaMobile, Cloudflare, DigitalOcean, Dreamhost, GCS, HuaweiOBS, IBMCOS, IDrive, IONOS, LyveCloud, Leviia, Liara, Linode, Minio, Netease, Petabox, RackCorp, Rclone, Scaleway, SeaweedFS, StackPath, Storj, Synology, TencentCOS, Wasabi, Qiniu and others).
+--s3-bucket-acl
+Canned ACL used when creating buckets.
+For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
+Note that this ACL is applied when only when creating buckets. If it isn't set then "acl" is used instead.
+If the "acl" and "bucket_acl" are empty strings then no X-Amz-Acl: header is added and the default (private) will be used.
Properties:
-- Config: region
-- Env Var: RCLONE_S3_REGION
-- Provider: Synology
+- Config: bucket_acl
+- Env Var: RCLONE_S3_BUCKET_ACL
- Type: string
- Required: false
- Examples:
-- "eu-001"
-
-- Europe Region 1
-
-- "eu-002"
+
- "private"
-- Europe Region 2
+- Owner gets FULL_CONTROL.
+- No one else has access rights (default).
-- "us-001"
+
- "public-read"
-- US Region 1
+- Owner gets FULL_CONTROL.
+- The AllUsers group gets READ access.
-- "us-002"
+
- "public-read-write"
-- US Region 2
+- Owner gets FULL_CONTROL.
+- The AllUsers group gets READ and WRITE access.
+- Granting this on a bucket is generally not recommended.
-- "tw-001"
+
- "authenticated-read"
-- Asia (Taiwan)
+- Owner gets FULL_CONTROL.
+- The AuthenticatedUsers group gets READ access.
---s3-region
-Region to connect to.
-Leave blank if you are using an S3 clone and you don't have a region.
+--s3-requester-pays
+Enables requester pays option when interacting with S3 bucket.
Properties:
-- Config: region
-- Env Var: RCLONE_S3_REGION
-- Provider: !AWS,Alibaba,ArvanCloud,ChinaMobile,Cloudflare,IONOS,Petabox,Liara,Qiniu,RackCorp,Scaleway,Storj,Synology,TencentCOS,HuaweiOBS,IDrive
+- Config: requester_pays
+- Env Var: RCLONE_S3_REQUESTER_PAYS
+- Provider: AWS
+- Type: bool
+- Default: false
+
+--s3-sse-customer-algorithm
+If using SSE-C, the server-side encryption algorithm used when storing this object in S3.
+Properties:
+
+- Config: sse_customer_algorithm
+- Env Var: RCLONE_S3_SSE_CUSTOMER_ALGORITHM
+- Provider: AWS,Ceph,ChinaMobile,Minio
- Type: string
- Required: false
- Examples:
- ""
-- Use this if unsure.
-- Will use v4 signatures and an empty region.
+- None
-- "other-v2-signature"
+
- "AES256"
-- Use this only if v4 signatures don't work.
-- E.g. pre Jewel/v10 CEPH.
+- AES256
---s3-endpoint
-Endpoint for S3 API.
-Leave blank if using AWS to use the default endpoint for the region.
-Properties:
-
-- Config: endpoint
-- Env Var: RCLONE_S3_ENDPOINT
-- Provider: AWS
-- Type: string
-- Required: false
-
---s3-endpoint
-Endpoint for China Mobile Ecloud Elastic Object Storage (EOS) API.
+--s3-sse-customer-key
+To use SSE-C you may provide the secret encryption key used to encrypt/decrypt your data.
+Alternatively you can provide --sse-customer-key-base64.
Properties:
-- Config: endpoint
-- Env Var: RCLONE_S3_ENDPOINT
-- Provider: ChinaMobile
+- Config: sse_customer_key
+- Env Var: RCLONE_S3_SSE_CUSTOMER_KEY
+- Provider: AWS,Ceph,ChinaMobile,Minio
- Type: string
- Required: false
- Examples:
-- "eos-wuxi-1.cmecloud.cn"
-
-- The default endpoint - a good choice if you are unsure.
-- East China (Suzhou)
-
-- "eos-jinan-1.cmecloud.cn"
-
-- East China (Jinan)
-
-- "eos-ningbo-1.cmecloud.cn"
-
-- East China (Hangzhou)
-
-- "eos-shanghai-1.cmecloud.cn"
-
-- East China (Shanghai-1)
-
-- "eos-zhengzhou-1.cmecloud.cn"
-
-- Central China (Zhengzhou)
-
-- "eos-hunan-1.cmecloud.cn"
-
-- Central China (Changsha-1)
-
-- "eos-zhuzhou-1.cmecloud.cn"
-
-- Central China (Changsha-2)
-
-- "eos-guangzhou-1.cmecloud.cn"
-
-- South China (Guangzhou-2)
-
-- "eos-dongguan-1.cmecloud.cn"
-
-- South China (Guangzhou-3)
-
-- "eos-beijing-1.cmecloud.cn"
-
-- North China (Beijing-1)
-
-- "eos-beijing-2.cmecloud.cn"
-
-- North China (Beijing-2)
-
-- "eos-beijing-4.cmecloud.cn"
-
-- North China (Beijing-3)
-
-- "eos-huhehaote-1.cmecloud.cn"
-
-- North China (Huhehaote)
-
-- "eos-chengdu-1.cmecloud.cn"
-
-- Southwest China (Chengdu)
-
-- "eos-chongqing-1.cmecloud.cn"
-
-- Southwest China (Chongqing)
-
-- "eos-guiyang-1.cmecloud.cn"
-
-- Southwest China (Guiyang)
-
-- "eos-xian-1.cmecloud.cn"
-
-- Nouthwest China (Xian)
-
-- "eos-yunnan.cmecloud.cn"
-
-- Yunnan China (Kunming)
-
-- "eos-yunnan-2.cmecloud.cn"
-
-- Yunnan China (Kunming-2)
-
-- "eos-tianjin-1.cmecloud.cn"
-
-- Tianjin China (Tianjin)
-
-- "eos-jilin-1.cmecloud.cn"
-
-- Jilin China (Changchun)
-
-- "eos-hubei-1.cmecloud.cn"
-
-- Hubei China (Xiangyan)
-
-- "eos-jiangxi-1.cmecloud.cn"
-
-- Jiangxi China (Nanchang)
-
-- "eos-gansu-1.cmecloud.cn"
-
-- Gansu China (Lanzhou)
-
-- "eos-shanxi-1.cmecloud.cn"
-
-- Shanxi China (Taiyuan)
-
-- "eos-liaoning-1.cmecloud.cn"
-
-- Liaoning China (Shenyang)
-
-- "eos-hebei-1.cmecloud.cn"
-
-- Hebei China (Shijiazhuang)
-
-- "eos-fujian-1.cmecloud.cn"
-
-- Fujian China (Xiamen)
-
-- "eos-guangxi-1.cmecloud.cn"
-
-- Guangxi China (Nanning)
-
-- "eos-anhui-1.cmecloud.cn"
+
- ""
-- Anhui China (Huainan)
+- None
---s3-endpoint
-Endpoint for Arvan Cloud Object Storage (AOS) API.
+--s3-sse-customer-key-base64
+If using SSE-C you must provide the secret encryption key encoded in base64 format to encrypt/decrypt your data.
+Alternatively you can provide --sse-customer-key.
Properties:
-- Config: endpoint
-- Env Var: RCLONE_S3_ENDPOINT
-- Provider: ArvanCloud
+- Config: sse_customer_key_base64
+- Env Var: RCLONE_S3_SSE_CUSTOMER_KEY_BASE64
+- Provider: AWS,Ceph,ChinaMobile,Minio
- Type: string
- Required: false
- Examples:
-- "s3.ir-thr-at1.arvanstorage.ir"
-
-- The default endpoint - a good choice if you are unsure.
-- Tehran Iran (Simin)
-
-- "s3.ir-tbz-sh1.arvanstorage.ir"
+
- ""
-- Tabriz Iran (Shahriar)
+- None
---s3-endpoint
-Endpoint for IBM COS S3 API.
-Specify if using an IBM COS On Premise.
+--s3-sse-customer-key-md5
+If using SSE-C you may provide the secret encryption key MD5 checksum (optional).
+If you leave it blank, this is calculated automatically from the sse_customer_key provided.
Properties:
-- Config: endpoint
-- Env Var: RCLONE_S3_ENDPOINT
-- Provider: IBMCOS
+- Config: sse_customer_key_md5
+- Env Var: RCLONE_S3_SSE_CUSTOMER_KEY_MD5
+- Provider: AWS,Ceph,ChinaMobile,Minio
- Type: string
- Required: false
- Examples:
-- "s3.us.cloud-object-storage.appdomain.cloud"
-
-- US Cross Region Endpoint
-
-- "s3.dal.us.cloud-object-storage.appdomain.cloud"
+
- ""
-- US Cross Region Dallas Endpoint
-
-- "s3.wdc.us.cloud-object-storage.appdomain.cloud"
-
-- US Cross Region Washington DC Endpoint
-
-- "s3.sjc.us.cloud-object-storage.appdomain.cloud"
-
-- US Cross Region San Jose Endpoint
-
-- "s3.private.us.cloud-object-storage.appdomain.cloud"
-
-- US Cross Region Private Endpoint
-
-- "s3.private.dal.us.cloud-object-storage.appdomain.cloud"
-
-- US Cross Region Dallas Private Endpoint
-
-- "s3.private.wdc.us.cloud-object-storage.appdomain.cloud"
-
-- US Cross Region Washington DC Private Endpoint
-
-- "s3.private.sjc.us.cloud-object-storage.appdomain.cloud"
-
-- US Cross Region San Jose Private Endpoint
-
-- "s3.us-east.cloud-object-storage.appdomain.cloud"
-
-- US Region East Endpoint
-
-- "s3.private.us-east.cloud-object-storage.appdomain.cloud"
-
-- US Region East Private Endpoint
-
-- "s3.us-south.cloud-object-storage.appdomain.cloud"
-
-- US Region South Endpoint
-
-- "s3.private.us-south.cloud-object-storage.appdomain.cloud"
-
-- US Region South Private Endpoint
-
-- "s3.eu.cloud-object-storage.appdomain.cloud"
-
-- EU Cross Region Endpoint
-
-- "s3.fra.eu.cloud-object-storage.appdomain.cloud"
-
-- EU Cross Region Frankfurt Endpoint
-
-- "s3.mil.eu.cloud-object-storage.appdomain.cloud"
-
-- EU Cross Region Milan Endpoint
-
-- "s3.ams.eu.cloud-object-storage.appdomain.cloud"
-
-- EU Cross Region Amsterdam Endpoint
-
-- "s3.private.eu.cloud-object-storage.appdomain.cloud"
-
-- EU Cross Region Private Endpoint
-
-- "s3.private.fra.eu.cloud-object-storage.appdomain.cloud"
-
-- EU Cross Region Frankfurt Private Endpoint
-
-- "s3.private.mil.eu.cloud-object-storage.appdomain.cloud"
-
-- EU Cross Region Milan Private Endpoint
-
-- "s3.private.ams.eu.cloud-object-storage.appdomain.cloud"
-
-- EU Cross Region Amsterdam Private Endpoint
-
-- "s3.eu-gb.cloud-object-storage.appdomain.cloud"
-
-- Great Britain Endpoint
-
-- "s3.private.eu-gb.cloud-object-storage.appdomain.cloud"
-
-- Great Britain Private Endpoint
-
-- "s3.eu-de.cloud-object-storage.appdomain.cloud"
-
-- EU Region DE Endpoint
-
-- "s3.private.eu-de.cloud-object-storage.appdomain.cloud"
-
-- EU Region DE Private Endpoint
-
-- "s3.ap.cloud-object-storage.appdomain.cloud"
-
-- APAC Cross Regional Endpoint
-
-- "s3.tok.ap.cloud-object-storage.appdomain.cloud"
-
-- APAC Cross Regional Tokyo Endpoint
-
-- "s3.hkg.ap.cloud-object-storage.appdomain.cloud"
-
-- APAC Cross Regional HongKong Endpoint
-
-- "s3.seo.ap.cloud-object-storage.appdomain.cloud"
-
-- APAC Cross Regional Seoul Endpoint
-
-- "s3.private.ap.cloud-object-storage.appdomain.cloud"
-
-- APAC Cross Regional Private Endpoint
-
-- "s3.private.tok.ap.cloud-object-storage.appdomain.cloud"
-
-- APAC Cross Regional Tokyo Private Endpoint
-
-- "s3.private.hkg.ap.cloud-object-storage.appdomain.cloud"
-
-- APAC Cross Regional HongKong Private Endpoint
-
-- "s3.private.seo.ap.cloud-object-storage.appdomain.cloud"
-
-- APAC Cross Regional Seoul Private Endpoint
-
-- "s3.jp-tok.cloud-object-storage.appdomain.cloud"
-
-- APAC Region Japan Endpoint
-
-- "s3.private.jp-tok.cloud-object-storage.appdomain.cloud"
-
-- APAC Region Japan Private Endpoint
-
-- "s3.au-syd.cloud-object-storage.appdomain.cloud"
-
-- APAC Region Australia Endpoint
-
-- "s3.private.au-syd.cloud-object-storage.appdomain.cloud"
-
-- APAC Region Australia Private Endpoint
-
-- "s3.ams03.cloud-object-storage.appdomain.cloud"
-
-- Amsterdam Single Site Endpoint
-
-- "s3.private.ams03.cloud-object-storage.appdomain.cloud"
-
-- Amsterdam Single Site Private Endpoint
-
-- "s3.che01.cloud-object-storage.appdomain.cloud"
-
-- Chennai Single Site Endpoint
-
-- "s3.private.che01.cloud-object-storage.appdomain.cloud"
-
-- Chennai Single Site Private Endpoint
-
-- "s3.mel01.cloud-object-storage.appdomain.cloud"
-
-- Melbourne Single Site Endpoint
-
-- "s3.private.mel01.cloud-object-storage.appdomain.cloud"
-
-- Melbourne Single Site Private Endpoint
-
-- "s3.osl01.cloud-object-storage.appdomain.cloud"
-
-- Oslo Single Site Endpoint
-
-- "s3.private.osl01.cloud-object-storage.appdomain.cloud"
-
-- Oslo Single Site Private Endpoint
-
-- "s3.tor01.cloud-object-storage.appdomain.cloud"
-
-- Toronto Single Site Endpoint
-
-- "s3.private.tor01.cloud-object-storage.appdomain.cloud"
-
-- Toronto Single Site Private Endpoint
-
-- "s3.seo01.cloud-object-storage.appdomain.cloud"
-
-- Seoul Single Site Endpoint
-
-- "s3.private.seo01.cloud-object-storage.appdomain.cloud"
-
-- Seoul Single Site Private Endpoint
-
-- "s3.mon01.cloud-object-storage.appdomain.cloud"
-
-- Montreal Single Site Endpoint
-
-- "s3.private.mon01.cloud-object-storage.appdomain.cloud"
-
-- Montreal Single Site Private Endpoint
-
-- "s3.mex01.cloud-object-storage.appdomain.cloud"
-
-- Mexico Single Site Endpoint
-
-- "s3.private.mex01.cloud-object-storage.appdomain.cloud"
-
-- Mexico Single Site Private Endpoint
-
-- "s3.sjc04.cloud-object-storage.appdomain.cloud"
-
-- San Jose Single Site Endpoint
-
-- "s3.private.sjc04.cloud-object-storage.appdomain.cloud"
-
-- San Jose Single Site Private Endpoint
-
-- "s3.mil01.cloud-object-storage.appdomain.cloud"
-
-- Milan Single Site Endpoint
-
-- "s3.private.mil01.cloud-object-storage.appdomain.cloud"
-
-- Milan Single Site Private Endpoint
-
-- "s3.hkg02.cloud-object-storage.appdomain.cloud"
-
-- Hong Kong Single Site Endpoint
-
-- "s3.private.hkg02.cloud-object-storage.appdomain.cloud"
-
-- Hong Kong Single Site Private Endpoint
-
-- "s3.par01.cloud-object-storage.appdomain.cloud"
-
-- Paris Single Site Endpoint
-
-- "s3.private.par01.cloud-object-storage.appdomain.cloud"
-
-- Paris Single Site Private Endpoint
-
-- "s3.sng01.cloud-object-storage.appdomain.cloud"
-
-- Singapore Single Site Endpoint
-
-- "s3.private.sng01.cloud-object-storage.appdomain.cloud"
-
-- Singapore Single Site Private Endpoint
+- None
---s3-endpoint
-Endpoint for IONOS S3 Object Storage.
-Specify the endpoint from the same region.
+--s3-upload-cutoff
+Cutoff for switching to chunked upload.
+Any files larger than this will be uploaded in chunks of chunk_size. The minimum is 0 and the maximum is 5 GiB.
Properties:
-- Config: endpoint
-- Env Var: RCLONE_S3_ENDPOINT
-- Provider: IONOS
-- Type: string
-- Required: false
-- Examples:
-
-- "s3-eu-central-1.ionoscloud.com"
-
-- Frankfurt, Germany
-
-- "s3-eu-central-2.ionoscloud.com"
-
-- Berlin, Germany
-
-- "s3-eu-south-2.ionoscloud.com"
-
-- Logrono, Spain
-
-
+- Config: upload_cutoff
+- Env Var: RCLONE_S3_UPLOAD_CUTOFF
+- Type: SizeSuffix
+- Default: 200Mi
---s3-endpoint
-Endpoint for Petabox S3 Object Storage.
-Specify the endpoint from the same region.
+--s3-chunk-size
+Chunk size to use for uploading.
+When uploading files larger than upload_cutoff or files with unknown size (e.g. from "rclone rcat" or uploaded with "rclone mount" or google photos or google docs) they will be uploaded as multipart uploads using this chunk size.
+Note that "--s3-upload-concurrency" chunks of this size are buffered in memory per transfer.
+If you are transferring large files over high-speed links and you have enough memory, then increasing this will speed up the transfers.
+Rclone will automatically increase the chunk size when uploading a large file of known size to stay below the 10,000 chunks limit.
+Files of unknown size are uploaded with the configured chunk_size. Since the default chunk size is 5 MiB and there can be at most 10,000 chunks, this means that by default the maximum size of a file you can stream upload is 48 GiB. If you wish to stream upload larger files then you will need to increase chunk_size.
+Increasing the chunk size decreases the accuracy of the progress statistics displayed with "-P" flag. Rclone treats chunk as sent when it's buffered by the AWS SDK, when in fact it may still be uploading. A bigger chunk size means a bigger AWS SDK buffer and progress reporting more deviating from the truth.
Properties:
-- Config: endpoint
-- Env Var: RCLONE_S3_ENDPOINT
-- Provider: Petabox
-- Type: string
-- Required: true
-- Examples:
-
-- "s3.petabox.io"
-
-- US East (N. Virginia)
-
-- "s3.us-east-1.petabox.io"
-
-- US East (N. Virginia)
-
-- "s3.eu-central-1.petabox.io"
-
-- Europe (Frankfurt)
-
-- "s3.ap-southeast-1.petabox.io"
-
-- Asia Pacific (Singapore)
-
-- "s3.me-south-1.petabox.io"
-
-- Middle East (Bahrain)
-
-- "s3.sa-east-1.petabox.io"
-
-- South America (São Paulo)
-
-
+- Config: chunk_size
+- Env Var: RCLONE_S3_CHUNK_SIZE
+- Type: SizeSuffix
+- Default: 5Mi
---s3-endpoint
-Endpoint for Leviia Object Storage API.
+--s3-max-upload-parts
+Maximum number of parts in a multipart upload.
+This option defines the maximum number of multipart chunks to use when doing a multipart upload.
+This can be useful if a service does not support the AWS S3 specification of 10,000 chunks.
+Rclone will automatically increase the chunk size when uploading a large file of a known size to stay below this number of chunks limit.
Properties:
-- Config: endpoint
-- Env Var: RCLONE_S3_ENDPOINT
-- Provider: Leviia
-- Type: string
-- Required: false
-- Examples:
+
- Config: max_upload_parts
+- Env Var: RCLONE_S3_MAX_UPLOAD_PARTS
+- Type: int
+- Default: 10000
+
+--s3-copy-cutoff
+Cutoff for switching to multipart copy.
+Any files larger than this that need to be server-side copied will be copied in chunks of this size.
+The minimum is 0 and the maximum is 5 GiB.
+Properties:
-- "s3.leviia.com"
+
- Config: copy_cutoff
+- Env Var: RCLONE_S3_COPY_CUTOFF
+- Type: SizeSuffix
+- Default: 4.656Gi
+
+--s3-disable-checksum
+Don't store MD5 checksum with object metadata.
+Normally rclone will calculate the MD5 checksum of the input before uploading it so it can add it to metadata on the object. This is great for data integrity checking but can cause long delays for large files to start uploading.
+Properties:
-- The default endpoint
-- Leviia
-
-
+Config: disable_checksum
+Env Var: RCLONE_S3_DISABLE_CHECKSUM
+Type: bool
+Default: false
---s3-endpoint
-Endpoint for Liara Object Storage API.
+--s3-shared-credentials-file
+Path to the shared credentials file.
+If env_auth = true then rclone can use a shared credentials file.
+If this variable is empty rclone will look for the "AWS_SHARED_CREDENTIALS_FILE" env variable. If the env value is empty it will default to the current user's home directory.
+Linux/OSX: "$HOME/.aws/credentials"
+Windows: "%USERPROFILE%\.aws\credentials"
Properties:
-- Config: endpoint
-- Env Var: RCLONE_S3_ENDPOINT
-- Provider: Liara
+- Config: shared_credentials_file
+- Env Var: RCLONE_S3_SHARED_CREDENTIALS_FILE
- Type: string
- Required: false
-- Examples:
-
-- "storage.iran.liara.space"
-
-- The default endpoint
-- Iran
-
-
---s3-endpoint
-Endpoint for OSS API.
+--s3-profile
+Profile to use in the shared credentials file.
+If env_auth = true then rclone can use a shared credentials file. This variable controls which profile is used in that file.
+If empty it will default to the environment variable "AWS_PROFILE" or "default" if that environment variable is also not set.
Properties:
-- Config: endpoint
-- Env Var: RCLONE_S3_ENDPOINT
-- Provider: Alibaba
+- Config: profile
+- Env Var: RCLONE_S3_PROFILE
- Type: string
- Required: false
-- Examples:
-
-- "oss-accelerate.aliyuncs.com"
-
-- Global Accelerate
-
-- "oss-accelerate-overseas.aliyuncs.com"
-
-- Global Accelerate (outside mainland China)
-
-- "oss-cn-hangzhou.aliyuncs.com"
-
-- East China 1 (Hangzhou)
-
-- "oss-cn-shanghai.aliyuncs.com"
-
-- East China 2 (Shanghai)
-
-- "oss-cn-qingdao.aliyuncs.com"
-
-- North China 1 (Qingdao)
-
-- "oss-cn-beijing.aliyuncs.com"
-
-- North China 2 (Beijing)
-
-- "oss-cn-zhangjiakou.aliyuncs.com"
-
-- North China 3 (Zhangjiakou)
-
-- "oss-cn-huhehaote.aliyuncs.com"
-
-- North China 5 (Hohhot)
-
-- "oss-cn-wulanchabu.aliyuncs.com"
-
-- North China 6 (Ulanqab)
-
-- "oss-cn-shenzhen.aliyuncs.com"
-
-- South China 1 (Shenzhen)
-
-- "oss-cn-heyuan.aliyuncs.com"
-
-- South China 2 (Heyuan)
-
-- "oss-cn-guangzhou.aliyuncs.com"
-
-- South China 3 (Guangzhou)
-
-- "oss-cn-chengdu.aliyuncs.com"
-
-- West China 1 (Chengdu)
-
-- "oss-cn-hongkong.aliyuncs.com"
-
-- Hong Kong (Hong Kong)
-
-- "oss-us-west-1.aliyuncs.com"
-
-- US West 1 (Silicon Valley)
-
-- "oss-us-east-1.aliyuncs.com"
-
-- US East 1 (Virginia)
-
-- "oss-ap-southeast-1.aliyuncs.com"
-
-- Southeast Asia Southeast 1 (Singapore)
-
-- "oss-ap-southeast-2.aliyuncs.com"
-
-- Asia Pacific Southeast 2 (Sydney)
-
-- "oss-ap-southeast-3.aliyuncs.com"
-
-- Southeast Asia Southeast 3 (Kuala Lumpur)
-
-- "oss-ap-southeast-5.aliyuncs.com"
-
-- Asia Pacific Southeast 5 (Jakarta)
-
-- "oss-ap-northeast-1.aliyuncs.com"
-
-- Asia Pacific Northeast 1 (Japan)
-
-- "oss-ap-south-1.aliyuncs.com"
-
-- Asia Pacific South 1 (Mumbai)
-
-- "oss-eu-central-1.aliyuncs.com"
-
-- Central Europe 1 (Frankfurt)
-
-- "oss-eu-west-1.aliyuncs.com"
-
-- West Europe (London)
-
-- "oss-me-east-1.aliyuncs.com"
-
-- Middle East 1 (Dubai)
-
-
---s3-endpoint
-Endpoint for OBS API.
+--s3-session-token
+An AWS session token.
Properties:
-- Config: endpoint
-- Env Var: RCLONE_S3_ENDPOINT
-- Provider: HuaweiOBS
+- Config: session_token
+- Env Var: RCLONE_S3_SESSION_TOKEN
- Type: string
- Required: false
-- Examples:
-
-- "obs.af-south-1.myhuaweicloud.com"
-
-- AF-Johannesburg
-
-- "obs.ap-southeast-2.myhuaweicloud.com"
-
-- AP-Bangkok
-
-- "obs.ap-southeast-3.myhuaweicloud.com"
-
-- AP-Singapore
-
-- "obs.cn-east-3.myhuaweicloud.com"
-
-- CN East-Shanghai1
-
-- "obs.cn-east-2.myhuaweicloud.com"
-
-- CN East-Shanghai2
-
-- "obs.cn-north-1.myhuaweicloud.com"
-
-- CN North-Beijing1
-
-- "obs.cn-north-4.myhuaweicloud.com"
-
-- CN North-Beijing4
-
-- "obs.cn-south-1.myhuaweicloud.com"
-
-- CN South-Guangzhou
-
-- "obs.ap-southeast-1.myhuaweicloud.com"
-
-- CN-Hong Kong
-
-- "obs.sa-argentina-1.myhuaweicloud.com"
+
+--s3-upload-concurrency
+Concurrency for multipart uploads.
+This is the number of chunks of the same file that are uploaded concurrently.
+If you are uploading small numbers of large files over high-speed links and these uploads do not fully utilize your bandwidth, then increasing this may help to speed up the transfers.
+Properties:
-- LA-Buenos Aires1
-
-- "obs.sa-peru-1.myhuaweicloud.com"
+
- Config: upload_concurrency
+- Env Var: RCLONE_S3_UPLOAD_CONCURRENCY
+- Type: int
+- Default: 4
+
+--s3-force-path-style
+If true use path style access if false use virtual hosted style.
+If this is true (the default) then rclone will use path style access, if false then rclone will use virtual path style. See the AWS S3 docs for more info.
+Some providers (e.g. AWS, Aliyun OSS, Netease COS, or Tencent COS) require this set to false - rclone will do this automatically based on the provider setting.
+Properties:
-- LA-Lima1
-
-"obs.na-mexico-1.myhuaweicloud.com"
+ Config: force_path_style
+Env Var: RCLONE_S3_FORCE_PATH_STYLE
+Type: bool
+Default: true
+
+--s3-v2-auth
+If true use v2 authentication.
+If this is false (the default) then rclone will use v4 authentication. If it is set then rclone will use v2 authentication.
+Use this only if v4 signatures don't work, e.g. pre Jewel/v10 CEPH.
+Properties:
-- LA-Mexico City1
-
-"obs.sa-chile-1.myhuaweicloud.com"
+ Config: v2_auth
+Env Var: RCLONE_S3_V2_AUTH
+Type: bool
+Default: false
+
+--s3-use-accelerate-endpoint
+If true use the AWS S3 accelerated endpoint.
+See: AWS S3 Transfer acceleration
+Properties:
-- LA-Santiago2
-
-"obs.sa-brazil-1.myhuaweicloud.com"
+ Config: use_accelerate_endpoint
+Env Var: RCLONE_S3_USE_ACCELERATE_ENDPOINT
+Provider: AWS
+Type: bool
+Default: false
+
+--s3-leave-parts-on-error
+If true avoid calling abort upload on a failure, leaving all successfully uploaded parts on S3 for manual recovery.
+It should be set to true for resuming uploads across different sessions.
+WARNING: Storing parts of an incomplete multipart upload counts towards space usage on S3 and will add additional costs if not cleaned up.
+Properties:
-- LA-Sao Paulo1
-
-"obs.ru-northwest-2.myhuaweicloud.com"
+ Config: leave_parts_on_error
+Env Var: RCLONE_S3_LEAVE_PARTS_ON_ERROR
+Provider: AWS
+Type: bool
+Default: false
+
+--s3-list-chunk
+Size of listing chunk (response list for each ListObject S3 request).
+This option is also known as "MaxKeys", "max-items", or "page-size" from the AWS S3 specification. Most services truncate the response list to 1000 objects even if requested more than that. In AWS S3 this is a global maximum and cannot be changed, see AWS S3. In Ceph, this can be increased with the "rgw list buckets max chunk" option.
+Properties:
-- RU-Moscow2
-
-
+Config: list_chunk
+Env Var: RCLONE_S3_LIST_CHUNK
+Type: int
+Default: 1000
---s3-endpoint
-Endpoint for Scaleway Object Storage.
+--s3-list-version
+Version of ListObjects to use: 1,2 or 0 for auto.
+When S3 originally launched it only provided the ListObjects call to enumerate objects in a bucket.
+However in May 2016 the ListObjectsV2 call was introduced. This is much higher performance and should be used if at all possible.
+If set to the default, 0, rclone will guess according to the provider set which list objects method to call. If it guesses wrong, then it may be set manually here.
Properties:
-- Config: endpoint
-- Env Var: RCLONE_S3_ENDPOINT
-- Provider: Scaleway
-- Type: string
-- Required: false
-- Examples:
+
- Config: list_version
+- Env Var: RCLONE_S3_LIST_VERSION
+- Type: int
+- Default: 0
+
+--s3-list-url-encode
+Whether to url encode listings: true/false/unset
+Some providers support URL encoding listings and where this is available this is more reliable when using control characters in file names. If this is set to unset (the default) then rclone will choose according to the provider setting what to apply, but you can override rclone's choice here.
+Properties:
-- "s3.nl-ams.scw.cloud"
+
- Config: list_url_encode
+- Env Var: RCLONE_S3_LIST_URL_ENCODE
+- Type: Tristate
+- Default: unset
+
+--s3-no-check-bucket
+If set, don't attempt to check the bucket exists or create it.
+This can be useful when trying to minimise the number of transactions rclone does if you know the bucket exists already.
+It can also be needed if the user you are using does not have bucket creation permissions. Before v1.52.0 this would have passed silently due to a bug.
+Properties:
-- Amsterdam Endpoint
-
-"s3.fr-par.scw.cloud"
+ Config: no_check_bucket
+Env Var: RCLONE_S3_NO_CHECK_BUCKET
+Type: bool
+Default: false
+
+--s3-no-head
+If set, don't HEAD uploaded objects to check integrity.
+This can be useful when trying to minimise the number of transactions rclone does.
+Setting it means that if rclone receives a 200 OK message after uploading an object with PUT then it will assume that it got uploaded properly.
+In particular it will assume:
-- Paris Endpoint
-
-"s3.pl-waw.scw.cloud"
+ the metadata, including modtime, storage class and content type was as uploaded
+the size was as uploaded
+
+It reads the following items from the response for a single part PUT:
-- Warsaw Endpoint
-
-
+the MD5SUM
+The uploaded date
---s3-endpoint
-Endpoint for StackPath Object Storage.
+For multipart uploads these items aren't read.
+If an source object of unknown length is uploaded then rclone will do a HEAD request.
+Setting this flag increases the chance for undetected upload failures, in particular an incorrect size, so it isn't recommended for normal operation. In practice the chance of an undetected upload failure is very small even with this flag.
Properties:
-- Config: endpoint
-- Env Var: RCLONE_S3_ENDPOINT
-- Provider: StackPath
-- Type: string
-- Required: false
-- Examples:
-
-- "s3.us-east-2.stackpathstorage.com"
-
-- US East Endpoint
-
-- "s3.us-west-1.stackpathstorage.com"
+
- Config: no_head
+- Env Var: RCLONE_S3_NO_HEAD
+- Type: bool
+- Default: false
+
+--s3-no-head-object
+If set, do not do HEAD before GET when getting objects.
+Properties:
-- US West Endpoint
-
-- "s3.eu-central-1.stackpathstorage.com"
+
- Config: no_head_object
+- Env Var: RCLONE_S3_NO_HEAD_OBJECT
+- Type: bool
+- Default: false
+
+--s3-encoding
+The encoding for the backend.
+See the encoding section in the overview for more info.
+Properties:
-- EU Endpoint
-
-
+Config: encoding
+Env Var: RCLONE_S3_ENCODING
+Type: Encoding
+Default: Slash,InvalidUtf8,Dot
---s3-endpoint
-Endpoint for Google Cloud Storage.
+--s3-memory-pool-flush-time
+How often internal memory buffer pools will be flushed. (no longer used)
Properties:
-- Config: endpoint
-- Env Var: RCLONE_S3_ENDPOINT
-- Provider: GCS
-- Type: string
-- Required: false
-- Examples:
+
- Config: memory_pool_flush_time
+- Env Var: RCLONE_S3_MEMORY_POOL_FLUSH_TIME
+- Type: Duration
+- Default: 1m0s
+
+--s3-memory-pool-use-mmap
+Whether to use mmap buffers in internal memory pool. (no longer used)
+Properties:
-- "https://storage.googleapis.com"
+
- Config: memory_pool_use_mmap
+- Env Var: RCLONE_S3_MEMORY_POOL_USE_MMAP
+- Type: bool
+- Default: false
+
+--s3-disable-http2
+Disable usage of http2 for S3 backends.
+There is currently an unsolved issue with the s3 (specifically minio) backend and HTTP/2. HTTP/2 is enabled by default for the s3 backend but can be disabled here. When the issue is solved this flag will be removed.
+See: https://github.com/rclone/rclone/issues/4673, https://github.com/rclone/rclone/issues/3631
+Properties:
-- Google Cloud Storage endpoint
-
-
+Config: disable_http2
+Env Var: RCLONE_S3_DISABLE_HTTP2
+Type: bool
+Default: false
---s3-endpoint
-Endpoint for Storj Gateway.
+--s3-download-url
+Custom endpoint for downloads. This is usually set to a CloudFront CDN URL as AWS S3 offers cheaper egress for data downloaded through the CloudFront network.
Properties:
-- Config: endpoint
-- Env Var: RCLONE_S3_ENDPOINT
-- Provider: Storj
+- Config: download_url
+- Env Var: RCLONE_S3_DOWNLOAD_URL
- Type: string
- Required: false
-- Examples:
-
-- "gateway.storjshare.io"
+
+--s3-directory-markers
+Upload an empty object with a trailing slash when a new directory is created
+Empty folders are unsupported for bucket based remotes, this option creates an empty object ending with "/", to persist the folder.
+Properties:
-- Global Hosted Gateway
-
-
+Config: directory_markers
+Env Var: RCLONE_S3_DIRECTORY_MARKERS
+Type: bool
+Default: false
---s3-endpoint
-Endpoint for Synology C2 Object Storage API.
+--s3-use-multipart-etag
+Whether to use ETag in multipart uploads for verification
+This should be true, false or left unset to use the default for the provider.
Properties:
-- Config: endpoint
-- Env Var: RCLONE_S3_ENDPOINT
-- Provider: Synology
-- Type: string
-- Required: false
-- Examples:
-
-- "eu-001.s3.synologyc2.net"
-
-- EU Endpoint 1
-
-- "eu-002.s3.synologyc2.net"
-
-- EU Endpoint 2
-
-- "us-001.s3.synologyc2.net"
-
-- US Endpoint 1
-
-- "us-002.s3.synologyc2.net"
-
-- US Endpoint 2
-
-- "tw-001.s3.synologyc2.net"
-
-- TW Endpoint 1
-
-
+- Config: use_multipart_etag
+- Env Var: RCLONE_S3_USE_MULTIPART_ETAG
+- Type: Tristate
+- Default: unset
---s3-endpoint
-Endpoint for Tencent COS API.
+--s3-use-presigned-request
+Whether to use a presigned request or PutObject for single part uploads
+If this is false rclone will use PutObject from the AWS SDK to upload an object.
+Versions of rclone < 1.59 use presigned requests to upload a single part object and setting this flag to true will re-enable that functionality. This shouldn't be necessary except in exceptional circumstances or for testing.
Properties:
-- Config: endpoint
-- Env Var: RCLONE_S3_ENDPOINT
-- Provider: TencentCOS
-- Type: string
-- Required: false
-- Examples:
-
-- "cos.ap-beijing.myqcloud.com"
-
-- Beijing Region
-
-- "cos.ap-nanjing.myqcloud.com"
-
-- Nanjing Region
-
-- "cos.ap-shanghai.myqcloud.com"
-
-- Shanghai Region
-
-- "cos.ap-guangzhou.myqcloud.com"
-
-- Guangzhou Region
-
-- "cos.ap-nanjing.myqcloud.com"
-
-- Nanjing Region
-
-- "cos.ap-chengdu.myqcloud.com"
-
-- Chengdu Region
-
-- "cos.ap-chongqing.myqcloud.com"
-
-- Chongqing Region
-
-- "cos.ap-hongkong.myqcloud.com"
-
-- Hong Kong (China) Region
-
-- "cos.ap-singapore.myqcloud.com"
-
-- Singapore Region
-
-- "cos.ap-mumbai.myqcloud.com"
-
-- Mumbai Region
-
-- "cos.ap-seoul.myqcloud.com"
-
-- Seoul Region
-
-- "cos.ap-bangkok.myqcloud.com"
-
-- Bangkok Region
-
-- "cos.ap-tokyo.myqcloud.com"
-
-- Tokyo Region
-
-- "cos.na-siliconvalley.myqcloud.com"
-
-- Silicon Valley Region
-
-- "cos.na-ashburn.myqcloud.com"
-
-- Virginia Region
-
-- "cos.na-toronto.myqcloud.com"
-
-- Toronto Region
-
-- "cos.eu-frankfurt.myqcloud.com"
-
-- Frankfurt Region
-
-- "cos.eu-moscow.myqcloud.com"
-
-- Moscow Region
-
-- "cos.accelerate.myqcloud.com"
-
-- Use Tencent COS Accelerate Endpoint
-
-
+- Config: use_presigned_request
+- Env Var: RCLONE_S3_USE_PRESIGNED_REQUEST
+- Type: bool
+- Default: false
---s3-endpoint
-Endpoint for RackCorp Object Storage.
+--s3-versions
+Include old versions in directory listings.
Properties:
-- Config: endpoint
-- Env Var: RCLONE_S3_ENDPOINT
-- Provider: RackCorp
-- Type: string
-- Required: false
-- Examples:
-
-- "s3.rackcorp.com"
-
-- Global (AnyCast) Endpoint
-
-- "au.s3.rackcorp.com"
-
-- Australia (Anycast) Endpoint
-
-- "au-nsw.s3.rackcorp.com"
-
-- Sydney (Australia) Endpoint
-
-- "au-qld.s3.rackcorp.com"
-
-- Brisbane (Australia) Endpoint
-
-- "au-vic.s3.rackcorp.com"
-
-- Melbourne (Australia) Endpoint
-
-- "au-wa.s3.rackcorp.com"
-
-- Perth (Australia) Endpoint
-
-- "ph.s3.rackcorp.com"
-
-- Manila (Philippines) Endpoint
-
-- "th.s3.rackcorp.com"
-
-- Bangkok (Thailand) Endpoint
-
-- "hk.s3.rackcorp.com"
-
-- HK (Hong Kong) Endpoint
-
-- "mn.s3.rackcorp.com"
-
-- Ulaanbaatar (Mongolia) Endpoint
-
-- "kg.s3.rackcorp.com"
-
-- Bishkek (Kyrgyzstan) Endpoint
-
-- "id.s3.rackcorp.com"
-
-- Jakarta (Indonesia) Endpoint
-
-- "jp.s3.rackcorp.com"
-
-- Tokyo (Japan) Endpoint
-
-- "sg.s3.rackcorp.com"
-
-- SG (Singapore) Endpoint
-
-- "de.s3.rackcorp.com"
-
-- Frankfurt (Germany) Endpoint
-
-- "us.s3.rackcorp.com"
-
-- USA (AnyCast) Endpoint
-
-- "us-east-1.s3.rackcorp.com"
-
-- New York (USA) Endpoint
-
-- "us-west-1.s3.rackcorp.com"
-
-- Freemont (USA) Endpoint
-
-- "nz.s3.rackcorp.com"
-
-- Auckland (New Zealand) Endpoint
-
-
+- Config: versions
+- Env Var: RCLONE_S3_VERSIONS
+- Type: bool
+- Default: false
---s3-endpoint
-Endpoint for Qiniu Object Storage.
+--s3-version-at
+Show file versions as they were at the specified time.
+The parameter should be a date, "2006-01-02", datetime "2006-01-02 15:04:05" or a duration for that long ago, eg "100d" or "1h".
+Note that when using this no file write operations are permitted, so you can't upload files or delete them.
+See the time option docs for valid formats.
Properties:
-- Config: endpoint
-- Env Var: RCLONE_S3_ENDPOINT
-- Provider: Qiniu
-- Type: string
-- Required: false
-- Examples:
-
-- "s3-cn-east-1.qiniucs.com"
-
-- East China Endpoint 1
-
-- "s3-cn-east-2.qiniucs.com"
-
-- East China Endpoint 2
-
-- "s3-cn-north-1.qiniucs.com"
-
-- North China Endpoint 1
-
-- "s3-cn-south-1.qiniucs.com"
-
-- South China Endpoint 1
-
-- "s3-us-north-1.qiniucs.com"
-
-- North America Endpoint 1
-
-- "s3-ap-southeast-1.qiniucs.com"
-
-- Southeast Asia Endpoint 1
-
-- "s3-ap-northeast-1.qiniucs.com"
-
-- Northeast Asia Endpoint 1
-
-
+- Config: version_at
+- Env Var: RCLONE_S3_VERSION_AT
+- Type: Time
+- Default: off
---s3-endpoint
-Endpoint for S3 API.
-Required when using an S3 clone.
+--s3-decompress
+If set this will decompress gzip encoded objects.
+It is possible to upload objects to S3 with "Content-Encoding: gzip" set. Normally rclone will download these files as compressed objects.
+If this flag is set then rclone will decompress these files with "Content-Encoding: gzip" as they are received. This means that rclone can't check the size and hash but the file contents will be decompressed.
Properties:
-- Config: endpoint
-- Env Var: RCLONE_S3_ENDPOINT
-- Provider: !AWS,ArvanCloud,IBMCOS,IDrive,IONOS,TencentCOS,HuaweiOBS,Alibaba,ChinaMobile,GCS,Liara,Scaleway,StackPath,Storj,Synology,RackCorp,Qiniu,Petabox
-- Type: string
-- Required: false
-- Examples:
-
-- "objects-us-east-1.dream.io"
-
-- Dream Objects endpoint
-
-- "syd1.digitaloceanspaces.com"
-
-- DigitalOcean Spaces Sydney 1
-
-- "sfo3.digitaloceanspaces.com"
-
-- DigitalOcean Spaces San Francisco 3
-
-- "fra1.digitaloceanspaces.com"
-
-- DigitalOcean Spaces Frankfurt 1
-
-- "nyc3.digitaloceanspaces.com"
-
-- DigitalOcean Spaces New York 3
-
-- "ams3.digitaloceanspaces.com"
-
-- DigitalOcean Spaces Amsterdam 3
-
-- "sgp1.digitaloceanspaces.com"
-
-- DigitalOcean Spaces Singapore 1
-
-- "localhost:8333"
-
-- SeaweedFS S3 localhost
-
-- "s3.us-east-1.lyvecloud.seagate.com"
-
-- Seagate Lyve Cloud US East 1 (Virginia)
-
-- "s3.us-west-1.lyvecloud.seagate.com"
-
-- Seagate Lyve Cloud US West 1 (California)
-
-- "s3.ap-southeast-1.lyvecloud.seagate.com"
-
-- Seagate Lyve Cloud AP Southeast 1 (Singapore)
-
-- "s3.wasabisys.com"
-
-- Wasabi US East 1 (N. Virginia)
-
-- "s3.us-east-2.wasabisys.com"
-
-- Wasabi US East 2 (N. Virginia)
-
-- "s3.us-central-1.wasabisys.com"
-
-- Wasabi US Central 1 (Texas)
-
-- "s3.us-west-1.wasabisys.com"
-
-- Wasabi US West 1 (Oregon)
-
-- "s3.ca-central-1.wasabisys.com"
-
-- Wasabi CA Central 1 (Toronto)
-
-- "s3.eu-central-1.wasabisys.com"
-
-- Wasabi EU Central 1 (Amsterdam)
-
-- "s3.eu-central-2.wasabisys.com"
-
-- Wasabi EU Central 2 (Frankfurt)
-
-- "s3.eu-west-1.wasabisys.com"
-
-- Wasabi EU West 1 (London)
-
-- "s3.eu-west-2.wasabisys.com"
-
-- Wasabi EU West 2 (Paris)
-
-- "s3.ap-northeast-1.wasabisys.com"
-
-- Wasabi AP Northeast 1 (Tokyo) endpoint
-
-- "s3.ap-northeast-2.wasabisys.com"
-
-- Wasabi AP Northeast 2 (Osaka) endpoint
-
-- "s3.ap-southeast-1.wasabisys.com"
-
-- Wasabi AP Southeast 1 (Singapore)
-
-- "s3.ap-southeast-2.wasabisys.com"
-
-- Wasabi AP Southeast 2 (Sydney)
-
-- "storage.iran.liara.space"
-
-- Liara Iran endpoint
-
-- "s3.ir-thr-at1.arvanstorage.ir"
-
-- ArvanCloud Tehran Iran (Simin) endpoint
-
-- "s3.ir-tbz-sh1.arvanstorage.ir"
-
-- ArvanCloud Tabriz Iran (Shahriar) endpoint
-
-
+- Config: decompress
+- Env Var: RCLONE_S3_DECOMPRESS
+- Type: bool
+- Default: false
---s3-location-constraint
-Location constraint - must be set to match the Region.
-Used when creating buckets only.
+--s3-might-gzip
+Set this if the backend might gzip objects.
+Normally providers will not alter objects when they are downloaded. If an object was not uploaded with Content-Encoding: gzip then it won't be set on download.
+However some providers may gzip objects even if they weren't uploaded with Content-Encoding: gzip (eg Cloudflare).
+A symptom of this would be receiving errors like
+ERROR corrupted on transfer: sizes differ NNN vs MMM
+If you set this flag and rclone downloads an object with Content-Encoding: gzip set and chunked transfer encoding, then rclone will decompress the object on the fly.
+If this is set to unset (the default) then rclone will choose according to the provider setting what to apply, but you can override rclone's choice here.
Properties:
-- Config: location_constraint
-- Env Var: RCLONE_S3_LOCATION_CONSTRAINT
-- Provider: AWS
-- Type: string
-- Required: false
-- Examples:
-
-- ""
-
-- Empty for US Region, Northern Virginia, or Pacific Northwest
-
-- "us-east-2"
-
-- US East (Ohio) Region
-
-- "us-west-1"
-
-- US West (Northern California) Region
-
-- "us-west-2"
-
-- US West (Oregon) Region
-
-- "ca-central-1"
-
-- Canada (Central) Region
-
-- "eu-west-1"
-
-- EU (Ireland) Region
-
-- "eu-west-2"
-
-- EU (London) Region
-
-- "eu-west-3"
-
-- EU (Paris) Region
-
-- "eu-north-1"
-
-- EU (Stockholm) Region
-
-- "eu-south-1"
-
-- EU (Milan) Region
-
-- "EU"
-
-- EU Region
-
-- "ap-southeast-1"
-
-- Asia Pacific (Singapore) Region
-
-- "ap-southeast-2"
-
-- Asia Pacific (Sydney) Region
-
-- "ap-northeast-1"
-
-- Asia Pacific (Tokyo) Region
-
-- "ap-northeast-2"
-
-- Asia Pacific (Seoul) Region
-
-- "ap-northeast-3"
-
-- Asia Pacific (Osaka-Local) Region
-
-- "ap-south-1"
-
-- Asia Pacific (Mumbai) Region
-
-- "ap-east-1"
-
-- Asia Pacific (Hong Kong) Region
-
-- "sa-east-1"
-
-- South America (Sao Paulo) Region
-
-- "me-south-1"
-
-- Middle East (Bahrain) Region
-
-- "af-south-1"
-
-- Africa (Cape Town) Region
-
-- "cn-north-1"
-
-- China (Beijing) Region
-
-- "cn-northwest-1"
-
-- China (Ningxia) Region
-
-- "us-gov-east-1"
-
-- AWS GovCloud (US-East) Region
-
-- "us-gov-west-1"
-
-- AWS GovCloud (US) Region
-
-
+- Config: might_gzip
+- Env Var: RCLONE_S3_MIGHT_GZIP
+- Type: Tristate
+- Default: unset
---s3-location-constraint
-Location constraint - must match endpoint.
-Used when creating buckets only.
+--s3-use-accept-encoding-gzip
+Whether to send Accept-Encoding: gzip header.
+By default, rclone will append Accept-Encoding: gzip to the request to download compressed objects whenever possible.
+However some providers such as Google Cloud Storage may alter the HTTP headers, breaking the signature of the request.
+A symptom of this would be receiving errors like
+SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided.
+In this case, you might want to try disabling this option.
Properties:
-- Config: location_constraint
-- Env Var: RCLONE_S3_LOCATION_CONSTRAINT
-- Provider: ChinaMobile
-- Type: string
-- Required: false
-- Examples:
-
-- "wuxi1"
-
-- East China (Suzhou)
-
-- "jinan1"
-
-- East China (Jinan)
-
-- "ningbo1"
-
-- East China (Hangzhou)
-
-- "shanghai1"
-
-- East China (Shanghai-1)
-
-- "zhengzhou1"
-
-- Central China (Zhengzhou)
-
-- "hunan1"
-
-- Central China (Changsha-1)
-
-- "zhuzhou1"
-
-- Central China (Changsha-2)
-
-- "guangzhou1"
-
-- South China (Guangzhou-2)
-
-- "dongguan1"
-
-- South China (Guangzhou-3)
-
-- "beijing1"
-
-- North China (Beijing-1)
-
-- "beijing2"
-
-- North China (Beijing-2)
-
-- "beijing4"
-
-- North China (Beijing-3)
-
-- "huhehaote1"
-
-- North China (Huhehaote)
-
-- "chengdu1"
-
-- Southwest China (Chengdu)
-
-- "chongqing1"
-
-- Southwest China (Chongqing)
-
-- "guiyang1"
-
-- Southwest China (Guiyang)
-
-- "xian1"
-
-- Nouthwest China (Xian)
-
-- "yunnan"
-
-- Yunnan China (Kunming)
-
-- "yunnan2"
-
-- Yunnan China (Kunming-2)
-
-- "tianjin1"
-
-- Tianjin China (Tianjin)
-
-- "jilin1"
-
-- Jilin China (Changchun)
-
-- "hubei1"
-
-- Hubei China (Xiangyan)
-
-- "jiangxi1"
-
-- Jiangxi China (Nanchang)
-
-- "gansu1"
-
-- Gansu China (Lanzhou)
-
-- "shanxi1"
-
-- Shanxi China (Taiyuan)
-
-- "liaoning1"
-
-- Liaoning China (Shenyang)
-
-- "hebei1"
-
-- Hebei China (Shijiazhuang)
-
-- "fujian1"
-
-- Fujian China (Xiamen)
-
-- "guangxi1"
-
-- Guangxi China (Nanning)
-
-- "anhui1"
-
-- Anhui China (Huainan)
-
-
+- Config: use_accept_encoding_gzip
+- Env Var: RCLONE_S3_USE_ACCEPT_ENCODING_GZIP
+- Type: Tristate
+- Default: unset
---s3-location-constraint
-Location constraint - must match endpoint.
-Used when creating buckets only.
+--s3-no-system-metadata
+Suppress setting and reading of system metadata
Properties:
-- Config: location_constraint
-- Env Var: RCLONE_S3_LOCATION_CONSTRAINT
-- Provider: ArvanCloud
-- Type: string
-- Required: false
-- Examples:
-
-- "ir-thr-at1"
-
-- Tehran Iran (Simin)
-
-- "ir-tbz-sh1"
-
-- Tabriz Iran (Shahriar)
-
-
+- Config: no_system_metadata
+- Env Var: RCLONE_S3_NO_SYSTEM_METADATA
+- Type: bool
+- Default: false
---s3-location-constraint
-Location constraint - must match endpoint when using IBM Cloud Public.
-For on-prem COS, do not make a selection from this list, hit enter.
+--s3-sts-endpoint
+Endpoint for STS.
+Leave blank if using AWS to use the default endpoint for the region.
Properties:
-- Config: location_constraint
-- Env Var: RCLONE_S3_LOCATION_CONSTRAINT
-- Provider: IBMCOS
+- Config: sts_endpoint
+- Env Var: RCLONE_S3_STS_ENDPOINT
+- Provider: AWS
- Type: string
- Required: false
-- Examples:
-
-- "us-standard"
-
-- US Cross Region Standard
-
-- "us-vault"
-
-- US Cross Region Vault
-
-- "us-cold"
-
-- US Cross Region Cold
-
-- "us-flex"
-
-- US Cross Region Flex
-
-- "us-east-standard"
-
-- US East Region Standard
-
-- "us-east-vault"
-
-- US East Region Vault
-
-- "us-east-cold"
-
-- US East Region Cold
-
-- "us-east-flex"
-
-- US East Region Flex
-
-- "us-south-standard"
-
-- US South Region Standard
-
-- "us-south-vault"
-
-- US South Region Vault
-
-- "us-south-cold"
-
-- US South Region Cold
-
-- "us-south-flex"
-
-- US South Region Flex
-
-- "eu-standard"
-
-- EU Cross Region Standard
-
-- "eu-vault"
-
-- EU Cross Region Vault
-
-- "eu-cold"
-
-- EU Cross Region Cold
-
-- "eu-flex"
-
-- EU Cross Region Flex
-
-- "eu-gb-standard"
-
-- Great Britain Standard
-
-- "eu-gb-vault"
-
-- Great Britain Vault
-
-- "eu-gb-cold"
-
-- Great Britain Cold
-
-- "eu-gb-flex"
-
-- Great Britain Flex
-
-- "ap-standard"
-
-- APAC Standard
-
-- "ap-vault"
-
-- APAC Vault
-
-- "ap-cold"
-
-- APAC Cold
-
-- "ap-flex"
-
-- APAC Flex
-
-- "mel01-standard"
-
-- Melbourne Standard
-
-- "mel01-vault"
-
-- Melbourne Vault
-
-- "mel01-cold"
+
+--s3-use-already-exists
+Set if rclone should report BucketAlreadyExists errors on bucket creation.
+At some point during the evolution of the s3 protocol, AWS started returning an AlreadyOwnedByYou error when attempting to create a bucket that the user already owned, rather than a BucketAlreadyExists error.
+Unfortunately exactly what has been implemented by s3 clones is a little inconsistent, some return AlreadyOwnedByYou, some return BucketAlreadyExists and some return no error at all.
+This is important to rclone because it ensures the bucket exists by creating it on quite a lot of operations (unless --s3-no-check-bucket is used).
+If rclone knows the provider can return AlreadyOwnedByYou or returns no error then it can report BucketAlreadyExists errors when the user attempts to create a bucket not owned by them. Otherwise rclone ignores the BucketAlreadyExists error which can lead to confusion.
+This should be automatically set correctly for all providers rclone knows about - please make a bug report if not.
+Properties:
-- Melbourne Cold
-
-- "mel01-flex"
+
- Config: use_already_exists
+- Env Var: RCLONE_S3_USE_ALREADY_EXISTS
+- Type: Tristate
+- Default: unset
+
+--s3-use-multipart-uploads
+Set if rclone should use multipart uploads.
+You can change this if you want to disable the use of multipart uploads. This shouldn't be necessary in normal operation.
+This should be automatically set correctly for all providers rclone knows about - please make a bug report if not.
+Properties:
-- Melbourne Flex
-
-"tor01-standard"
-
-- Toronto Standard
-
-"tor01-vault"
-
-- Toronto Vault
-
-"tor01-cold"
-
-- Toronto Cold
-
-"tor01-flex"
-
-- Toronto Flex
-
-
-
---s3-location-constraint
-Location constraint - the location where your bucket will be located and your data stored.
-Properties:
-
-- Config: location_constraint
-- Env Var: RCLONE_S3_LOCATION_CONSTRAINT
-- Provider: RackCorp
-- Type: string
-- Required: false
-- Examples:
-
-- "global"
-
-- Global CDN Region
-
-- "au"
-
-- Australia (All locations)
-
-- "au-nsw"
-
-- NSW (Australia) Region
-
-- "au-qld"
-
-- QLD (Australia) Region
-
-- "au-vic"
-
-- VIC (Australia) Region
-
-- "au-wa"
-
-- Perth (Australia) Region
-
-- "ph"
-
-- Manila (Philippines) Region
-
-- "th"
-
-- Bangkok (Thailand) Region
-
-- "hk"
-
-- HK (Hong Kong) Region
-
-- "mn"
-
-- Ulaanbaatar (Mongolia) Region
-
-- "kg"
-
-- Bishkek (Kyrgyzstan) Region
-
-- "id"
-
-- Jakarta (Indonesia) Region
-
-- "jp"
-
-- Tokyo (Japan) Region
-
-- "sg"
-
-- SG (Singapore) Region
-
-- "de"
-
-- Frankfurt (Germany) Region
-
-- "us"
-
-- USA (AnyCast) Region
-
-- "us-east-1"
-
-- New York (USA) Region
-
-- "us-west-1"
-
-- Freemont (USA) Region
-
-- "nz"
-
-- Auckland (New Zealand) Region
-
-
-
---s3-location-constraint
-Location constraint - must be set to match the Region.
-Used when creating buckets only.
-Properties:
-
-- Config: location_constraint
-- Env Var: RCLONE_S3_LOCATION_CONSTRAINT
-- Provider: Qiniu
-- Type: string
-- Required: false
-- Examples:
-
-- "cn-east-1"
-
-- East China Region 1
-
-- "cn-east-2"
-
-- East China Region 2
-
-- "cn-north-1"
-
-- North China Region 1
-
-- "cn-south-1"
-
-- South China Region 1
-
-- "us-north-1"
-
-- North America Region 1
-
-- "ap-southeast-1"
-
-- Southeast Asia Region 1
-
-- "ap-northeast-1"
-
-- Northeast Asia Region 1
-
-
-
---s3-location-constraint
-Location constraint - must be set to match the Region.
-Leave blank if not sure. Used when creating buckets only.
-Properties:
-
-- Config: location_constraint
-- Env Var: RCLONE_S3_LOCATION_CONSTRAINT
-- Provider: !AWS,Alibaba,ArvanCloud,HuaweiOBS,ChinaMobile,Cloudflare,IBMCOS,IDrive,IONOS,Leviia,Liara,Qiniu,RackCorp,Scaleway,StackPath,Storj,TencentCOS,Petabox
-- Type: string
-- Required: false
-
---s3-acl
-Canned ACL used when creating buckets and storing or copying objects.
-This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.
-For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
-Note that this ACL is applied when server-side copying objects as S3 doesn't copy the ACL from the source but rather writes a fresh one.
-If the acl is an empty string then no X-Amz-Acl: header is added and the default (private) will be used.
-Properties:
-
-- Config: acl
-- Env Var: RCLONE_S3_ACL
-- Provider: !Storj,Synology,Cloudflare
-- Type: string
-- Required: false
-- Examples:
-
-- "default"
-
-- Owner gets Full_CONTROL.
-- No one else has access rights (default).
-
-- "private"
-
-- Owner gets FULL_CONTROL.
-- No one else has access rights (default).
-
-- "public-read"
-
-- Owner gets FULL_CONTROL.
-- The AllUsers group gets READ access.
-
-- "public-read-write"
-
-- Owner gets FULL_CONTROL.
-- The AllUsers group gets READ and WRITE access.
-- Granting this on a bucket is generally not recommended.
-
-- "authenticated-read"
-
-- Owner gets FULL_CONTROL.
-- The AuthenticatedUsers group gets READ access.
-
-- "bucket-owner-read"
-
-- Object owner gets FULL_CONTROL.
-- Bucket owner gets READ access.
-- If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
-
-- "bucket-owner-full-control"
-
-- Both the object owner and the bucket owner get FULL_CONTROL over the object.
-- If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
-
-- "private"
-
-- Owner gets FULL_CONTROL.
-- No one else has access rights (default).
-- This acl is available on IBM Cloud (Infra), IBM Cloud (Storage), On-Premise COS.
-
-- "public-read"
-
-- Owner gets FULL_CONTROL.
-- The AllUsers group gets READ access.
-- This acl is available on IBM Cloud (Infra), IBM Cloud (Storage), On-Premise IBM COS.
-
-- "public-read-write"
-
-- Owner gets FULL_CONTROL.
-- The AllUsers group gets READ and WRITE access.
-- This acl is available on IBM Cloud (Infra), On-Premise IBM COS.
-
-- "authenticated-read"
-
-- Owner gets FULL_CONTROL.
-- The AuthenticatedUsers group gets READ access.
-- Not supported on Buckets.
-- This acl is available on IBM Cloud (Infra) and On-Premise IBM COS.
-
-
-
---s3-server-side-encryption
-The server-side encryption algorithm used when storing this object in S3.
-Properties:
-
-- Config: server_side_encryption
-- Env Var: RCLONE_S3_SERVER_SIDE_ENCRYPTION
-- Provider: AWS,Ceph,ChinaMobile,Minio
-- Type: string
-- Required: false
-- Examples:
-
-- ""
-
-- None
-
-- "AES256"
-
-- AES256
-
-- "aws:kms"
-
-- aws:kms
-
-
-
---s3-sse-kms-key-id
-If using KMS ID you must provide the ARN of Key.
-Properties:
-
-- Config: sse_kms_key_id
-- Env Var: RCLONE_S3_SSE_KMS_KEY_ID
-- Provider: AWS,Ceph,Minio
-- Type: string
-- Required: false
-- Examples:
-
-- ""
-
-- None
-
-- "arn:aws:kms:us-east-1:*"
-
-- arn:aws:kms:*
-
-
+- Config: use_multipart_uploads
+- Env Var: RCLONE_S3_USE_MULTIPART_UPLOADS
+- Type: Tristate
+- Default: unset
---s3-storage-class
-The storage class to use when storing new objects in S3.
-Properties:
-
-- Config: storage_class
-- Env Var: RCLONE_S3_STORAGE_CLASS
-- Provider: AWS
-- Type: string
-- Required: false
-- Examples:
-
-- ""
-
-- Default
-
-- "STANDARD"
-
-- Standard storage class
-
-- "REDUCED_REDUNDANCY"
-
-- Reduced redundancy storage class
-
-- "STANDARD_IA"
-
-- Standard Infrequent Access storage class
-
-- "ONEZONE_IA"
-
-- One Zone Infrequent Access storage class
-
-- "GLACIER"
-
-- Glacier storage class
-
-- "DEEP_ARCHIVE"
-
-- Glacier Deep Archive storage class
-
-- "INTELLIGENT_TIERING"
-
-- Intelligent-Tiering storage class
-
-- "GLACIER_IR"
+
Metadata
+User metadata is stored as x-amz-meta- keys. S3 metadata keys are case insensitive and are always returned in lower case.
+Here are the possible system metadata items for the s3 backend.
+
+
+
+
+
+Name
+Help
+Type
+Example
+Read Only
+
+
+
+
+btime
+Time of file birth (creation) read from Last-Modified header
+RFC 3339
+2006-01-02T15:04:05.999999999Z07:00
+Y
+
+
+cache-control
+Cache-Control header
+string
+no-cache
+N
+
+
+content-disposition
+Content-Disposition header
+string
+inline
+N
+
+
+content-encoding
+Content-Encoding header
+string
+gzip
+N
+
+
+content-language
+Content-Language header
+string
+en-US
+N
+
+
+content-type
+Content-Type header
+string
+text/plain
+N
+
+
+mtime
+Time of last modification, read from rclone metadata
+RFC 3339
+2006-01-02T15:04:05.999999999Z07:00
+N
+
+
+tier
+Tier of the object
+string
+GLACIER
+Y
+
+
+
+See the metadata docs for more info.
+Backend commands
+Here are the commands specific to the s3 backend.
+Run them with
+rclone backend COMMAND remote:
+The help below will explain what arguments each command takes.
+See the backend command for more info on how to pass options and arguments.
+These can be run on a running backend using the rc command backend/command.
+restore
+Restore objects from GLACIER to normal storage
+rclone backend restore remote: [options] [<arguments>+]
+This command can be used to restore one or more objects from GLACIER to normal storage.
+Usage Examples:
+rclone backend restore s3:bucket/path/to/object -o priority=PRIORITY -o lifetime=DAYS
+rclone backend restore s3:bucket/path/to/directory -o priority=PRIORITY -o lifetime=DAYS
+rclone backend restore s3:bucket -o priority=PRIORITY -o lifetime=DAYS
+This flag also obeys the filters. Test first with --interactive/-i or --dry-run flags
+rclone --interactive backend restore --include "*.txt" s3:bucket/path -o priority=Standard -o lifetime=1
+All the objects shown will be marked for restore, then
+rclone backend restore --include "*.txt" s3:bucket/path -o priority=Standard -o lifetime=1
+It returns a list of status dictionaries with Remote and Status keys. The Status will be OK if it was successful or an error message if not.
+[
+ {
+ "Status": "OK",
+ "Remote": "test.txt"
+ },
+ {
+ "Status": "OK",
+ "Remote": "test/file4.txt"
+ }
+]
+Options:
-- Glacier Instant Retrieval storage class
-
-
+- "description": The optional description for the job.
+- "lifetime": Lifetime of the active copy in days
+- "priority": Priority of restore: Standard|Expedited|Bulk
---s3-storage-class
-The storage class to use when storing new objects in OSS.
-Properties:
-
-- Config: storage_class
-- Env Var: RCLONE_S3_STORAGE_CLASS
-- Provider: Alibaba
-- Type: string
-- Required: false
-- Examples:
-
-- ""
-
-- Default
-
-- "STANDARD"
-
-- Standard storage class
-
-- "GLACIER"
-
-- Archive storage mode
-
-- "STANDARD_IA"
+
restore-status
+Show the restore status for objects being restored from GLACIER to normal storage
+rclone backend restore-status remote: [options] [<arguments>+]
+This command can be used to show the status for objects being restored from GLACIER to normal storage.
+Usage Examples:
+rclone backend restore-status s3:bucket/path/to/object
+rclone backend restore-status s3:bucket/path/to/directory
+rclone backend restore-status -o all s3:bucket/path/to/directory
+This command does not obey the filters.
+It returns a list of status dictionaries.
+[
+ {
+ "Remote": "file.txt",
+ "VersionID": null,
+ "RestoreStatus": {
+ "IsRestoreInProgress": true,
+ "RestoreExpiryDate": "2023-09-06T12:29:19+01:00"
+ },
+ "StorageClass": "GLACIER"
+ },
+ {
+ "Remote": "test.pdf",
+ "VersionID": null,
+ "RestoreStatus": {
+ "IsRestoreInProgress": false,
+ "RestoreExpiryDate": "2023-09-06T12:29:19+01:00"
+ },
+ "StorageClass": "DEEP_ARCHIVE"
+ }
+]
+Options:
-- Infrequent access storage mode
-
-
+- "all": if set then show all objects, not just ones with restore status
---s3-storage-class
-The storage class to use when storing new objects in ChinaMobile.
-Properties:
-
-- Config: storage_class
-- Env Var: RCLONE_S3_STORAGE_CLASS
-- Provider: ChinaMobile
-- Type: string
-- Required: false
-- Examples:
-
-- ""
-
-- Default
-
-- "STANDARD"
-
-- Standard storage class
-
-- "GLACIER"
-
-- Archive storage mode
-
-- "STANDARD_IA"
+
list-multipart-uploads
+List the unfinished multipart uploads
+rclone backend list-multipart-uploads remote: [options] [<arguments>+]
+This command lists the unfinished multipart uploads in JSON format.
+rclone backend list-multipart s3:bucket/path/to/object
+It returns a dictionary of buckets with values as lists of unfinished multipart uploads.
+You can call it with no bucket in which case it lists all bucket, with a bucket or with a bucket and path.
+{
+ "rclone": [
+ {
+ "Initiated": "2020-06-26T14:20:36Z",
+ "Initiator": {
+ "DisplayName": "XXX",
+ "ID": "arn:aws:iam::XXX:user/XXX"
+ },
+ "Key": "KEY",
+ "Owner": {
+ "DisplayName": null,
+ "ID": "XXX"
+ },
+ "StorageClass": "STANDARD",
+ "UploadId": "XXX"
+ }
+ ],
+ "rclone-1000files": [],
+ "rclone-dst": []
+}
+cleanup
+Remove unfinished multipart uploads.
+rclone backend cleanup remote: [options] [<arguments>+]
+This command removes unfinished multipart uploads of age greater than max-age which defaults to 24 hours.
+Note that you can use --interactive/-i or --dry-run with this command to see what it would do.
+rclone backend cleanup s3:bucket/path/to/object
+rclone backend cleanup -o max-age=7w s3:bucket/path/to/object
+Durations are parsed as per the rest of rclone, 2h, 7d, 7w etc.
+Options:
-- Infrequent access storage mode
-
-
+- "max-age": Max age of upload to delete
---s3-storage-class
-The storage class to use when storing new objects in Liara
-Properties:
-
-- Config: storage_class
-- Env Var: RCLONE_S3_STORAGE_CLASS
-- Provider: Liara
-- Type: string
-- Required: false
-- Examples:
-
-- "STANDARD"
-
-- Standard storage class
-
-
-
---s3-storage-class
-The storage class to use when storing new objects in ArvanCloud.
-Properties:
-
-- Config: storage_class
-- Env Var: RCLONE_S3_STORAGE_CLASS
-- Provider: ArvanCloud
-- Type: string
-- Required: false
-- Examples:
-
-- "STANDARD"
-
-- Standard storage class
-
-
-
---s3-storage-class
-The storage class to use when storing new objects in Tencent COS.
-Properties:
-
-- Config: storage_class
-- Env Var: RCLONE_S3_STORAGE_CLASS
-- Provider: TencentCOS
-- Type: string
-- Required: false
-- Examples:
-
-- ""
-
-- Default
-
-- "STANDARD"
-
-- Standard storage class
-
-- "ARCHIVE"
-
-- Archive storage mode
-
-- "STANDARD_IA"
-
-- Infrequent access storage mode
-
-
-
---s3-storage-class
-The storage class to use when storing new objects in S3.
-Properties:
-
-- Config: storage_class
-- Env Var: RCLONE_S3_STORAGE_CLASS
-- Provider: Scaleway
-- Type: string
-- Required: false
-- Examples:
-
-- ""
-
-- Default.
-
-- "STANDARD"
-
-- The Standard class for any upload.
-- Suitable for on-demand content like streaming or CDN.
-- Available in all regions.
-
-- "GLACIER"
-
-- Archived storage.
-- Prices are lower, but it needs to be restored first to be accessed.
-- Available in FR-PAR and NL-AMS regions.
-
-- "ONEZONE_IA"
-
-- One Zone - Infrequent Access.
-- A good choice for storing secondary backup copies or easily re-creatable data.
-- Available in the FR-PAR region only.
-
-
-
---s3-storage-class
-The storage class to use when storing new objects in Qiniu.
-Properties:
-
-- Config: storage_class
-- Env Var: RCLONE_S3_STORAGE_CLASS
-- Provider: Qiniu
-- Type: string
-- Required: false
-- Examples:
-
-- "STANDARD"
-
-- Standard storage class
-
-- "LINE"
-
-- Infrequent access storage mode
-
-- "GLACIER"
-
-- Archive storage mode
-
-- "DEEP_ARCHIVE"
-
-- Deep archive storage mode
-
-
-
-Advanced options
-Here are the Advanced options specific to s3 (Amazon S3 Compliant Storage Providers including AWS, Alibaba, ArvanCloud, Ceph, China Mobile, Cloudflare, GCS, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, IDrive e2, IONOS Cloud, Leviia, Liara, Lyve Cloud, Minio, Netease, Petabox, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Synology, Tencent COS, Qiniu and Wasabi).
---s3-bucket-acl
-Canned ACL used when creating buckets.
-For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
-Note that this ACL is applied when only when creating buckets. If it isn't set then "acl" is used instead.
-If the "acl" and "bucket_acl" are empty strings then no X-Amz-Acl: header is added and the default (private) will be used.
-Properties:
-
-- Config: bucket_acl
-- Env Var: RCLONE_S3_BUCKET_ACL
-- Type: string
-- Required: false
-- Examples:
-
-- "private"
-
-- Owner gets FULL_CONTROL.
-- No one else has access rights (default).
-
-- "public-read"
-
-- Owner gets FULL_CONTROL.
-- The AllUsers group gets READ access.
-
-- "public-read-write"
-
-- Owner gets FULL_CONTROL.
-- The AllUsers group gets READ and WRITE access.
-- Granting this on a bucket is generally not recommended.
-
-- "authenticated-read"
-
-- Owner gets FULL_CONTROL.
-- The AuthenticatedUsers group gets READ access.
-
-
-
---s3-requester-pays
-Enables requester pays option when interacting with S3 bucket.
-Properties:
-
-- Config: requester_pays
-- Env Var: RCLONE_S3_REQUESTER_PAYS
-- Provider: AWS
-- Type: bool
-- Default: false
-
---s3-sse-customer-algorithm
-If using SSE-C, the server-side encryption algorithm used when storing this object in S3.
-Properties:
-
-- Config: sse_customer_algorithm
-- Env Var: RCLONE_S3_SSE_CUSTOMER_ALGORITHM
-- Provider: AWS,Ceph,ChinaMobile,Minio
-- Type: string
-- Required: false
-- Examples:
-
-- ""
-
-- None
-
-- "AES256"
-
-- AES256
-
-
-
---s3-sse-customer-key
-To use SSE-C you may provide the secret encryption key used to encrypt/decrypt your data.
-Alternatively you can provide --sse-customer-key-base64.
-Properties:
-
-- Config: sse_customer_key
-- Env Var: RCLONE_S3_SSE_CUSTOMER_KEY
-- Provider: AWS,Ceph,ChinaMobile,Minio
-- Type: string
-- Required: false
-- Examples:
-
-- ""
-
-- None
-
-
-
---s3-sse-customer-key-base64
-If using SSE-C you must provide the secret encryption key encoded in base64 format to encrypt/decrypt your data.
-Alternatively you can provide --sse-customer-key.
-Properties:
-
-- Config: sse_customer_key_base64
-- Env Var: RCLONE_S3_SSE_CUSTOMER_KEY_BASE64
-- Provider: AWS,Ceph,ChinaMobile,Minio
-- Type: string
-- Required: false
-- Examples:
-
-- ""
-
-- None
-
-
-
---s3-sse-customer-key-md5
-If using SSE-C you may provide the secret encryption key MD5 checksum (optional).
-If you leave it blank, this is calculated automatically from the sse_customer_key provided.
-Properties:
-
-- Config: sse_customer_key_md5
-- Env Var: RCLONE_S3_SSE_CUSTOMER_KEY_MD5
-- Provider: AWS,Ceph,ChinaMobile,Minio
-- Type: string
-- Required: false
-- Examples:
-
-- ""
-
-- None
-
-
-
---s3-upload-cutoff
-Cutoff for switching to chunked upload.
-Any files larger than this will be uploaded in chunks of chunk_size. The minimum is 0 and the maximum is 5 GiB.
-Properties:
-
-- Config: upload_cutoff
-- Env Var: RCLONE_S3_UPLOAD_CUTOFF
-- Type: SizeSuffix
-- Default: 200Mi
-
---s3-chunk-size
-Chunk size to use for uploading.
-When uploading files larger than upload_cutoff or files with unknown size (e.g. from "rclone rcat" or uploaded with "rclone mount" or google photos or google docs) they will be uploaded as multipart uploads using this chunk size.
-Note that "--s3-upload-concurrency" chunks of this size are buffered in memory per transfer.
-If you are transferring large files over high-speed links and you have enough memory, then increasing this will speed up the transfers.
-Rclone will automatically increase the chunk size when uploading a large file of known size to stay below the 10,000 chunks limit.
-Files of unknown size are uploaded with the configured chunk_size. Since the default chunk size is 5 MiB and there can be at most 10,000 chunks, this means that by default the maximum size of a file you can stream upload is 48 GiB. If you wish to stream upload larger files then you will need to increase chunk_size.
-Increasing the chunk size decreases the accuracy of the progress statistics displayed with "-P" flag. Rclone treats chunk as sent when it's buffered by the AWS SDK, when in fact it may still be uploading. A bigger chunk size means a bigger AWS SDK buffer and progress reporting more deviating from the truth.
-Properties:
-
-- Config: chunk_size
-- Env Var: RCLONE_S3_CHUNK_SIZE
-- Type: SizeSuffix
-- Default: 5Mi
-
---s3-max-upload-parts
-Maximum number of parts in a multipart upload.
-This option defines the maximum number of multipart chunks to use when doing a multipart upload.
-This can be useful if a service does not support the AWS S3 specification of 10,000 chunks.
-Rclone will automatically increase the chunk size when uploading a large file of a known size to stay below this number of chunks limit.
-Properties:
-
-- Config: max_upload_parts
-- Env Var: RCLONE_S3_MAX_UPLOAD_PARTS
-- Type: int
-- Default: 10000
-
---s3-copy-cutoff
-Cutoff for switching to multipart copy.
-Any files larger than this that need to be server-side copied will be copied in chunks of this size.
-The minimum is 0 and the maximum is 5 GiB.
-Properties:
-
-- Config: copy_cutoff
-- Env Var: RCLONE_S3_COPY_CUTOFF
-- Type: SizeSuffix
-- Default: 4.656Gi
-
---s3-disable-checksum
-Don't store MD5 checksum with object metadata.
-Normally rclone will calculate the MD5 checksum of the input before uploading it so it can add it to metadata on the object. This is great for data integrity checking but can cause long delays for large files to start uploading.
-Properties:
-
-- Config: disable_checksum
-- Env Var: RCLONE_S3_DISABLE_CHECKSUM
-- Type: bool
-- Default: false
-
---s3-shared-credentials-file
-Path to the shared credentials file.
-If env_auth = true then rclone can use a shared credentials file.
-If this variable is empty rclone will look for the "AWS_SHARED_CREDENTIALS_FILE" env variable. If the env value is empty it will default to the current user's home directory.
-Linux/OSX: "$HOME/.aws/credentials"
-Windows: "%USERPROFILE%\.aws\credentials"
-Properties:
-
-- Config: shared_credentials_file
-- Env Var: RCLONE_S3_SHARED_CREDENTIALS_FILE
-- Type: string
-- Required: false
-
---s3-profile
-Profile to use in the shared credentials file.
-If env_auth = true then rclone can use a shared credentials file. This variable controls which profile is used in that file.
-If empty it will default to the environment variable "AWS_PROFILE" or "default" if that environment variable is also not set.
-Properties:
-
-- Config: profile
-- Env Var: RCLONE_S3_PROFILE
-- Type: string
-- Required: false
-
---s3-session-token
-An AWS session token.
-Properties:
-
-- Config: session_token
-- Env Var: RCLONE_S3_SESSION_TOKEN
-- Type: string
-- Required: false
-
---s3-upload-concurrency
-Concurrency for multipart uploads.
-This is the number of chunks of the same file that are uploaded concurrently.
-If you are uploading small numbers of large files over high-speed links and these uploads do not fully utilize your bandwidth, then increasing this may help to speed up the transfers.
-Properties:
-
-- Config: upload_concurrency
-- Env Var: RCLONE_S3_UPLOAD_CONCURRENCY
-- Type: int
-- Default: 4
-
---s3-force-path-style
-If true use path style access if false use virtual hosted style.
-If this is true (the default) then rclone will use path style access, if false then rclone will use virtual path style. See the AWS S3 docs for more info.
-Some providers (e.g. AWS, Aliyun OSS, Netease COS, or Tencent COS) require this set to false - rclone will do this automatically based on the provider setting.
-Properties:
-
-- Config: force_path_style
-- Env Var: RCLONE_S3_FORCE_PATH_STYLE
-- Type: bool
-- Default: true
-
---s3-v2-auth
-If true use v2 authentication.
-If this is false (the default) then rclone will use v4 authentication. If it is set then rclone will use v2 authentication.
-Use this only if v4 signatures don't work, e.g. pre Jewel/v10 CEPH.
-Properties:
-
-- Config: v2_auth
-- Env Var: RCLONE_S3_V2_AUTH
-- Type: bool
-- Default: false
-
---s3-use-accelerate-endpoint
-If true use the AWS S3 accelerated endpoint.
-See: AWS S3 Transfer acceleration
-Properties:
-
-- Config: use_accelerate_endpoint
-- Env Var: RCLONE_S3_USE_ACCELERATE_ENDPOINT
-- Provider: AWS
-- Type: bool
-- Default: false
-
---s3-leave-parts-on-error
-If true avoid calling abort upload on a failure, leaving all successfully uploaded parts on S3 for manual recovery.
-It should be set to true for resuming uploads across different sessions.
-WARNING: Storing parts of an incomplete multipart upload counts towards space usage on S3 and will add additional costs if not cleaned up.
-Properties:
-
-- Config: leave_parts_on_error
-- Env Var: RCLONE_S3_LEAVE_PARTS_ON_ERROR
-- Provider: AWS
-- Type: bool
-- Default: false
-
---s3-list-chunk
-Size of listing chunk (response list for each ListObject S3 request).
-This option is also known as "MaxKeys", "max-items", or "page-size" from the AWS S3 specification. Most services truncate the response list to 1000 objects even if requested more than that. In AWS S3 this is a global maximum and cannot be changed, see AWS S3. In Ceph, this can be increased with the "rgw list buckets max chunk" option.
-Properties:
-
-- Config: list_chunk
-- Env Var: RCLONE_S3_LIST_CHUNK
-- Type: int
-- Default: 1000
-
---s3-list-version
-Version of ListObjects to use: 1,2 or 0 for auto.
-When S3 originally launched it only provided the ListObjects call to enumerate objects in a bucket.
-However in May 2016 the ListObjectsV2 call was introduced. This is much higher performance and should be used if at all possible.
-If set to the default, 0, rclone will guess according to the provider set which list objects method to call. If it guesses wrong, then it may be set manually here.
-Properties:
-
-- Config: list_version
-- Env Var: RCLONE_S3_LIST_VERSION
-- Type: int
-- Default: 0
-
---s3-list-url-encode
-Whether to url encode listings: true/false/unset
-Some providers support URL encoding listings and where this is available this is more reliable when using control characters in file names. If this is set to unset (the default) then rclone will choose according to the provider setting what to apply, but you can override rclone's choice here.
-Properties:
-
-- Config: list_url_encode
-- Env Var: RCLONE_S3_LIST_URL_ENCODE
-- Type: Tristate
-- Default: unset
-
---s3-no-check-bucket
-If set, don't attempt to check the bucket exists or create it.
-This can be useful when trying to minimise the number of transactions rclone does if you know the bucket exists already.
-It can also be needed if the user you are using does not have bucket creation permissions. Before v1.52.0 this would have passed silently due to a bug.
-Properties:
-
-- Config: no_check_bucket
-- Env Var: RCLONE_S3_NO_CHECK_BUCKET
-- Type: bool
-- Default: false
-
---s3-no-head
-If set, don't HEAD uploaded objects to check integrity.
-This can be useful when trying to minimise the number of transactions rclone does.
-Setting it means that if rclone receives a 200 OK message after uploading an object with PUT then it will assume that it got uploaded properly.
-In particular it will assume:
-
-- the metadata, including modtime, storage class and content type was as uploaded
-- the size was as uploaded
-
-It reads the following items from the response for a single part PUT:
-
-- the MD5SUM
-- The uploaded date
-
-For multipart uploads these items aren't read.
-If an source object of unknown length is uploaded then rclone will do a HEAD request.
-Setting this flag increases the chance for undetected upload failures, in particular an incorrect size, so it isn't recommended for normal operation. In practice the chance of an undetected upload failure is very small even with this flag.
-Properties:
-
-- Config: no_head
-- Env Var: RCLONE_S3_NO_HEAD
-- Type: bool
-- Default: false
-
---s3-no-head-object
-If set, do not do HEAD before GET when getting objects.
-Properties:
-
-- Config: no_head_object
-- Env Var: RCLONE_S3_NO_HEAD_OBJECT
-- Type: bool
-- Default: false
-
---s3-encoding
-The encoding for the backend.
-See the encoding section in the overview for more info.
-Properties:
-
-- Config: encoding
-- Env Var: RCLONE_S3_ENCODING
-- Type: MultiEncoder
-- Default: Slash,InvalidUtf8,Dot
-
---s3-memory-pool-flush-time
-How often internal memory buffer pools will be flushed. (no longer used)
-Properties:
-
-- Config: memory_pool_flush_time
-- Env Var: RCLONE_S3_MEMORY_POOL_FLUSH_TIME
-- Type: Duration
-- Default: 1m0s
-
---s3-memory-pool-use-mmap
-Whether to use mmap buffers in internal memory pool. (no longer used)
-Properties:
-
-- Config: memory_pool_use_mmap
-- Env Var: RCLONE_S3_MEMORY_POOL_USE_MMAP
-- Type: bool
-- Default: false
-
---s3-disable-http2
-Disable usage of http2 for S3 backends.
-There is currently an unsolved issue with the s3 (specifically minio) backend and HTTP/2. HTTP/2 is enabled by default for the s3 backend but can be disabled here. When the issue is solved this flag will be removed.
-See: https://github.com/rclone/rclone/issues/4673, https://github.com/rclone/rclone/issues/3631
-Properties:
-
-- Config: disable_http2
-- Env Var: RCLONE_S3_DISABLE_HTTP2
-- Type: bool
-- Default: false
-
---s3-download-url
-Custom endpoint for downloads. This is usually set to a CloudFront CDN URL as AWS S3 offers cheaper egress for data downloaded through the CloudFront network.
-Properties:
-
-- Config: download_url
-- Env Var: RCLONE_S3_DOWNLOAD_URL
-- Type: string
-- Required: false
-
---s3-directory-markers
-Upload an empty object with a trailing slash when a new directory is created
-Empty folders are unsupported for bucket based remotes, this option creates an empty object ending with "/", to persist the folder.
-Properties:
-
-- Config: directory_markers
-- Env Var: RCLONE_S3_DIRECTORY_MARKERS
-- Type: bool
-- Default: false
-
---s3-use-multipart-etag
-Whether to use ETag in multipart uploads for verification
-This should be true, false or left unset to use the default for the provider.
-Properties:
-
-- Config: use_multipart_etag
-- Env Var: RCLONE_S3_USE_MULTIPART_ETAG
-- Type: Tristate
-- Default: unset
-
---s3-use-presigned-request
-Whether to use a presigned request or PutObject for single part uploads
-If this is false rclone will use PutObject from the AWS SDK to upload an object.
-Versions of rclone < 1.59 use presigned requests to upload a single part object and setting this flag to true will re-enable that functionality. This shouldn't be necessary except in exceptional circumstances or for testing.
-Properties:
-
-- Config: use_presigned_request
-- Env Var: RCLONE_S3_USE_PRESIGNED_REQUEST
-- Type: bool
-- Default: false
-
---s3-versions
-Include old versions in directory listings.
-Properties:
-
-- Config: versions
-- Env Var: RCLONE_S3_VERSIONS
-- Type: bool
-- Default: false
-
---s3-version-at
-Show file versions as they were at the specified time.
-The parameter should be a date, "2006-01-02", datetime "2006-01-02 15:04:05" or a duration for that long ago, eg "100d" or "1h".
-Note that when using this no file write operations are permitted, so you can't upload files or delete them.
-See the time option docs for valid formats.
-Properties:
-
-- Config: version_at
-- Env Var: RCLONE_S3_VERSION_AT
-- Type: Time
-- Default: off
-
---s3-decompress
-If set this will decompress gzip encoded objects.
-It is possible to upload objects to S3 with "Content-Encoding: gzip" set. Normally rclone will download these files as compressed objects.
-If this flag is set then rclone will decompress these files with "Content-Encoding: gzip" as they are received. This means that rclone can't check the size and hash but the file contents will be decompressed.
-Properties:
-
-- Config: decompress
-- Env Var: RCLONE_S3_DECOMPRESS
-- Type: bool
-- Default: false
-
---s3-might-gzip
-Set this if the backend might gzip objects.
-Normally providers will not alter objects when they are downloaded. If an object was not uploaded with Content-Encoding: gzip then it won't be set on download.
-However some providers may gzip objects even if they weren't uploaded with Content-Encoding: gzip (eg Cloudflare).
-A symptom of this would be receiving errors like
-ERROR corrupted on transfer: sizes differ NNN vs MMM
-If you set this flag and rclone downloads an object with Content-Encoding: gzip set and chunked transfer encoding, then rclone will decompress the object on the fly.
-If this is set to unset (the default) then rclone will choose according to the provider setting what to apply, but you can override rclone's choice here.
-Properties:
-
-- Config: might_gzip
-- Env Var: RCLONE_S3_MIGHT_GZIP
-- Type: Tristate
-- Default: unset
-
---s3-use-accept-encoding-gzip
-Whether to send Accept-Encoding: gzip header.
-By default, rclone will append Accept-Encoding: gzip to the request to download compressed objects whenever possible.
-However some providers such as Google Cloud Storage may alter the HTTP headers, breaking the signature of the request.
-A symptom of this would be receiving errors like
-SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided.
-In this case, you might want to try disabling this option.
-Properties:
-
-- Config: use_accept_encoding_gzip
-- Env Var: RCLONE_S3_USE_ACCEPT_ENCODING_GZIP
-- Type: Tristate
-- Default: unset
-
---s3-no-system-metadata
-Suppress setting and reading of system metadata
-Properties:
-
-- Config: no_system_metadata
-- Env Var: RCLONE_S3_NO_SYSTEM_METADATA
-- Type: bool
-- Default: false
-
---s3-sts-endpoint
-Endpoint for STS.
-Leave blank if using AWS to use the default endpoint for the region.
-Properties:
-
-- Config: sts_endpoint
-- Env Var: RCLONE_S3_STS_ENDPOINT
-- Provider: AWS
-- Type: string
-- Required: false
-
-Metadata
-User metadata is stored as x-amz-meta- keys. S3 metadata keys are case insensitive and are always returned in lower case.
-Here are the possible system metadata items for the s3 backend.
-
-
-
-
-
-Name
-Help
-Type
-Example
-Read Only
-
-
-
-
-btime
-Time of file birth (creation) read from Last-Modified header
-RFC 3339
-2006-01-02T15:04:05.999999999Z07:00
-Y
-
-
-cache-control
-Cache-Control header
-string
-no-cache
-N
-
-
-content-disposition
-Content-Disposition header
-string
-inline
-N
-
-
-content-encoding
-Content-Encoding header
-string
-gzip
-N
-
-
-content-language
-Content-Language header
-string
-en-US
-N
-
-
-content-type
-Content-Type header
-string
-text/plain
-N
-
-
-mtime
-Time of last modification, read from rclone metadata
-RFC 3339
-2006-01-02T15:04:05.999999999Z07:00
-N
-
-
-tier
-Tier of the object
-string
-GLACIER
-Y
-
-
-
-See the metadata docs for more info.
-Backend commands
-Here are the commands specific to the s3 backend.
-Run them with
-rclone backend COMMAND remote:
-The help below will explain what arguments each command takes.
-See the backend command for more info on how to pass options and arguments.
-These can be run on a running backend using the rc command backend/command.
-restore
-Restore objects from GLACIER to normal storage
-rclone backend restore remote: [options] [<arguments>+]
-This command can be used to restore one or more objects from GLACIER to normal storage.
-Usage Examples:
-rclone backend restore s3:bucket/path/to/object -o priority=PRIORITY -o lifetime=DAYS
-rclone backend restore s3:bucket/path/to/directory -o priority=PRIORITY -o lifetime=DAYS
-rclone backend restore s3:bucket -o priority=PRIORITY -o lifetime=DAYS
-This flag also obeys the filters. Test first with --interactive/-i or --dry-run flags
-rclone --interactive backend restore --include "*.txt" s3:bucket/path -o priority=Standard -o lifetime=1
-All the objects shown will be marked for restore, then
-rclone backend restore --include "*.txt" s3:bucket/path -o priority=Standard -o lifetime=1
-It returns a list of status dictionaries with Remote and Status keys. The Status will be OK if it was successful or an error message if not.
-[
- {
- "Status": "OK",
- "Remote": "test.txt"
- },
- {
- "Status": "OK",
- "Remote": "test/file4.txt"
- }
-]
-Options:
-
-- "description": The optional description for the job.
-- "lifetime": Lifetime of the active copy in days
-- "priority": Priority of restore: Standard|Expedited|Bulk
-
-restore-status
-Show the restore status for objects being restored from GLACIER to normal storage
-rclone backend restore-status remote: [options] [<arguments>+]
-This command can be used to show the status for objects being restored from GLACIER to normal storage.
+cleanup-hidden
+Remove old versions of files.
+rclone backend cleanup-hidden remote: [options] [<arguments>+]
+This command removes any old hidden versions of files on a versions enabled bucket.
+Note that you can use --interactive/-i or --dry-run with this command to see what it would do.
+rclone backend cleanup-hidden s3:bucket/path/to/dir
+versioning
+Set/get versioning support for a bucket.
+rclone backend versioning remote: [options] [<arguments>+]
+This command sets versioning support if a parameter is passed and then returns the current versioning status for the bucket supplied.
+rclone backend versioning s3:bucket # read status only
+rclone backend versioning s3:bucket Enabled
+rclone backend versioning s3:bucket Suspended
+It may return "Enabled", "Suspended" or "Unversioned". Note that once versioning has been enabled the status can't be set back to "Unversioned".
+set
+Set command for updating the config parameters.
+rclone backend set remote: [options] [<arguments>+]
+This set command can be used to update the config parameters for a running s3 backend.
Usage Examples:
-rclone backend restore-status s3:bucket/path/to/object
-rclone backend restore-status s3:bucket/path/to/directory
-rclone backend restore-status -o all s3:bucket/path/to/directory
-This command does not obey the filters.
-It returns a list of status dictionaries.
-[
- {
- "Remote": "file.txt",
- "VersionID": null,
- "RestoreStatus": {
- "IsRestoreInProgress": true,
- "RestoreExpiryDate": "2023-09-06T12:29:19+01:00"
- },
- "StorageClass": "GLACIER"
- },
- {
- "Remote": "test.pdf",
- "VersionID": null,
- "RestoreStatus": {
- "IsRestoreInProgress": false,
- "RestoreExpiryDate": "2023-09-06T12:29:19+01:00"
- },
- "StorageClass": "DEEP_ARCHIVE"
- }
-]
-Options:
-
-- "all": if set then show all objects, not just ones with restore status
-
-list-multipart-uploads
-List the unfinished multipart uploads
-rclone backend list-multipart-uploads remote: [options] [<arguments>+]
-This command lists the unfinished multipart uploads in JSON format.
-rclone backend list-multipart s3:bucket/path/to/object
-It returns a dictionary of buckets with values as lists of unfinished multipart uploads.
-You can call it with no bucket in which case it lists all bucket, with a bucket or with a bucket and path.
+rclone backend set s3: [-o opt_name=opt_value] [-o opt_name2=opt_value2]
+rclone rc backend/command command=set fs=s3: [-o opt_name=opt_value] [-o opt_name2=opt_value2]
+rclone rc backend/command command=set fs=s3: -o session_token=X -o access_key_id=X -o secret_access_key=X
+The option keys are named as they are in the config file.
+This rebuilds the connection to the s3 backend when it is called with the new parameters. Only new parameters need be passed as the values will default to those currently in use.
+It doesn't return anything.
+Anonymous access to public buckets
+If you want to use rclone to access a public bucket, configure with a blank access_key_id and secret_access_key. Your config should end up looking like this:
+[anons3]
+type = s3
+provider = AWS
+env_auth = false
+access_key_id =
+secret_access_key =
+region = us-east-1
+endpoint =
+location_constraint =
+acl = private
+server_side_encryption =
+storage_class =
+Then use it as normal with the name of the public bucket, e.g.
+rclone lsd anons3:1000genomes
+You will be able to list and copy data but not upload it.
+Providers
+AWS S3
+This is the provider used as main example and described in the configuration section above.
+AWS Snowball Edge
+AWS Snowball is a hardware appliance used for transferring bulk data back to AWS. Its main software interface is S3 object storage.
+To use rclone with AWS Snowball Edge devices, configure as standard for an 'S3 Compatible Service'.
+If using rclone pre v1.59 be sure to set upload_cutoff = 0 otherwise you will run into authentication header issues as the snowball device does not support query parameter based authentication.
+With rclone v1.59 or later setting upload_cutoff should not be necessary.
+eg.
+[snowball]
+type = s3
+provider = Other
+access_key_id = YOUR_ACCESS_KEY
+secret_access_key = YOUR_SECRET_KEY
+endpoint = http://[IP of Snowball]:8080
+upload_cutoff = 0
+Ceph
+Ceph is an open-source, unified, distributed storage system designed for excellent performance, reliability and scalability. It has an S3 compatible object storage interface.
+To use rclone with Ceph, configure as above but leave the region blank and set the endpoint. You should end up with something like this in your config:
+[ceph]
+type = s3
+provider = Ceph
+env_auth = false
+access_key_id = XXX
+secret_access_key = YYY
+region =
+endpoint = https://ceph.endpoint.example.com
+location_constraint =
+acl =
+server_side_encryption =
+storage_class =
+If you are using an older version of CEPH (e.g. 10.2.x Jewel) and a version of rclone before v1.59 then you may need to supply the parameter --s3-upload-cutoff 0 or put this in the config file as upload_cutoff 0 to work around a bug which causes uploading of small files to fail.
+Note also that Ceph sometimes puts / in the passwords it gives users. If you read the secret access key using the command line tools you will get a JSON blob with the / escaped as \/. Make sure you only write / in the secret access key.
+Eg the dump from Ceph looks something like this (irrelevant keys removed).
{
- "rclone": [
+ "user_id": "xxx",
+ "display_name": "xxxx",
+ "keys": [
+ {
+ "user": "xxx",
+ "access_key": "xxxxxx",
+ "secret_key": "xxxxxx\/xxxx"
+ }
+ ],
+}
+Because this is a json dump, it is encoding the / as \/, so if you use the secret key as xxxxxx/xxxx it will work fine.
+Cloudflare R2
+Cloudflare R2 Storage allows developers to store large amounts of unstructured data without the costly egress bandwidth fees associated with typical cloud storage services.
+Here is an example of making a Cloudflare R2 configuration. First run:
+rclone config
+This will guide you through an interactive setup process.
+Note that all buckets are private, and all are stored in the same "auto" region. It is necessary to use Cloudflare workers to share the content of a bucket publicly.
+No remotes found, make a new one?
+n) New remote
+s) Set configuration password
+q) Quit config
+n/s/q> n
+name> r2
+Option Storage.
+Type of storage to configure.
+Choose a number from below, or type in your own value.
+...
+XX / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, China Mobile, Cloudflare, ArvanCloud, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, Lyve Cloud, Minio, Netease, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Synology, Tencent COS and Wasabi
+ \ (s3)
+...
+Storage> s3
+Option provider.
+Choose your S3 provider.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+...
+XX / Cloudflare R2 Storage
+ \ (Cloudflare)
+...
+provider> Cloudflare
+Option env_auth.
+Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
+Only applies if access_key_id and secret_access_key is blank.
+Choose a number from below, or type in your own boolean value (true or false).
+Press Enter for the default (false).
+ 1 / Enter AWS credentials in the next step.
+ \ (false)
+ 2 / Get AWS credentials from the environment (env vars or IAM).
+ \ (true)
+env_auth> 1
+Option access_key_id.
+AWS Access Key ID.
+Leave blank for anonymous access or runtime credentials.
+Enter a value. Press Enter to leave empty.
+access_key_id> ACCESS_KEY
+Option secret_access_key.
+AWS Secret Access Key (password).
+Leave blank for anonymous access or runtime credentials.
+Enter a value. Press Enter to leave empty.
+secret_access_key> SECRET_ACCESS_KEY
+Option region.
+Region to connect to.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+ 1 / R2 buckets are automatically distributed across Cloudflare's data centers for low latency.
+ \ (auto)
+region> 1
+Option endpoint.
+Endpoint for S3 API.
+Required when using an S3 clone.
+Enter a value. Press Enter to leave empty.
+endpoint> https://ACCOUNT_ID.r2.cloudflarestorage.com
+Edit advanced config?
+y) Yes
+n) No (default)
+y/n> n
+--------------------
+y) Yes this is OK (default)
+e) Edit this remote
+d) Delete this remote
+y/e/d> y
+This will leave your config looking something like:
+[r2]
+type = s3
+provider = Cloudflare
+access_key_id = ACCESS_KEY
+secret_access_key = SECRET_ACCESS_KEY
+region = auto
+endpoint = https://ACCOUNT_ID.r2.cloudflarestorage.com
+acl = private
+Now run rclone lsf r2: to see your buckets and rclone lsf r2:bucket to look within a bucket.
+Dreamhost
+Dreamhost DreamObjects is an object storage system based on CEPH.
+To use rclone with Dreamhost, configure as above but leave the region blank and set the endpoint. You should end up with something like this in your config:
+[dreamobjects]
+type = s3
+provider = DreamHost
+env_auth = false
+access_key_id = your_access_key
+secret_access_key = your_secret_key
+region =
+endpoint = objects-us-west-1.dream.io
+location_constraint =
+acl = private
+server_side_encryption =
+storage_class =
+Google Cloud Storage
+GoogleCloudStorage is an S3-interoperable object storage service from Google Cloud Platform.
+To connect to Google Cloud Storage you will need an access key and secret key. These can be retrieved by creating an HMAC key.
+[gs]
+type = s3
+provider = GCS
+access_key_id = your_access_key
+secret_access_key = your_secret_key
+endpoint = https://storage.googleapis.com
+Note that --s3-versions does not work with GCS when it needs to do directory paging. Rclone will return the error:
+s3 protocol error: received versions listing with IsTruncated set with no NextKeyMarker
+This is Google bug #312292516.
+DigitalOcean Spaces
+Spaces is an S3-interoperable object storage service from cloud provider DigitalOcean.
+To connect to DigitalOcean Spaces you will need an access key and secret key. These can be retrieved on the "Applications & API" page of the DigitalOcean control panel. They will be needed when prompted by rclone config for your access_key_id and secret_access_key.
+When prompted for a region or location_constraint, press enter to use the default value. The region must be included in the endpoint setting (e.g. nyc3.digitaloceanspaces.com). The default values can be used for other settings.
+Going through the whole process of creating a new remote by running rclone config, each prompt should be answered as shown below:
+Storage> s3
+env_auth> 1
+access_key_id> YOUR_ACCESS_KEY
+secret_access_key> YOUR_SECRET_KEY
+region>
+endpoint> nyc3.digitaloceanspaces.com
+location_constraint>
+acl>
+storage_class>
+The resulting configuration file should look like:
+[spaces]
+type = s3
+provider = DigitalOcean
+env_auth = false
+access_key_id = YOUR_ACCESS_KEY
+secret_access_key = YOUR_SECRET_KEY
+region =
+endpoint = nyc3.digitaloceanspaces.com
+location_constraint =
+acl =
+server_side_encryption =
+storage_class =
+Once configured, you can create a new Space and begin copying files. For example:
+rclone mkdir spaces:my-new-space
+rclone copy /path/to/files spaces:my-new-space
+Huawei OBS
+Object Storage Service (OBS) provides stable, secure, efficient, and easy-to-use cloud storage that lets you store virtually any volume of unstructured data in any format and access it from anywhere.
+OBS provides an S3 interface, you can copy and modify the following configuration and add it to your rclone configuration file.
+[obs]
+type = s3
+provider = HuaweiOBS
+access_key_id = your-access-key-id
+secret_access_key = your-secret-access-key
+region = af-south-1
+endpoint = obs.af-south-1.myhuaweicloud.com
+acl = private
+Or you can also configure via the interactive command line:
+No remotes found, make a new one?
+n) New remote
+s) Set configuration password
+q) Quit config
+n/s/q> n
+name> obs
+Option Storage.
+Type of storage to configure.
+Choose a number from below, or type in your own value.
+[snip]
+ 5 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, China Mobile, Cloudflare, ArvanCloud, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, Lyve Cloud, Minio, Netease, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Synology, Tencent COS and Wasabi
+ \ (s3)
+[snip]
+Storage> 5
+Option provider.
+Choose your S3 provider.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+[snip]
+ 9 / Huawei Object Storage Service
+ \ (HuaweiOBS)
+[snip]
+provider> 9
+Option env_auth.
+Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
+Only applies if access_key_id and secret_access_key is blank.
+Choose a number from below, or type in your own boolean value (true or false).
+Press Enter for the default (false).
+ 1 / Enter AWS credentials in the next step.
+ \ (false)
+ 2 / Get AWS credentials from the environment (env vars or IAM).
+ \ (true)
+env_auth> 1
+Option access_key_id.
+AWS Access Key ID.
+Leave blank for anonymous access or runtime credentials.
+Enter a value. Press Enter to leave empty.
+access_key_id> your-access-key-id
+Option secret_access_key.
+AWS Secret Access Key (password).
+Leave blank for anonymous access or runtime credentials.
+Enter a value. Press Enter to leave empty.
+secret_access_key> your-secret-access-key
+Option region.
+Region to connect to.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+ 1 / AF-Johannesburg
+ \ (af-south-1)
+ 2 / AP-Bangkok
+ \ (ap-southeast-2)
+[snip]
+region> 1
+Option endpoint.
+Endpoint for OBS API.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+ 1 / AF-Johannesburg
+ \ (obs.af-south-1.myhuaweicloud.com)
+ 2 / AP-Bangkok
+ \ (obs.ap-southeast-2.myhuaweicloud.com)
+[snip]
+endpoint> 1
+Option acl.
+Canned ACL used when creating buckets and storing or copying objects.
+This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.
+For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
+Note that this ACL is applied when server-side copying objects as S3
+doesn't copy the ACL from the source but rather writes a fresh one.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+ / Owner gets FULL_CONTROL.
+ 1 | No one else has access rights (default).
+ \ (private)
+[snip]
+acl> 1
+Edit advanced config?
+y) Yes
+n) No (default)
+y/n>
+--------------------
+[obs]
+type = s3
+provider = HuaweiOBS
+access_key_id = your-access-key-id
+secret_access_key = your-secret-access-key
+region = af-south-1
+endpoint = obs.af-south-1.myhuaweicloud.com
+acl = private
+--------------------
+y) Yes this is OK (default)
+e) Edit this remote
+d) Delete this remote
+y/e/d> y
+Current remotes:
+
+Name Type
+==== ====
+obs s3
+
+e) Edit existing remote
+n) New remote
+d) Delete remote
+r) Rename remote
+c) Copy remote
+s) Set configuration password
+q) Quit config
+e/n/d/r/c/s/q> q
+IBM COS (S3)
+Information stored with IBM Cloud Object Storage is encrypted and dispersed across multiple geographic locations, and accessed through an implementation of the S3 API. This service makes use of the distributed storage technologies provided by IBM’s Cloud Object Storage System (formerly Cleversafe). For more information visit: (http://www.ibm.com/cloud/object-storage)
+To configure access to IBM COS S3, follow the steps below:
+
+- Run rclone config and select n for a new remote.
+
+ 2018/02/14 14:13:11 NOTICE: Config file "C:\\Users\\a\\.config\\rclone\\rclone.conf" not found - using defaults
+ No remotes found, make a new one?
+ n) New remote
+ s) Set configuration password
+ q) Quit config
+ n/s/q> n
+
+- Enter the name for the configuration
+
+ name> <YOUR NAME>
+
+- Select "s3" storage.
+
+Choose a number from below, or type in your own value
+ 1 / Alias for an existing remote
+ \ "alias"
+ 2 / Amazon Drive
+ \ "amazon cloud drive"
+ 3 / Amazon S3 Complaint Storage Providers (Dreamhost, Ceph, ChinaMobile, Liara, ArvanCloud, Minio, IBM COS)
+ \ "s3"
+ 4 / Backblaze B2
+ \ "b2"
+[snip]
+ 23 / HTTP
+ \ "http"
+Storage> 3
+
+- Select IBM COS as the S3 Storage Provider.
+
+Choose the S3 provider.
+Choose a number from below, or type in your own value
+ 1 / Choose this option to configure Storage to AWS S3
+ \ "AWS"
+ 2 / Choose this option to configure Storage to Ceph Systems
+ \ "Ceph"
+ 3 / Choose this option to configure Storage to Dreamhost
+ \ "Dreamhost"
+ 4 / Choose this option to the configure Storage to IBM COS S3
+ \ "IBMCOS"
+ 5 / Choose this option to the configure Storage to Minio
+ \ "Minio"
+ Provider>4
+
+- Enter the Access Key and Secret.
+
+ AWS Access Key ID - leave blank for anonymous access or runtime credentials.
+ access_key_id> <>
+ AWS Secret Access Key (password) - leave blank for anonymous access or runtime credentials.
+ secret_access_key> <>
+
+- Specify the endpoint for IBM COS. For Public IBM COS, choose from the option below. For On Premise IBM COS, enter an endpoint address.
+
+ Endpoint for IBM COS S3 API.
+ Specify if using an IBM COS On Premise.
+ Choose a number from below, or type in your own value
+ 1 / US Cross Region Endpoint
+ \ "s3-api.us-geo.objectstorage.softlayer.net"
+ 2 / US Cross Region Dallas Endpoint
+ \ "s3-api.dal.us-geo.objectstorage.softlayer.net"
+ 3 / US Cross Region Washington DC Endpoint
+ \ "s3-api.wdc-us-geo.objectstorage.softlayer.net"
+ 4 / US Cross Region San Jose Endpoint
+ \ "s3-api.sjc-us-geo.objectstorage.softlayer.net"
+ 5 / US Cross Region Private Endpoint
+ \ "s3-api.us-geo.objectstorage.service.networklayer.com"
+ 6 / US Cross Region Dallas Private Endpoint
+ \ "s3-api.dal-us-geo.objectstorage.service.networklayer.com"
+ 7 / US Cross Region Washington DC Private Endpoint
+ \ "s3-api.wdc-us-geo.objectstorage.service.networklayer.com"
+ 8 / US Cross Region San Jose Private Endpoint
+ \ "s3-api.sjc-us-geo.objectstorage.service.networklayer.com"
+ 9 / US Region East Endpoint
+ \ "s3.us-east.objectstorage.softlayer.net"
+ 10 / US Region East Private Endpoint
+ \ "s3.us-east.objectstorage.service.networklayer.com"
+ 11 / US Region South Endpoint
+[snip]
+ 34 / Toronto Single Site Private Endpoint
+ \ "s3.tor01.objectstorage.service.networklayer.com"
+ endpoint>1
+
+- Specify a IBM COS Location Constraint. The location constraint must match endpoint when using IBM Cloud Public. For on-prem COS, do not make a selection from this list, hit enter
+
+ 1 / US Cross Region Standard
+ \ "us-standard"
+ 2 / US Cross Region Vault
+ \ "us-vault"
+ 3 / US Cross Region Cold
+ \ "us-cold"
+ 4 / US Cross Region Flex
+ \ "us-flex"
+ 5 / US East Region Standard
+ \ "us-east-standard"
+ 6 / US East Region Vault
+ \ "us-east-vault"
+ 7 / US East Region Cold
+ \ "us-east-cold"
+ 8 / US East Region Flex
+ \ "us-east-flex"
+ 9 / US South Region Standard
+ \ "us-south-standard"
+ 10 / US South Region Vault
+ \ "us-south-vault"
+[snip]
+ 32 / Toronto Flex
+ \ "tor01-flex"
+location_constraint>1
+
+- Specify a canned ACL. IBM Cloud (Storage) supports "public-read" and "private". IBM Cloud(Infra) supports all the canned ACLs. On-Premise COS supports all the canned ACLs.
+
+Canned ACL used when creating buckets and/or storing objects in S3.
+For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
+Choose a number from below, or type in your own value
+ 1 / Owner gets FULL_CONTROL. No one else has access rights (default). This acl is available on IBM Cloud (Infra), IBM Cloud (Storage), On-Premise COS
+ \ "private"
+ 2 / Owner gets FULL_CONTROL. The AllUsers group gets READ access. This acl is available on IBM Cloud (Infra), IBM Cloud (Storage), On-Premise IBM COS
+ \ "public-read"
+ 3 / Owner gets FULL_CONTROL. The AllUsers group gets READ and WRITE access. This acl is available on IBM Cloud (Infra), On-Premise IBM COS
+ \ "public-read-write"
+ 4 / Owner gets FULL_CONTROL. The AuthenticatedUsers group gets READ access. Not supported on Buckets. This acl is available on IBM Cloud (Infra) and On-Premise IBM COS
+ \ "authenticated-read"
+acl> 1
+
+- Review the displayed configuration and accept to save the "remote" then quit. The config file should look like this
+
+ [xxx]
+ type = s3
+ Provider = IBMCOS
+ access_key_id = xxx
+ secret_access_key = yyy
+ endpoint = s3-api.us-geo.objectstorage.softlayer.net
+ location_constraint = us-standard
+ acl = private
+
+- Execute rclone commands
+
+ 1) Create a bucket.
+ rclone mkdir IBM-COS-XREGION:newbucket
+ 2) List available buckets.
+ rclone lsd IBM-COS-XREGION:
+ -1 2017-11-08 21:16:22 -1 test
+ -1 2018-02-14 20:16:39 -1 newbucket
+ 3) List contents of a bucket.
+ rclone ls IBM-COS-XREGION:newbucket
+ 18685952 test.exe
+ 4) Copy a file from local to remote.
+ rclone copy /Users/file.txt IBM-COS-XREGION:newbucket
+ 5) Copy a file from remote to local.
+ rclone copy IBM-COS-XREGION:newbucket/file.txt .
+ 6) Delete a file on remote.
+ rclone delete IBM-COS-XREGION:newbucket/file.txt
+IDrive e2
+Here is an example of making an IDrive e2 configuration. First run:
+rclone config
+This will guide you through an interactive setup process.
+No remotes found, make a new one?
+n) New remote
+s) Set configuration password
+q) Quit config
+n/s/q> n
+
+Enter name for new remote.
+name> e2
+
+Option Storage.
+Type of storage to configure.
+Choose a number from below, or type in your own value.
+[snip]
+XX / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, China Mobile, Cloudflare, ArvanCloud, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, IDrive e2, Lyve Cloud, Minio, Netease, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Synology, Tencent COS and Wasabi
+ \ (s3)
+[snip]
+Storage> s3
+
+Option provider.
+Choose your S3 provider.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+[snip]
+XX / IDrive e2
+ \ (IDrive)
+[snip]
+provider> IDrive
+
+Option env_auth.
+Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
+Only applies if access_key_id and secret_access_key is blank.
+Choose a number from below, or type in your own boolean value (true or false).
+Press Enter for the default (false).
+ 1 / Enter AWS credentials in the next step.
+ \ (false)
+ 2 / Get AWS credentials from the environment (env vars or IAM).
+ \ (true)
+env_auth>
+
+Option access_key_id.
+AWS Access Key ID.
+Leave blank for anonymous access or runtime credentials.
+Enter a value. Press Enter to leave empty.
+access_key_id> YOUR_ACCESS_KEY
+
+Option secret_access_key.
+AWS Secret Access Key (password).
+Leave blank for anonymous access or runtime credentials.
+Enter a value. Press Enter to leave empty.
+secret_access_key> YOUR_SECRET_KEY
+
+Option acl.
+Canned ACL used when creating buckets and storing or copying objects.
+This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.
+For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
+Note that this ACL is applied when server-side copying objects as S3
+doesn't copy the ACL from the source but rather writes a fresh one.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+ / Owner gets FULL_CONTROL.
+ 1 | No one else has access rights (default).
+ \ (private)
+ / Owner gets FULL_CONTROL.
+ 2 | The AllUsers group gets READ access.
+ \ (public-read)
+ / Owner gets FULL_CONTROL.
+ 3 | The AllUsers group gets READ and WRITE access.
+ | Granting this on a bucket is generally not recommended.
+ \ (public-read-write)
+ / Owner gets FULL_CONTROL.
+ 4 | The AuthenticatedUsers group gets READ access.
+ \ (authenticated-read)
+ / Object owner gets FULL_CONTROL.
+ 5 | Bucket owner gets READ access.
+ | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
+ \ (bucket-owner-read)
+ / Both the object owner and the bucket owner get FULL_CONTROL over the object.
+ 6 | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
+ \ (bucket-owner-full-control)
+acl>
+
+Edit advanced config?
+y) Yes
+n) No (default)
+y/n>
+
+Configuration complete.
+Options:
+- type: s3
+- provider: IDrive
+- access_key_id: YOUR_ACCESS_KEY
+- secret_access_key: YOUR_SECRET_KEY
+- endpoint: q9d9.la12.idrivee2-5.com
+Keep this "e2" remote?
+y) Yes this is OK (default)
+e) Edit this remote
+d) Delete this remote
+y/e/d> y
+IONOS Cloud
+IONOS S3 Object Storage is a service offered by IONOS for storing and accessing unstructured data. To connect to the service, you will need an access key and a secret key. These can be found in the Data Center Designer, by selecting Manager resources > Object Storage Key Manager.
+Here is an example of a configuration. First, run rclone config. This will walk you through an interactive setup process. Type n to add the new remote, and then enter a name:
+Enter name for new remote.
+name> ionos-fra
+Type s3 to choose the connection type:
+Option Storage.
+Type of storage to configure.
+Choose a number from below, or type in your own value.
+[snip]
+XX / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, China Mobile, Cloudflare, ArvanCloud, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, IDrive e2, IONOS Cloud, Lyve Cloud, Minio, Netease, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Synology, Tencent COS and Wasabi
+ \ (s3)
+[snip]
+Storage> s3
+Type IONOS:
+Option provider.
+Choose your S3 provider.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+[snip]
+XX / IONOS Cloud
+ \ (IONOS)
+[snip]
+provider> IONOS
+Press Enter to choose the default option Enter AWS credentials in the next step:
+Option env_auth.
+Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
+Only applies if access_key_id and secret_access_key is blank.
+Choose a number from below, or type in your own boolean value (true or false).
+Press Enter for the default (false).
+ 1 / Enter AWS credentials in the next step.
+ \ (false)
+ 2 / Get AWS credentials from the environment (env vars or IAM).
+ \ (true)
+env_auth>
+Enter your Access Key and Secret key. These can be retrieved in the Data Center Designer, click on the menu “Manager resources†/ "Object Storage Key Manager".
+Option access_key_id.
+AWS Access Key ID.
+Leave blank for anonymous access or runtime credentials.
+Enter a value. Press Enter to leave empty.
+access_key_id> YOUR_ACCESS_KEY
+
+Option secret_access_key.
+AWS Secret Access Key (password).
+Leave blank for anonymous access or runtime credentials.
+Enter a value. Press Enter to leave empty.
+secret_access_key> YOUR_SECRET_KEY
+Choose the region where your bucket is located:
+Option region.
+Region where your bucket will be created and your data stored.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+ 1 / Frankfurt, Germany
+ \ (de)
+ 2 / Berlin, Germany
+ \ (eu-central-2)
+ 3 / Logrono, Spain
+ \ (eu-south-2)
+region> 2
+Choose the endpoint from the same region:
+Option endpoint.
+Endpoint for IONOS S3 Object Storage.
+Specify the endpoint from the same region.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+ 1 / Frankfurt, Germany
+ \ (s3-eu-central-1.ionoscloud.com)
+ 2 / Berlin, Germany
+ \ (s3-eu-central-2.ionoscloud.com)
+ 3 / Logrono, Spain
+ \ (s3-eu-south-2.ionoscloud.com)
+endpoint> 1
+Press Enter to choose the default option or choose the desired ACL setting:
+Option acl.
+Canned ACL used when creating buckets and storing or copying objects.
+This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.
+For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
+Note that this ACL is applied when server-side copying objects as S3
+doesn't copy the ACL from the source but rather writes a fresh one.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+ / Owner gets FULL_CONTROL.
+ 1 | No one else has access rights (default).
+ \ (private)
+ / Owner gets FULL_CONTROL.
+[snip]
+acl>
+Press Enter to skip the advanced config:
+Edit advanced config?
+y) Yes
+n) No (default)
+y/n>
+Press Enter to save the configuration, and then q to quit the configuration process:
+Configuration complete.
+Options:
+- type: s3
+- provider: IONOS
+- access_key_id: YOUR_ACCESS_KEY
+- secret_access_key: YOUR_SECRET_KEY
+- endpoint: s3-eu-central-1.ionoscloud.com
+Keep this "ionos-fra" remote?
+y) Yes this is OK (default)
+e) Edit this remote
+d) Delete this remote
+y/e/d> y
+Done! Now you can try some commands (for macOS, use ./rclone instead of rclone).
+
+- Create a bucket (the name must be unique within the whole IONOS S3)
+
+rclone mkdir ionos-fra:my-bucket
+
+- List available buckets
+
+rclone lsd ionos-fra:
+
+- Copy a file from local to remote
+
+rclone copy /Users/file.txt ionos-fra:my-bucket
+
+- List contents of a bucket
+
+rclone ls ionos-fra:my-bucket
+
+- Copy a file from remote to local
+
+rclone copy ionos-fra:my-bucket/file.txt
+Minio
+Minio is an object storage server built for cloud application developers and devops.
+It is very easy to install and provides an S3 compatible server which can be used by rclone.
+To use it, install Minio following the instructions here.
+When it configures itself Minio will print something like this
+Endpoint: http://192.168.1.106:9000 http://172.23.0.1:9000
+AccessKey: USWUXHGYZQYFYFFIT3RE
+SecretKey: MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03
+Region: us-east-1
+SQS ARNs: arn:minio:sqs:us-east-1:1:redis arn:minio:sqs:us-east-1:2:redis
+
+Browser Access:
+ http://192.168.1.106:9000 http://172.23.0.1:9000
+
+Command-line Access: https://docs.minio.io/docs/minio-client-quickstart-guide
+ $ mc config host add myminio http://192.168.1.106:9000 USWUXHGYZQYFYFFIT3RE MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03
+
+Object API (Amazon S3 compatible):
+ Go: https://docs.minio.io/docs/golang-client-quickstart-guide
+ Java: https://docs.minio.io/docs/java-client-quickstart-guide
+ Python: https://docs.minio.io/docs/python-client-quickstart-guide
+ JavaScript: https://docs.minio.io/docs/javascript-client-quickstart-guide
+ .NET: https://docs.minio.io/docs/dotnet-client-quickstart-guide
+
+Drive Capacity: 26 GiB Free, 165 GiB Total
+These details need to go into rclone config like this. Note that it is important to put the region in as stated above.
+env_auth> 1
+access_key_id> USWUXHGYZQYFYFFIT3RE
+secret_access_key> MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03
+region> us-east-1
+endpoint> http://192.168.1.106:9000
+location_constraint>
+server_side_encryption>
+Which makes the config file look like this
+[minio]
+type = s3
+provider = Minio
+env_auth = false
+access_key_id = USWUXHGYZQYFYFFIT3RE
+secret_access_key = MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03
+region = us-east-1
+endpoint = http://192.168.1.106:9000
+location_constraint =
+server_side_encryption =
+So once set up, for example, to copy files into a bucket
+rclone copy /path/to/files minio:bucket
+Qiniu Cloud Object Storage (Kodo)
+Qiniu Cloud Object Storage (Kodo), a completely independent-researched core technology which is proven by repeated customer experience has occupied absolute leading market leader position. Kodo can be widely applied to mass data management.
+To configure access to Qiniu Kodo, follow the steps below:
+
+- Run
rclone config and select n for a new remote.
+
+rclone config
+No remotes found, make a new one?
+n) New remote
+s) Set configuration password
+q) Quit config
+n/s/q> n
+
+- Give the name of the configuration. For example, name it 'qiniu'.
+
+name> qiniu
+
+- Select
s3 storage.
+
+Choose a number from below, or type in your own value
+ 1 / 1Fichier
+ \ (fichier)
+ 2 / Akamai NetStorage
+ \ (netstorage)
+ 3 / Alias for an existing remote
+ \ (alias)
+ 4 / Amazon Drive
+ \ (amazon cloud drive)
+ 5 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, China Mobile, Cloudflare, ArvanCloud, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, IDrive e2, Liara, Lyve Cloud, Minio, Netease, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Synology, Tencent COS, Qiniu and Wasabi
+ \ (s3)
+[snip]
+Storage> s3
+
+- Select
Qiniu provider.
+
+Choose a number from below, or type in your own value
+1 / Amazon Web Services (AWS) S3
+ \ "AWS"
+[snip]
+22 / Qiniu Object Storage (Kodo)
+ \ (Qiniu)
+[snip]
+provider> Qiniu
+
+- Enter your SecretId and SecretKey of Qiniu Kodo.
+
+Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
+Only applies if access_key_id and secret_access_key is blank.
+Enter a boolean value (true or false). Press Enter for the default ("false").
+Choose a number from below, or type in your own value
+ 1 / Enter AWS credentials in the next step
+ \ "false"
+ 2 / Get AWS credentials from the environment (env vars or IAM)
+ \ "true"
+env_auth> 1
+AWS Access Key ID.
+Leave blank for anonymous access or runtime credentials.
+Enter a string value. Press Enter for the default ("").
+access_key_id> AKIDxxxxxxxxxx
+AWS Secret Access Key (password)
+Leave blank for anonymous access or runtime credentials.
+Enter a string value. Press Enter for the default ("").
+secret_access_key> xxxxxxxxxxx
+
+- Select endpoint for Qiniu Kodo. This is the standard endpoint for different region.
+
+ / The default endpoint - a good choice if you are unsure.
+ 1 | East China Region 1.
+ | Needs location constraint cn-east-1.
+ \ (cn-east-1)
+ / East China Region 2.
+ 2 | Needs location constraint cn-east-2.
+ \ (cn-east-2)
+ / North China Region 1.
+ 3 | Needs location constraint cn-north-1.
+ \ (cn-north-1)
+ / South China Region 1.
+ 4 | Needs location constraint cn-south-1.
+ \ (cn-south-1)
+ / North America Region.
+ 5 | Needs location constraint us-north-1.
+ \ (us-north-1)
+ / Southeast Asia Region 1.
+ 6 | Needs location constraint ap-southeast-1.
+ \ (ap-southeast-1)
+ / Northeast Asia Region 1.
+ 7 | Needs location constraint ap-northeast-1.
+ \ (ap-northeast-1)
+[snip]
+endpoint> 1
+
+Option endpoint.
+Endpoint for Qiniu Object Storage.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+ 1 / East China Endpoint 1
+ \ (s3-cn-east-1.qiniucs.com)
+ 2 / East China Endpoint 2
+ \ (s3-cn-east-2.qiniucs.com)
+ 3 / North China Endpoint 1
+ \ (s3-cn-north-1.qiniucs.com)
+ 4 / South China Endpoint 1
+ \ (s3-cn-south-1.qiniucs.com)
+ 5 / North America Endpoint 1
+ \ (s3-us-north-1.qiniucs.com)
+ 6 / Southeast Asia Endpoint 1
+ \ (s3-ap-southeast-1.qiniucs.com)
+ 7 / Northeast Asia Endpoint 1
+ \ (s3-ap-northeast-1.qiniucs.com)
+endpoint> 1
+
+Option location_constraint.
+Location constraint - must be set to match the Region.
+Used when creating buckets only.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+ 1 / East China Region 1
+ \ (cn-east-1)
+ 2 / East China Region 2
+ \ (cn-east-2)
+ 3 / North China Region 1
+ \ (cn-north-1)
+ 4 / South China Region 1
+ \ (cn-south-1)
+ 5 / North America Region 1
+ \ (us-north-1)
+ 6 / Southeast Asia Region 1
+ \ (ap-southeast-1)
+ 7 / Northeast Asia Region 1
+ \ (ap-northeast-1)
+location_constraint> 1
+
+- Choose acl and storage class.
+
+Note that this ACL is applied when server-side copying objects as S3
+doesn't copy the ACL from the source but rather writes a fresh one.
+Enter a string value. Press Enter for the default ("").
+Choose a number from below, or type in your own value
+ / Owner gets FULL_CONTROL.
+ 1 | No one else has access rights (default).
+ \ (private)
+ / Owner gets FULL_CONTROL.
+ 2 | The AllUsers group gets READ access.
+ \ (public-read)
+[snip]
+acl> 2
+The storage class to use when storing new objects in Tencent COS.
+Enter a string value. Press Enter for the default ("").
+Choose a number from below, or type in your own value
+ 1 / Standard storage class
+ \ (STANDARD)
+ 2 / Infrequent access storage mode
+ \ (LINE)
+ 3 / Archive storage mode
+ \ (GLACIER)
+ 4 / Deep archive storage mode
+ \ (DEEP_ARCHIVE)
+[snip]
+storage_class> 1
+Edit advanced config? (y/n)
+y) Yes
+n) No (default)
+y/n> n
+Remote config
+--------------------
+[qiniu]
+- type: s3
+- provider: Qiniu
+- access_key_id: xxx
+- secret_access_key: xxx
+- region: cn-east-1
+- endpoint: s3-cn-east-1.qiniucs.com
+- location_constraint: cn-east-1
+- acl: public-read
+- storage_class: STANDARD
+--------------------
+y) Yes this is OK (default)
+e) Edit this remote
+d) Delete this remote
+y/e/d> y
+Current remotes:
+
+Name Type
+==== ====
+qiniu s3
+RackCorp
+RackCorp Object Storage is an S3 compatible object storage platform from your friendly cloud provider RackCorp. The service is fast, reliable, well priced and located in many strategic locations unserviced by others, to ensure you can maintain data sovereignty.
+Before you can use RackCorp Object Storage, you'll need to "sign up" for an account on our "portal". Next you can create an access key, a secret key and buckets, in your location of choice with ease. These details are required for the next steps of configuration, when rclone config asks for your access_key_id and secret_access_key.
+Your config should end up looking a bit like this:
+[RCS3-demo-config]
+type = s3
+provider = RackCorp
+env_auth = true
+access_key_id = YOURACCESSKEY
+secret_access_key = YOURSECRETACCESSKEY
+region = au-nsw
+endpoint = s3.rackcorp.com
+location_constraint = au-nsw
+Rclone Serve S3
+Rclone can serve any remote over the S3 protocol. For details see the rclone serve s3 documentation.
+For example, to serve remote:path over s3, run the server like this:
+rclone serve s3 --auth-key ACCESS_KEY_ID,SECRET_ACCESS_KEY remote:path
+This will be compatible with an rclone remote which is defined like this:
+[serves3]
+type = s3
+provider = Rclone
+endpoint = http://127.0.0.1:8080/
+access_key_id = ACCESS_KEY_ID
+secret_access_key = SECRET_ACCESS_KEY
+use_multipart_uploads = false
+Note that setting disable_multipart_uploads = true is to work around a bug which will be fixed in due course.
+Scaleway
+Scaleway The Object Storage platform allows you to store anything from backups, logs and web assets to documents and photos. Files can be dropped from the Scaleway console or transferred through our API and CLI or using any S3-compatible tool.
+Scaleway provides an S3 interface which can be configured for use with rclone like this:
+[scaleway]
+type = s3
+provider = Scaleway
+env_auth = false
+endpoint = s3.nl-ams.scw.cloud
+access_key_id = SCWXXXXXXXXXXXXXX
+secret_access_key = 1111111-2222-3333-44444-55555555555555
+region = nl-ams
+location_constraint =
+acl = private
+server_side_encryption =
+storage_class =
+C14 Cold Storage is the low-cost S3 Glacier alternative from Scaleway and it works the same way as on S3 by accepting the "GLACIER" storage_class. So you can configure your remote with the storage_class = GLACIER option to upload directly to C14. Don't forget that in this state you can't read files back after, you will need to restore them to "STANDARD" storage_class first before being able to read them (see "restore" section above)
+Seagate Lyve Cloud
+Seagate Lyve Cloud is an S3 compatible object storage platform from Seagate intended for enterprise use.
+Here is a config run through for a remote called remote - you may choose a different name of course. Note that to create an access key and secret key you will need to create a service account first.
+$ rclone config
+No remotes found, make a new one?
+n) New remote
+s) Set configuration password
+q) Quit config
+n/s/q> n
+name> remote
+Choose s3 backend
+Type of storage to configure.
+Choose a number from below, or type in your own value.
+[snip]
+XX / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, ChinaMobile, ArvanCloud, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, Liara, Lyve Cloud, Minio, RackCorp, SeaweedFS, and Tencent COS
+ \ (s3)
+[snip]
+Storage> s3
+Choose LyveCloud as S3 provider
+Choose your S3 provider.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+[snip]
+XX / Seagate Lyve Cloud
+ \ (LyveCloud)
+[snip]
+provider> LyveCloud
+Take the default (just press enter) to enter access key and secret in the config file.
+Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
+Only applies if access_key_id and secret_access_key is blank.
+Choose a number from below, or type in your own boolean value (true or false).
+Press Enter for the default (false).
+ 1 / Enter AWS credentials in the next step.
+ \ (false)
+ 2 / Get AWS credentials from the environment (env vars or IAM).
+ \ (true)
+env_auth>
+AWS Access Key ID.
+Leave blank for anonymous access or runtime credentials.
+Enter a value. Press Enter to leave empty.
+access_key_id> XXX
+AWS Secret Access Key (password).
+Leave blank for anonymous access or runtime credentials.
+Enter a value. Press Enter to leave empty.
+secret_access_key> YYY
+Leave region blank
+Region to connect to.
+Leave blank if you are using an S3 clone and you don't have a region.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+ / Use this if unsure.
+ 1 | Will use v4 signatures and an empty region.
+ \ ()
+ / Use this only if v4 signatures don't work.
+ 2 | E.g. pre Jewel/v10 CEPH.
+ \ (other-v2-signature)
+region>
+Choose an endpoint from the list
+Endpoint for S3 API.
+Required when using an S3 clone.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+ 1 / Seagate Lyve Cloud US East 1 (Virginia)
+ \ (s3.us-east-1.lyvecloud.seagate.com)
+ 2 / Seagate Lyve Cloud US West 1 (California)
+ \ (s3.us-west-1.lyvecloud.seagate.com)
+ 3 / Seagate Lyve Cloud AP Southeast 1 (Singapore)
+ \ (s3.ap-southeast-1.lyvecloud.seagate.com)
+endpoint> 1
+Leave location constraint blank
+Location constraint - must be set to match the Region.
+Leave blank if not sure. Used when creating buckets only.
+Enter a value. Press Enter to leave empty.
+location_constraint>
+Choose default ACL (private).
+Canned ACL used when creating buckets and storing or copying objects.
+This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.
+For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
+Note that this ACL is applied when server-side copying objects as S3
+doesn't copy the ACL from the source but rather writes a fresh one.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+ / Owner gets FULL_CONTROL.
+ 1 | No one else has access rights (default).
+ \ (private)
+[snip]
+acl>
+And the config file should end up looking like this:
+[remote]
+type = s3
+provider = LyveCloud
+access_key_id = XXX
+secret_access_key = YYY
+endpoint = s3.us-east-1.lyvecloud.seagate.com
+SeaweedFS
+SeaweedFS is a distributed storage system for blobs, objects, files, and data lake, with O(1) disk seek and a scalable file metadata store. It has an S3 compatible object storage interface. SeaweedFS can also act as a gateway to remote S3 compatible object store to cache data and metadata with asynchronous write back, for fast local speed and minimize access cost.
+Assuming the SeaweedFS are configured with weed shell as such:
+> s3.bucket.create -name foo
+> s3.configure -access_key=any -secret_key=any -buckets=foo -user=me -actions=Read,Write,List,Tagging,Admin -apply
+{
+ "identities": [
{
- "Initiated": "2020-06-26T14:20:36Z",
- "Initiator": {
- "DisplayName": "XXX",
- "ID": "arn:aws:iam::XXX:user/XXX"
- },
- "Key": "KEY",
- "Owner": {
- "DisplayName": null,
- "ID": "XXX"
- },
- "StorageClass": "STANDARD",
- "UploadId": "XXX"
+ "name": "me",
+ "credentials": [
+ {
+ "accessKey": "any",
+ "secretKey": "any"
+ }
+ ],
+ "actions": [
+ "Read:foo",
+ "Write:foo",
+ "List:foo",
+ "Tagging:foo",
+ "Admin:foo"
+ ]
}
- ],
- "rclone-1000files": [],
- "rclone-dst": []
+ ]
}
-cleanup
-Remove unfinished multipart uploads.
-rclone backend cleanup remote: [options] [<arguments>+]
-This command removes unfinished multipart uploads of age greater than max-age which defaults to 24 hours.
-Note that you can use --interactive/-i or --dry-run with this command to see what it would do.
-rclone backend cleanup s3:bucket/path/to/object
-rclone backend cleanup -o max-age=7w s3:bucket/path/to/object
-Durations are parsed as per the rest of rclone, 2h, 7d, 7w etc.
-Options:
-
-- "max-age": Max age of upload to delete
-
-cleanup-hidden
-Remove old versions of files.
-rclone backend cleanup-hidden remote: [options] [<arguments>+]
-This command removes any old hidden versions of files on a versions enabled bucket.
-Note that you can use --interactive/-i or --dry-run with this command to see what it would do.
-rclone backend cleanup-hidden s3:bucket/path/to/dir
-versioning
-Set/get versioning support for a bucket.
-rclone backend versioning remote: [options] [<arguments>+]
-This command sets versioning support if a parameter is passed and then returns the current versioning status for the bucket supplied.
-rclone backend versioning s3:bucket # read status only
-rclone backend versioning s3:bucket Enabled
-rclone backend versioning s3:bucket Suspended
-It may return "Enabled", "Suspended" or "Unversioned". Note that once versioning has been enabled the status can't be set back to "Unversioned".
-set
-Set command for updating the config parameters.
-rclone backend set remote: [options] [<arguments>+]
-This set command can be used to update the config parameters for a running s3 backend.
-Usage Examples:
-rclone backend set s3: [-o opt_name=opt_value] [-o opt_name2=opt_value2]
-rclone rc backend/command command=set fs=s3: [-o opt_name=opt_value] [-o opt_name2=opt_value2]
-rclone rc backend/command command=set fs=s3: -o session_token=X -o access_key_id=X -o secret_access_key=X
-The option keys are named as they are in the config file.
-This rebuilds the connection to the s3 backend when it is called with the new parameters. Only new parameters need be passed as the values will default to those currently in use.
-It doesn't return anything.
-Anonymous access to public buckets
-If you want to use rclone to access a public bucket, configure with a blank access_key_id and secret_access_key. Your config should end up looking like this:
-[anons3]
+To use rclone with SeaweedFS, above configuration should end up with something like this in your config:
+[seaweedfs_s3]
+type = s3
+provider = SeaweedFS
+access_key_id = any
+secret_access_key = any
+endpoint = localhost:8333
+So once set up, for example to copy files into a bucket
+rclone copy /path/to/files seaweedfs_s3:foo
+Wasabi
+Wasabi is a cloud-based object storage service for a broad range of applications and use cases. Wasabi is designed for individuals and organizations that require a high-performance, reliable, and secure data storage infrastructure at minimal cost.
+Wasabi provides an S3 interface which can be configured for use with rclone like this.
+No remotes found, make a new one?
+n) New remote
+s) Set configuration password
+n/s> n
+name> wasabi
+Type of storage to configure.
+Choose a number from below, or type in your own value
+[snip]
+XX / Amazon S3 (also Dreamhost, Ceph, ChinaMobile, ArvanCloud, Minio, Liara)
+ \ "s3"
+[snip]
+Storage> s3
+Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars). Only applies if access_key_id and secret_access_key is blank.
+Choose a number from below, or type in your own value
+ 1 / Enter AWS credentials in the next step
+ \ "false"
+ 2 / Get AWS credentials from the environment (env vars or IAM)
+ \ "true"
+env_auth> 1
+AWS Access Key ID - leave blank for anonymous access or runtime credentials.
+access_key_id> YOURACCESSKEY
+AWS Secret Access Key (password) - leave blank for anonymous access or runtime credentials.
+secret_access_key> YOURSECRETACCESSKEY
+Region to connect to.
+Choose a number from below, or type in your own value
+ / The default endpoint - a good choice if you are unsure.
+ 1 | US Region, Northern Virginia, or Pacific Northwest.
+ | Leave location constraint empty.
+ \ "us-east-1"
+[snip]
+region> us-east-1
+Endpoint for S3 API.
+Leave blank if using AWS to use the default endpoint for the region.
+Specify if using an S3 clone such as Ceph.
+endpoint> s3.wasabisys.com
+Location constraint - must be set to match the Region. Used when creating buckets only.
+Choose a number from below, or type in your own value
+ 1 / Empty for US Region, Northern Virginia, or Pacific Northwest.
+ \ ""
+[snip]
+location_constraint>
+Canned ACL used when creating buckets and/or storing objects in S3.
+For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
+Choose a number from below, or type in your own value
+ 1 / Owner gets FULL_CONTROL. No one else has access rights (default).
+ \ "private"
+[snip]
+acl>
+The server-side encryption algorithm used when storing this object in S3.
+Choose a number from below, or type in your own value
+ 1 / None
+ \ ""
+ 2 / AES256
+ \ "AES256"
+server_side_encryption>
+The storage class to use when storing objects in S3.
+Choose a number from below, or type in your own value
+ 1 / Default
+ \ ""
+ 2 / Standard storage class
+ \ "STANDARD"
+ 3 / Reduced redundancy storage class
+ \ "REDUCED_REDUNDANCY"
+ 4 / Standard Infrequent Access storage class
+ \ "STANDARD_IA"
+storage_class>
+Remote config
+--------------------
+[wasabi]
+env_auth = false
+access_key_id = YOURACCESSKEY
+secret_access_key = YOURSECRETACCESSKEY
+region = us-east-1
+endpoint = s3.wasabisys.com
+location_constraint =
+acl =
+server_side_encryption =
+storage_class =
+--------------------
+y) Yes this is OK
+e) Edit this remote
+d) Delete this remote
+y/e/d> y
+This will leave the config file looking like this.
+[wasabi]
+type = s3
+provider = Wasabi
+env_auth = false
+access_key_id = YOURACCESSKEY
+secret_access_key = YOURSECRETACCESSKEY
+region =
+endpoint = s3.wasabisys.com
+location_constraint =
+acl =
+server_side_encryption =
+storage_class =
+Alibaba OSS
+Here is an example of making an Alibaba Cloud (Aliyun) OSS configuration. First run:
+rclone config
+This will guide you through an interactive setup process.
+No remotes found, make a new one?
+n) New remote
+s) Set configuration password
+q) Quit config
+n/s/q> n
+name> oss
+Type of storage to configure.
+Enter a string value. Press Enter for the default ("").
+Choose a number from below, or type in your own value
+[snip]
+ 4 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, ChinaMobile, ArvanCloud, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, Liara, Minio, and Tencent COS
+ \ "s3"
+[snip]
+Storage> s3
+Choose your S3 provider.
+Enter a string value. Press Enter for the default ("").
+Choose a number from below, or type in your own value
+ 1 / Amazon Web Services (AWS) S3
+ \ "AWS"
+ 2 / Alibaba Cloud Object Storage System (OSS) formerly Aliyun
+ \ "Alibaba"
+ 3 / Ceph Object Storage
+ \ "Ceph"
+[snip]
+provider> Alibaba
+Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
+Only applies if access_key_id and secret_access_key is blank.
+Enter a boolean value (true or false). Press Enter for the default ("false").
+Choose a number from below, or type in your own value
+ 1 / Enter AWS credentials in the next step
+ \ "false"
+ 2 / Get AWS credentials from the environment (env vars or IAM)
+ \ "true"
+env_auth> 1
+AWS Access Key ID.
+Leave blank for anonymous access or runtime credentials.
+Enter a string value. Press Enter for the default ("").
+access_key_id> accesskeyid
+AWS Secret Access Key (password)
+Leave blank for anonymous access or runtime credentials.
+Enter a string value. Press Enter for the default ("").
+secret_access_key> secretaccesskey
+Endpoint for OSS API.
+Enter a string value. Press Enter for the default ("").
+Choose a number from below, or type in your own value
+ 1 / East China 1 (Hangzhou)
+ \ "oss-cn-hangzhou.aliyuncs.com"
+ 2 / East China 2 (Shanghai)
+ \ "oss-cn-shanghai.aliyuncs.com"
+ 3 / North China 1 (Qingdao)
+ \ "oss-cn-qingdao.aliyuncs.com"
+[snip]
+endpoint> 1
+Canned ACL used when creating buckets and storing or copying objects.
+
+Note that this ACL is applied when server-side copying objects as S3
+doesn't copy the ACL from the source but rather writes a fresh one.
+Enter a string value. Press Enter for the default ("").
+Choose a number from below, or type in your own value
+ 1 / Owner gets FULL_CONTROL. No one else has access rights (default).
+ \ "private"
+ 2 / Owner gets FULL_CONTROL. The AllUsers group gets READ access.
+ \ "public-read"
+ / Owner gets FULL_CONTROL. The AllUsers group gets READ and WRITE access.
+[snip]
+acl> 1
+The storage class to use when storing new objects in OSS.
+Enter a string value. Press Enter for the default ("").
+Choose a number from below, or type in your own value
+ 1 / Default
+ \ ""
+ 2 / Standard storage class
+ \ "STANDARD"
+ 3 / Archive storage mode.
+ \ "GLACIER"
+ 4 / Infrequent access storage mode.
+ \ "STANDARD_IA"
+storage_class> 1
+Edit advanced config? (y/n)
+y) Yes
+n) No
+y/n> n
+Remote config
+--------------------
+[oss]
+type = s3
+provider = Alibaba
+env_auth = false
+access_key_id = accesskeyid
+secret_access_key = secretaccesskey
+endpoint = oss-cn-hangzhou.aliyuncs.com
+acl = private
+storage_class = Standard
+--------------------
+y) Yes this is OK
+e) Edit this remote
+d) Delete this remote
+y/e/d> y
+China Mobile Ecloud Elastic Object Storage (EOS)
+Here is an example of making an China Mobile Ecloud Elastic Object Storage (EOS) configuration. First run:
+rclone config
+This will guide you through an interactive setup process.
+No remotes found, make a new one?
+n) New remote
+s) Set configuration password
+q) Quit config
+n/s/q> n
+name> ChinaMobile
+Option Storage.
+Type of storage to configure.
+Choose a number from below, or type in your own value.
+ ...
+ 5 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, ChinaMobile, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, Lyve Cloud, Minio, RackCorp, SeaweedFS, and Tencent COS
+ \ (s3)
+ ...
+Storage> s3
+Option provider.
+Choose your S3 provider.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+ ...
+ 4 / China Mobile Ecloud Elastic Object Storage (EOS)
+ \ (ChinaMobile)
+ ...
+provider> ChinaMobile
+Option env_auth.
+Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
+Only applies if access_key_id and secret_access_key is blank.
+Choose a number from below, or type in your own boolean value (true or false).
+Press Enter for the default (false).
+ 1 / Enter AWS credentials in the next step.
+ \ (false)
+ 2 / Get AWS credentials from the environment (env vars or IAM).
+ \ (true)
+env_auth>
+Option access_key_id.
+AWS Access Key ID.
+Leave blank for anonymous access or runtime credentials.
+Enter a value. Press Enter to leave empty.
+access_key_id> accesskeyid
+Option secret_access_key.
+AWS Secret Access Key (password).
+Leave blank for anonymous access or runtime credentials.
+Enter a value. Press Enter to leave empty.
+secret_access_key> secretaccesskey
+Option endpoint.
+Endpoint for China Mobile Ecloud Elastic Object Storage (EOS) API.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+ / The default endpoint - a good choice if you are unsure.
+ 1 | East China (Suzhou)
+ \ (eos-wuxi-1.cmecloud.cn)
+ 2 / East China (Jinan)
+ \ (eos-jinan-1.cmecloud.cn)
+ 3 / East China (Hangzhou)
+ \ (eos-ningbo-1.cmecloud.cn)
+ 4 / East China (Shanghai-1)
+ \ (eos-shanghai-1.cmecloud.cn)
+ 5 / Central China (Zhengzhou)
+ \ (eos-zhengzhou-1.cmecloud.cn)
+ 6 / Central China (Changsha-1)
+ \ (eos-hunan-1.cmecloud.cn)
+ 7 / Central China (Changsha-2)
+ \ (eos-zhuzhou-1.cmecloud.cn)
+ 8 / South China (Guangzhou-2)
+ \ (eos-guangzhou-1.cmecloud.cn)
+ 9 / South China (Guangzhou-3)
+ \ (eos-dongguan-1.cmecloud.cn)
+10 / North China (Beijing-1)
+ \ (eos-beijing-1.cmecloud.cn)
+11 / North China (Beijing-2)
+ \ (eos-beijing-2.cmecloud.cn)
+12 / North China (Beijing-3)
+ \ (eos-beijing-4.cmecloud.cn)
+13 / North China (Huhehaote)
+ \ (eos-huhehaote-1.cmecloud.cn)
+14 / Southwest China (Chengdu)
+ \ (eos-chengdu-1.cmecloud.cn)
+15 / Southwest China (Chongqing)
+ \ (eos-chongqing-1.cmecloud.cn)
+16 / Southwest China (Guiyang)
+ \ (eos-guiyang-1.cmecloud.cn)
+17 / Nouthwest China (Xian)
+ \ (eos-xian-1.cmecloud.cn)
+18 / Yunnan China (Kunming)
+ \ (eos-yunnan.cmecloud.cn)
+19 / Yunnan China (Kunming-2)
+ \ (eos-yunnan-2.cmecloud.cn)
+20 / Tianjin China (Tianjin)
+ \ (eos-tianjin-1.cmecloud.cn)
+21 / Jilin China (Changchun)
+ \ (eos-jilin-1.cmecloud.cn)
+22 / Hubei China (Xiangyan)
+ \ (eos-hubei-1.cmecloud.cn)
+23 / Jiangxi China (Nanchang)
+ \ (eos-jiangxi-1.cmecloud.cn)
+24 / Gansu China (Lanzhou)
+ \ (eos-gansu-1.cmecloud.cn)
+25 / Shanxi China (Taiyuan)
+ \ (eos-shanxi-1.cmecloud.cn)
+26 / Liaoning China (Shenyang)
+ \ (eos-liaoning-1.cmecloud.cn)
+27 / Hebei China (Shijiazhuang)
+ \ (eos-hebei-1.cmecloud.cn)
+28 / Fujian China (Xiamen)
+ \ (eos-fujian-1.cmecloud.cn)
+29 / Guangxi China (Nanning)
+ \ (eos-guangxi-1.cmecloud.cn)
+30 / Anhui China (Huainan)
+ \ (eos-anhui-1.cmecloud.cn)
+endpoint> 1
+Option location_constraint.
+Location constraint - must match endpoint.
+Used when creating buckets only.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+ 1 / East China (Suzhou)
+ \ (wuxi1)
+ 2 / East China (Jinan)
+ \ (jinan1)
+ 3 / East China (Hangzhou)
+ \ (ningbo1)
+ 4 / East China (Shanghai-1)
+ \ (shanghai1)
+ 5 / Central China (Zhengzhou)
+ \ (zhengzhou1)
+ 6 / Central China (Changsha-1)
+ \ (hunan1)
+ 7 / Central China (Changsha-2)
+ \ (zhuzhou1)
+ 8 / South China (Guangzhou-2)
+ \ (guangzhou1)
+ 9 / South China (Guangzhou-3)
+ \ (dongguan1)
+10 / North China (Beijing-1)
+ \ (beijing1)
+11 / North China (Beijing-2)
+ \ (beijing2)
+12 / North China (Beijing-3)
+ \ (beijing4)
+13 / North China (Huhehaote)
+ \ (huhehaote1)
+14 / Southwest China (Chengdu)
+ \ (chengdu1)
+15 / Southwest China (Chongqing)
+ \ (chongqing1)
+16 / Southwest China (Guiyang)
+ \ (guiyang1)
+17 / Nouthwest China (Xian)
+ \ (xian1)
+18 / Yunnan China (Kunming)
+ \ (yunnan)
+19 / Yunnan China (Kunming-2)
+ \ (yunnan2)
+20 / Tianjin China (Tianjin)
+ \ (tianjin1)
+21 / Jilin China (Changchun)
+ \ (jilin1)
+22 / Hubei China (Xiangyan)
+ \ (hubei1)
+23 / Jiangxi China (Nanchang)
+ \ (jiangxi1)
+24 / Gansu China (Lanzhou)
+ \ (gansu1)
+25 / Shanxi China (Taiyuan)
+ \ (shanxi1)
+26 / Liaoning China (Shenyang)
+ \ (liaoning1)
+27 / Hebei China (Shijiazhuang)
+ \ (hebei1)
+28 / Fujian China (Xiamen)
+ \ (fujian1)
+29 / Guangxi China (Nanning)
+ \ (guangxi1)
+30 / Anhui China (Huainan)
+ \ (anhui1)
+location_constraint> 1
+Option acl.
+Canned ACL used when creating buckets and storing or copying objects.
+This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.
+For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
+Note that this ACL is applied when server-side copying objects as S3
+doesn't copy the ACL from the source but rather writes a fresh one.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+ / Owner gets FULL_CONTROL.
+ 1 | No one else has access rights (default).
+ \ (private)
+ / Owner gets FULL_CONTROL.
+ 2 | The AllUsers group gets READ access.
+ \ (public-read)
+ / Owner gets FULL_CONTROL.
+ 3 | The AllUsers group gets READ and WRITE access.
+ | Granting this on a bucket is generally not recommended.
+ \ (public-read-write)
+ / Owner gets FULL_CONTROL.
+ 4 | The AuthenticatedUsers group gets READ access.
+ \ (authenticated-read)
+ / Object owner gets FULL_CONTROL.
+acl> private
+Option server_side_encryption.
+The server-side encryption algorithm used when storing this object in S3.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+ 1 / None
+ \ ()
+ 2 / AES256
+ \ (AES256)
+server_side_encryption>
+Option storage_class.
+The storage class to use when storing new objects in ChinaMobile.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+ 1 / Default
+ \ ()
+ 2 / Standard storage class
+ \ (STANDARD)
+ 3 / Archive storage mode
+ \ (GLACIER)
+ 4 / Infrequent access storage mode
+ \ (STANDARD_IA)
+storage_class>
+Edit advanced config?
+y) Yes
+n) No (default)
+y/n> n
+--------------------
+[ChinaMobile]
type = s3
-provider = AWS
+provider = ChinaMobile
+access_key_id = accesskeyid
+secret_access_key = secretaccesskey
+endpoint = eos-wuxi-1.cmecloud.cn
+location_constraint = wuxi1
+acl = private
+--------------------
+y) Yes this is OK (default)
+e) Edit this remote
+d) Delete this remote
+y/e/d> y
+Leviia Cloud Object Storage
+Leviia Object Storage, backup and secure your data in a 100% French cloud, independent of GAFAM..
+To configure access to Leviia, follow the steps below:
+
+- Run
rclone config and select n for a new remote.
+
+rclone config
+No remotes found, make a new one?
+n) New remote
+s) Set configuration password
+q) Quit config
+n/s/q> n
+
+- Give the name of the configuration. For example, name it 'leviia'.
+
+name> leviia
+
+- Select
s3 storage.
+
+Choose a number from below, or type in your own value
+ 1 / 1Fichier
+ \ (fichier)
+ 2 / Akamai NetStorage
+ \ (netstorage)
+ 3 / Alias for an existing remote
+ \ (alias)
+ 4 / Amazon Drive
+ \ (amazon cloud drive)
+ 5 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, China Mobile, Cloudflare, ArvanCloud, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, IDrive e2, Liara, Lyve Cloud, Minio, Netease, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Synology, Tencent COS, Qiniu and Wasabi
+ \ (s3)
+[snip]
+Storage> s3
+
+- Select
Leviia provider.
+
+Choose a number from below, or type in your own value
+1 / Amazon Web Services (AWS) S3
+ \ "AWS"
+[snip]
+15 / Leviia Object Storage
+ \ (Leviia)
+[snip]
+provider> Leviia
+
+- Enter your SecretId and SecretKey of Leviia.
+
+Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
+Only applies if access_key_id and secret_access_key is blank.
+Enter a boolean value (true or false). Press Enter for the default ("false").
+Choose a number from below, or type in your own value
+ 1 / Enter AWS credentials in the next step
+ \ "false"
+ 2 / Get AWS credentials from the environment (env vars or IAM)
+ \ "true"
+env_auth> 1
+AWS Access Key ID.
+Leave blank for anonymous access or runtime credentials.
+Enter a string value. Press Enter for the default ("").
+access_key_id> ZnIx.xxxxxxxxxxxxxxx
+AWS Secret Access Key (password)
+Leave blank for anonymous access or runtime credentials.
+Enter a string value. Press Enter for the default ("").
+secret_access_key> xxxxxxxxxxx
+
+- Select endpoint for Leviia.
+
+ / The default endpoint
+ 1 | Leviia.
+ \ (s3.leviia.com)
+[snip]
+endpoint> 1
+
+- Choose acl.
+
+Note that this ACL is applied when server-side copying objects as S3
+doesn't copy the ACL from the source but rather writes a fresh one.
+Enter a string value. Press Enter for the default ("").
+Choose a number from below, or type in your own value
+ / Owner gets FULL_CONTROL.
+ 1 | No one else has access rights (default).
+ \ (private)
+ / Owner gets FULL_CONTROL.
+ 2 | The AllUsers group gets READ access.
+ \ (public-read)
+[snip]
+acl> 1
+Edit advanced config? (y/n)
+y) Yes
+n) No (default)
+y/n> n
+Remote config
+--------------------
+[leviia]
+- type: s3
+- provider: Leviia
+- access_key_id: ZnIx.xxxxxxx
+- secret_access_key: xxxxxxxx
+- endpoint: s3.leviia.com
+- acl: private
+--------------------
+y) Yes this is OK (default)
+e) Edit this remote
+d) Delete this remote
+y/e/d> y
+Current remotes:
+
+Name Type
+==== ====
+leviia s3
+Liara
+Here is an example of making a Liara Object Storage configuration. First run:
+rclone config
+This will guide you through an interactive setup process.
+No remotes found, make a new one?
+n) New remote
+s) Set configuration password
+n/s> n
+name> Liara
+Type of storage to configure.
+Choose a number from below, or type in your own value
+[snip]
+XX / Amazon S3 (also Dreamhost, Ceph, ChinaMobile, ArvanCloud, Liara, Minio)
+ \ "s3"
+[snip]
+Storage> s3
+Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars). Only applies if access_key_id and secret_access_key is blank.
+Choose a number from below, or type in your own value
+ 1 / Enter AWS credentials in the next step
+ \ "false"
+ 2 / Get AWS credentials from the environment (env vars or IAM)
+ \ "true"
+env_auth> 1
+AWS Access Key ID - leave blank for anonymous access or runtime credentials.
+access_key_id> YOURACCESSKEY
+AWS Secret Access Key (password) - leave blank for anonymous access or runtime credentials.
+secret_access_key> YOURSECRETACCESSKEY
+Region to connect to.
+Choose a number from below, or type in your own value
+ / The default endpoint
+ 1 | US Region, Northern Virginia, or Pacific Northwest.
+ | Leave location constraint empty.
+ \ "us-east-1"
+[snip]
+region>
+Endpoint for S3 API.
+Leave blank if using Liara to use the default endpoint for the region.
+Specify if using an S3 clone such as Ceph.
+endpoint> storage.iran.liara.space
+Canned ACL used when creating buckets and/or storing objects in S3.
+For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
+Choose a number from below, or type in your own value
+ 1 / Owner gets FULL_CONTROL. No one else has access rights (default).
+ \ "private"
+[snip]
+acl>
+The server-side encryption algorithm used when storing this object in S3.
+Choose a number from below, or type in your own value
+ 1 / None
+ \ ""
+ 2 / AES256
+ \ "AES256"
+server_side_encryption>
+The storage class to use when storing objects in S3.
+Choose a number from below, or type in your own value
+ 1 / Default
+ \ ""
+ 2 / Standard storage class
+ \ "STANDARD"
+storage_class>
+Remote config
+--------------------
+[Liara]
env_auth = false
-access_key_id =
-secret_access_key =
-region = us-east-1
-endpoint =
+access_key_id = YOURACCESSKEY
+secret_access_key = YOURSECRETACCESSKEY
+endpoint = storage.iran.liara.space
+location_constraint =
+acl =
+server_side_encryption =
+storage_class =
+--------------------
+y) Yes this is OK
+e) Edit this remote
+d) Delete this remote
+y/e/d> y
+This will leave the config file looking like this.
+[Liara]
+type = s3
+provider = Liara
+env_auth = false
+access_key_id = YOURACCESSKEY
+secret_access_key = YOURSECRETACCESSKEY
+region =
+endpoint = storage.iran.liara.space
+location_constraint =
+acl =
+server_side_encryption =
+storage_class =
+Linode
+Here is an example of making a Linode Object Storage configuration. First run:
+rclone config
+This will guide you through an interactive setup process.
+No remotes found, make a new one?
+n) New remote
+s) Set configuration password
+q) Quit config
+n/s/q> n
+
+Enter name for new remote.
+name> linode
+
+Option Storage.
+Type of storage to configure.
+Choose a number from below, or type in your own value.
+[snip]
+ X / Amazon S3 Compliant Storage Providers including AWS, ...Linode, ...and others
+ \ (s3)
+[snip]
+Storage> s3
+
+Option provider.
+Choose your S3 provider.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+[snip]
+XX / Linode Object Storage
+ \ (Linode)
+[snip]
+provider> Linode
+
+Option env_auth.
+Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
+Only applies if access_key_id and secret_access_key is blank.
+Choose a number from below, or type in your own boolean value (true or false).
+Press Enter for the default (false).
+ 1 / Enter AWS credentials in the next step.
+ \ (false)
+ 2 / Get AWS credentials from the environment (env vars or IAM).
+ \ (true)
+env_auth>
+
+Option access_key_id.
+AWS Access Key ID.
+Leave blank for anonymous access or runtime credentials.
+Enter a value. Press Enter to leave empty.
+access_key_id> ACCESS_KEY
+
+Option secret_access_key.
+AWS Secret Access Key (password).
+Leave blank for anonymous access or runtime credentials.
+Enter a value. Press Enter to leave empty.
+secret_access_key> SECRET_ACCESS_KEY
+
+Option endpoint.
+Endpoint for Linode Object Storage API.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+ 1 / Atlanta, GA (USA), us-southeast-1
+ \ (us-southeast-1.linodeobjects.com)
+ 2 / Chicago, IL (USA), us-ord-1
+ \ (us-ord-1.linodeobjects.com)
+ 3 / Frankfurt (Germany), eu-central-1
+ \ (eu-central-1.linodeobjects.com)
+ 4 / Milan (Italy), it-mil-1
+ \ (it-mil-1.linodeobjects.com)
+ 5 / Newark, NJ (USA), us-east-1
+ \ (us-east-1.linodeobjects.com)
+ 6 / Paris (France), fr-par-1
+ \ (fr-par-1.linodeobjects.com)
+ 7 / Seattle, WA (USA), us-sea-1
+ \ (us-sea-1.linodeobjects.com)
+ 8 / Singapore ap-south-1
+ \ (ap-south-1.linodeobjects.com)
+ 9 / Stockholm (Sweden), se-sto-1
+ \ (se-sto-1.linodeobjects.com)
+10 / Washington, DC, (USA), us-iad-1
+ \ (us-iad-1.linodeobjects.com)
+endpoint> 3
+
+Option acl.
+Canned ACL used when creating buckets and storing or copying objects.
+This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.
+For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
+Note that this ACL is applied when server-side copying objects as S3
+doesn't copy the ACL from the source but rather writes a fresh one.
+If the acl is an empty string then no X-Amz-Acl: header is added and
+the default (private) will be used.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+ / Owner gets FULL_CONTROL.
+ 1 | No one else has access rights (default).
+ \ (private)
+[snip]
+acl>
+
+Edit advanced config?
+y) Yes
+n) No (default)
+y/n> n
+
+Configuration complete.
+Options:
+- type: s3
+- provider: Linode
+- access_key_id: ACCESS_KEY
+- secret_access_key: SECRET_ACCESS_KEY
+- endpoint: eu-central-1.linodeobjects.com
+Keep this "linode" remote?
+y) Yes this is OK (default)
+e) Edit this remote
+d) Delete this remote
+y/e/d> y
+This will leave the config file looking like this.
+[linode]
+type = s3
+provider = Linode
+access_key_id = ACCESS_KEY
+secret_access_key = SECRET_ACCESS_KEY
+endpoint = eu-central-1.linodeobjects.com
+ArvanCloud
+ArvanCloud ArvanCloud Object Storage goes beyond the limited traditional file storage. It gives you access to backup and archived files and allows sharing. Files like profile image in the app, images sent by users or scanned documents can be stored securely and easily in our Object Storage service.
+ArvanCloud provides an S3 interface which can be configured for use with rclone like this.
+No remotes found, make a new one?
+n) New remote
+s) Set configuration password
+n/s> n
+name> ArvanCloud
+Type of storage to configure.
+Choose a number from below, or type in your own value
+[snip]
+XX / Amazon S3 (also Dreamhost, Ceph, ChinaMobile, ArvanCloud, Liara, Minio)
+ \ "s3"
+[snip]
+Storage> s3
+Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars). Only applies if access_key_id and secret_access_key is blank.
+Choose a number from below, or type in your own value
+ 1 / Enter AWS credentials in the next step
+ \ "false"
+ 2 / Get AWS credentials from the environment (env vars or IAM)
+ \ "true"
+env_auth> 1
+AWS Access Key ID - leave blank for anonymous access or runtime credentials.
+access_key_id> YOURACCESSKEY
+AWS Secret Access Key (password) - leave blank for anonymous access or runtime credentials.
+secret_access_key> YOURSECRETACCESSKEY
+Region to connect to.
+Choose a number from below, or type in your own value
+ / The default endpoint - a good choice if you are unsure.
+ 1 | US Region, Northern Virginia, or Pacific Northwest.
+ | Leave location constraint empty.
+ \ "us-east-1"
+[snip]
+region>
+Endpoint for S3 API.
+Leave blank if using ArvanCloud to use the default endpoint for the region.
+Specify if using an S3 clone such as Ceph.
+endpoint> s3.arvanstorage.com
+Location constraint - must be set to match the Region. Used when creating buckets only.
+Choose a number from below, or type in your own value
+ 1 / Empty for Iran-Tehran Region.
+ \ ""
+[snip]
+location_constraint>
+Canned ACL used when creating buckets and/or storing objects in S3.
+For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
+Choose a number from below, or type in your own value
+ 1 / Owner gets FULL_CONTROL. No one else has access rights (default).
+ \ "private"
+[snip]
+acl>
+The server-side encryption algorithm used when storing this object in S3.
+Choose a number from below, or type in your own value
+ 1 / None
+ \ ""
+ 2 / AES256
+ \ "AES256"
+server_side_encryption>
+The storage class to use when storing objects in S3.
+Choose a number from below, or type in your own value
+ 1 / Default
+ \ ""
+ 2 / Standard storage class
+ \ "STANDARD"
+storage_class>
+Remote config
+--------------------
+[ArvanCloud]
+env_auth = false
+access_key_id = YOURACCESSKEY
+secret_access_key = YOURSECRETACCESSKEY
+region = ir-thr-at1
+endpoint = s3.arvanstorage.com
location_constraint =
-acl = private
+acl =
server_side_encryption =
-storage_class =
-Then use it as normal with the name of the public bucket, e.g.
-rclone lsd anons3:1000genomes
-You will be able to list and copy data but not upload it.
-Providers
-AWS S3
-This is the provider used as main example and described in the configuration section above.
-AWS Snowball Edge
-AWS Snowball is a hardware appliance used for transferring bulk data back to AWS. Its main software interface is S3 object storage.
-To use rclone with AWS Snowball Edge devices, configure as standard for an 'S3 Compatible Service'.
-If using rclone pre v1.59 be sure to set upload_cutoff = 0 otherwise you will run into authentication header issues as the snowball device does not support query parameter based authentication.
-With rclone v1.59 or later setting upload_cutoff should not be necessary.
-eg.
-[snowball]
-type = s3
-provider = Other
-access_key_id = YOUR_ACCESS_KEY
-secret_access_key = YOUR_SECRET_KEY
-endpoint = http://[IP of Snowball]:8080
-upload_cutoff = 0
-Ceph
-Ceph is an open-source, unified, distributed storage system designed for excellent performance, reliability and scalability. It has an S3 compatible object storage interface.
-To use rclone with Ceph, configure as above but leave the region blank and set the endpoint. You should end up with something like this in your config:
-[ceph]
+storage_class =
+--------------------
+y) Yes this is OK
+e) Edit this remote
+d) Delete this remote
+y/e/d> y
+This will leave the config file looking like this.
+[ArvanCloud]
type = s3
-provider = Ceph
+provider = ArvanCloud
env_auth = false
-access_key_id = XXX
-secret_access_key = YYY
+access_key_id = YOURACCESSKEY
+secret_access_key = YOURSECRETACCESSKEY
region =
-endpoint = https://ceph.endpoint.example.com
+endpoint = s3.arvanstorage.com
location_constraint =
acl =
server_side_encryption =
storage_class =
-If you are using an older version of CEPH (e.g. 10.2.x Jewel) and a version of rclone before v1.59 then you may need to supply the parameter --s3-upload-cutoff 0 or put this in the config file as upload_cutoff 0 to work around a bug which causes uploading of small files to fail.
-Note also that Ceph sometimes puts / in the passwords it gives users. If you read the secret access key using the command line tools you will get a JSON blob with the / escaped as \/. Make sure you only write / in the secret access key.
-Eg the dump from Ceph looks something like this (irrelevant keys removed).
-{
- "user_id": "xxx",
- "display_name": "xxxx",
- "keys": [
- {
- "user": "xxx",
- "access_key": "xxxxxx",
- "secret_key": "xxxxxx\/xxxx"
- }
- ],
-}
-Because this is a json dump, it is encoding the / as \/, so if you use the secret key as xxxxxx/xxxx it will work fine.
-Cloudflare R2
-Cloudflare R2 Storage allows developers to store large amounts of unstructured data without the costly egress bandwidth fees associated with typical cloud storage services.
-Here is an example of making a Cloudflare R2 configuration. First run:
-rclone config
+Tencent COS
+Tencent Cloud Object Storage (COS) is a distributed storage service offered by Tencent Cloud for unstructured data. It is secure, stable, massive, convenient, low-delay and low-cost.
+To configure access to Tencent COS, follow the steps below:
+
+- Run
rclone config and select n for a new remote.
+
+rclone config
+No remotes found, make a new one?
+n) New remote
+s) Set configuration password
+q) Quit config
+n/s/q> n
+
+- Give the name of the configuration. For example, name it 'cos'.
+
+name> cos
+
+- Select
s3 storage.
+
+Choose a number from below, or type in your own value
+1 / 1Fichier
+ \ "fichier"
+ 2 / Alias for an existing remote
+ \ "alias"
+ 3 / Amazon Drive
+ \ "amazon cloud drive"
+ 4 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, ChinaMobile, ArvanCloud, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, Liara, Minio, and Tencent COS
+ \ "s3"
+[snip]
+Storage> s3
+
+- Select
TencentCOS provider.
+
+Choose a number from below, or type in your own value
+1 / Amazon Web Services (AWS) S3
+ \ "AWS"
+[snip]
+11 / Tencent Cloud Object Storage (COS)
+ \ "TencentCOS"
+[snip]
+provider> TencentCOS
+
+- Enter your SecretId and SecretKey of Tencent Cloud.
+
+Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
+Only applies if access_key_id and secret_access_key is blank.
+Enter a boolean value (true or false). Press Enter for the default ("false").
+Choose a number from below, or type in your own value
+ 1 / Enter AWS credentials in the next step
+ \ "false"
+ 2 / Get AWS credentials from the environment (env vars or IAM)
+ \ "true"
+env_auth> 1
+AWS Access Key ID.
+Leave blank for anonymous access or runtime credentials.
+Enter a string value. Press Enter for the default ("").
+access_key_id> AKIDxxxxxxxxxx
+AWS Secret Access Key (password)
+Leave blank for anonymous access or runtime credentials.
+Enter a string value. Press Enter for the default ("").
+secret_access_key> xxxxxxxxxxx
+
+- Select endpoint for Tencent COS. This is the standard endpoint for different region.
+
+ 1 / Beijing Region.
+ \ "cos.ap-beijing.myqcloud.com"
+ 2 / Nanjing Region.
+ \ "cos.ap-nanjing.myqcloud.com"
+ 3 / Shanghai Region.
+ \ "cos.ap-shanghai.myqcloud.com"
+ 4 / Guangzhou Region.
+ \ "cos.ap-guangzhou.myqcloud.com"
+[snip]
+endpoint> 4
+
+- Choose acl and storage class.
+
+Note that this ACL is applied when server-side copying objects as S3
+doesn't copy the ACL from the source but rather writes a fresh one.
+Enter a string value. Press Enter for the default ("").
+Choose a number from below, or type in your own value
+ 1 / Owner gets Full_CONTROL. No one else has access rights (default).
+ \ "default"
+[snip]
+acl> 1
+The storage class to use when storing new objects in Tencent COS.
+Enter a string value. Press Enter for the default ("").
+Choose a number from below, or type in your own value
+ 1 / Default
+ \ ""
+[snip]
+storage_class> 1
+Edit advanced config? (y/n)
+y) Yes
+n) No (default)
+y/n> n
+Remote config
+--------------------
+[cos]
+type = s3
+provider = TencentCOS
+env_auth = false
+access_key_id = xxx
+secret_access_key = xxx
+endpoint = cos.ap-guangzhou.myqcloud.com
+acl = default
+--------------------
+y) Yes this is OK (default)
+e) Edit this remote
+d) Delete this remote
+y/e/d> y
+Current remotes:
+
+Name Type
+==== ====
+cos s3
+Netease NOS
+For Netease NOS configure as per the configurator rclone config setting the provider Netease. This will automatically set force_path_style = false which is necessary for it to run properly.
+Petabox
+Here is an example of making a Petabox configuration. First run:
+rclone config
This will guide you through an interactive setup process.
-Note that all buckets are private, and all are stored in the same "auto" region. It is necessary to use Cloudflare workers to share the content of a bucket publicly.
No remotes found, make a new one?
n) New remote
s) Set configuration password
-q) Quit config
-n/s/q> n
-name> r2
+n/s> n
+
+Enter name for new remote.
+name> My Petabox Storage
+
Option Storage.
Type of storage to configure.
Choose a number from below, or type in your own value.
-...
-XX / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, China Mobile, Cloudflare, ArvanCloud, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, Lyve Cloud, Minio, Netease, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Synology, Tencent COS and Wasabi
- \ (s3)
-...
+[snip]
+XX / Amazon S3 Compliant Storage Providers including AWS, ...
+ \ "s3"
+[snip]
Storage> s3
+
Option provider.
Choose your S3 provider.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
-...
-XX / Cloudflare R2 Storage
- \ (Cloudflare)
-...
-provider> Cloudflare
+[snip]
+XX / Petabox Object Storage
+ \ (Petabox)
+[snip]
+provider> Petabox
+
Option env_auth.
Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
Only applies if access_key_id and secret_access_key is blank.
@@ -16309,137 +16967,116 @@ Cloudflare R2
2 / Get AWS credentials from the environment (env vars or IAM).
\ (true)
env_auth> 1
+
Option access_key_id.
AWS Access Key ID.
Leave blank for anonymous access or runtime credentials.
Enter a value. Press Enter to leave empty.
-access_key_id> ACCESS_KEY
+access_key_id> YOUR_ACCESS_KEY_ID
+
Option secret_access_key.
AWS Secret Access Key (password).
Leave blank for anonymous access or runtime credentials.
Enter a value. Press Enter to leave empty.
-secret_access_key> SECRET_ACCESS_KEY
+secret_access_key> YOUR_SECRET_ACCESS_KEY
+
Option region.
-Region to connect to.
+Region where your bucket will be created and your data stored.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
- 1 / R2 buckets are automatically distributed across Cloudflare's data centers for low latency.
- \ (auto)
+ 1 / US East (N. Virginia)
+ \ (us-east-1)
+ 2 / Europe (Frankfurt)
+ \ (eu-central-1)
+ 3 / Asia Pacific (Singapore)
+ \ (ap-southeast-1)
+ 4 / Middle East (Bahrain)
+ \ (me-south-1)
+ 5 / South America (São Paulo)
+ \ (sa-east-1)
region> 1
+
Option endpoint.
-Endpoint for S3 API.
-Required when using an S3 clone.
-Enter a value. Press Enter to leave empty.
-endpoint> https://ACCOUNT_ID.r2.cloudflarestorage.com
+Endpoint for Petabox S3 Object Storage.
+Specify the endpoint from the same region.
+Choose a number from below, or type in your own value.
+ 1 / US East (N. Virginia)
+ \ (s3.petabox.io)
+ 2 / US East (N. Virginia)
+ \ (s3.us-east-1.petabox.io)
+ 3 / Europe (Frankfurt)
+ \ (s3.eu-central-1.petabox.io)
+ 4 / Asia Pacific (Singapore)
+ \ (s3.ap-southeast-1.petabox.io)
+ 5 / Middle East (Bahrain)
+ \ (s3.me-south-1.petabox.io)
+ 6 / South America (São Paulo)
+ \ (s3.sa-east-1.petabox.io)
+endpoint> 1
+
+Option acl.
+Canned ACL used when creating buckets and storing or copying objects.
+This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.
+For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
+Note that this ACL is applied when server-side copying objects as S3
+doesn't copy the ACL from the source but rather writes a fresh one.
+If the acl is an empty string then no X-Amz-Acl: header is added and
+the default (private) will be used.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+ / Owner gets FULL_CONTROL.
+ 1 | No one else has access rights (default).
+ \ (private)
+ / Owner gets FULL_CONTROL.
+ 2 | The AllUsers group gets READ access.
+ \ (public-read)
+ / Owner gets FULL_CONTROL.
+ 3 | The AllUsers group gets READ and WRITE access.
+ | Granting this on a bucket is generally not recommended.
+ \ (public-read-write)
+ / Owner gets FULL_CONTROL.
+ 4 | The AuthenticatedUsers group gets READ access.
+ \ (authenticated-read)
+ / Object owner gets FULL_CONTROL.
+ 5 | Bucket owner gets READ access.
+ | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
+ \ (bucket-owner-read)
+ / Both the object owner and the bucket owner get FULL_CONTROL over the object.
+ 6 | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
+ \ (bucket-owner-full-control)
+acl> 1
+
Edit advanced config?
y) Yes
n) No (default)
-y/n> n
---------------------
+y/n> No
+
+Configuration complete.
+Options:
+- type: s3
+- provider: Petabox
+- access_key_id: YOUR_ACCESS_KEY_ID
+- secret_access_key: YOUR_SECRET_ACCESS_KEY
+- region: us-east-1
+- endpoint: s3.petabox.io
+Keep this "My Petabox Storage" remote?
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y
-This will leave your config looking something like:
-[r2]
-type = s3
-provider = Cloudflare
-access_key_id = ACCESS_KEY
-secret_access_key = SECRET_ACCESS_KEY
-region = auto
-endpoint = https://ACCOUNT_ID.r2.cloudflarestorage.com
-acl = private
-Now run rclone lsf r2: to see your buckets and rclone lsf r2:bucket to look within a bucket.
-Dreamhost
-Dreamhost DreamObjects is an object storage system based on CEPH.
-To use rclone with Dreamhost, configure as above but leave the region blank and set the endpoint. You should end up with something like this in your config:
-[dreamobjects]
-type = s3
-provider = DreamHost
-env_auth = false
-access_key_id = your_access_key
-secret_access_key = your_secret_key
-region =
-endpoint = objects-us-west-1.dream.io
-location_constraint =
-acl = private
-server_side_encryption =
-storage_class =
-Google Cloud Storage
-GoogleCloudStorage is an S3-interoperable object storage service from Google Cloud Platform.
-To connect to Google Cloud Storage you will need an access key and secret key. These can be retrieved by creating an HMAC key.
-[gs]
-type = s3
-provider = GCS
-access_key_id = your_access_key
-secret_access_key = your_secret_key
-endpoint = https://storage.googleapis.com
-DigitalOcean Spaces
-Spaces is an S3-interoperable object storage service from cloud provider DigitalOcean.
-To connect to DigitalOcean Spaces you will need an access key and secret key. These can be retrieved on the "Applications & API" page of the DigitalOcean control panel. They will be needed when prompted by rclone config for your access_key_id and secret_access_key.
-When prompted for a region or location_constraint, press enter to use the default value. The region must be included in the endpoint setting (e.g. nyc3.digitaloceanspaces.com). The default values can be used for other settings.
-Going through the whole process of creating a new remote by running rclone config, each prompt should be answered as shown below:
-Storage> s3
-env_auth> 1
-access_key_id> YOUR_ACCESS_KEY
-secret_access_key> YOUR_SECRET_KEY
-region>
-endpoint> nyc3.digitaloceanspaces.com
-location_constraint>
-acl>
-storage_class>
-The resulting configuration file should look like:
-[spaces]
-type = s3
-provider = DigitalOcean
-env_auth = false
-access_key_id = YOUR_ACCESS_KEY
-secret_access_key = YOUR_SECRET_KEY
-region =
-endpoint = nyc3.digitaloceanspaces.com
-location_constraint =
-acl =
-server_side_encryption =
-storage_class =
-Once configured, you can create a new Space and begin copying files. For example:
-rclone mkdir spaces:my-new-space
-rclone copy /path/to/files spaces:my-new-space
-Huawei OBS
-Object Storage Service (OBS) provides stable, secure, efficient, and easy-to-use cloud storage that lets you store virtually any volume of unstructured data in any format and access it from anywhere.
-OBS provides an S3 interface, you can copy and modify the following configuration and add it to your rclone configuration file.
-[obs]
+This will leave the config file looking like this.
+[My Petabox Storage]
type = s3
-provider = HuaweiOBS
-access_key_id = your-access-key-id
-secret_access_key = your-secret-access-key
-region = af-south-1
-endpoint = obs.af-south-1.myhuaweicloud.com
-acl = private
-Or you can also configure via the interactive command line:
-No remotes found, make a new one?
-n) New remote
-s) Set configuration password
-q) Quit config
-n/s/q> n
-name> obs
-Option Storage.
-Type of storage to configure.
-Choose a number from below, or type in your own value.
-[snip]
- 5 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, China Mobile, Cloudflare, ArvanCloud, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, Lyve Cloud, Minio, Netease, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Synology, Tencent COS and Wasabi
- \ (s3)
-[snip]
-Storage> 5
-Option provider.
-Choose your S3 provider.
-Choose a number from below, or type in your own value.
-Press Enter to leave empty.
-[snip]
- 9 / Huawei Object Storage Service
- \ (HuaweiOBS)
-[snip]
-provider> 9
-Option env_auth.
+provider = Petabox
+access_key_id = YOUR_ACCESS_KEY_ID
+secret_access_key = YOUR_SECRET_ACCESS_KEY
+region = us-east-1
+endpoint = s3.petabox.io
+Storj
+Storj is a decentralized cloud storage which can be used through its native protocol or an S3 compatible gateway.
+The S3 compatible gateway is configured using rclone config with a type of s3 and with a provider name of Storj. Here is an example run of the configurator.
+Type of storage to configure.
+Storage> s3
Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
Only applies if access_key_id and secret_access_key is blank.
Choose a number from below, or type in your own boolean value (true or false).
@@ -16453,5857 +17090,5131 @@ Huawei OBS
AWS Access Key ID.
Leave blank for anonymous access or runtime credentials.
Enter a value. Press Enter to leave empty.
-access_key_id> your-access-key-id
+access_key_id> XXXX (as shown when creating the access grant)
Option secret_access_key.
AWS Secret Access Key (password).
Leave blank for anonymous access or runtime credentials.
-Enter a value. Press Enter to leave empty.
-secret_access_key> your-secret-access-key
-Option region.
-Region to connect to.
+Enter a value. Press Enter to leave empty.
+secret_access_key> XXXX (as shown when creating the access grant)
+Option endpoint.
+Endpoint of the Shared Gateway.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+ 1 / EU1 Shared Gateway
+ \ (gateway.eu1.storjshare.io)
+ 2 / US1 Shared Gateway
+ \ (gateway.us1.storjshare.io)
+ 3 / Asia-Pacific Shared Gateway
+ \ (gateway.ap1.storjshare.io)
+endpoint> 1 (as shown when creating the access grant)
+Edit advanced config?
+y) Yes
+n) No (default)
+y/n> n
+Note that s3 credentials are generated when you create an access grant.
+Backend quirks
+
+--chunk-size is forced to be 64 MiB or greater. This will use more memory than the default of 5 MiB.- Server side copy is disabled as it isn't currently supported in the gateway.
+- GetTier and SetTier are not supported.
+
+Backend bugs
+Due to issue #39 uploading multipart files via the S3 gateway causes them to lose their metadata. For rclone's purpose this means that the modification time is not stored, nor is any MD5SUM (if one is available from the source).
+This has the following consequences:
+
+- Using
rclone rcat will fail as the medatada doesn't match after upload
+- Uploading files with
rclone mount will fail for the same reason
+
+- This can worked around by using
--vfs-cache-mode writes or --vfs-cache-mode full or setting --s3-upload-cutoff large
+
+- Files uploaded via a multipart upload won't have their modtimes
+
+- This will mean that
rclone sync will likely keep trying to upload files bigger than --s3-upload-cutoff
+- This can be worked around with
--checksum or --size-only or setting --s3-upload-cutoff large
+- The maximum value for
--s3-upload-cutoff is 5GiB though
+
+
+One general purpose workaround is to set --s3-upload-cutoff 5G. This means that rclone will upload files smaller than 5GiB as single parts. Note that this can be set in the config file with upload_cutoff = 5G or configured in the advanced settings. If you regularly transfer files larger than 5G then using --checksum or --size-only in rclone sync is the recommended workaround.
+Comparison with the native protocol
+Use the the native protocol to take advantage of client-side encryption as well as to achieve the best possible download performance. Uploads will be erasure-coded locally, thus a 1gb upload will result in 2.68gb of data being uploaded to storage nodes across the network.
+Use this backend and the S3 compatible Hosted Gateway to increase upload performance and reduce the load on your systems and network. Uploads will be encrypted and erasure-coded server-side, thus a 1GB upload will result in only in 1GB of data being uploaded to storage nodes across the network.
+For more detailed comparison please check the documentation of the storj backend.
+Limitations
+rclone about is not supported by the S3 backend. Backends without this capability cannot determine free space for an rclone mount or use policy mfs (most free space) as a member of an rclone union remote.
+See List of backends that do not support rclone about and rclone about
+Synology C2 Object Storage
+Synology C2 Object Storage provides a secure, S3-compatible, and cost-effective cloud storage solution without API request, download fees, and deletion penalty.
+The S3 compatible gateway is configured using rclone config with a type of s3 and with a provider name of Synology. Here is an example run of the configurator.
+First run:
+rclone config
+This will guide you through an interactive setup process.
+No remotes found, make a new one?
+n) New remote
+s) Set configuration password
+q) Quit config
+
+n/s/q> n
+
+Enter name for new remote.1
+name> syno
+
+Type of storage to configure.
+Enter a string value. Press Enter for the default ("").
+Choose a number from below, or type in your own value
+
+ 5 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, China Mobile, Cloudflare, GCS, ArvanCloud, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, IDrive e2, IONOS Cloud, Liara, Lyve Cloud, Minio, Netease, Petabox, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Synology, Tencent COS, Qiniu and Wasabi
+ \ "s3"
+
+Storage> s3
+
+Choose your S3 provider.
+Enter a string value. Press Enter for the default ("").
+Choose a number from below, or type in your own value
+ 24 / Synology C2 Object Storage
+ \ (Synology)
+
+provider> Synology
+
+Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
+Only applies if access_key_id and secret_access_key is blank.
+Enter a boolean value (true or false). Press Enter for the default ("false").
+Choose a number from below, or type in your own value
+ 1 / Enter AWS credentials in the next step
+ \ "false"
+ 2 / Get AWS credentials from the environment (env vars or IAM)
+ \ "true"
+
+env_auth> 1
+
+AWS Access Key ID.
+Leave blank for anonymous access or runtime credentials.
+Enter a string value. Press Enter for the default ("").
+
+access_key_id> accesskeyid
+
+AWS Secret Access Key (password)
+Leave blank for anonymous access or runtime credentials.
+Enter a string value. Press Enter for the default ("").
+
+secret_access_key> secretaccesskey
+
+Region where your data stored.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
- 1 / AF-Johannesburg
- \ (af-south-1)
- 2 / AP-Bangkok
- \ (ap-southeast-2)
-[snip]
-region> 1
+ 1 / Europe Region 1
+ \ (eu-001)
+ 2 / Europe Region 2
+ \ (eu-002)
+ 3 / US Region 1
+ \ (us-001)
+ 4 / US Region 2
+ \ (us-002)
+ 5 / Asia (Taiwan)
+ \ (tw-001)
+
+region > 1
+
Option endpoint.
-Endpoint for OBS API.
+Endpoint for Synology C2 Object Storage API.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
- 1 / AF-Johannesburg
- \ (obs.af-south-1.myhuaweicloud.com)
- 2 / AP-Bangkok
- \ (obs.ap-southeast-2.myhuaweicloud.com)
-[snip]
+ 1 / EU Endpoint 1
+ \ (eu-001.s3.synologyc2.net)
+ 2 / US Endpoint 1
+ \ (us-001.s3.synologyc2.net)
+ 3 / TW Endpoint 1
+ \ (tw-001.s3.synologyc2.net)
+
endpoint> 1
-Option acl.
-Canned ACL used when creating buckets and storing or copying objects.
-This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.
-For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
-Note that this ACL is applied when server-side copying objects as S3
-doesn't copy the ACL from the source but rather writes a fresh one.
-Choose a number from below, or type in your own value.
-Press Enter to leave empty.
- / Owner gets FULL_CONTROL.
- 1 | No one else has access rights (default).
- \ (private)
-[snip]
-acl> 1
-Edit advanced config?
+
+Option location_constraint.
+Location constraint - must be set to match the Region.
+Leave blank if not sure. Used when creating buckets only.
+Enter a value. Press Enter to leave empty.
+location_constraint>
+
+Edit advanced config? (y/n)
y) Yes
-n) No (default)
-y/n>
---------------------
-[obs]
-type = s3
-provider = HuaweiOBS
-access_key_id = your-access-key-id
-secret_access_key = your-secret-access-key
-region = af-south-1
-endpoint = obs.af-south-1.myhuaweicloud.com
-acl = private
---------------------
+n) No
+y/n> y
+
+Option no_check_bucket.
+If set, don't attempt to check the bucket exists or create it.
+This can be useful when trying to minimise the number of transactions
+rclone does if you know the bucket exists already.
+It can also be needed if the user you are using does not have bucket
+creation permissions. Before v1.52.0 this would have passed silently
+due to a bug.
+Enter a boolean value (true or false). Press Enter for the default (true).
+
+no_check_bucket> true
+
+Configuration complete.
+Options:
+- type: s3
+- provider: Synology
+- region: eu-001
+- endpoint: eu-001.s3.synologyc2.net
+- no_check_bucket: true
+Keep this "syno" remote?
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
+
y/e/d> y
-Current remotes:
-Name Type
-==== ====
-obs s3
+# Backblaze B2
-e) Edit existing remote
-n) New remote
-d) Delete remote
-r) Rename remote
-c) Copy remote
-s) Set configuration password
-q) Quit config
-e/n/d/r/c/s/q> q
-IBM COS (S3)
-Information stored with IBM Cloud Object Storage is encrypted and dispersed across multiple geographic locations, and accessed through an implementation of the S3 API. This service makes use of the distributed storage technologies provided by IBM’s Cloud Object Storage System (formerly Cleversafe). For more information visit: (http://www.ibm.com/cloud/object-storage)
-To configure access to IBM COS S3, follow the steps below:
-
-- Run rclone config and select n for a new remote.
-
- 2018/02/14 14:13:11 NOTICE: Config file "C:\\Users\\a\\.config\\rclone\\rclone.conf" not found - using defaults
- No remotes found, make a new one?
- n) New remote
- s) Set configuration password
- q) Quit config
- n/s/q> n
-
-- Enter the name for the configuration
-
- name> <YOUR NAME>
-
-- Select "s3" storage.
-
-Choose a number from below, or type in your own value
- 1 / Alias for an existing remote
- \ "alias"
- 2 / Amazon Drive
- \ "amazon cloud drive"
- 3 / Amazon S3 Complaint Storage Providers (Dreamhost, Ceph, ChinaMobile, Liara, ArvanCloud, Minio, IBM COS)
- \ "s3"
- 4 / Backblaze B2
- \ "b2"
-[snip]
- 23 / HTTP
- \ "http"
-Storage> 3
-
-- Select IBM COS as the S3 Storage Provider.
-
-Choose the S3 provider.
-Choose a number from below, or type in your own value
- 1 / Choose this option to configure Storage to AWS S3
- \ "AWS"
- 2 / Choose this option to configure Storage to Ceph Systems
- \ "Ceph"
- 3 / Choose this option to configure Storage to Dreamhost
- \ "Dreamhost"
- 4 / Choose this option to the configure Storage to IBM COS S3
- \ "IBMCOS"
- 5 / Choose this option to the configure Storage to Minio
- \ "Minio"
- Provider>4
-
-- Enter the Access Key and Secret.
-
- AWS Access Key ID - leave blank for anonymous access or runtime credentials.
- access_key_id> <>
- AWS Secret Access Key (password) - leave blank for anonymous access or runtime credentials.
- secret_access_key> <>
-
-- Specify the endpoint for IBM COS. For Public IBM COS, choose from the option below. For On Premise IBM COS, enter an endpoint address.
-
- Endpoint for IBM COS S3 API.
- Specify if using an IBM COS On Premise.
- Choose a number from below, or type in your own value
- 1 / US Cross Region Endpoint
- \ "s3-api.us-geo.objectstorage.softlayer.net"
- 2 / US Cross Region Dallas Endpoint
- \ "s3-api.dal.us-geo.objectstorage.softlayer.net"
- 3 / US Cross Region Washington DC Endpoint
- \ "s3-api.wdc-us-geo.objectstorage.softlayer.net"
- 4 / US Cross Region San Jose Endpoint
- \ "s3-api.sjc-us-geo.objectstorage.softlayer.net"
- 5 / US Cross Region Private Endpoint
- \ "s3-api.us-geo.objectstorage.service.networklayer.com"
- 6 / US Cross Region Dallas Private Endpoint
- \ "s3-api.dal-us-geo.objectstorage.service.networklayer.com"
- 7 / US Cross Region Washington DC Private Endpoint
- \ "s3-api.wdc-us-geo.objectstorage.service.networklayer.com"
- 8 / US Cross Region San Jose Private Endpoint
- \ "s3-api.sjc-us-geo.objectstorage.service.networklayer.com"
- 9 / US Region East Endpoint
- \ "s3.us-east.objectstorage.softlayer.net"
- 10 / US Region East Private Endpoint
- \ "s3.us-east.objectstorage.service.networklayer.com"
- 11 / US Region South Endpoint
-[snip]
- 34 / Toronto Single Site Private Endpoint
- \ "s3.tor01.objectstorage.service.networklayer.com"
- endpoint>1
-
-- Specify a IBM COS Location Constraint. The location constraint must match endpoint when using IBM Cloud Public. For on-prem COS, do not make a selection from this list, hit enter
-
- 1 / US Cross Region Standard
- \ "us-standard"
- 2 / US Cross Region Vault
- \ "us-vault"
- 3 / US Cross Region Cold
- \ "us-cold"
- 4 / US Cross Region Flex
- \ "us-flex"
- 5 / US East Region Standard
- \ "us-east-standard"
- 6 / US East Region Vault
- \ "us-east-vault"
- 7 / US East Region Cold
- \ "us-east-cold"
- 8 / US East Region Flex
- \ "us-east-flex"
- 9 / US South Region Standard
- \ "us-south-standard"
- 10 / US South Region Vault
- \ "us-south-vault"
-[snip]
- 32 / Toronto Flex
- \ "tor01-flex"
-location_constraint>1
-
-- Specify a canned ACL. IBM Cloud (Storage) supports "public-read" and "private". IBM Cloud(Infra) supports all the canned ACLs. On-Premise COS supports all the canned ACLs.
-
-Canned ACL used when creating buckets and/or storing objects in S3.
-For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
-Choose a number from below, or type in your own value
- 1 / Owner gets FULL_CONTROL. No one else has access rights (default). This acl is available on IBM Cloud (Infra), IBM Cloud (Storage), On-Premise COS
- \ "private"
- 2 / Owner gets FULL_CONTROL. The AllUsers group gets READ access. This acl is available on IBM Cloud (Infra), IBM Cloud (Storage), On-Premise IBM COS
- \ "public-read"
- 3 / Owner gets FULL_CONTROL. The AllUsers group gets READ and WRITE access. This acl is available on IBM Cloud (Infra), On-Premise IBM COS
- \ "public-read-write"
- 4 / Owner gets FULL_CONTROL. The AuthenticatedUsers group gets READ access. Not supported on Buckets. This acl is available on IBM Cloud (Infra) and On-Premise IBM COS
- \ "authenticated-read"
-acl> 1
-
-- Review the displayed configuration and accept to save the "remote" then quit. The config file should look like this
-
- [xxx]
- type = s3
- Provider = IBMCOS
- access_key_id = xxx
- secret_access_key = yyy
- endpoint = s3-api.us-geo.objectstorage.softlayer.net
- location_constraint = us-standard
- acl = private
-
-- Execute rclone commands
-
- 1) Create a bucket.
- rclone mkdir IBM-COS-XREGION:newbucket
- 2) List available buckets.
- rclone lsd IBM-COS-XREGION:
- -1 2017-11-08 21:16:22 -1 test
- -1 2018-02-14 20:16:39 -1 newbucket
- 3) List contents of a bucket.
- rclone ls IBM-COS-XREGION:newbucket
- 18685952 test.exe
- 4) Copy a file from local to remote.
- rclone copy /Users/file.txt IBM-COS-XREGION:newbucket
- 5) Copy a file from remote to local.
- rclone copy IBM-COS-XREGION:newbucket/file.txt .
- 6) Delete a file on remote.
- rclone delete IBM-COS-XREGION:newbucket/file.txt
-IDrive e2
-Here is an example of making an IDrive e2 configuration. First run:
-rclone config
-This will guide you through an interactive setup process.
-No remotes found, make a new one?
-n) New remote
-s) Set configuration password
-q) Quit config
-n/s/q> n
+B2 is [Backblaze's cloud storage system](https://www.backblaze.com/b2/).
+
+Paths are specified as `remote:bucket` (or `remote:` for the `lsd`
+command.) You may put subdirectories in too, e.g. `remote:bucket/path/to/dir`.
+
+## Configuration
+
+Here is an example of making a b2 configuration. First run
+
+ rclone config
+
+This will guide you through an interactive setup process. To authenticate
+you will either need your Account ID (a short hex number) and Master
+Application Key (a long hex number) OR an Application Key, which is the
+recommended method. See below for further details on generating and using
+an Application Key.
+
+No remotes found, make a new one? n) New remote q) Quit config n/q> n name> remote Type of storage to configure. Choose a number from below, or type in your own value [snip] XX / Backblaze B2 Â "b2" [snip] Storage> b2 Account ID or Application Key ID account> 123456789abc Application Key key> 0123456789abcdef0123456789abcdef0123456789 Endpoint for the service - leave blank normally. endpoint> Remote config -------------------- [remote] account = 123456789abc key = 0123456789abcdef0123456789abcdef0123456789 endpoint = -------------------- y) Yes this is OK e) Edit this remote d) Delete this remote y/e/d> y
+
+This remote is called `remote` and can now be used like this
+
+See all buckets
+
+ rclone lsd remote:
+
+Create a new bucket
+
+ rclone mkdir remote:bucket
+
+List the contents of a bucket
+
+ rclone ls remote:bucket
+
+Sync `/home/local/directory` to the remote bucket, deleting any
+excess files in the bucket.
+
+ rclone sync --interactive /home/local/directory remote:bucket
+
+### Application Keys
+
+B2 supports multiple [Application Keys for different access permission
+to B2 Buckets](https://www.backblaze.com/b2/docs/application_keys.html).
+
+You can use these with rclone too; you will need to use rclone version 1.43
+or later.
+
+Follow Backblaze's docs to create an Application Key with the required
+permission and add the `applicationKeyId` as the `account` and the
+`Application Key` itself as the `key`.
+
+Note that you must put the _applicationKeyId_ as the `account` – you
+can't use the master Account ID. If you try then B2 will return 401
+errors.
+
+### --fast-list
+
+This remote supports `--fast-list` which allows you to use fewer
+transactions in exchange for more memory. See the [rclone
+docs](https://rclone.org/docs/#fast-list) for more details.
+
+### Modification times
+
+The modification time is stored as metadata on the object as
+`X-Bz-Info-src_last_modified_millis` as milliseconds since 1970-01-01
+in the Backblaze standard. Other tools should be able to use this as
+a modified time.
+
+Modified times are used in syncing and are fully supported. Note that
+if a modification time needs to be updated on an object then it will
+create a new version of the object.
+
+### Restricted filename characters
+
+In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters)
+the following characters are also replaced:
+
+| Character | Value | Replacement |
+| --------- |:-----:|:-----------:|
+| \ | 0x5C | ï¼¼ |
+
+Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8),
+as they can't be used in JSON strings.
+
+Note that in 2020-05 Backblaze started allowing \ characters in file
+names. Rclone hasn't changed its encoding as this could cause syncs to
+re-transfer files. If you want rclone not to replace \ then see the
+`--b2-encoding` flag below and remove the `BackSlash` from the
+string. This can be set in the config.
+
+### SHA1 checksums
+
+The SHA1 checksums of the files are checked on upload and download and
+will be used in the syncing process.
+
+Large files (bigger than the limit in `--b2-upload-cutoff`) which are
+uploaded in chunks will store their SHA1 on the object as
+`X-Bz-Info-large_file_sha1` as recommended by Backblaze.
+
+For a large file to be uploaded with an SHA1 checksum, the source
+needs to support SHA1 checksums. The local disk supports SHA1
+checksums so large file transfers from local disk will have an SHA1.
+See [the overview](https://rclone.org/overview/#features) for exactly which remotes
+support SHA1.
+
+Sources which don't support SHA1, in particular `crypt` will upload
+large files without SHA1 checksums. This may be fixed in the future
+(see [#1767](https://github.com/rclone/rclone/issues/1767)).
+
+Files sizes below `--b2-upload-cutoff` will always have an SHA1
+regardless of the source.
+
+### Transfers
+
+Backblaze recommends that you do lots of transfers simultaneously for
+maximum speed. In tests from my SSD equipped laptop the optimum
+setting is about `--transfers 32` though higher numbers may be used
+for a slight speed improvement. The optimum number for you may vary
+depending on your hardware, how big the files are, how much you want
+to load your computer, etc. The default of `--transfers 4` is
+definitely too low for Backblaze B2 though.
+
+Note that uploading big files (bigger than 200 MiB by default) will use
+a 96 MiB RAM buffer by default. There can be at most `--transfers` of
+these in use at any moment, so this sets the upper limit on the memory
+used.
+
+### Versions
+
+When rclone uploads a new version of a file it creates a [new version
+of it](https://www.backblaze.com/b2/docs/file_versions.html).
+Likewise when you delete a file, the old version will be marked hidden
+and still be available. Conversely, you may opt in to a "hard delete"
+of files with the `--b2-hard-delete` flag which would permanently remove
+the file instead of hiding it.
+
+Old versions of files, where available, are visible using the
+`--b2-versions` flag.
+
+It is also possible to view a bucket as it was at a certain point in time,
+using the `--b2-version-at` flag. This will show the file versions as they
+were at that time, showing files that have been deleted afterwards, and
+hiding files that were created since.
+
+If you wish to remove all the old versions then you can use the
+`rclone cleanup remote:bucket` command which will delete all the old
+versions of files, leaving the current ones intact. You can also
+supply a path and only old versions under that path will be deleted,
+e.g. `rclone cleanup remote:bucket/path/to/stuff`.
+
+Note that `cleanup` will remove partially uploaded files from the bucket
+if they are more than a day old.
+
+When you `purge` a bucket, the current and the old versions will be
+deleted then the bucket will be deleted.
+
+However `delete` will cause the current versions of the files to
+become hidden old versions.
+
+Here is a session showing the listing and retrieval of an old
+version followed by a `cleanup` of the old versions.
+
+Show current version and all the versions with `--b2-versions` flag.
+
+$ rclone -q ls b2:cleanup-test 9 one.txt
+$ rclone -q --b2-versions ls b2:cleanup-test 9 one.txt 8 one-v2016-07-04-141032-000.txt 16 one-v2016-07-04-141003-000.txt 15 one-v2016-07-02-155621-000.txt
+
+Retrieve an old version
+
+$ rclone -q --b2-versions copy b2:cleanup-test/one-v2016-07-04-141003-000.txt /tmp
+$ ls -l /tmp/one-v2016-07-04-141003-000.txt -rw-rw-r-- 1 ncw ncw 16 Jul 2 17:46 /tmp/one-v2016-07-04-141003-000.txt
+
+Clean up all the old versions and show that they've gone.
+
+$ rclone -q cleanup b2:cleanup-test
+$ rclone -q ls b2:cleanup-test 9 one.txt
+$ rclone -q --b2-versions ls b2:cleanup-test 9 one.txt
+
+#### Versions naming caveat
+
+When using `--b2-versions` flag rclone is relying on the file name
+to work out whether the objects are versions or not. Versions' names
+are created by inserting timestamp between file name and its extension.
+ 9 file.txt
+ 8 file-v2023-07-17-161032-000.txt
+ 16 file-v2023-06-15-141003-000.txt
+If there are real files present with the same names as versions, then
+behaviour of `--b2-versions` can be unpredictable.
+
+### Data usage
+
+It is useful to know how many requests are sent to the server in different scenarios.
+
+All copy commands send the following 4 requests:
+
+/b2api/v1/b2_authorize_account /b2api/v1/b2_create_bucket /b2api/v1/b2_list_buckets /b2api/v1/b2_list_file_names
+
+The `b2_list_file_names` request will be sent once for every 1k files
+in the remote path, providing the checksum and modification time of
+the listed files. As of version 1.33 issue
+[#818](https://github.com/rclone/rclone/issues/818) causes extra requests
+to be sent when using B2 with Crypt. When a copy operation does not
+require any files to be uploaded, no more requests will be sent.
+
+Uploading files that do not require chunking, will send 2 requests per
+file upload:
+
+/b2api/v1/b2_get_upload_url /b2api/v1/b2_upload_file/
+
+Uploading files requiring chunking, will send 2 requests (one each to
+start and finish the upload) and another 2 requests for each chunk:
+
+/b2api/v1/b2_start_large_file /b2api/v1/b2_get_upload_part_url /b2api/v1/b2_upload_part/ /b2api/v1/b2_finish_large_file
+
+#### Versions
+
+Versions can be viewed with the `--b2-versions` flag. When it is set
+rclone will show and act on older versions of files. For example
+
+Listing without `--b2-versions`
+
+$ rclone -q ls b2:cleanup-test 9 one.txt
+
+And with
+
+$ rclone -q --b2-versions ls b2:cleanup-test 9 one.txt 8 one-v2016-07-04-141032-000.txt 16 one-v2016-07-04-141003-000.txt 15 one-v2016-07-02-155621-000.txt
+
+Showing that the current version is unchanged but older versions can
+be seen. These have the UTC date that they were uploaded to the
+server to the nearest millisecond appended to them.
+
+Note that when using `--b2-versions` no file write operations are
+permitted, so you can't upload files or delete them.
+
+### B2 and rclone link
-Enter name for new remote.
-name> e2
+Rclone supports generating file share links for private B2 buckets.
+They can either be for a file for example:
+
+./rclone link B2:bucket/path/to/file.txt https://f002.backblazeb2.com/file/bucket/path/to/file.txt?Authorization=xxxxxxxx
+
+or if run on a directory you will get:
+
+./rclone link B2:bucket/path https://f002.backblazeb2.com/file/bucket/path?Authorization=xxxxxxxx
+
+you can then use the authorization token (the part of the url from the
+ `?Authorization=` on) on any file path under that directory. For example:
+
+https://f002.backblazeb2.com/file/bucket/path/to/file1?Authorization=xxxxxxxx https://f002.backblazeb2.com/file/bucket/path/file2?Authorization=xxxxxxxx https://f002.backblazeb2.com/file/bucket/path/folder/file3?Authorization=xxxxxxxx
+
-Option Storage.
-Type of storage to configure.
-Choose a number from below, or type in your own value.
-[snip]
-XX / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, China Mobile, Cloudflare, ArvanCloud, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, IDrive e2, Lyve Cloud, Minio, Netease, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Synology, Tencent COS and Wasabi
- \ (s3)
-[snip]
-Storage> s3
+### Standard options
-Option provider.
-Choose your S3 provider.
-Choose a number from below, or type in your own value.
-Press Enter to leave empty.
-[snip]
-XX / IDrive e2
- \ (IDrive)
-[snip]
-provider> IDrive
+Here are the Standard options specific to b2 (Backblaze B2).
-Option env_auth.
-Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
-Only applies if access_key_id and secret_access_key is blank.
-Choose a number from below, or type in your own boolean value (true or false).
-Press Enter for the default (false).
- 1 / Enter AWS credentials in the next step.
- \ (false)
- 2 / Get AWS credentials from the environment (env vars or IAM).
- \ (true)
-env_auth>
+#### --b2-account
-Option access_key_id.
-AWS Access Key ID.
-Leave blank for anonymous access or runtime credentials.
-Enter a value. Press Enter to leave empty.
-access_key_id> YOUR_ACCESS_KEY
+Account ID or Application Key ID.
-Option secret_access_key.
-AWS Secret Access Key (password).
-Leave blank for anonymous access or runtime credentials.
-Enter a value. Press Enter to leave empty.
-secret_access_key> YOUR_SECRET_KEY
+Properties:
-Option acl.
-Canned ACL used when creating buckets and storing or copying objects.
-This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.
-For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
-Note that this ACL is applied when server-side copying objects as S3
-doesn't copy the ACL from the source but rather writes a fresh one.
-Choose a number from below, or type in your own value.
-Press Enter to leave empty.
- / Owner gets FULL_CONTROL.
- 1 | No one else has access rights (default).
- \ (private)
- / Owner gets FULL_CONTROL.
- 2 | The AllUsers group gets READ access.
- \ (public-read)
- / Owner gets FULL_CONTROL.
- 3 | The AllUsers group gets READ and WRITE access.
- | Granting this on a bucket is generally not recommended.
- \ (public-read-write)
- / Owner gets FULL_CONTROL.
- 4 | The AuthenticatedUsers group gets READ access.
- \ (authenticated-read)
- / Object owner gets FULL_CONTROL.
- 5 | Bucket owner gets READ access.
- | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
- \ (bucket-owner-read)
- / Both the object owner and the bucket owner get FULL_CONTROL over the object.
- 6 | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
- \ (bucket-owner-full-control)
-acl>
+- Config: account
+- Env Var: RCLONE_B2_ACCOUNT
+- Type: string
+- Required: true
-Edit advanced config?
-y) Yes
-n) No (default)
-y/n>
+#### --b2-key
-Configuration complete.
-Options:
-- type: s3
-- provider: IDrive
-- access_key_id: YOUR_ACCESS_KEY
-- secret_access_key: YOUR_SECRET_KEY
-- endpoint: q9d9.la12.idrivee2-5.com
-Keep this "e2" remote?
-y) Yes this is OK (default)
-e) Edit this remote
-d) Delete this remote
-y/e/d> y
-IONOS Cloud
-IONOS S3 Object Storage is a service offered by IONOS for storing and accessing unstructured data. To connect to the service, you will need an access key and a secret key. These can be found in the Data Center Designer, by selecting Manager resources > Object Storage Key Manager.
-Here is an example of a configuration. First, run rclone config. This will walk you through an interactive setup process. Type n to add the new remote, and then enter a name:
-Enter name for new remote.
-name> ionos-fra
-Type s3 to choose the connection type:
-Option Storage.
-Type of storage to configure.
-Choose a number from below, or type in your own value.
-[snip]
-XX / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, China Mobile, Cloudflare, ArvanCloud, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, IDrive e2, IONOS Cloud, Lyve Cloud, Minio, Netease, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Synology, Tencent COS and Wasabi
- \ (s3)
-[snip]
-Storage> s3
-Type IONOS:
-Option provider.
-Choose your S3 provider.
-Choose a number from below, or type in your own value.
-Press Enter to leave empty.
-[snip]
-XX / IONOS Cloud
- \ (IONOS)
-[snip]
-provider> IONOS
-Press Enter to choose the default option Enter AWS credentials in the next step:
-Option env_auth.
-Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
-Only applies if access_key_id and secret_access_key is blank.
-Choose a number from below, or type in your own boolean value (true or false).
-Press Enter for the default (false).
- 1 / Enter AWS credentials in the next step.
- \ (false)
- 2 / Get AWS credentials from the environment (env vars or IAM).
- \ (true)
-env_auth>
-Enter your Access Key and Secret key. These can be retrieved in the Data Center Designer, click on the menu “Manager resources†/ "Object Storage Key Manager".
-Option access_key_id.
-AWS Access Key ID.
-Leave blank for anonymous access or runtime credentials.
-Enter a value. Press Enter to leave empty.
-access_key_id> YOUR_ACCESS_KEY
+Application Key.
-Option secret_access_key.
-AWS Secret Access Key (password).
-Leave blank for anonymous access or runtime credentials.
-Enter a value. Press Enter to leave empty.
-secret_access_key> YOUR_SECRET_KEY
-Choose the region where your bucket is located:
-Option region.
-Region where your bucket will be created and your data stored.
-Choose a number from below, or type in your own value.
-Press Enter to leave empty.
- 1 / Frankfurt, Germany
- \ (de)
- 2 / Berlin, Germany
- \ (eu-central-2)
- 3 / Logrono, Spain
- \ (eu-south-2)
-region> 2
-Choose the endpoint from the same region:
-Option endpoint.
-Endpoint for IONOS S3 Object Storage.
-Specify the endpoint from the same region.
-Choose a number from below, or type in your own value.
-Press Enter to leave empty.
- 1 / Frankfurt, Germany
- \ (s3-eu-central-1.ionoscloud.com)
- 2 / Berlin, Germany
- \ (s3-eu-central-2.ionoscloud.com)
- 3 / Logrono, Spain
- \ (s3-eu-south-2.ionoscloud.com)
-endpoint> 1
-Press Enter to choose the default option or choose the desired ACL setting:
-Option acl.
-Canned ACL used when creating buckets and storing or copying objects.
-This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.
-For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
-Note that this ACL is applied when server-side copying objects as S3
-doesn't copy the ACL from the source but rather writes a fresh one.
-Choose a number from below, or type in your own value.
-Press Enter to leave empty.
- / Owner gets FULL_CONTROL.
- 1 | No one else has access rights (default).
- \ (private)
- / Owner gets FULL_CONTROL.
-[snip]
-acl>
-Press Enter to skip the advanced config:
-Edit advanced config?
-y) Yes
-n) No (default)
-y/n>
-Press Enter to save the configuration, and then q to quit the configuration process:
-Configuration complete.
-Options:
-- type: s3
-- provider: IONOS
-- access_key_id: YOUR_ACCESS_KEY
-- secret_access_key: YOUR_SECRET_KEY
-- endpoint: s3-eu-central-1.ionoscloud.com
-Keep this "ionos-fra" remote?
-y) Yes this is OK (default)
-e) Edit this remote
-d) Delete this remote
-y/e/d> y
-Done! Now you can try some commands (for macOS, use ./rclone instead of rclone).
-
-- Create a bucket (the name must be unique within the whole IONOS S3)
-
-rclone mkdir ionos-fra:my-bucket
-
-- List available buckets
-
-rclone lsd ionos-fra:
-
-- Copy a file from local to remote
-
-rclone copy /Users/file.txt ionos-fra:my-bucket
-
-- List contents of a bucket
-
-rclone ls ionos-fra:my-bucket
-
-- Copy a file from remote to local
-
-rclone copy ionos-fra:my-bucket/file.txt
-Minio
-Minio is an object storage server built for cloud application developers and devops.
-It is very easy to install and provides an S3 compatible server which can be used by rclone.
-To use it, install Minio following the instructions here.
-When it configures itself Minio will print something like this
-Endpoint: http://192.168.1.106:9000 http://172.23.0.1:9000
-AccessKey: USWUXHGYZQYFYFFIT3RE
-SecretKey: MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03
-Region: us-east-1
-SQS ARNs: arn:minio:sqs:us-east-1:1:redis arn:minio:sqs:us-east-1:2:redis
+Properties:
-Browser Access:
- http://192.168.1.106:9000 http://172.23.0.1:9000
+- Config: key
+- Env Var: RCLONE_B2_KEY
+- Type: string
+- Required: true
-Command-line Access: https://docs.minio.io/docs/minio-client-quickstart-guide
- $ mc config host add myminio http://192.168.1.106:9000 USWUXHGYZQYFYFFIT3RE MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03
+#### --b2-hard-delete
-Object API (Amazon S3 compatible):
- Go: https://docs.minio.io/docs/golang-client-quickstart-guide
- Java: https://docs.minio.io/docs/java-client-quickstart-guide
- Python: https://docs.minio.io/docs/python-client-quickstart-guide
- JavaScript: https://docs.minio.io/docs/javascript-client-quickstart-guide
- .NET: https://docs.minio.io/docs/dotnet-client-quickstart-guide
+Permanently delete files on remote removal, otherwise hide files.
-Drive Capacity: 26 GiB Free, 165 GiB Total
-These details need to go into rclone config like this. Note that it is important to put the region in as stated above.
-env_auth> 1
-access_key_id> USWUXHGYZQYFYFFIT3RE
-secret_access_key> MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03
-region> us-east-1
-endpoint> http://192.168.1.106:9000
-location_constraint>
-server_side_encryption>
-Which makes the config file look like this
-[minio]
-type = s3
-provider = Minio
-env_auth = false
-access_key_id = USWUXHGYZQYFYFFIT3RE
-secret_access_key = MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03
-region = us-east-1
-endpoint = http://192.168.1.106:9000
-location_constraint =
-server_side_encryption =
-So once set up, for example, to copy files into a bucket
-rclone copy /path/to/files minio:bucket
-Qiniu Cloud Object Storage (Kodo)
-Qiniu Cloud Object Storage (Kodo), a completely independent-researched core technology which is proven by repeated customer experience has occupied absolute leading market leader position. Kodo can be widely applied to mass data management.
-To configure access to Qiniu Kodo, follow the steps below:
-
-- Run
rclone config and select n for a new remote.
-
-rclone config
-No remotes found, make a new one?
-n) New remote
-s) Set configuration password
-q) Quit config
-n/s/q> n
-
-- Give the name of the configuration. For example, name it 'qiniu'.
-
-name> qiniu
-
-- Select
s3 storage.
-
-Choose a number from below, or type in your own value
- 1 / 1Fichier
- \ (fichier)
- 2 / Akamai NetStorage
- \ (netstorage)
- 3 / Alias for an existing remote
- \ (alias)
- 4 / Amazon Drive
- \ (amazon cloud drive)
- 5 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, China Mobile, Cloudflare, ArvanCloud, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, IDrive e2, Liara, Lyve Cloud, Minio, Netease, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Synology, Tencent COS, Qiniu and Wasabi
- \ (s3)
-[snip]
-Storage> s3
-
-- Select
Qiniu provider.
-
-Choose a number from below, or type in your own value
-1 / Amazon Web Services (AWS) S3
- \ "AWS"
-[snip]
-22 / Qiniu Object Storage (Kodo)
- \ (Qiniu)
-[snip]
-provider> Qiniu
-
-- Enter your SecretId and SecretKey of Qiniu Kodo.
-
-Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
-Only applies if access_key_id and secret_access_key is blank.
-Enter a boolean value (true or false). Press Enter for the default ("false").
-Choose a number from below, or type in your own value
- 1 / Enter AWS credentials in the next step
- \ "false"
- 2 / Get AWS credentials from the environment (env vars or IAM)
- \ "true"
-env_auth> 1
-AWS Access Key ID.
-Leave blank for anonymous access or runtime credentials.
-Enter a string value. Press Enter for the default ("").
-access_key_id> AKIDxxxxxxxxxx
-AWS Secret Access Key (password)
-Leave blank for anonymous access or runtime credentials.
-Enter a string value. Press Enter for the default ("").
-secret_access_key> xxxxxxxxxxx
-
-- Select endpoint for Qiniu Kodo. This is the standard endpoint for different region.
-
- / The default endpoint - a good choice if you are unsure.
- 1 | East China Region 1.
- | Needs location constraint cn-east-1.
- \ (cn-east-1)
- / East China Region 2.
- 2 | Needs location constraint cn-east-2.
- \ (cn-east-2)
- / North China Region 1.
- 3 | Needs location constraint cn-north-1.
- \ (cn-north-1)
- / South China Region 1.
- 4 | Needs location constraint cn-south-1.
- \ (cn-south-1)
- / North America Region.
- 5 | Needs location constraint us-north-1.
- \ (us-north-1)
- / Southeast Asia Region 1.
- 6 | Needs location constraint ap-southeast-1.
- \ (ap-southeast-1)
- / Northeast Asia Region 1.
- 7 | Needs location constraint ap-northeast-1.
- \ (ap-northeast-1)
-[snip]
-endpoint> 1
+Properties:
+
+- Config: hard_delete
+- Env Var: RCLONE_B2_HARD_DELETE
+- Type: bool
+- Default: false
+
+### Advanced options
+
+Here are the Advanced options specific to b2 (Backblaze B2).
+
+#### --b2-endpoint
+
+Endpoint for the service.
+
+Leave blank normally.
+
+Properties:
+
+- Config: endpoint
+- Env Var: RCLONE_B2_ENDPOINT
+- Type: string
+- Required: false
+
+#### --b2-test-mode
+
+A flag string for X-Bz-Test-Mode header for debugging.
+
+This is for debugging purposes only. Setting it to one of the strings
+below will cause b2 to return specific errors:
+
+ * "fail_some_uploads"
+ * "expire_some_account_authorization_tokens"
+ * "force_cap_exceeded"
+
+These will be set in the "X-Bz-Test-Mode" header which is documented
+in the [b2 integrations checklist](https://www.backblaze.com/b2/docs/integration_checklist.html).
+
+Properties:
+
+- Config: test_mode
+- Env Var: RCLONE_B2_TEST_MODE
+- Type: string
+- Required: false
+
+#### --b2-versions
+
+Include old versions in directory listings.
+
+Note that when using this no file write operations are permitted,
+so you can't upload files or delete them.
+
+Properties:
+
+- Config: versions
+- Env Var: RCLONE_B2_VERSIONS
+- Type: bool
+- Default: false
+
+#### --b2-version-at
+
+Show file versions as they were at the specified time.
+
+Note that when using this no file write operations are permitted,
+so you can't upload files or delete them.
+
+Properties:
+
+- Config: version_at
+- Env Var: RCLONE_B2_VERSION_AT
+- Type: Time
+- Default: off
+
+#### --b2-upload-cutoff
+
+Cutoff for switching to chunked upload.
+
+Files above this size will be uploaded in chunks of "--b2-chunk-size".
+
+This value should be set no larger than 4.657 GiB (== 5 GB).
+
+Properties:
+
+- Config: upload_cutoff
+- Env Var: RCLONE_B2_UPLOAD_CUTOFF
+- Type: SizeSuffix
+- Default: 200Mi
+
+#### --b2-copy-cutoff
+
+Cutoff for switching to multipart copy.
+
+Any files larger than this that need to be server-side copied will be
+copied in chunks of this size.
+
+The minimum is 0 and the maximum is 4.6 GiB.
+
+Properties:
+
+- Config: copy_cutoff
+- Env Var: RCLONE_B2_COPY_CUTOFF
+- Type: SizeSuffix
+- Default: 4Gi
+
+#### --b2-chunk-size
+
+Upload chunk size.
+
+When uploading large files, chunk the file into this size.
+
+Must fit in memory. These chunks are buffered in memory and there
+might a maximum of "--transfers" chunks in progress at once.
+
+5,000,000 Bytes is the minimum size.
+
+Properties:
+
+- Config: chunk_size
+- Env Var: RCLONE_B2_CHUNK_SIZE
+- Type: SizeSuffix
+- Default: 96Mi
+
+#### --b2-upload-concurrency
+
+Concurrency for multipart uploads.
+
+This is the number of chunks of the same file that are uploaded
+concurrently.
+
+Note that chunks are stored in memory and there may be up to
+"--transfers" * "--b2-upload-concurrency" chunks stored at once
+in memory.
+
+Properties:
+
+- Config: upload_concurrency
+- Env Var: RCLONE_B2_UPLOAD_CONCURRENCY
+- Type: int
+- Default: 4
-Option endpoint.
-Endpoint for Qiniu Object Storage.
-Choose a number from below, or type in your own value.
-Press Enter to leave empty.
- 1 / East China Endpoint 1
- \ (s3-cn-east-1.qiniucs.com)
- 2 / East China Endpoint 2
- \ (s3-cn-east-2.qiniucs.com)
- 3 / North China Endpoint 1
- \ (s3-cn-north-1.qiniucs.com)
- 4 / South China Endpoint 1
- \ (s3-cn-south-1.qiniucs.com)
- 5 / North America Endpoint 1
- \ (s3-us-north-1.qiniucs.com)
- 6 / Southeast Asia Endpoint 1
- \ (s3-ap-southeast-1.qiniucs.com)
- 7 / Northeast Asia Endpoint 1
- \ (s3-ap-northeast-1.qiniucs.com)
-endpoint> 1
+#### --b2-disable-checksum
-Option location_constraint.
-Location constraint - must be set to match the Region.
-Used when creating buckets only.
-Choose a number from below, or type in your own value.
-Press Enter to leave empty.
- 1 / East China Region 1
- \ (cn-east-1)
- 2 / East China Region 2
- \ (cn-east-2)
- 3 / North China Region 1
- \ (cn-north-1)
- 4 / South China Region 1
- \ (cn-south-1)
- 5 / North America Region 1
- \ (us-north-1)
- 6 / Southeast Asia Region 1
- \ (ap-southeast-1)
- 7 / Northeast Asia Region 1
- \ (ap-northeast-1)
-location_constraint> 1
-
-- Choose acl and storage class.
-
-Note that this ACL is applied when server-side copying objects as S3
-doesn't copy the ACL from the source but rather writes a fresh one.
-Enter a string value. Press Enter for the default ("").
-Choose a number from below, or type in your own value
- / Owner gets FULL_CONTROL.
- 1 | No one else has access rights (default).
- \ (private)
- / Owner gets FULL_CONTROL.
- 2 | The AllUsers group gets READ access.
- \ (public-read)
-[snip]
-acl> 2
-The storage class to use when storing new objects in Tencent COS.
-Enter a string value. Press Enter for the default ("").
-Choose a number from below, or type in your own value
- 1 / Standard storage class
- \ (STANDARD)
- 2 / Infrequent access storage mode
- \ (LINE)
- 3 / Archive storage mode
- \ (GLACIER)
- 4 / Deep archive storage mode
- \ (DEEP_ARCHIVE)
-[snip]
-storage_class> 1
-Edit advanced config? (y/n)
-y) Yes
-n) No (default)
-y/n> n
-Remote config
---------------------
-[qiniu]
-- type: s3
-- provider: Qiniu
-- access_key_id: xxx
-- secret_access_key: xxx
-- region: cn-east-1
-- endpoint: s3-cn-east-1.qiniucs.com
-- location_constraint: cn-east-1
-- acl: public-read
-- storage_class: STANDARD
---------------------
-y) Yes this is OK (default)
-e) Edit this remote
-d) Delete this remote
-y/e/d> y
-Current remotes:
+Disable checksums for large (> upload cutoff) files.
-Name Type
-==== ====
-qiniu s3
-RackCorp
-RackCorp Object Storage is an S3 compatible object storage platform from your friendly cloud provider RackCorp. The service is fast, reliable, well priced and located in many strategic locations unserviced by others, to ensure you can maintain data sovereignty.
-Before you can use RackCorp Object Storage, you'll need to "sign up" for an account on our "portal". Next you can create an access key, a secret key and buckets, in your location of choice with ease. These details are required for the next steps of configuration, when rclone config asks for your access_key_id and secret_access_key.
-Your config should end up looking a bit like this:
-[RCS3-demo-config]
-type = s3
-provider = RackCorp
-env_auth = true
-access_key_id = YOURACCESSKEY
-secret_access_key = YOURSECRETACCESSKEY
-region = au-nsw
-endpoint = s3.rackcorp.com
-location_constraint = au-nsw
-Scaleway
-Scaleway The Object Storage platform allows you to store anything from backups, logs and web assets to documents and photos. Files can be dropped from the Scaleway console or transferred through our API and CLI or using any S3-compatible tool.
-Scaleway provides an S3 interface which can be configured for use with rclone like this:
-[scaleway]
-type = s3
-provider = Scaleway
-env_auth = false
-endpoint = s3.nl-ams.scw.cloud
-access_key_id = SCWXXXXXXXXXXXXXX
-secret_access_key = 1111111-2222-3333-44444-55555555555555
-region = nl-ams
-location_constraint =
-acl = private
-server_side_encryption =
-storage_class =
-C14 Cold Storage is the low-cost S3 Glacier alternative from Scaleway and it works the same way as on S3 by accepting the "GLACIER" storage_class. So you can configure your remote with the storage_class = GLACIER option to upload directly to C14. Don't forget that in this state you can't read files back after, you will need to restore them to "STANDARD" storage_class first before being able to read them (see "restore" section above)
-Seagate Lyve Cloud
-Seagate Lyve Cloud is an S3 compatible object storage platform from Seagate intended for enterprise use.
-Here is a config run through for a remote called remote - you may choose a different name of course. Note that to create an access key and secret key you will need to create a service account first.
-$ rclone config
-No remotes found, make a new one?
-n) New remote
-s) Set configuration password
-q) Quit config
-n/s/q> n
-name> remote
-Choose s3 backend
-Type of storage to configure.
-Choose a number from below, or type in your own value.
-[snip]
-XX / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, ChinaMobile, ArvanCloud, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, Liara, Lyve Cloud, Minio, RackCorp, SeaweedFS, and Tencent COS
- \ (s3)
-[snip]
-Storage> s3
-Choose LyveCloud as S3 provider
-Choose your S3 provider.
-Choose a number from below, or type in your own value.
-Press Enter to leave empty.
-[snip]
-XX / Seagate Lyve Cloud
- \ (LyveCloud)
-[snip]
-provider> LyveCloud
-Take the default (just press enter) to enter access key and secret in the config file.
-Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
-Only applies if access_key_id and secret_access_key is blank.
-Choose a number from below, or type in your own boolean value (true or false).
-Press Enter for the default (false).
- 1 / Enter AWS credentials in the next step.
- \ (false)
- 2 / Get AWS credentials from the environment (env vars or IAM).
- \ (true)
-env_auth>
-AWS Access Key ID.
-Leave blank for anonymous access or runtime credentials.
-Enter a value. Press Enter to leave empty.
-access_key_id> XXX
-AWS Secret Access Key (password).
-Leave blank for anonymous access or runtime credentials.
-Enter a value. Press Enter to leave empty.
-secret_access_key> YYY
-Leave region blank
-Region to connect to.
-Leave blank if you are using an S3 clone and you don't have a region.
-Choose a number from below, or type in your own value.
-Press Enter to leave empty.
- / Use this if unsure.
- 1 | Will use v4 signatures and an empty region.
- \ ()
- / Use this only if v4 signatures don't work.
- 2 | E.g. pre Jewel/v10 CEPH.
- \ (other-v2-signature)
-region>
-Choose an endpoint from the list
-Endpoint for S3 API.
-Required when using an S3 clone.
-Choose a number from below, or type in your own value.
-Press Enter to leave empty.
- 1 / Seagate Lyve Cloud US East 1 (Virginia)
- \ (s3.us-east-1.lyvecloud.seagate.com)
- 2 / Seagate Lyve Cloud US West 1 (California)
- \ (s3.us-west-1.lyvecloud.seagate.com)
- 3 / Seagate Lyve Cloud AP Southeast 1 (Singapore)
- \ (s3.ap-southeast-1.lyvecloud.seagate.com)
-endpoint> 1
-Leave location constraint blank
-Location constraint - must be set to match the Region.
-Leave blank if not sure. Used when creating buckets only.
-Enter a value. Press Enter to leave empty.
-location_constraint>
-Choose default ACL (private).
-Canned ACL used when creating buckets and storing or copying objects.
-This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.
-For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
-Note that this ACL is applied when server-side copying objects as S3
-doesn't copy the ACL from the source but rather writes a fresh one.
-Choose a number from below, or type in your own value.
-Press Enter to leave empty.
- / Owner gets FULL_CONTROL.
- 1 | No one else has access rights (default).
- \ (private)
-[snip]
-acl>
-And the config file should end up looking like this:
-[remote]
-type = s3
-provider = LyveCloud
-access_key_id = XXX
-secret_access_key = YYY
-endpoint = s3.us-east-1.lyvecloud.seagate.com
-SeaweedFS
-SeaweedFS is a distributed storage system for blobs, objects, files, and data lake, with O(1) disk seek and a scalable file metadata store. It has an S3 compatible object storage interface. SeaweedFS can also act as a gateway to remote S3 compatible object store to cache data and metadata with asynchronous write back, for fast local speed and minimize access cost.
-Assuming the SeaweedFS are configured with weed shell as such:
-> s3.bucket.create -name foo
-> s3.configure -access_key=any -secret_key=any -buckets=foo -user=me -actions=Read,Write,List,Tagging,Admin -apply
-{
- "identities": [
- {
- "name": "me",
- "credentials": [
- {
- "accessKey": "any",
- "secretKey": "any"
- }
- ],
- "actions": [
- "Read:foo",
- "Write:foo",
- "List:foo",
- "Tagging:foo",
- "Admin:foo"
- ]
- }
- ]
-}
-To use rclone with SeaweedFS, above configuration should end up with something like this in your config:
-[seaweedfs_s3]
-type = s3
-provider = SeaweedFS
-access_key_id = any
-secret_access_key = any
-endpoint = localhost:8333
-So once set up, for example to copy files into a bucket
-rclone copy /path/to/files seaweedfs_s3:foo
-Wasabi
-Wasabi is a cloud-based object storage service for a broad range of applications and use cases. Wasabi is designed for individuals and organizations that require a high-performance, reliable, and secure data storage infrastructure at minimal cost.
-Wasabi provides an S3 interface which can be configured for use with rclone like this.
-No remotes found, make a new one?
-n) New remote
-s) Set configuration password
-n/s> n
-name> wasabi
-Type of storage to configure.
-Choose a number from below, or type in your own value
-[snip]
-XX / Amazon S3 (also Dreamhost, Ceph, ChinaMobile, ArvanCloud, Minio, Liara)
- \ "s3"
-[snip]
-Storage> s3
-Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars). Only applies if access_key_id and secret_access_key is blank.
-Choose a number from below, or type in your own value
- 1 / Enter AWS credentials in the next step
- \ "false"
- 2 / Get AWS credentials from the environment (env vars or IAM)
- \ "true"
-env_auth> 1
-AWS Access Key ID - leave blank for anonymous access or runtime credentials.
-access_key_id> YOURACCESSKEY
-AWS Secret Access Key (password) - leave blank for anonymous access or runtime credentials.
-secret_access_key> YOURSECRETACCESSKEY
-Region to connect to.
-Choose a number from below, or type in your own value
- / The default endpoint - a good choice if you are unsure.
- 1 | US Region, Northern Virginia, or Pacific Northwest.
- | Leave location constraint empty.
- \ "us-east-1"
-[snip]
-region> us-east-1
-Endpoint for S3 API.
-Leave blank if using AWS to use the default endpoint for the region.
-Specify if using an S3 clone such as Ceph.
-endpoint> s3.wasabisys.com
-Location constraint - must be set to match the Region. Used when creating buckets only.
-Choose a number from below, or type in your own value
- 1 / Empty for US Region, Northern Virginia, or Pacific Northwest.
- \ ""
-[snip]
-location_constraint>
-Canned ACL used when creating buckets and/or storing objects in S3.
-For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
-Choose a number from below, or type in your own value
- 1 / Owner gets FULL_CONTROL. No one else has access rights (default).
- \ "private"
-[snip]
-acl>
-The server-side encryption algorithm used when storing this object in S3.
-Choose a number from below, or type in your own value
- 1 / None
- \ ""
- 2 / AES256
- \ "AES256"
-server_side_encryption>
-The storage class to use when storing objects in S3.
-Choose a number from below, or type in your own value
- 1 / Default
- \ ""
- 2 / Standard storage class
- \ "STANDARD"
- 3 / Reduced redundancy storage class
- \ "REDUCED_REDUNDANCY"
- 4 / Standard Infrequent Access storage class
- \ "STANDARD_IA"
-storage_class>
-Remote config
---------------------
-[wasabi]
-env_auth = false
-access_key_id = YOURACCESSKEY
-secret_access_key = YOURSECRETACCESSKEY
-region = us-east-1
-endpoint = s3.wasabisys.com
-location_constraint =
-acl =
-server_side_encryption =
-storage_class =
---------------------
-y) Yes this is OK
-e) Edit this remote
-d) Delete this remote
-y/e/d> y
-This will leave the config file looking like this.
-[wasabi]
-type = s3
-provider = Wasabi
-env_auth = false
-access_key_id = YOURACCESSKEY
-secret_access_key = YOURSECRETACCESSKEY
-region =
-endpoint = s3.wasabisys.com
-location_constraint =
-acl =
-server_side_encryption =
-storage_class =
-Alibaba OSS
-Here is an example of making an Alibaba Cloud (Aliyun) OSS configuration. First run:
-rclone config
-This will guide you through an interactive setup process.
-No remotes found, make a new one?
-n) New remote
-s) Set configuration password
-q) Quit config
-n/s/q> n
-name> oss
-Type of storage to configure.
-Enter a string value. Press Enter for the default ("").
-Choose a number from below, or type in your own value
-[snip]
- 4 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, ChinaMobile, ArvanCloud, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, Liara, Minio, and Tencent COS
- \ "s3"
-[snip]
-Storage> s3
-Choose your S3 provider.
-Enter a string value. Press Enter for the default ("").
-Choose a number from below, or type in your own value
- 1 / Amazon Web Services (AWS) S3
- \ "AWS"
- 2 / Alibaba Cloud Object Storage System (OSS) formerly Aliyun
- \ "Alibaba"
- 3 / Ceph Object Storage
- \ "Ceph"
-[snip]
-provider> Alibaba
-Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
-Only applies if access_key_id and secret_access_key is blank.
-Enter a boolean value (true or false). Press Enter for the default ("false").
-Choose a number from below, or type in your own value
- 1 / Enter AWS credentials in the next step
- \ "false"
- 2 / Get AWS credentials from the environment (env vars or IAM)
- \ "true"
-env_auth> 1
-AWS Access Key ID.
-Leave blank for anonymous access or runtime credentials.
-Enter a string value. Press Enter for the default ("").
-access_key_id> accesskeyid
-AWS Secret Access Key (password)
-Leave blank for anonymous access or runtime credentials.
-Enter a string value. Press Enter for the default ("").
-secret_access_key> secretaccesskey
-Endpoint for OSS API.
-Enter a string value. Press Enter for the default ("").
-Choose a number from below, or type in your own value
- 1 / East China 1 (Hangzhou)
- \ "oss-cn-hangzhou.aliyuncs.com"
- 2 / East China 2 (Shanghai)
- \ "oss-cn-shanghai.aliyuncs.com"
- 3 / North China 1 (Qingdao)
- \ "oss-cn-qingdao.aliyuncs.com"
-[snip]
-endpoint> 1
-Canned ACL used when creating buckets and storing or copying objects.
+Normally rclone will calculate the SHA1 checksum of the input before
+uploading it so it can add it to metadata on the object. This is great
+for data integrity checking but can cause long delays for large files
+to start uploading.
+
+Properties:
+
+- Config: disable_checksum
+- Env Var: RCLONE_B2_DISABLE_CHECKSUM
+- Type: bool
+- Default: false
+
+#### --b2-download-url
+
+Custom endpoint for downloads.
+
+This is usually set to a Cloudflare CDN URL as Backblaze offers
+free egress for data downloaded through the Cloudflare network.
+Rclone works with private buckets by sending an "Authorization" header.
+If the custom endpoint rewrites the requests for authentication,
+e.g., in Cloudflare Workers, this header needs to be handled properly.
+Leave blank if you want to use the endpoint provided by Backblaze.
+
+The URL provided here SHOULD have the protocol and SHOULD NOT have
+a trailing slash or specify the /file/bucket subpath as rclone will
+request files with "{download_url}/file/{bucket_name}/{path}".
+
+Example:
+> https://mysubdomain.mydomain.tld
+(No trailing "/", "file" or "bucket")
+
+Properties:
+
+- Config: download_url
+- Env Var: RCLONE_B2_DOWNLOAD_URL
+- Type: string
+- Required: false
+
+#### --b2-download-auth-duration
+
+Time before the authorization token will expire in s or suffix ms|s|m|h|d.
+
+The duration before the download authorization token will expire.
+The minimum value is 1 second. The maximum value is one week.
+
+Properties:
+
+- Config: download_auth_duration
+- Env Var: RCLONE_B2_DOWNLOAD_AUTH_DURATION
+- Type: Duration
+- Default: 1w
+
+#### --b2-memory-pool-flush-time
+
+How often internal memory buffer pools will be flushed. (no longer used)
+
+Properties:
+
+- Config: memory_pool_flush_time
+- Env Var: RCLONE_B2_MEMORY_POOL_FLUSH_TIME
+- Type: Duration
+- Default: 1m0s
+
+#### --b2-memory-pool-use-mmap
+
+Whether to use mmap buffers in internal memory pool. (no longer used)
+
+Properties:
+
+- Config: memory_pool_use_mmap
+- Env Var: RCLONE_B2_MEMORY_POOL_USE_MMAP
+- Type: bool
+- Default: false
+
+#### --b2-lifecycle
+
+Set the number of days deleted files should be kept when creating a bucket.
+
+On bucket creation, this parameter is used to create a lifecycle rule
+for the entire bucket.
+
+If lifecycle is 0 (the default) it does not create a lifecycle rule so
+the default B2 behaviour applies. This is to create versions of files
+on delete and overwrite and to keep them indefinitely.
+
+If lifecycle is >0 then it creates a single rule setting the number of
+days before a file that is deleted or overwritten is deleted
+permanently. This is known as daysFromHidingToDeleting in the b2 docs.
+
+The minimum value for this parameter is 1 day.
+
+You can also enable hard_delete in the config also which will mean
+deletions won't cause versions but overwrites will still cause
+versions to be made.
+
+See: [rclone backend lifecycle](#lifecycle) for setting lifecycles after bucket creation.
+
+
+Properties:
+
+- Config: lifecycle
+- Env Var: RCLONE_B2_LIFECYCLE
+- Type: int
+- Default: 0
+
+#### --b2-encoding
+
+The encoding for the backend.
+
+See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info.
+
+Properties:
+
+- Config: encoding
+- Env Var: RCLONE_B2_ENCODING
+- Type: Encoding
+- Default: Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot
+
+## Backend commands
+
+Here are the commands specific to the b2 backend.
+
+Run them with
-Note that this ACL is applied when server-side copying objects as S3
-doesn't copy the ACL from the source but rather writes a fresh one.
-Enter a string value. Press Enter for the default ("").
-Choose a number from below, or type in your own value
- 1 / Owner gets FULL_CONTROL. No one else has access rights (default).
- \ "private"
- 2 / Owner gets FULL_CONTROL. The AllUsers group gets READ access.
- \ "public-read"
- / Owner gets FULL_CONTROL. The AllUsers group gets READ and WRITE access.
-[snip]
-acl> 1
-The storage class to use when storing new objects in OSS.
-Enter a string value. Press Enter for the default ("").
-Choose a number from below, or type in your own value
- 1 / Default
- \ ""
- 2 / Standard storage class
- \ "STANDARD"
- 3 / Archive storage mode.
- \ "GLACIER"
- 4 / Infrequent access storage mode.
- \ "STANDARD_IA"
-storage_class> 1
-Edit advanced config? (y/n)
-y) Yes
-n) No
-y/n> n
-Remote config
---------------------
-[oss]
-type = s3
-provider = Alibaba
-env_auth = false
-access_key_id = accesskeyid
-secret_access_key = secretaccesskey
-endpoint = oss-cn-hangzhou.aliyuncs.com
-acl = private
-storage_class = Standard
---------------------
-y) Yes this is OK
-e) Edit this remote
-d) Delete this remote
-y/e/d> y
-China Mobile Ecloud Elastic Object Storage (EOS)
-Here is an example of making an China Mobile Ecloud Elastic Object Storage (EOS) configuration. First run:
-rclone config
-This will guide you through an interactive setup process.
-No remotes found, make a new one?
-n) New remote
-s) Set configuration password
-q) Quit config
-n/s/q> n
-name> ChinaMobile
-Option Storage.
-Type of storage to configure.
-Choose a number from below, or type in your own value.
- ...
- 5 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, ChinaMobile, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, Lyve Cloud, Minio, RackCorp, SeaweedFS, and Tencent COS
- \ (s3)
- ...
-Storage> s3
-Option provider.
-Choose your S3 provider.
-Choose a number from below, or type in your own value.
-Press Enter to leave empty.
- ...
- 4 / China Mobile Ecloud Elastic Object Storage (EOS)
- \ (ChinaMobile)
- ...
-provider> ChinaMobile
-Option env_auth.
-Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
-Only applies if access_key_id and secret_access_key is blank.
-Choose a number from below, or type in your own boolean value (true or false).
-Press Enter for the default (false).
- 1 / Enter AWS credentials in the next step.
- \ (false)
- 2 / Get AWS credentials from the environment (env vars or IAM).
- \ (true)
-env_auth>
-Option access_key_id.
-AWS Access Key ID.
-Leave blank for anonymous access or runtime credentials.
-Enter a value. Press Enter to leave empty.
-access_key_id> accesskeyid
-Option secret_access_key.
-AWS Secret Access Key (password).
-Leave blank for anonymous access or runtime credentials.
-Enter a value. Press Enter to leave empty.
-secret_access_key> secretaccesskey
-Option endpoint.
-Endpoint for China Mobile Ecloud Elastic Object Storage (EOS) API.
-Choose a number from below, or type in your own value.
-Press Enter to leave empty.
- / The default endpoint - a good choice if you are unsure.
- 1 | East China (Suzhou)
- \ (eos-wuxi-1.cmecloud.cn)
- 2 / East China (Jinan)
- \ (eos-jinan-1.cmecloud.cn)
- 3 / East China (Hangzhou)
- \ (eos-ningbo-1.cmecloud.cn)
- 4 / East China (Shanghai-1)
- \ (eos-shanghai-1.cmecloud.cn)
- 5 / Central China (Zhengzhou)
- \ (eos-zhengzhou-1.cmecloud.cn)
- 6 / Central China (Changsha-1)
- \ (eos-hunan-1.cmecloud.cn)
- 7 / Central China (Changsha-2)
- \ (eos-zhuzhou-1.cmecloud.cn)
- 8 / South China (Guangzhou-2)
- \ (eos-guangzhou-1.cmecloud.cn)
- 9 / South China (Guangzhou-3)
- \ (eos-dongguan-1.cmecloud.cn)
-10 / North China (Beijing-1)
- \ (eos-beijing-1.cmecloud.cn)
-11 / North China (Beijing-2)
- \ (eos-beijing-2.cmecloud.cn)
-12 / North China (Beijing-3)
- \ (eos-beijing-4.cmecloud.cn)
-13 / North China (Huhehaote)
- \ (eos-huhehaote-1.cmecloud.cn)
-14 / Southwest China (Chengdu)
- \ (eos-chengdu-1.cmecloud.cn)
-15 / Southwest China (Chongqing)
- \ (eos-chongqing-1.cmecloud.cn)
-16 / Southwest China (Guiyang)
- \ (eos-guiyang-1.cmecloud.cn)
-17 / Nouthwest China (Xian)
- \ (eos-xian-1.cmecloud.cn)
-18 / Yunnan China (Kunming)
- \ (eos-yunnan.cmecloud.cn)
-19 / Yunnan China (Kunming-2)
- \ (eos-yunnan-2.cmecloud.cn)
-20 / Tianjin China (Tianjin)
- \ (eos-tianjin-1.cmecloud.cn)
-21 / Jilin China (Changchun)
- \ (eos-jilin-1.cmecloud.cn)
-22 / Hubei China (Xiangyan)
- \ (eos-hubei-1.cmecloud.cn)
-23 / Jiangxi China (Nanchang)
- \ (eos-jiangxi-1.cmecloud.cn)
-24 / Gansu China (Lanzhou)
- \ (eos-gansu-1.cmecloud.cn)
-25 / Shanxi China (Taiyuan)
- \ (eos-shanxi-1.cmecloud.cn)
-26 / Liaoning China (Shenyang)
- \ (eos-liaoning-1.cmecloud.cn)
-27 / Hebei China (Shijiazhuang)
- \ (eos-hebei-1.cmecloud.cn)
-28 / Fujian China (Xiamen)
- \ (eos-fujian-1.cmecloud.cn)
-29 / Guangxi China (Nanning)
- \ (eos-guangxi-1.cmecloud.cn)
-30 / Anhui China (Huainan)
- \ (eos-anhui-1.cmecloud.cn)
-endpoint> 1
-Option location_constraint.
-Location constraint - must match endpoint.
-Used when creating buckets only.
-Choose a number from below, or type in your own value.
-Press Enter to leave empty.
- 1 / East China (Suzhou)
- \ (wuxi1)
- 2 / East China (Jinan)
- \ (jinan1)
- 3 / East China (Hangzhou)
- \ (ningbo1)
- 4 / East China (Shanghai-1)
- \ (shanghai1)
- 5 / Central China (Zhengzhou)
- \ (zhengzhou1)
- 6 / Central China (Changsha-1)
- \ (hunan1)
- 7 / Central China (Changsha-2)
- \ (zhuzhou1)
- 8 / South China (Guangzhou-2)
- \ (guangzhou1)
- 9 / South China (Guangzhou-3)
- \ (dongguan1)
-10 / North China (Beijing-1)
- \ (beijing1)
-11 / North China (Beijing-2)
- \ (beijing2)
-12 / North China (Beijing-3)
- \ (beijing4)
-13 / North China (Huhehaote)
- \ (huhehaote1)
-14 / Southwest China (Chengdu)
- \ (chengdu1)
-15 / Southwest China (Chongqing)
- \ (chongqing1)
-16 / Southwest China (Guiyang)
- \ (guiyang1)
-17 / Nouthwest China (Xian)
- \ (xian1)
-18 / Yunnan China (Kunming)
- \ (yunnan)
-19 / Yunnan China (Kunming-2)
- \ (yunnan2)
-20 / Tianjin China (Tianjin)
- \ (tianjin1)
-21 / Jilin China (Changchun)
- \ (jilin1)
-22 / Hubei China (Xiangyan)
- \ (hubei1)
-23 / Jiangxi China (Nanchang)
- \ (jiangxi1)
-24 / Gansu China (Lanzhou)
- \ (gansu1)
-25 / Shanxi China (Taiyuan)
- \ (shanxi1)
-26 / Liaoning China (Shenyang)
- \ (liaoning1)
-27 / Hebei China (Shijiazhuang)
- \ (hebei1)
-28 / Fujian China (Xiamen)
- \ (fujian1)
-29 / Guangxi China (Nanning)
- \ (guangxi1)
-30 / Anhui China (Huainan)
- \ (anhui1)
-location_constraint> 1
-Option acl.
-Canned ACL used when creating buckets and storing or copying objects.
-This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.
-For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
-Note that this ACL is applied when server-side copying objects as S3
-doesn't copy the ACL from the source but rather writes a fresh one.
-Choose a number from below, or type in your own value.
-Press Enter to leave empty.
- / Owner gets FULL_CONTROL.
- 1 | No one else has access rights (default).
- \ (private)
- / Owner gets FULL_CONTROL.
- 2 | The AllUsers group gets READ access.
- \ (public-read)
- / Owner gets FULL_CONTROL.
- 3 | The AllUsers group gets READ and WRITE access.
- | Granting this on a bucket is generally not recommended.
- \ (public-read-write)
- / Owner gets FULL_CONTROL.
- 4 | The AuthenticatedUsers group gets READ access.
- \ (authenticated-read)
- / Object owner gets FULL_CONTROL.
-acl> private
-Option server_side_encryption.
-The server-side encryption algorithm used when storing this object in S3.
-Choose a number from below, or type in your own value.
-Press Enter to leave empty.
- 1 / None
- \ ()
- 2 / AES256
- \ (AES256)
-server_side_encryption>
-Option storage_class.
-The storage class to use when storing new objects in ChinaMobile.
-Choose a number from below, or type in your own value.
-Press Enter to leave empty.
- 1 / Default
- \ ()
- 2 / Standard storage class
- \ (STANDARD)
- 3 / Archive storage mode
- \ (GLACIER)
- 4 / Infrequent access storage mode
- \ (STANDARD_IA)
-storage_class>
-Edit advanced config?
-y) Yes
-n) No (default)
-y/n> n
---------------------
-[ChinaMobile]
-type = s3
-provider = ChinaMobile
-access_key_id = accesskeyid
-secret_access_key = secretaccesskey
-endpoint = eos-wuxi-1.cmecloud.cn
-location_constraint = wuxi1
-acl = private
---------------------
-y) Yes this is OK (default)
-e) Edit this remote
-d) Delete this remote
-y/e/d> y
-Leviia Cloud Object Storage
-Leviia Object Storage, backup and secure your data in a 100% French cloud, independent of GAFAM..
-To configure access to Leviia, follow the steps below:
-
-- Run
rclone config and select n for a new remote.
-
-rclone config
-No remotes found, make a new one?
-n) New remote
-s) Set configuration password
-q) Quit config
-n/s/q> n
-
-- Give the name of the configuration. For example, name it 'leviia'.
-
-name> leviia
-
-- Select
s3 storage.
-
-Choose a number from below, or type in your own value
- 1 / 1Fichier
- \ (fichier)
- 2 / Akamai NetStorage
- \ (netstorage)
- 3 / Alias for an existing remote
- \ (alias)
- 4 / Amazon Drive
- \ (amazon cloud drive)
- 5 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, China Mobile, Cloudflare, ArvanCloud, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, IDrive e2, Liara, Lyve Cloud, Minio, Netease, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Synology, Tencent COS, Qiniu and Wasabi
- \ (s3)
-[snip]
-Storage> s3
-
-- Select
Leviia provider.
-
-Choose a number from below, or type in your own value
-1 / Amazon Web Services (AWS) S3
- \ "AWS"
-[snip]
-15 / Leviia Object Storage
- \ (Leviia)
-[snip]
-provider> Leviia
-
-- Enter your SecretId and SecretKey of Leviia.
-
-Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
-Only applies if access_key_id and secret_access_key is blank.
-Enter a boolean value (true or false). Press Enter for the default ("false").
-Choose a number from below, or type in your own value
- 1 / Enter AWS credentials in the next step
- \ "false"
- 2 / Get AWS credentials from the environment (env vars or IAM)
- \ "true"
-env_auth> 1
-AWS Access Key ID.
-Leave blank for anonymous access or runtime credentials.
-Enter a string value. Press Enter for the default ("").
-access_key_id> ZnIx.xxxxxxxxxxxxxxx
-AWS Secret Access Key (password)
-Leave blank for anonymous access or runtime credentials.
-Enter a string value. Press Enter for the default ("").
-secret_access_key> xxxxxxxxxxx
-
-- Select endpoint for Leviia.
-
- / The default endpoint
- 1 | Leviia.
- \ (s3.leviia.com)
-[snip]
-endpoint> 1
-
-- Choose acl.
-
-Note that this ACL is applied when server-side copying objects as S3
-doesn't copy the ACL from the source but rather writes a fresh one.
-Enter a string value. Press Enter for the default ("").
-Choose a number from below, or type in your own value
- / Owner gets FULL_CONTROL.
- 1 | No one else has access rights (default).
- \ (private)
- / Owner gets FULL_CONTROL.
- 2 | The AllUsers group gets READ access.
- \ (public-read)
-[snip]
-acl> 1
-Edit advanced config? (y/n)
-y) Yes
-n) No (default)
-y/n> n
-Remote config
---------------------
-[leviia]
-- type: s3
-- provider: Leviia
-- access_key_id: ZnIx.xxxxxxx
-- secret_access_key: xxxxxxxx
-- endpoint: s3.leviia.com
-- acl: private
---------------------
-y) Yes this is OK (default)
-e) Edit this remote
-d) Delete this remote
-y/e/d> y
-Current remotes:
+ rclone backend COMMAND remote:
+
+The help below will explain what arguments each command takes.
+
+See the [backend](https://rclone.org/commands/rclone_backend/) command for more
+info on how to pass options and arguments.
+
+These can be run on a running backend using the rc command
+[backend/command](https://rclone.org/rc/#backend-command).
+
+### lifecycle
+
+Read or set the lifecycle for a bucket
+
+ rclone backend lifecycle remote: [options] [<arguments>+]
+
+This command can be used to read or set the lifecycle for a bucket.
+
+Usage Examples:
+
+To show the current lifecycle rules:
+
+ rclone backend lifecycle b2:bucket
+
+This will dump something like this showing the lifecycle rules.
+
+ [
+ {
+ "daysFromHidingToDeleting": 1,
+ "daysFromUploadingToHiding": null,
+ "fileNamePrefix": ""
+ }
+ ]
+
+If there are no lifecycle rules (the default) then it will just return [].
+
+To reset the current lifecycle rules:
+
+ rclone backend lifecycle b2:bucket -o daysFromHidingToDeleting=30
+ rclone backend lifecycle b2:bucket -o daysFromUploadingToHiding=5 -o daysFromHidingToDeleting=1
+
+This will run and then print the new lifecycle rules as above.
+
+Rclone only lets you set lifecycles for the whole bucket with the
+fileNamePrefix = "".
+
+You can't disable versioning with B2. The best you can do is to set
+the daysFromHidingToDeleting to 1 day. You can enable hard_delete in
+the config also which will mean deletions won't cause versions but
+overwrites will still cause versions to be made.
+
+ rclone backend lifecycle b2:bucket -o daysFromHidingToDeleting=1
+
+See: https://www.backblaze.com/docs/cloud-storage-lifecycle-rules
+
+
+Options:
+
+- "daysFromHidingToDeleting": After a file has been hidden for this many days it is deleted. 0 is off.
+- "daysFromUploadingToHiding": This many days after uploading a file is hidden
+
+
+
+## Limitations
+
+`rclone about` is not supported by the B2 backend. Backends without
+this capability cannot determine free space for an rclone mount or
+use policy `mfs` (most free space) as a member of an rclone union
+remote.
+
+See [List of backends that do not support rclone about](https://rclone.org/overview/#optional-features) and [rclone about](https://rclone.org/commands/rclone_about/)
+
+# Box
+
+Paths are specified as `remote:path`
+
+Paths may be as deep as required, e.g. `remote:directory/subdirectory`.
+
+The initial setup for Box involves getting a token from Box which you
+can do either in your browser, or with a config.json downloaded from Box
+to use JWT authentication. `rclone config` walks you through it.
+
+## Configuration
+
+Here is an example of how to make a remote called `remote`. First run:
+
+ rclone config
+
+This will guide you through an interactive setup process:
+
+No remotes found, make a new one? n) New remote s) Set configuration password q) Quit config n/s/q> n name> remote Type of storage to configure. Choose a number from below, or type in your own value [snip] XX / Box  "box" [snip] Storage> box Box App Client Id - leave blank normally. client_id> Box App Client Secret - leave blank normally. client_secret> Box App config.json location Leave blank normally. Enter a string value. Press Enter for the default (""). box_config_file> Box App Primary Access Token Leave blank normally. Enter a string value. Press Enter for the default (""). access_token>
+Enter a string value. Press Enter for the default ("user"). Choose a number from below, or type in your own value 1 / Rclone should act on behalf of a user  "user" 2 / Rclone should act on behalf of a service account  "enterprise" box_sub_type> Remote config Use web browser to automatically authenticate rclone with remote? * Say Y if the machine running rclone has a web browser you can use * Say N if running rclone on a (remote) machine without web browser access If not sure try Y. If Y failed, try N. y) Yes n) No y/n> y If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth Log in and authorize rclone for access Waiting for code... Got code -------------------- [remote] client_id = client_secret = token = {"access_token":"XXX","token_type":"bearer","refresh_token":"XXX","expiry":"XXX"} -------------------- y) Yes this is OK e) Edit this remote d) Delete this remote y/e/d> y
+
+See the [remote setup docs](https://rclone.org/remote_setup/) for how to set it up on a
+machine with no Internet browser available.
+
+Note that rclone runs a webserver on your local machine to collect the
+token as returned from Box. This only runs from the moment it opens
+your browser to the moment you get back the verification code. This
+is on `http://127.0.0.1:53682/` and this it may require you to unblock
+it temporarily if you are running a host firewall.
+
+Once configured you can then use `rclone` like this,
+
+List directories in top level of your Box
+
+ rclone lsd remote:
+
+List all the files in your Box
+
+ rclone ls remote:
+
+To copy a local directory to an Box directory called backup
+
+ rclone copy /home/source remote:backup
+
+### Using rclone with an Enterprise account with SSO
+
+If you have an "Enterprise" account type with Box with single sign on
+(SSO), you need to create a password to use Box with rclone. This can
+be done at your Enterprise Box account by going to Settings, "Account"
+Tab, and then set the password in the "Authentication" field.
+
+Once you have done this, you can setup your Enterprise Box account
+using the same procedure detailed above in the, using the password you
+have just set.
+
+### Invalid refresh token
+
+According to the [box docs](https://developer.box.com/v2.0/docs/oauth-20#section-6-using-the-access-and-refresh-tokens):
+
+> Each refresh_token is valid for one use in 60 days.
+
+This means that if you
+
+ * Don't use the box remote for 60 days
+ * Copy the config file with a box refresh token in and use it in two places
+ * Get an error on a token refresh
+
+then rclone will return an error which includes the text `Invalid
+refresh token`.
+
+To fix this you will need to use oauth2 again to update the refresh
+token. You can use the methods in [the remote setup
+docs](https://rclone.org/remote_setup/), bearing in mind that if you use the copy the
+config file method, you should not use that remote on the computer you
+did the authentication on.
-Name Type
-==== ====
-leviia s3
-Liara
-Here is an example of making a Liara Object Storage configuration. First run:
-rclone config
-This will guide you through an interactive setup process.
-No remotes found, make a new one?
-n) New remote
-s) Set configuration password
-n/s> n
-name> Liara
-Type of storage to configure.
-Choose a number from below, or type in your own value
-[snip]
-XX / Amazon S3 (also Dreamhost, Ceph, ChinaMobile, ArvanCloud, Liara, Minio)
- \ "s3"
-[snip]
-Storage> s3
-Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars). Only applies if access_key_id and secret_access_key is blank.
-Choose a number from below, or type in your own value
- 1 / Enter AWS credentials in the next step
- \ "false"
- 2 / Get AWS credentials from the environment (env vars or IAM)
- \ "true"
-env_auth> 1
-AWS Access Key ID - leave blank for anonymous access or runtime credentials.
-access_key_id> YOURACCESSKEY
-AWS Secret Access Key (password) - leave blank for anonymous access or runtime credentials.
-secret_access_key> YOURSECRETACCESSKEY
-Region to connect to.
-Choose a number from below, or type in your own value
- / The default endpoint
- 1 | US Region, Northern Virginia, or Pacific Northwest.
- | Leave location constraint empty.
- \ "us-east-1"
-[snip]
-region>
-Endpoint for S3 API.
-Leave blank if using Liara to use the default endpoint for the region.
-Specify if using an S3 clone such as Ceph.
-endpoint> storage.iran.liara.space
-Canned ACL used when creating buckets and/or storing objects in S3.
-For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
-Choose a number from below, or type in your own value
- 1 / Owner gets FULL_CONTROL. No one else has access rights (default).
- \ "private"
-[snip]
-acl>
-The server-side encryption algorithm used when storing this object in S3.
-Choose a number from below, or type in your own value
- 1 / None
- \ ""
- 2 / AES256
- \ "AES256"
-server_side_encryption>
-The storage class to use when storing objects in S3.
-Choose a number from below, or type in your own value
- 1 / Default
- \ ""
- 2 / Standard storage class
- \ "STANDARD"
-storage_class>
-Remote config
---------------------
-[Liara]
-env_auth = false
-access_key_id = YOURACCESSKEY
-secret_access_key = YOURSECRETACCESSKEY
-endpoint = storage.iran.liara.space
-location_constraint =
-acl =
-server_side_encryption =
-storage_class =
---------------------
-y) Yes this is OK
-e) Edit this remote
-d) Delete this remote
-y/e/d> y
-This will leave the config file looking like this.
-[Liara]
-type = s3
-provider = Liara
-env_auth = false
-access_key_id = YOURACCESSKEY
-secret_access_key = YOURSECRETACCESSKEY
-region =
-endpoint = storage.iran.liara.space
-location_constraint =
-acl =
-server_side_encryption =
-storage_class =
-ArvanCloud
-ArvanCloud ArvanCloud Object Storage goes beyond the limited traditional file storage. It gives you access to backup and archived files and allows sharing. Files like profile image in the app, images sent by users or scanned documents can be stored securely and easily in our Object Storage service.
-ArvanCloud provides an S3 interface which can be configured for use with rclone like this.
-No remotes found, make a new one?
-n) New remote
-s) Set configuration password
-n/s> n
-name> ArvanCloud
-Type of storage to configure.
-Choose a number from below, or type in your own value
-[snip]
-XX / Amazon S3 (also Dreamhost, Ceph, ChinaMobile, ArvanCloud, Liara, Minio)
- \ "s3"
-[snip]
-Storage> s3
-Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars). Only applies if access_key_id and secret_access_key is blank.
-Choose a number from below, or type in your own value
- 1 / Enter AWS credentials in the next step
- \ "false"
- 2 / Get AWS credentials from the environment (env vars or IAM)
- \ "true"
-env_auth> 1
-AWS Access Key ID - leave blank for anonymous access or runtime credentials.
-access_key_id> YOURACCESSKEY
-AWS Secret Access Key (password) - leave blank for anonymous access or runtime credentials.
-secret_access_key> YOURSECRETACCESSKEY
-Region to connect to.
-Choose a number from below, or type in your own value
- / The default endpoint - a good choice if you are unsure.
- 1 | US Region, Northern Virginia, or Pacific Northwest.
- | Leave location constraint empty.
- \ "us-east-1"
-[snip]
-region>
-Endpoint for S3 API.
-Leave blank if using ArvanCloud to use the default endpoint for the region.
-Specify if using an S3 clone such as Ceph.
-endpoint> s3.arvanstorage.com
-Location constraint - must be set to match the Region. Used when creating buckets only.
-Choose a number from below, or type in your own value
- 1 / Empty for Iran-Tehran Region.
- \ ""
-[snip]
-location_constraint>
-Canned ACL used when creating buckets and/or storing objects in S3.
-For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
-Choose a number from below, or type in your own value
- 1 / Owner gets FULL_CONTROL. No one else has access rights (default).
- \ "private"
-[snip]
-acl>
-The server-side encryption algorithm used when storing this object in S3.
-Choose a number from below, or type in your own value
- 1 / None
- \ ""
- 2 / AES256
- \ "AES256"
-server_side_encryption>
-The storage class to use when storing objects in S3.
-Choose a number from below, or type in your own value
- 1 / Default
- \ ""
- 2 / Standard storage class
- \ "STANDARD"
-storage_class>
-Remote config
---------------------
-[ArvanCloud]
-env_auth = false
-access_key_id = YOURACCESSKEY
-secret_access_key = YOURSECRETACCESSKEY
-region = ir-thr-at1
-endpoint = s3.arvanstorage.com
-location_constraint =
-acl =
-server_side_encryption =
-storage_class =
---------------------
-y) Yes this is OK
-e) Edit this remote
-d) Delete this remote
-y/e/d> y
-This will leave the config file looking like this.
-[ArvanCloud]
-type = s3
-provider = ArvanCloud
-env_auth = false
-access_key_id = YOURACCESSKEY
-secret_access_key = YOURSECRETACCESSKEY
-region =
-endpoint = s3.arvanstorage.com
-location_constraint =
-acl =
-server_side_encryption =
-storage_class =
-Tencent COS
-Tencent Cloud Object Storage (COS) is a distributed storage service offered by Tencent Cloud for unstructured data. It is secure, stable, massive, convenient, low-delay and low-cost.
-To configure access to Tencent COS, follow the steps below:
-
-- Run
rclone config and select n for a new remote.
-
-rclone config
-No remotes found, make a new one?
-n) New remote
-s) Set configuration password
-q) Quit config
-n/s/q> n
-
-- Give the name of the configuration. For example, name it 'cos'.
-
-name> cos
-
-- Select
s3 storage.
-
-Choose a number from below, or type in your own value
-1 / 1Fichier
- \ "fichier"
- 2 / Alias for an existing remote
- \ "alias"
- 3 / Amazon Drive
- \ "amazon cloud drive"
- 4 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, ChinaMobile, ArvanCloud, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, Liara, Minio, and Tencent COS
- \ "s3"
-[snip]
-Storage> s3
-
-- Select
TencentCOS provider.
-
-Choose a number from below, or type in your own value
-1 / Amazon Web Services (AWS) S3
- \ "AWS"
-[snip]
-11 / Tencent Cloud Object Storage (COS)
- \ "TencentCOS"
-[snip]
-provider> TencentCOS
-
-- Enter your SecretId and SecretKey of Tencent Cloud.
-
-Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
-Only applies if access_key_id and secret_access_key is blank.
-Enter a boolean value (true or false). Press Enter for the default ("false").
-Choose a number from below, or type in your own value
- 1 / Enter AWS credentials in the next step
- \ "false"
- 2 / Get AWS credentials from the environment (env vars or IAM)
- \ "true"
-env_auth> 1
-AWS Access Key ID.
-Leave blank for anonymous access or runtime credentials.
-Enter a string value. Press Enter for the default ("").
-access_key_id> AKIDxxxxxxxxxx
-AWS Secret Access Key (password)
-Leave blank for anonymous access or runtime credentials.
-Enter a string value. Press Enter for the default ("").
-secret_access_key> xxxxxxxxxxx
-
-- Select endpoint for Tencent COS. This is the standard endpoint for different region.
+Here is how to do it.
+
+$ rclone config Current remotes:
+Name Type ==== ==== remote box
+
+- Edit existing remote
+- New remote
+- Delete remote
+- Rename remote
+- Copy remote
+- Set configuration password
+- Quit config e/n/d/r/c/s/q> e Choose a number from below, or type in an existing value 1 > remote remote> remote -------------------- [remote] type = box token = {"access_token":"XXX","token_type":"bearer","refresh_token":"XXX","expiry":"2017-07-08T23:40:08.059167677+01:00"} -------------------- Edit remote Value "client_id" = "" Edit? (y/n)>
+- Yes
+- No y/n> n Value "client_secret" = "" Edit? (y/n)>
+- Yes
+- No y/n> n Remote config Already have a token - refresh?
+- Yes
+- No y/n> y Use web browser to automatically authenticate rclone with remote?
- 1 / Beijing Region.
- \ "cos.ap-beijing.myqcloud.com"
- 2 / Nanjing Region.
- \ "cos.ap-nanjing.myqcloud.com"
- 3 / Shanghai Region.
- \ "cos.ap-shanghai.myqcloud.com"
- 4 / Guangzhou Region.
- \ "cos.ap-guangzhou.myqcloud.com"
-[snip]
-endpoint> 4
-
-- Choose acl and storage class.
+
+- Say Y if the machine running rclone has a web browser you can use
+- Say N if running rclone on a (remote) machine without web browser access If not sure try Y. If Y failed, try N.
+
+
+- Yes
+- No y/n> y If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth Log in and authorize rclone for access Waiting for code... Got code -------------------- [remote] type = box token = {"access_token":"YYY","token_type":"bearer","refresh_token":"YYY","expiry":"2017-07-23T12:22:29.259137901+01:00"} --------------------
+- Yes this is OK
+- Edit this remote
+- Delete this remote y/e/d> y
-Note that this ACL is applied when server-side copying objects as S3
-doesn't copy the ACL from the source but rather writes a fresh one.
-Enter a string value. Press Enter for the default ("").
-Choose a number from below, or type in your own value
- 1 / Owner gets Full_CONTROL. No one else has access rights (default).
- \ "default"
-[snip]
-acl> 1
-The storage class to use when storing new objects in Tencent COS.
-Enter a string value. Press Enter for the default ("").
-Choose a number from below, or type in your own value
- 1 / Default
- \ ""
-[snip]
-storage_class> 1
-Edit advanced config? (y/n)
-y) Yes
-n) No (default)
-y/n> n
-Remote config
---------------------
-[cos]
-type = s3
-provider = TencentCOS
-env_auth = false
-access_key_id = xxx
-secret_access_key = xxx
-endpoint = cos.ap-guangzhou.myqcloud.com
-acl = default
---------------------
-y) Yes this is OK (default)
-e) Edit this remote
-d) Delete this remote
-y/e/d> y
-Current remotes:
+
+### Modification times and hashes
-Name Type
-==== ====
-cos s3
-Netease NOS
-For Netease NOS configure as per the configurator rclone config setting the provider Netease. This will automatically set force_path_style = false which is necessary for it to run properly.
-Petabox
-Here is an example of making a Petabox configuration. First run:
-rclone config
-This will guide you through an interactive setup process.
-No remotes found, make a new one?
-n) New remote
-s) Set configuration password
-n/s> n
+Box allows modification times to be set on objects accurate to 1
+second. These will be used to detect whether objects need syncing or
+not.
-Enter name for new remote.
-name> My Petabox Storage
+Box supports SHA1 type hashes, so you can use the `--checksum`
+flag.
-Option Storage.
-Type of storage to configure.
-Choose a number from below, or type in your own value.
-[snip]
-XX / Amazon S3 Compliant Storage Providers including AWS, ...
- \ "s3"
-[snip]
-Storage> s3
+### Restricted filename characters
-Option provider.
-Choose your S3 provider.
-Choose a number from below, or type in your own value.
-Press Enter to leave empty.
-[snip]
-XX / Petabox Object Storage
- \ (Petabox)
-[snip]
-provider> Petabox
+In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters)
+the following characters are also replaced:
-Option env_auth.
-Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
-Only applies if access_key_id and secret_access_key is blank.
-Choose a number from below, or type in your own boolean value (true or false).
-Press Enter for the default (false).
- 1 / Enter AWS credentials in the next step.
- \ (false)
- 2 / Get AWS credentials from the environment (env vars or IAM).
- \ (true)
-env_auth> 1
+| Character | Value | Replacement |
+| --------- |:-----:|:-----------:|
+| \ | 0x5C | ï¼¼ |
-Option access_key_id.
-AWS Access Key ID.
-Leave blank for anonymous access or runtime credentials.
-Enter a value. Press Enter to leave empty.
-access_key_id> YOUR_ACCESS_KEY_ID
+File names can also not end with the following characters.
+These only get replaced if they are the last character in the name:
-Option secret_access_key.
-AWS Secret Access Key (password).
-Leave blank for anonymous access or runtime credentials.
-Enter a value. Press Enter to leave empty.
-secret_access_key> YOUR_SECRET_ACCESS_KEY
+| Character | Value | Replacement |
+| --------- |:-----:|:-----------:|
+| SP | 0x20 | â |
-Option region.
-Region where your bucket will be created and your data stored.
-Choose a number from below, or type in your own value.
-Press Enter to leave empty.
- 1 / US East (N. Virginia)
- \ (us-east-1)
- 2 / Europe (Frankfurt)
- \ (eu-central-1)
- 3 / Asia Pacific (Singapore)
- \ (ap-southeast-1)
- 4 / Middle East (Bahrain)
- \ (me-south-1)
- 5 / South America (São Paulo)
- \ (sa-east-1)
-region> 1
+Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8),
+as they can't be used in JSON strings.
-Option endpoint.
-Endpoint for Petabox S3 Object Storage.
-Specify the endpoint from the same region.
-Choose a number from below, or type in your own value.
- 1 / US East (N. Virginia)
- \ (s3.petabox.io)
- 2 / US East (N. Virginia)
- \ (s3.us-east-1.petabox.io)
- 3 / Europe (Frankfurt)
- \ (s3.eu-central-1.petabox.io)
- 4 / Asia Pacific (Singapore)
- \ (s3.ap-southeast-1.petabox.io)
- 5 / Middle East (Bahrain)
- \ (s3.me-south-1.petabox.io)
- 6 / South America (São Paulo)
- \ (s3.sa-east-1.petabox.io)
-endpoint> 1
+### Transfers
+
+For files above 50 MiB rclone will use a chunked transfer. Rclone will
+upload up to `--transfers` chunks at the same time (shared among all
+the multipart uploads). Chunks are buffered in memory and are
+normally 8 MiB so increasing `--transfers` will increase memory use.
+
+### Deleting files
+
+Depending on the enterprise settings for your user, the item will
+either be actually deleted from Box or moved to the trash.
+
+Emptying the trash is supported via the rclone however cleanup command
+however this deletes every trashed file and folder individually so it
+may take a very long time.
+Emptying the trash via the WebUI does not have this limitation
+so it is advised to empty the trash via the WebUI.
+
+### Root folder ID
+
+You can set the `root_folder_id` for rclone. This is the directory
+(identified by its `Folder ID`) that rclone considers to be the root
+of your Box drive.
+
+Normally you will leave this blank and rclone will determine the
+correct root to use itself.
+
+However you can set this to restrict rclone to a specific folder
+hierarchy.
+
+In order to do this you will have to find the `Folder ID` of the
+directory you wish rclone to display. This will be the last segment
+of the URL when you open the relevant folder in the Box web
+interface.
+
+So if the folder you want rclone to use has a URL which looks like
+`https://app.box.com/folder/11xxxxxxxxx8`
+in the browser, then you use `11xxxxxxxxx8` as
+the `root_folder_id` in the config.
+
+
+### Standard options
+
+Here are the Standard options specific to box (Box).
+
+#### --box-client-id
+
+OAuth Client Id.
+
+Leave blank normally.
+
+Properties:
+
+- Config: client_id
+- Env Var: RCLONE_BOX_CLIENT_ID
+- Type: string
+- Required: false
+
+#### --box-client-secret
+
+OAuth Client Secret.
+
+Leave blank normally.
+
+Properties:
+
+- Config: client_secret
+- Env Var: RCLONE_BOX_CLIENT_SECRET
+- Type: string
+- Required: false
+
+#### --box-box-config-file
+
+Box App config.json location
+
+Leave blank normally.
+
+Leading `~` will be expanded in the file name as will environment variables such as `${RCLONE_CONFIG_DIR}`.
+
+Properties:
+
+- Config: box_config_file
+- Env Var: RCLONE_BOX_BOX_CONFIG_FILE
+- Type: string
+- Required: false
-Option acl.
-Canned ACL used when creating buckets and storing or copying objects.
-This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.
-For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
-Note that this ACL is applied when server-side copying objects as S3
-doesn't copy the ACL from the source but rather writes a fresh one.
-If the acl is an empty string then no X-Amz-Acl: header is added and
-the default (private) will be used.
-Choose a number from below, or type in your own value.
-Press Enter to leave empty.
- / Owner gets FULL_CONTROL.
- 1 | No one else has access rights (default).
- \ (private)
- / Owner gets FULL_CONTROL.
- 2 | The AllUsers group gets READ access.
- \ (public-read)
- / Owner gets FULL_CONTROL.
- 3 | The AllUsers group gets READ and WRITE access.
- | Granting this on a bucket is generally not recommended.
- \ (public-read-write)
- / Owner gets FULL_CONTROL.
- 4 | The AuthenticatedUsers group gets READ access.
- \ (authenticated-read)
- / Object owner gets FULL_CONTROL.
- 5 | Bucket owner gets READ access.
- | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
- \ (bucket-owner-read)
- / Both the object owner and the bucket owner get FULL_CONTROL over the object.
- 6 | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
- \ (bucket-owner-full-control)
-acl> 1
+#### --box-access-token
-Edit advanced config?
-y) Yes
-n) No (default)
-y/n> No
+Box App Primary Access Token
-Configuration complete.
-Options:
-- type: s3
-- provider: Petabox
-- access_key_id: YOUR_ACCESS_KEY_ID
-- secret_access_key: YOUR_SECRET_ACCESS_KEY
-- region: us-east-1
-- endpoint: s3.petabox.io
-Keep this "My Petabox Storage" remote?
-y) Yes this is OK (default)
-e) Edit this remote
-d) Delete this remote
-y/e/d> y
-This will leave the config file looking like this.
-[My Petabox Storage]
-type = s3
-provider = Petabox
-access_key_id = YOUR_ACCESS_KEY_ID
-secret_access_key = YOUR_SECRET_ACCESS_KEY
-region = us-east-1
-endpoint = s3.petabox.io
-Storj
-Storj is a decentralized cloud storage which can be used through its native protocol or an S3 compatible gateway.
-The S3 compatible gateway is configured using rclone config with a type of s3 and with a provider name of Storj. Here is an example run of the configurator.
-Type of storage to configure.
-Storage> s3
-Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
-Only applies if access_key_id and secret_access_key is blank.
-Choose a number from below, or type in your own boolean value (true or false).
-Press Enter for the default (false).
- 1 / Enter AWS credentials in the next step.
- \ (false)
- 2 / Get AWS credentials from the environment (env vars or IAM).
- \ (true)
-env_auth> 1
-Option access_key_id.
-AWS Access Key ID.
-Leave blank for anonymous access or runtime credentials.
-Enter a value. Press Enter to leave empty.
-access_key_id> XXXX (as shown when creating the access grant)
-Option secret_access_key.
-AWS Secret Access Key (password).
-Leave blank for anonymous access or runtime credentials.
-Enter a value. Press Enter to leave empty.
-secret_access_key> XXXX (as shown when creating the access grant)
-Option endpoint.
-Endpoint of the Shared Gateway.
-Choose a number from below, or type in your own value.
-Press Enter to leave empty.
- 1 / EU1 Shared Gateway
- \ (gateway.eu1.storjshare.io)
- 2 / US1 Shared Gateway
- \ (gateway.us1.storjshare.io)
- 3 / Asia-Pacific Shared Gateway
- \ (gateway.ap1.storjshare.io)
-endpoint> 1 (as shown when creating the access grant)
-Edit advanced config?
-y) Yes
-n) No (default)
-y/n> n
-Note that s3 credentials are generated when you create an access grant.
-Backend quirks
-
---chunk-size is forced to be 64 MiB or greater. This will use more memory than the default of 5 MiB.- Server side copy is disabled as it isn't currently supported in the gateway.
-- GetTier and SetTier are not supported.
-
-Backend bugs
-Due to issue #39 uploading multipart files via the S3 gateway causes them to lose their metadata. For rclone's purpose this means that the modification time is not stored, nor is any MD5SUM (if one is available from the source).
-This has the following consequences:
-
-- Using
rclone rcat will fail as the medatada doesn't match after upload
-- Uploading files with
rclone mount will fail for the same reason
-
-- This can worked around by using
--vfs-cache-mode writes or --vfs-cache-mode full or setting --s3-upload-cutoff large
-
-- Files uploaded via a multipart upload won't have their modtimes
-
-- This will mean that
rclone sync will likely keep trying to upload files bigger than --s3-upload-cutoff
-- This can be worked around with
--checksum or --size-only or setting --s3-upload-cutoff large
-- The maximum value for
--s3-upload-cutoff is 5GiB though
-
-
-One general purpose workaround is to set --s3-upload-cutoff 5G. This means that rclone will upload files smaller than 5GiB as single parts. Note that this can be set in the config file with upload_cutoff = 5G or configured in the advanced settings. If you regularly transfer files larger than 5G then using --checksum or --size-only in rclone sync is the recommended workaround.
-Comparison with the native protocol
-Use the the native protocol to take advantage of client-side encryption as well as to achieve the best possible download performance. Uploads will be erasure-coded locally, thus a 1gb upload will result in 2.68gb of data being uploaded to storage nodes across the network.
-Use this backend and the S3 compatible Hosted Gateway to increase upload performance and reduce the load on your systems and network. Uploads will be encrypted and erasure-coded server-side, thus a 1GB upload will result in only in 1GB of data being uploaded to storage nodes across the network.
-For more detailed comparison please check the documentation of the storj backend.
-Limitations
-rclone about is not supported by the S3 backend. Backends without this capability cannot determine free space for an rclone mount or use policy mfs (most free space) as a member of an rclone union remote.
-See List of backends that do not support rclone about and rclone about
-Synology C2 Object Storage
-Synology C2 Object Storage provides a secure, S3-compatible, and cost-effective cloud storage solution without API request, download fees, and deletion penalty.
-The S3 compatible gateway is configured using rclone config with a type of s3 and with a provider name of Synology. Here is an example run of the configurator.
-First run:
-rclone config
-This will guide you through an interactive setup process.
-No remotes found, make a new one?
-n) New remote
-s) Set configuration password
-q) Quit config
+Leave blank normally.
-n/s/q> n
+Properties:
-Enter name for new remote.1
-name> syno
+- Config: access_token
+- Env Var: RCLONE_BOX_ACCESS_TOKEN
+- Type: string
+- Required: false
-Type of storage to configure.
-Enter a string value. Press Enter for the default ("").
-Choose a number from below, or type in your own value
+#### --box-box-sub-type
- 5 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, China Mobile, Cloudflare, GCS, ArvanCloud, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, IDrive e2, IONOS Cloud, Liara, Lyve Cloud, Minio, Netease, Petabox, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Synology, Tencent COS, Qiniu and Wasabi
- \ "s3"
-Storage> s3
-Choose your S3 provider.
-Enter a string value. Press Enter for the default ("").
-Choose a number from below, or type in your own value
- 24 / Synology C2 Object Storage
- \ (Synology)
+Properties:
-provider> Synology
+- Config: box_sub_type
+- Env Var: RCLONE_BOX_BOX_SUB_TYPE
+- Type: string
+- Default: "user"
+- Examples:
+ - "user"
+ - Rclone should act on behalf of a user.
+ - "enterprise"
+ - Rclone should act on behalf of a service account.
-Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
-Only applies if access_key_id and secret_access_key is blank.
-Enter a boolean value (true or false). Press Enter for the default ("false").
-Choose a number from below, or type in your own value
- 1 / Enter AWS credentials in the next step
- \ "false"
- 2 / Get AWS credentials from the environment (env vars or IAM)
- \ "true"
+### Advanced options
-env_auth> 1
+Here are the Advanced options specific to box (Box).
-AWS Access Key ID.
-Leave blank for anonymous access or runtime credentials.
-Enter a string value. Press Enter for the default ("").
+#### --box-token
-access_key_id> accesskeyid
+OAuth Access Token as a JSON blob.
-AWS Secret Access Key (password)
-Leave blank for anonymous access or runtime credentials.
-Enter a string value. Press Enter for the default ("").
+Properties:
-secret_access_key> secretaccesskey
+- Config: token
+- Env Var: RCLONE_BOX_TOKEN
+- Type: string
+- Required: false
-Region where your data stored.
-Choose a number from below, or type in your own value.
-Press Enter to leave empty.
- 1 / Europe Region 1
- \ (eu-001)
- 2 / Europe Region 2
- \ (eu-002)
- 3 / US Region 1
- \ (us-001)
- 4 / US Region 2
- \ (us-002)
- 5 / Asia (Taiwan)
- \ (tw-001)
+#### --box-auth-url
-region > 1
+Auth server URL.
-Option endpoint.
-Endpoint for Synology C2 Object Storage API.
-Choose a number from below, or type in your own value.
-Press Enter to leave empty.
- 1 / EU Endpoint 1
- \ (eu-001.s3.synologyc2.net)
- 2 / US Endpoint 1
- \ (us-001.s3.synologyc2.net)
- 3 / TW Endpoint 1
- \ (tw-001.s3.synologyc2.net)
+Leave blank to use the provider defaults.
-endpoint> 1
+Properties:
-Option location_constraint.
-Location constraint - must be set to match the Region.
-Leave blank if not sure. Used when creating buckets only.
-Enter a value. Press Enter to leave empty.
-location_constraint>
+- Config: auth_url
+- Env Var: RCLONE_BOX_AUTH_URL
+- Type: string
+- Required: false
-Edit advanced config? (y/n)
-y) Yes
-n) No
-y/n> y
+#### --box-token-url
-Option no_check_bucket.
-If set, don't attempt to check the bucket exists or create it.
-This can be useful when trying to minimise the number of transactions
-rclone does if you know the bucket exists already.
-It can also be needed if the user you are using does not have bucket
-creation permissions. Before v1.52.0 this would have passed silently
-due to a bug.
-Enter a boolean value (true or false). Press Enter for the default (true).
+Token server url.
-no_check_bucket> true
+Leave blank to use the provider defaults.
-Configuration complete.
-Options:
-- type: s3
-- provider: Synology
-- region: eu-001
-- endpoint: eu-001.s3.synologyc2.net
-- no_check_bucket: true
-Keep this "syno" remote?
-y) Yes this is OK (default)
-e) Edit this remote
-d) Delete this remote
+Properties:
-y/e/d> y
+- Config: token_url
+- Env Var: RCLONE_BOX_TOKEN_URL
+- Type: string
+- Required: false
-# Backblaze B2
+#### --box-root-folder-id
-B2 is [Backblaze's cloud storage system](https://www.backblaze.com/b2/).
+Fill in for rclone to use a non root folder as its starting point.
-Paths are specified as `remote:bucket` (or `remote:` for the `lsd`
-command.) You may put subdirectories in too, e.g. `remote:bucket/path/to/dir`.
+Properties:
-## Configuration
+- Config: root_folder_id
+- Env Var: RCLONE_BOX_ROOT_FOLDER_ID
+- Type: string
+- Default: "0"
-Here is an example of making a b2 configuration. First run
+#### --box-upload-cutoff
- rclone config
+Cutoff for switching to multipart upload (>= 50 MiB).
-This will guide you through an interactive setup process. To authenticate
-you will either need your Account ID (a short hex number) and Master
-Application Key (a long hex number) OR an Application Key, which is the
-recommended method. See below for further details on generating and using
-an Application Key.
-
-No remotes found, make a new one? n) New remote q) Quit config n/q> n name> remote Type of storage to configure. Choose a number from below, or type in your own value [snip] XX / Backblaze B2 Â "b2" [snip] Storage> b2 Account ID or Application Key ID account> 123456789abc Application Key key> 0123456789abcdef0123456789abcdef0123456789 Endpoint for the service - leave blank normally. endpoint> Remote config -------------------- [remote] account = 123456789abc key = 0123456789abcdef0123456789abcdef0123456789 endpoint = -------------------- y) Yes this is OK e) Edit this remote d) Delete this remote y/e/d> y
-
-This remote is called `remote` and can now be used like this
+Properties:
-See all buckets
+- Config: upload_cutoff
+- Env Var: RCLONE_BOX_UPLOAD_CUTOFF
+- Type: SizeSuffix
+- Default: 50Mi
- rclone lsd remote:
+#### --box-commit-retries
-Create a new bucket
+Max number of times to try committing a multipart file.
- rclone mkdir remote:bucket
+Properties:
-List the contents of a bucket
+- Config: commit_retries
+- Env Var: RCLONE_BOX_COMMIT_RETRIES
+- Type: int
+- Default: 100
- rclone ls remote:bucket
+#### --box-list-chunk
-Sync `/home/local/directory` to the remote bucket, deleting any
-excess files in the bucket.
+Size of listing chunk 1-1000.
- rclone sync --interactive /home/local/directory remote:bucket
+Properties:
-### Application Keys
+- Config: list_chunk
+- Env Var: RCLONE_BOX_LIST_CHUNK
+- Type: int
+- Default: 1000
-B2 supports multiple [Application Keys for different access permission
-to B2 Buckets](https://www.backblaze.com/b2/docs/application_keys.html).
+#### --box-owned-by
-You can use these with rclone too; you will need to use rclone version 1.43
-or later.
+Only show items owned by the login (email address) passed in.
-Follow Backblaze's docs to create an Application Key with the required
-permission and add the `applicationKeyId` as the `account` and the
-`Application Key` itself as the `key`.
+Properties:
-Note that you must put the _applicationKeyId_ as the `account` – you
-can't use the master Account ID. If you try then B2 will return 401
-errors.
+- Config: owned_by
+- Env Var: RCLONE_BOX_OWNED_BY
+- Type: string
+- Required: false
-### --fast-list
+#### --box-impersonate
-This remote supports `--fast-list` which allows you to use fewer
-transactions in exchange for more memory. See the [rclone
-docs](https://rclone.org/docs/#fast-list) for more details.
+Impersonate this user ID when using a service account.
-### Modified time
+Setting this flag allows rclone, when using a JWT service account, to
+act on behalf of another user by setting the as-user header.
-The modified time is stored as metadata on the object as
-`X-Bz-Info-src_last_modified_millis` as milliseconds since 1970-01-01
-in the Backblaze standard. Other tools should be able to use this as
-a modified time.
+The user ID is the Box identifier for a user. User IDs can found for
+any user via the GET /users endpoint, which is only available to
+admins, or by calling the GET /users/me endpoint with an authenticated
+user session.
-Modified times are used in syncing and are fully supported. Note that
-if a modification time needs to be updated on an object then it will
-create a new version of the object.
+See: https://developer.box.com/guides/authentication/jwt/as-user/
-### Restricted filename characters
-In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters)
-the following characters are also replaced:
+Properties:
-| Character | Value | Replacement |
-| --------- |:-----:|:-----------:|
-| \ | 0x5C | ï¼¼ |
+- Config: impersonate
+- Env Var: RCLONE_BOX_IMPERSONATE
+- Type: string
+- Required: false
-Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8),
-as they can't be used in JSON strings.
+#### --box-encoding
-Note that in 2020-05 Backblaze started allowing \ characters in file
-names. Rclone hasn't changed its encoding as this could cause syncs to
-re-transfer files. If you want rclone not to replace \ then see the
-`--b2-encoding` flag below and remove the `BackSlash` from the
-string. This can be set in the config.
+The encoding for the backend.
-### SHA1 checksums
+See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info.
-The SHA1 checksums of the files are checked on upload and download and
-will be used in the syncing process.
+Properties:
-Large files (bigger than the limit in `--b2-upload-cutoff`) which are
-uploaded in chunks will store their SHA1 on the object as
-`X-Bz-Info-large_file_sha1` as recommended by Backblaze.
+- Config: encoding
+- Env Var: RCLONE_BOX_ENCODING
+- Type: Encoding
+- Default: Slash,BackSlash,Del,Ctl,RightSpace,InvalidUtf8,Dot
-For a large file to be uploaded with an SHA1 checksum, the source
-needs to support SHA1 checksums. The local disk supports SHA1
-checksums so large file transfers from local disk will have an SHA1.
-See [the overview](https://rclone.org/overview/#features) for exactly which remotes
-support SHA1.
-Sources which don't support SHA1, in particular `crypt` will upload
-large files without SHA1 checksums. This may be fixed in the future
-(see [#1767](https://github.com/rclone/rclone/issues/1767)).
-Files sizes below `--b2-upload-cutoff` will always have an SHA1
-regardless of the source.
+## Limitations
-### Transfers
+Note that Box is case insensitive so you can't have a file called
+"Hello.doc" and one called "hello.doc".
-Backblaze recommends that you do lots of transfers simultaneously for
-maximum speed. In tests from my SSD equipped laptop the optimum
-setting is about `--transfers 32` though higher numbers may be used
-for a slight speed improvement. The optimum number for you may vary
-depending on your hardware, how big the files are, how much you want
-to load your computer, etc. The default of `--transfers 4` is
-definitely too low for Backblaze B2 though.
+Box file names can't have the `\` character in. rclone maps this to
+and from an identical looking unicode equivalent `ï¼¼` (U+FF3C Fullwidth
+Reverse Solidus).
-Note that uploading big files (bigger than 200 MiB by default) will use
-a 96 MiB RAM buffer by default. There can be at most `--transfers` of
-these in use at any moment, so this sets the upper limit on the memory
-used.
+Box only supports filenames up to 255 characters in length.
-### Versions
+Box has [API rate limits](https://developer.box.com/guides/api-calls/permissions-and-errors/rate-limits/) that sometimes reduce the speed of rclone.
-When rclone uploads a new version of a file it creates a [new version
-of it](https://www.backblaze.com/b2/docs/file_versions.html).
-Likewise when you delete a file, the old version will be marked hidden
-and still be available. Conversely, you may opt in to a "hard delete"
-of files with the `--b2-hard-delete` flag which would permanently remove
-the file instead of hiding it.
+`rclone about` is not supported by the Box backend. Backends without
+this capability cannot determine free space for an rclone mount or
+use policy `mfs` (most free space) as a member of an rclone union
+remote.
-Old versions of files, where available, are visible using the
-`--b2-versions` flag.
+See [List of backends that do not support rclone about](https://rclone.org/overview/#optional-features) and [rclone about](https://rclone.org/commands/rclone_about/)
-It is also possible to view a bucket as it was at a certain point in time,
-using the `--b2-version-at` flag. This will show the file versions as they
-were at that time, showing files that have been deleted afterwards, and
-hiding files that were created since.
+## Get your own Box App ID
-If you wish to remove all the old versions then you can use the
-`rclone cleanup remote:bucket` command which will delete all the old
-versions of files, leaving the current ones intact. You can also
-supply a path and only old versions under that path will be deleted,
-e.g. `rclone cleanup remote:bucket/path/to/stuff`.
+Here is how to create your own Box App ID for rclone:
-Note that `cleanup` will remove partially uploaded files from the bucket
-if they are more than a day old.
+1. Go to the [Box Developer Console](https://app.box.com/developers/console)
+and login, then click `My Apps` on the sidebar. Click `Create New App`
+and select `Custom App`.
-When you `purge` a bucket, the current and the old versions will be
-deleted then the bucket will be deleted.
+2. In the first screen on the box that pops up, you can pretty much enter
+whatever you want. The `App Name` can be whatever. For `Purpose` choose
+automation to avoid having to fill out anything else. Click `Next`.
-However `delete` will cause the current versions of the files to
-become hidden old versions.
+3. In the second screen of the creation screen, select
+`User Authentication (OAuth 2.0)`. Then click `Create App`.
-Here is a session showing the listing and retrieval of an old
-version followed by a `cleanup` of the old versions.
+4. You should now be on the `Configuration` tab of your new app. If not,
+click on it at the top of the webpage. Copy down `Client ID`
+and `Client Secret`, you'll need those for rclone.
-Show current version and all the versions with `--b2-versions` flag.
-
-$ rclone -q ls b2:cleanup-test 9 one.txt
-$ rclone -q --b2-versions ls b2:cleanup-test 9 one.txt 8 one-v2016-07-04-141032-000.txt 16 one-v2016-07-04-141003-000.txt 15 one-v2016-07-02-155621-000.txt
-
-Retrieve an old version
-
-$ rclone -q --b2-versions copy b2:cleanup-test/one-v2016-07-04-141003-000.txt /tmp
-$ ls -l /tmp/one-v2016-07-04-141003-000.txt -rw-rw-r-- 1 ncw ncw 16 Jul 2 17:46 /tmp/one-v2016-07-04-141003-000.txt
-
-Clean up all the old versions and show that they've gone.
-
-$ rclone -q cleanup b2:cleanup-test
-$ rclone -q ls b2:cleanup-test 9 one.txt
-$ rclone -q --b2-versions ls b2:cleanup-test 9 one.txt
-
-#### Versions naming caveat
+5. Under "OAuth 2.0 Redirect URI", add `http://127.0.0.1:53682/`
-When using `--b2-versions` flag rclone is relying on the file name
-to work out whether the objects are versions or not. Versions' names
-are created by inserting timestamp between file name and its extension.
- 9 file.txt
- 8 file-v2023-07-17-161032-000.txt
- 16 file-v2023-06-15-141003-000.txt
-If there are real files present with the same names as versions, then
-behaviour of `--b2-versions` can be unpredictable.
+6. For `Application Scopes`, select `Read all files and folders stored in Box`
+and `Write all files and folders stored in box` (assuming you want to do both).
+Leave others unchecked. Click `Save Changes` at the top right.
-### Data usage
+# Cache
-It is useful to know how many requests are sent to the server in different scenarios.
+The `cache` remote wraps another existing remote and stores file structure
+and its data for long running tasks like `rclone mount`.
-All copy commands send the following 4 requests:
-
-/b2api/v1/b2_authorize_account /b2api/v1/b2_create_bucket /b2api/v1/b2_list_buckets /b2api/v1/b2_list_file_names
-
-The `b2_list_file_names` request will be sent once for every 1k files
-in the remote path, providing the checksum and modification time of
-the listed files. As of version 1.33 issue
-[#818](https://github.com/rclone/rclone/issues/818) causes extra requests
-to be sent when using B2 with Crypt. When a copy operation does not
-require any files to be uploaded, no more requests will be sent.
+## Status
+
+The cache backend code is working but it currently doesn't
+have a maintainer so there are [outstanding bugs](https://github.com/rclone/rclone/issues?q=is%3Aopen+is%3Aissue+label%3Abug+label%3A%22Remote%3A+Cache%22) which aren't getting fixed.
-Uploading files that do not require chunking, will send 2 requests per
-file upload:
-
-/b2api/v1/b2_get_upload_url /b2api/v1/b2_upload_file/
-
-Uploading files requiring chunking, will send 2 requests (one each to
-start and finish the upload) and another 2 requests for each chunk:
-
-/b2api/v1/b2_start_large_file /b2api/v1/b2_get_upload_part_url /b2api/v1/b2_upload_part/ /b2api/v1/b2_finish_large_file
-
-#### Versions
+The cache backend is due to be phased out in favour of the VFS caching
+layer eventually which is more tightly integrated into rclone.
-Versions can be viewed with the `--b2-versions` flag. When it is set
-rclone will show and act on older versions of files. For example
+Until this happens we recommend only using the cache backend if you
+find you can't work without it. There are many docs online describing
+the use of the cache backend to minimize API hits and by-and-large
+these are out of date and the cache backend isn't needed in those
+scenarios any more.
-Listing without `--b2-versions`
-
-$ rclone -q ls b2:cleanup-test 9 one.txt
-
-And with
-
-$ rclone -q --b2-versions ls b2:cleanup-test 9 one.txt 8 one-v2016-07-04-141032-000.txt 16 one-v2016-07-04-141003-000.txt 15 one-v2016-07-02-155621-000.txt
-
-Showing that the current version is unchanged but older versions can
-be seen. These have the UTC date that they were uploaded to the
-server to the nearest millisecond appended to them.
+## Configuration
-Note that when using `--b2-versions` no file write operations are
-permitted, so you can't upload files or delete them.
+To get started you just need to have an existing remote which can be configured
+with `cache`.
-### B2 and rclone link
+Here is an example of how to make a remote called `test-cache`. First run:
-Rclone supports generating file share links for private B2 buckets.
-They can either be for a file for example:
-
-./rclone link B2:bucket/path/to/file.txt https://f002.backblazeb2.com/file/bucket/path/to/file.txt?Authorization=xxxxxxxx
-
-or if run on a directory you will get:
-
-./rclone link B2:bucket/path https://f002.backblazeb2.com/file/bucket/path?Authorization=xxxxxxxx
-
-you can then use the authorization token (the part of the url from the
- `?Authorization=` on) on any file path under that directory. For example:
+ rclone config
+
+This will guide you through an interactive setup process:
-https://f002.backblazeb2.com/file/bucket/path/to/file1?Authorization=xxxxxxxx https://f002.backblazeb2.com/file/bucket/path/file2?Authorization=xxxxxxxx https://f002.backblazeb2.com/file/bucket/path/folder/file3?Authorization=xxxxxxxx
+No remotes found, make a new one? n) New remote r) Rename remote c) Copy remote s) Set configuration password q) Quit config n/r/c/s/q> n name> test-cache Type of storage to configure. Choose a number from below, or type in your own value [snip] XX / Cache a remote  "cache" [snip] Storage> cache Remote to cache. Normally should contain a ':' and a path, e.g. "myremote:path/to/dir", "myremote:bucket" or maybe "myremote:" (not recommended). remote> local:/test Optional: The URL of the Plex server plex_url> http://127.0.0.1:32400 Optional: The username of the Plex user plex_username> dummyusername Optional: The password of the Plex user y) Yes type in my own password g) Generate random password n) No leave this optional password blank y/g/n> y Enter the password: password: Confirm the password: password: The size of a chunk. Lower value good for slow connections but can affect seamless reading. Default: 5M Choose a number from below, or type in your own value 1 / 1 MiB  "1M" 2 / 5 MiB  "5M" 3 / 10 MiB  "10M" chunk_size> 2 How much time should object info (file size, file hashes, etc.) be stored in cache. Use a very high value if you don't plan on changing the source FS from outside the cache. Accepted units are: "s", "m", "h". Default: 5m Choose a number from below, or type in your own value 1 / 1 hour  "1h" 2 / 24 hours  "24h" 3 / 24 hours  "48h" info_age> 2 The maximum size of stored chunks. When the storage grows beyond this size, the oldest chunks will be deleted. Default: 10G Choose a number from below, or type in your own value 1 / 500 MiB  "500M" 2 / 1 GiB  "1G" 3 / 10 GiB  "10G" chunk_total_size> 3 Remote config -------------------- [test-cache] remote = local:/test plex_url = http://127.0.0.1:32400 plex_username = dummyusername plex_password = *** ENCRYPTED *** chunk_size = 5M info_age = 48h chunk_total_size = 10G
+You can then use it like this,
-### Standard options
-
-Here are the Standard options specific to b2 (Backblaze B2).
-
-#### --b2-account
+List directories in top level of your drive
-Account ID or Application Key ID.
+ rclone lsd test-cache:
-Properties:
+List all the files in your drive
-- Config: account
-- Env Var: RCLONE_B2_ACCOUNT
-- Type: string
-- Required: true
+ rclone ls test-cache:
-#### --b2-key
+To start a cached mount
-Application Key.
+ rclone mount --allow-other test-cache: /var/tmp/test-cache
-Properties:
+### Write Features ###
-- Config: key
-- Env Var: RCLONE_B2_KEY
-- Type: string
-- Required: true
+### Offline uploading ###
-#### --b2-hard-delete
+In an effort to make writing through cache more reliable, the backend
+now supports this feature which can be activated by specifying a
+`cache-tmp-upload-path`.
-Permanently delete files on remote removal, otherwise hide files.
+A files goes through these states when using this feature:
-Properties:
+1. An upload is started (usually by copying a file on the cache remote)
+2. When the copy to the temporary location is complete the file is part
+of the cached remote and looks and behaves like any other file (reading included)
+3. After `cache-tmp-wait-time` passes and the file is next in line, `rclone move`
+is used to move the file to the cloud provider
+4. Reading the file still works during the upload but most modifications on it will be prohibited
+5. Once the move is complete the file is unlocked for modifications as it
+becomes as any other regular file
+6. If the file is being read through `cache` when it's actually
+deleted from the temporary path then `cache` will simply swap the source
+to the cloud provider without interrupting the reading (small blip can happen though)
-- Config: hard_delete
-- Env Var: RCLONE_B2_HARD_DELETE
-- Type: bool
-- Default: false
+Files are uploaded in sequence and only one file is uploaded at a time.
+Uploads will be stored in a queue and be processed based on the order they were added.
+The queue and the temporary storage is persistent across restarts but
+can be cleared on startup with the `--cache-db-purge` flag.
-### Advanced options
+### Write Support ###
-Here are the Advanced options specific to b2 (Backblaze B2).
+Writes are supported through `cache`.
+One caveat is that a mounted cache remote does not add any retry or fallback
+mechanism to the upload operation. This will depend on the implementation
+of the wrapped remote. Consider using `Offline uploading` for reliable writes.
-#### --b2-endpoint
+One special case is covered with `cache-writes` which will cache the file
+data at the same time as the upload when it is enabled making it available
+from the cache store immediately once the upload is finished.
-Endpoint for the service.
+### Read Features ###
-Leave blank normally.
+#### Multiple connections ####
-Properties:
+To counter the high latency between a local PC where rclone is running
+and cloud providers, the cache remote can split multiple requests to the
+cloud provider for smaller file chunks and combines them together locally
+where they can be available almost immediately before the reader usually
+needs them.
-- Config: endpoint
-- Env Var: RCLONE_B2_ENDPOINT
-- Type: string
-- Required: false
+This is similar to buffering when media files are played online. Rclone
+will stay around the current marker but always try its best to stay ahead
+and prepare the data before.
-#### --b2-test-mode
+#### Plex Integration ####
-A flag string for X-Bz-Test-Mode header for debugging.
+There is a direct integration with Plex which allows cache to detect during reading
+if the file is in playback or not. This helps cache to adapt how it queries
+the cloud provider depending on what is needed for.
-This is for debugging purposes only. Setting it to one of the strings
-below will cause b2 to return specific errors:
+Scans will have a minimum amount of workers (1) while in a confirmed playback cache
+will deploy the configured number of workers.
- * "fail_some_uploads"
- * "expire_some_account_authorization_tokens"
- * "force_cap_exceeded"
+This integration opens the doorway to additional performance improvements
+which will be explored in the near future.
-These will be set in the "X-Bz-Test-Mode" header which is documented
-in the [b2 integrations checklist](https://www.backblaze.com/b2/docs/integration_checklist.html).
+**Note:** If Plex options are not configured, `cache` will function with its
+configured options without adapting any of its settings.
-Properties:
+How to enable? Run `rclone config` and add all the Plex options (endpoint, username
+and password) in your remote and it will be automatically enabled.
-- Config: test_mode
-- Env Var: RCLONE_B2_TEST_MODE
-- Type: string
-- Required: false
+Affected settings:
+- `cache-workers`: _Configured value_ during confirmed playback or _1_ all the other times
-#### --b2-versions
+##### Certificate Validation #####
-Include old versions in directory listings.
+When the Plex server is configured to only accept secure connections, it is
+possible to use `.plex.direct` URLs to ensure certificate validation succeeds.
+These URLs are used by Plex internally to connect to the Plex server securely.
-Note that when using this no file write operations are permitted,
-so you can't upload files or delete them.
+The format for these URLs is the following:
-Properties:
+`https://ip-with-dots-replaced.server-hash.plex.direct:32400/`
-- Config: versions
-- Env Var: RCLONE_B2_VERSIONS
-- Type: bool
-- Default: false
+The `ip-with-dots-replaced` part can be any IPv4 address, where the dots
+have been replaced with dashes, e.g. `127.0.0.1` becomes `127-0-0-1`.
-#### --b2-version-at
+To get the `server-hash` part, the easiest way is to visit
-Show file versions as they were at the specified time.
+https://plex.tv/api/resources?includeHttps=1&X-Plex-Token=your-plex-token
-Note that when using this no file write operations are permitted,
-so you can't upload files or delete them.
+This page will list all the available Plex servers for your account
+with at least one `.plex.direct` link for each. Copy one URL and replace
+the IP address with the desired address. This can be used as the
+`plex_url` value.
-Properties:
+### Known issues ###
-- Config: version_at
-- Env Var: RCLONE_B2_VERSION_AT
-- Type: Time
-- Default: off
+#### Mount and --dir-cache-time ####
-#### --b2-upload-cutoff
+--dir-cache-time controls the first layer of directory caching which works at the mount layer.
+Being an independent caching mechanism from the `cache` backend, it will manage its own entries
+based on the configured time.
-Cutoff for switching to chunked upload.
+To avoid getting in a scenario where dir cache has obsolete data and cache would have the correct
+one, try to set `--dir-cache-time` to a lower time than `--cache-info-age`. Default values are
+already configured in this way.
-Files above this size will be uploaded in chunks of "--b2-chunk-size".
+#### Windows support - Experimental ####
-This value should be set no larger than 4.657 GiB (== 5 GB).
+There are a couple of issues with Windows `mount` functionality that still require some investigations.
+It should be considered as experimental thus far as fixes come in for this OS.
-Properties:
+Most of the issues seem to be related to the difference between filesystems
+on Linux flavors and Windows as cache is heavily dependent on them.
-- Config: upload_cutoff
-- Env Var: RCLONE_B2_UPLOAD_CUTOFF
-- Type: SizeSuffix
-- Default: 200Mi
+Any reports or feedback on how cache behaves on this OS is greatly appreciated.
+
+- https://github.com/rclone/rclone/issues/1935
+- https://github.com/rclone/rclone/issues/1907
+- https://github.com/rclone/rclone/issues/1834
-#### --b2-copy-cutoff
+#### Risk of throttling ####
-Cutoff for switching to multipart copy.
+Future iterations of the cache backend will make use of the pooling functionality
+of the cloud provider to synchronize and at the same time make writing through it
+more tolerant to failures.
-Any files larger than this that need to be server-side copied will be
-copied in chunks of this size.
+There are a couple of enhancements in track to add these but in the meantime
+there is a valid concern that the expiring cache listings can lead to cloud provider
+throttles or bans due to repeated queries on it for very large mounts.
-The minimum is 0 and the maximum is 4.6 GiB.
+Some recommendations:
+- don't use a very small interval for entry information (`--cache-info-age`)
+- while writes aren't yet optimised, you can still write through `cache` which gives you the advantage
+of adding the file in the cache at the same time if configured to do so.
-Properties:
+Future enhancements:
-- Config: copy_cutoff
-- Env Var: RCLONE_B2_COPY_CUTOFF
-- Type: SizeSuffix
-- Default: 4Gi
+- https://github.com/rclone/rclone/issues/1937
+- https://github.com/rclone/rclone/issues/1936
-#### --b2-chunk-size
+#### cache and crypt ####
-Upload chunk size.
+One common scenario is to keep your data encrypted in the cloud provider
+using the `crypt` remote. `crypt` uses a similar technique to wrap around
+an existing remote and handles this translation in a seamless way.
-When uploading large files, chunk the file into this size.
+There is an issue with wrapping the remotes in this order:
+**cloud remote** -> **crypt** -> **cache**
-Must fit in memory. These chunks are buffered in memory and there
-might a maximum of "--transfers" chunks in progress at once.
+During testing, I experienced a lot of bans with the remotes in this order.
+I suspect it might be related to how crypt opens files on the cloud provider
+which makes it think we're downloading the full file instead of small chunks.
+Organizing the remotes in this order yields better results:
+**cloud remote** -> **cache** -> **crypt**
-5,000,000 Bytes is the minimum size.
+#### absolute remote paths ####
-Properties:
+`cache` can not differentiate between relative and absolute paths for the wrapped remote.
+Any path given in the `remote` config setting and on the command line will be passed to
+the wrapped remote as is, but for storing the chunks on disk the path will be made
+relative by removing any leading `/` character.
-- Config: chunk_size
-- Env Var: RCLONE_B2_CHUNK_SIZE
-- Type: SizeSuffix
-- Default: 96Mi
+This behavior is irrelevant for most backend types, but there are backends where a leading `/`
+changes the effective directory, e.g. in the `sftp` backend paths starting with a `/` are
+relative to the root of the SSH server and paths without are relative to the user home directory.
+As a result `sftp:bin` and `sftp:/bin` will share the same cache folder, even if they represent
+a different directory on the SSH server.
-#### --b2-upload-concurrency
+### Cache and Remote Control (--rc) ###
+Cache supports the new `--rc` mode in rclone and can be remote controlled through the following end points:
+By default, the listener is disabled if you do not add the flag.
-Concurrency for multipart uploads.
+### rc cache/expire
+Purge a remote from the cache backend. Supports either a directory or a file.
+It supports both encrypted and unencrypted file names if cache is wrapped by crypt.
-This is the number of chunks of the same file that are uploaded
-concurrently.
+Params:
+ - **remote** = path to remote **(required)**
+ - **withData** = true/false to delete cached data (chunks) as well _(optional, false by default)_
-Note that chunks are stored in memory and there may be up to
-"--transfers" * "--b2-upload-concurrency" chunks stored at once
-in memory.
-Properties:
+### Standard options
-- Config: upload_concurrency
-- Env Var: RCLONE_B2_UPLOAD_CONCURRENCY
-- Type: int
-- Default: 16
+Here are the Standard options specific to cache (Cache a remote).
-#### --b2-disable-checksum
+#### --cache-remote
-Disable checksums for large (> upload cutoff) files.
+Remote to cache.
-Normally rclone will calculate the SHA1 checksum of the input before
-uploading it so it can add it to metadata on the object. This is great
-for data integrity checking but can cause long delays for large files
-to start uploading.
+Normally should contain a ':' and a path, e.g. "myremote:path/to/dir",
+"myremote:bucket" or maybe "myremote:" (not recommended).
Properties:
-- Config: disable_checksum
-- Env Var: RCLONE_B2_DISABLE_CHECKSUM
-- Type: bool
-- Default: false
+- Config: remote
+- Env Var: RCLONE_CACHE_REMOTE
+- Type: string
+- Required: true
-#### --b2-download-url
+#### --cache-plex-url
-Custom endpoint for downloads.
+The URL of the Plex server.
-This is usually set to a Cloudflare CDN URL as Backblaze offers
-free egress for data downloaded through the Cloudflare network.
-Rclone works with private buckets by sending an "Authorization" header.
-If the custom endpoint rewrites the requests for authentication,
-e.g., in Cloudflare Workers, this header needs to be handled properly.
-Leave blank if you want to use the endpoint provided by Backblaze.
+Properties:
-The URL provided here SHOULD have the protocol and SHOULD NOT have
-a trailing slash or specify the /file/bucket subpath as rclone will
-request files with "{download_url}/file/{bucket_name}/{path}".
+- Config: plex_url
+- Env Var: RCLONE_CACHE_PLEX_URL
+- Type: string
+- Required: false
-Example:
-> https://mysubdomain.mydomain.tld
-(No trailing "/", "file" or "bucket")
+#### --cache-plex-username
+
+The username of the Plex user.
Properties:
-- Config: download_url
-- Env Var: RCLONE_B2_DOWNLOAD_URL
+- Config: plex_username
+- Env Var: RCLONE_CACHE_PLEX_USERNAME
- Type: string
- Required: false
-#### --b2-download-auth-duration
+#### --cache-plex-password
-Time before the authorization token will expire in s or suffix ms|s|m|h|d.
+The password of the Plex user.
-The duration before the download authorization token will expire.
-The minimum value is 1 second. The maximum value is one week.
+**NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/).
Properties:
-- Config: download_auth_duration
-- Env Var: RCLONE_B2_DOWNLOAD_AUTH_DURATION
-- Type: Duration
-- Default: 1w
+- Config: plex_password
+- Env Var: RCLONE_CACHE_PLEX_PASSWORD
+- Type: string
+- Required: false
-#### --b2-memory-pool-flush-time
+#### --cache-chunk-size
-How often internal memory buffer pools will be flushed. (no longer used)
+The size of a chunk (partial file data).
+
+Use lower numbers for slower connections. If the chunk size is
+changed, any downloaded chunks will be invalid and cache-chunk-path
+will need to be cleared or unexpected EOF errors will occur.
Properties:
-- Config: memory_pool_flush_time
-- Env Var: RCLONE_B2_MEMORY_POOL_FLUSH_TIME
-- Type: Duration
-- Default: 1m0s
+- Config: chunk_size
+- Env Var: RCLONE_CACHE_CHUNK_SIZE
+- Type: SizeSuffix
+- Default: 5Mi
+- Examples:
+ - "1M"
+ - 1 MiB
+ - "5M"
+ - 5 MiB
+ - "10M"
+ - 10 MiB
-#### --b2-memory-pool-use-mmap
+#### --cache-info-age
-Whether to use mmap buffers in internal memory pool. (no longer used)
+How long to cache file structure information (directory listings, file size, times, etc.).
+If all write operations are done through the cache then you can safely make
+this value very large as the cache store will also be updated in real time.
Properties:
-- Config: memory_pool_use_mmap
-- Env Var: RCLONE_B2_MEMORY_POOL_USE_MMAP
-- Type: bool
-- Default: false
+- Config: info_age
+- Env Var: RCLONE_CACHE_INFO_AGE
+- Type: Duration
+- Default: 6h0m0s
+- Examples:
+ - "1h"
+ - 1 hour
+ - "24h"
+ - 24 hours
+ - "48h"
+ - 48 hours
-#### --b2-encoding
+#### --cache-chunk-total-size
-The encoding for the backend.
+The total size that the chunks can take up on the local disk.
-See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info.
+If the cache exceeds this value then it will start to delete the
+oldest chunks until it goes under this value.
Properties:
-- Config: encoding
-- Env Var: RCLONE_B2_ENCODING
-- Type: MultiEncoder
-- Default: Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot
-
-
+- Config: chunk_total_size
+- Env Var: RCLONE_CACHE_CHUNK_TOTAL_SIZE
+- Type: SizeSuffix
+- Default: 10Gi
+- Examples:
+ - "500M"
+ - 500 MiB
+ - "1G"
+ - 1 GiB
+ - "10G"
+ - 10 GiB
-## Limitations
+### Advanced options
-`rclone about` is not supported by the B2 backend. Backends without
-this capability cannot determine free space for an rclone mount or
-use policy `mfs` (most free space) as a member of an rclone union
-remote.
+Here are the Advanced options specific to cache (Cache a remote).
-See [List of backends that do not support rclone about](https://rclone.org/overview/#optional-features) and [rclone about](https://rclone.org/commands/rclone_about/)
+#### --cache-plex-token
-# Box
+The plex token for authentication - auto set normally.
-Paths are specified as `remote:path`
+Properties:
-Paths may be as deep as required, e.g. `remote:directory/subdirectory`.
+- Config: plex_token
+- Env Var: RCLONE_CACHE_PLEX_TOKEN
+- Type: string
+- Required: false
-The initial setup for Box involves getting a token from Box which you
-can do either in your browser, or with a config.json downloaded from Box
-to use JWT authentication. `rclone config` walks you through it.
+#### --cache-plex-insecure
-## Configuration
+Skip all certificate verification when connecting to the Plex server.
-Here is an example of how to make a remote called `remote`. First run:
+Properties:
- rclone config
+- Config: plex_insecure
+- Env Var: RCLONE_CACHE_PLEX_INSECURE
+- Type: string
+- Required: false
-This will guide you through an interactive setup process:
-
-No remotes found, make a new one? n) New remote s) Set configuration password q) Quit config n/s/q> n name> remote Type of storage to configure. Choose a number from below, or type in your own value [snip] XX / Box  "box" [snip] Storage> box Box App Client Id - leave blank normally. client_id> Box App Client Secret - leave blank normally. client_secret> Box App config.json location Leave blank normally. Enter a string value. Press Enter for the default (""). box_config_file> Box App Primary Access Token Leave blank normally. Enter a string value. Press Enter for the default (""). access_token>
-Enter a string value. Press Enter for the default ("user"). Choose a number from below, or type in your own value 1 / Rclone should act on behalf of a user  "user" 2 / Rclone should act on behalf of a service account  "enterprise" box_sub_type> Remote config Use web browser to automatically authenticate rclone with remote? * Say Y if the machine running rclone has a web browser you can use * Say N if running rclone on a (remote) machine without web browser access If not sure try Y. If Y failed, try N. y) Yes n) No y/n> y If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth Log in and authorize rclone for access Waiting for code... Got code -------------------- [remote] client_id = client_secret = token = {"access_token":"XXX","token_type":"bearer","refresh_token":"XXX","expiry":"XXX"} -------------------- y) Yes this is OK e) Edit this remote d) Delete this remote y/e/d> y
-
-See the [remote setup docs](https://rclone.org/remote_setup/) for how to set it up on a
-machine with no Internet browser available.
+#### --cache-db-path
-Note that rclone runs a webserver on your local machine to collect the
-token as returned from Box. This only runs from the moment it opens
-your browser to the moment you get back the verification code. This
-is on `http://127.0.0.1:53682/` and this it may require you to unblock
-it temporarily if you are running a host firewall.
+Directory to store file structure metadata DB.
-Once configured you can then use `rclone` like this,
+The remote name is used as the DB file name.
-List directories in top level of your Box
+Properties:
- rclone lsd remote:
+- Config: db_path
+- Env Var: RCLONE_CACHE_DB_PATH
+- Type: string
+- Default: "$HOME/.cache/rclone/cache-backend"
-List all the files in your Box
+#### --cache-chunk-path
- rclone ls remote:
+Directory to cache chunk files.
-To copy a local directory to an Box directory called backup
+Path to where partial file data (chunks) are stored locally. The remote
+name is appended to the final path.
- rclone copy /home/source remote:backup
+This config follows the "--cache-db-path". If you specify a custom
+location for "--cache-db-path" and don't specify one for "--cache-chunk-path"
+then "--cache-chunk-path" will use the same path as "--cache-db-path".
-### Using rclone with an Enterprise account with SSO
+Properties:
-If you have an "Enterprise" account type with Box with single sign on
-(SSO), you need to create a password to use Box with rclone. This can
-be done at your Enterprise Box account by going to Settings, "Account"
-Tab, and then set the password in the "Authentication" field.
+- Config: chunk_path
+- Env Var: RCLONE_CACHE_CHUNK_PATH
+- Type: string
+- Default: "$HOME/.cache/rclone/cache-backend"
-Once you have done this, you can setup your Enterprise Box account
-using the same procedure detailed above in the, using the password you
-have just set.
+#### --cache-db-purge
-### Invalid refresh token
+Clear all the cached data for this remote on start.
-According to the [box docs](https://developer.box.com/v2.0/docs/oauth-20#section-6-using-the-access-and-refresh-tokens):
+Properties:
-> Each refresh_token is valid for one use in 60 days.
+- Config: db_purge
+- Env Var: RCLONE_CACHE_DB_PURGE
+- Type: bool
+- Default: false
-This means that if you
+#### --cache-chunk-clean-interval
- * Don't use the box remote for 60 days
- * Copy the config file with a box refresh token in and use it in two places
- * Get an error on a token refresh
+How often should the cache perform cleanups of the chunk storage.
-then rclone will return an error which includes the text `Invalid
-refresh token`.
+The default value should be ok for most people. If you find that the
+cache goes over "cache-chunk-total-size" too often then try to lower
+this value to force it to perform cleanups more often.
-To fix this you will need to use oauth2 again to update the refresh
-token. You can use the methods in [the remote setup
-docs](https://rclone.org/remote_setup/), bearing in mind that if you use the copy the
-config file method, you should not use that remote on the computer you
-did the authentication on.
+Properties:
-Here is how to do it.
-
-$ rclone config Current remotes:
-Name Type ==== ==== remote box
-
-- Edit existing remote
-- New remote
-- Delete remote
-- Rename remote
-- Copy remote
-- Set configuration password
-- Quit config e/n/d/r/c/s/q> e Choose a number from below, or type in an existing value 1 > remote remote> remote -------------------- [remote] type = box token = {"access_token":"XXX","token_type":"bearer","refresh_token":"XXX","expiry":"2017-07-08T23:40:08.059167677+01:00"} -------------------- Edit remote Value "client_id" = "" Edit? (y/n)>
-- Yes
-- No y/n> n Value "client_secret" = "" Edit? (y/n)>
-- Yes
-- No y/n> n Remote config Already have a token - refresh?
-- Yes
-- No y/n> y Use web browser to automatically authenticate rclone with remote?
-
-
-- Say Y if the machine running rclone has a web browser you can use
-- Say N if running rclone on a (remote) machine without web browser access If not sure try Y. If Y failed, try N.
-
-
-- Yes
-- No y/n> y If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth Log in and authorize rclone for access Waiting for code... Got code -------------------- [remote] type = box token = {"access_token":"YYY","token_type":"bearer","refresh_token":"YYY","expiry":"2017-07-23T12:22:29.259137901+01:00"} --------------------
-- Yes this is OK
-- Edit this remote
-- Delete this remote y/e/d> y
-
-
-### Modified time and hashes
+- Config: chunk_clean_interval
+- Env Var: RCLONE_CACHE_CHUNK_CLEAN_INTERVAL
+- Type: Duration
+- Default: 1m0s
-Box allows modification times to be set on objects accurate to 1
-second. These will be used to detect whether objects need syncing or
-not.
+#### --cache-read-retries
-Box supports SHA1 type hashes, so you can use the `--checksum`
-flag.
+How many times to retry a read from a cache storage.
-### Restricted filename characters
+Since reading from a cache stream is independent from downloading file
+data, readers can get to a point where there's no more data in the
+cache. Most of the times this can indicate a connectivity issue if
+cache isn't able to provide file data anymore.
-In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters)
-the following characters are also replaced:
+For really slow connections, increase this to a point where the stream is
+able to provide data but your experience will be very stuttering.
-| Character | Value | Replacement |
-| --------- |:-----:|:-----------:|
-| \ | 0x5C | ï¼¼ |
+Properties:
-File names can also not end with the following characters.
-These only get replaced if they are the last character in the name:
+- Config: read_retries
+- Env Var: RCLONE_CACHE_READ_RETRIES
+- Type: int
+- Default: 10
-| Character | Value | Replacement |
-| --------- |:-----:|:-----------:|
-| SP | 0x20 | â |
+#### --cache-workers
-Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8),
-as they can't be used in JSON strings.
+How many workers should run in parallel to download chunks.
-### Transfers
+Higher values will mean more parallel processing (better CPU needed)
+and more concurrent requests on the cloud provider. This impacts
+several aspects like the cloud provider API limits, more stress on the
+hardware that rclone runs on but it also means that streams will be
+more fluid and data will be available much more faster to readers.
-For files above 50 MiB rclone will use a chunked transfer. Rclone will
-upload up to `--transfers` chunks at the same time (shared among all
-the multipart uploads). Chunks are buffered in memory and are
-normally 8 MiB so increasing `--transfers` will increase memory use.
+**Note**: If the optional Plex integration is enabled then this
+setting will adapt to the type of reading performed and the value
+specified here will be used as a maximum number of workers to use.
-### Deleting files
+Properties:
-Depending on the enterprise settings for your user, the item will
-either be actually deleted from Box or moved to the trash.
+- Config: workers
+- Env Var: RCLONE_CACHE_WORKERS
+- Type: int
+- Default: 4
-Emptying the trash is supported via the rclone however cleanup command
-however this deletes every trashed file and folder individually so it
-may take a very long time.
-Emptying the trash via the WebUI does not have this limitation
-so it is advised to empty the trash via the WebUI.
+#### --cache-chunk-no-memory
-### Root folder ID
+Disable the in-memory cache for storing chunks during streaming.
-You can set the `root_folder_id` for rclone. This is the directory
-(identified by its `Folder ID`) that rclone considers to be the root
-of your Box drive.
+By default, cache will keep file data during streaming in RAM as well
+to provide it to readers as fast as possible.
-Normally you will leave this blank and rclone will determine the
-correct root to use itself.
+This transient data is evicted as soon as it is read and the number of
+chunks stored doesn't exceed the number of workers. However, depending
+on other settings like "cache-chunk-size" and "cache-workers" this footprint
+can increase if there are parallel streams too (multiple files being read
+at the same time).
-However you can set this to restrict rclone to a specific folder
-hierarchy.
+If the hardware permits it, use this feature to provide an overall better
+performance during streaming but it can also be disabled if RAM is not
+available on the local machine.
-In order to do this you will have to find the `Folder ID` of the
-directory you wish rclone to display. This will be the last segment
-of the URL when you open the relevant folder in the Box web
-interface.
+Properties:
-So if the folder you want rclone to use has a URL which looks like
-`https://app.box.com/folder/11xxxxxxxxx8`
-in the browser, then you use `11xxxxxxxxx8` as
-the `root_folder_id` in the config.
+- Config: chunk_no_memory
+- Env Var: RCLONE_CACHE_CHUNK_NO_MEMORY
+- Type: bool
+- Default: false
+#### --cache-rps
-### Standard options
+Limits the number of requests per second to the source FS (-1 to disable).
-Here are the Standard options specific to box (Box).
+This setting places a hard limit on the number of requests per second
+that cache will be doing to the cloud provider remote and try to
+respect that value by setting waits between reads.
-#### --box-client-id
+If you find that you're getting banned or limited on the cloud
+provider through cache and know that a smaller number of requests per
+second will allow you to work with it then you can use this setting
+for that.
-OAuth Client Id.
+A good balance of all the other settings should make this setting
+useless but it is available to set for more special cases.
-Leave blank normally.
+**NOTE**: This will limit the number of requests during streams but
+other API calls to the cloud provider like directory listings will
+still pass.
Properties:
-- Config: client_id
-- Env Var: RCLONE_BOX_CLIENT_ID
-- Type: string
-- Required: false
+- Config: rps
+- Env Var: RCLONE_CACHE_RPS
+- Type: int
+- Default: -1
-#### --box-client-secret
+#### --cache-writes
-OAuth Client Secret.
+Cache file data on writes through the FS.
-Leave blank normally.
+If you need to read files immediately after you upload them through
+cache you can enable this flag to have their data stored in the
+cache store at the same time during upload.
Properties:
-- Config: client_secret
-- Env Var: RCLONE_BOX_CLIENT_SECRET
-- Type: string
-- Required: false
+- Config: writes
+- Env Var: RCLONE_CACHE_WRITES
+- Type: bool
+- Default: false
-#### --box-box-config-file
+#### --cache-tmp-upload-path
-Box App config.json location
+Directory to keep temporary files until they are uploaded.
-Leave blank normally.
+This is the path where cache will use as a temporary storage for new
+files that need to be uploaded to the cloud provider.
-Leading `~` will be expanded in the file name as will environment variables such as `${RCLONE_CONFIG_DIR}`.
+Specifying a value will enable this feature. Without it, it is
+completely disabled and files will be uploaded directly to the cloud
+provider
Properties:
-- Config: box_config_file
-- Env Var: RCLONE_BOX_BOX_CONFIG_FILE
+- Config: tmp_upload_path
+- Env Var: RCLONE_CACHE_TMP_UPLOAD_PATH
- Type: string
- Required: false
-#### --box-access-token
+#### --cache-tmp-wait-time
-Box App Primary Access Token
+How long should files be stored in local cache before being uploaded.
-Leave blank normally.
+This is the duration that a file must wait in the temporary location
+_cache-tmp-upload-path_ before it is selected for upload.
-Properties:
+Note that only one file is uploaded at a time and it can take longer
+to start the upload if a queue formed for this purpose.
-- Config: access_token
-- Env Var: RCLONE_BOX_ACCESS_TOKEN
-- Type: string
-- Required: false
+Properties:
-#### --box-box-sub-type
+- Config: tmp_wait_time
+- Env Var: RCLONE_CACHE_TMP_WAIT_TIME
+- Type: Duration
+- Default: 15s
+#### --cache-db-wait-time
+How long to wait for the DB to be available - 0 is unlimited.
-Properties:
+Only one process can have the DB open at any one time, so rclone waits
+for this duration for the DB to become available before it gives an
+error.
-- Config: box_sub_type
-- Env Var: RCLONE_BOX_BOX_SUB_TYPE
-- Type: string
-- Default: "user"
-- Examples:
- - "user"
- - Rclone should act on behalf of a user.
- - "enterprise"
- - Rclone should act on behalf of a service account.
+If you set it to 0 then it will wait forever.
-### Advanced options
+Properties:
-Here are the Advanced options specific to box (Box).
+- Config: db_wait_time
+- Env Var: RCLONE_CACHE_DB_WAIT_TIME
+- Type: Duration
+- Default: 1s
-#### --box-token
+## Backend commands
-OAuth Access Token as a JSON blob.
+Here are the commands specific to the cache backend.
-Properties:
+Run them with
-- Config: token
-- Env Var: RCLONE_BOX_TOKEN
-- Type: string
-- Required: false
+ rclone backend COMMAND remote:
-#### --box-auth-url
+The help below will explain what arguments each command takes.
-Auth server URL.
+See the [backend](https://rclone.org/commands/rclone_backend/) command for more
+info on how to pass options and arguments.
-Leave blank to use the provider defaults.
+These can be run on a running backend using the rc command
+[backend/command](https://rclone.org/rc/#backend-command).
-Properties:
+### stats
-- Config: auth_url
-- Env Var: RCLONE_BOX_AUTH_URL
-- Type: string
-- Required: false
+Print stats on the cache backend in JSON format.
-#### --box-token-url
+ rclone backend stats remote: [options] [<arguments>+]
-Token server url.
-Leave blank to use the provider defaults.
-Properties:
+# Chunker
-- Config: token_url
-- Env Var: RCLONE_BOX_TOKEN_URL
-- Type: string
-- Required: false
+The `chunker` overlay transparently splits large files into smaller chunks
+during upload to wrapped remote and transparently assembles them back
+when the file is downloaded. This allows to effectively overcome size limits
+imposed by storage providers.
-#### --box-root-folder-id
+## Configuration
-Fill in for rclone to use a non root folder as its starting point.
+To use it, first set up the underlying remote following the configuration
+instructions for that remote. You can also use a local pathname instead of
+a remote.
-Properties:
+First check your chosen remote is working - we'll call it `remote:path` here.
+Note that anything inside `remote:path` will be chunked and anything outside
+won't. This means that if you are using a bucket-based remote (e.g. S3, B2, swift)
+then you should probably put the bucket in the remote `s3:bucket`.
-- Config: root_folder_id
-- Env Var: RCLONE_BOX_ROOT_FOLDER_ID
-- Type: string
-- Default: "0"
+Now configure `chunker` using `rclone config`. We will call this one `overlay`
+to separate it from the `remote` itself.
+
+No remotes found, make a new one? n) New remote s) Set configuration password q) Quit config n/s/q> n name> overlay Type of storage to configure. Choose a number from below, or type in your own value [snip] XX / Transparently chunk/split large files  "chunker" [snip] Storage> chunker Remote to chunk/unchunk. Normally should contain a ':' and a path, e.g. "myremote:path/to/dir", "myremote:bucket" or maybe "myremote:" (not recommended). Enter a string value. Press Enter for the default (""). remote> remote:path Files larger than chunk size will be split in chunks. Enter a size with suffix K,M,G,T. Press Enter for the default ("2G"). chunk_size> 100M Choose how chunker handles hash sums. All modes but "none" require metadata. Enter a string value. Press Enter for the default ("md5"). Choose a number from below, or type in your own value 1 / Pass any hash supported by wrapped remote for non-chunked files, return nothing otherwise  "none" 2 / MD5 for composite files  "md5" 3 / SHA1 for composite files  "sha1" 4 / MD5 for all files  "md5all" 5 / SHA1 for all files  "sha1all" 6 / Copying a file to chunker will request MD5 from the source falling back to SHA1 if unsupported  "md5quick" 7 / Similar to "md5quick" but prefers SHA1 over MD5  "sha1quick" hash_type> md5 Edit advanced config? (y/n) y) Yes n) No y/n> n Remote config -------------------- [overlay] type = chunker remote = remote:bucket chunk_size = 100M hash_type = md5 -------------------- y) Yes this is OK e) Edit this remote d) Delete this remote y/e/d> y
+
+### Specifying the remote
-#### --box-upload-cutoff
+In normal use, make sure the remote has a `:` in. If you specify the remote
+without a `:` then rclone will use a local directory of that name.
+So if you use a remote of `/path/to/secret/files` then rclone will
+chunk stuff in that directory. If you use a remote of `name` then rclone
+will put files in a directory called `name` in the current directory.
-Cutoff for switching to multipart upload (>= 50 MiB).
-Properties:
+### Chunking
-- Config: upload_cutoff
-- Env Var: RCLONE_BOX_UPLOAD_CUTOFF
-- Type: SizeSuffix
-- Default: 50Mi
+When rclone starts a file upload, chunker checks the file size. If it
+doesn't exceed the configured chunk size, chunker will just pass the file
+to the wrapped remote (however, see caveat below). If a file is large, chunker will transparently cut
+data in pieces with temporary names and stream them one by one, on the fly.
+Each data chunk will contain the specified number of bytes, except for the
+last one which may have less data. If file size is unknown in advance
+(this is called a streaming upload), chunker will internally create
+a temporary copy, record its size and repeat the above process.
-#### --box-commit-retries
+When upload completes, temporary chunk files are finally renamed.
+This scheme guarantees that operations can be run in parallel and look
+from outside as atomic.
+A similar method with hidden temporary chunks is used for other operations
+(copy/move/rename, etc.). If an operation fails, hidden chunks are normally
+destroyed, and the target composite file stays intact.
-Max number of times to try committing a multipart file.
+When a composite file download is requested, chunker transparently
+assembles it by concatenating data chunks in order. As the split is trivial
+one could even manually concatenate data chunks together to obtain the
+original content.
-Properties:
+When the `list` rclone command scans a directory on wrapped remote,
+the potential chunk files are accounted for, grouped and assembled into
+composite directory entries. Any temporary chunks are hidden.
-- Config: commit_retries
-- Env Var: RCLONE_BOX_COMMIT_RETRIES
-- Type: int
-- Default: 100
+List and other commands can sometimes come across composite files with
+missing or invalid chunks, e.g. shadowed by like-named directory or
+another file. This usually means that wrapped file system has been directly
+tampered with or damaged. If chunker detects a missing chunk it will
+by default print warning, skip the whole incomplete group of chunks but
+proceed with current command.
+You can set the `--chunker-fail-hard` flag to have commands abort with
+error message in such cases.
-#### --box-list-chunk
+**Caveat**: As it is now, chunker will always create a temporary file in the
+backend and then rename it, even if the file is below the chunk threshold.
+This will result in unnecessary API calls and can severely restrict throughput
+when handling transfers primarily composed of small files on some backends (e.g. Box).
+A workaround to this issue is to use chunker only for files above the chunk threshold
+via `--min-size` and then perform a separate call without chunker on the remaining
+files.
-Size of listing chunk 1-1000.
-Properties:
+#### Chunk names
-- Config: list_chunk
-- Env Var: RCLONE_BOX_LIST_CHUNK
-- Type: int
-- Default: 1000
+The default chunk name format is `*.rclone_chunk.###`, hence by default
+chunk names are `BIG_FILE_NAME.rclone_chunk.001`,
+`BIG_FILE_NAME.rclone_chunk.002` etc. You can configure another name format
+using the `name_format` configuration file option. The format uses asterisk
+`*` as a placeholder for the base file name and one or more consecutive
+hash characters `#` as a placeholder for sequential chunk number.
+There must be one and only one asterisk. The number of consecutive hash
+characters defines the minimum length of a string representing a chunk number.
+If decimal chunk number has less digits than the number of hashes, it is
+left-padded by zeros. If the decimal string is longer, it is left intact.
+By default numbering starts from 1 but there is another option that allows
+user to start from 0, e.g. for compatibility with legacy software.
-#### --box-owned-by
+For example, if name format is `big_*-##.part` and original file name is
+`data.txt` and numbering starts from 0, then the first chunk will be named
+`big_data.txt-00.part`, the 99th chunk will be `big_data.txt-98.part`
+and the 302nd chunk will become `big_data.txt-301.part`.
-Only show items owned by the login (email address) passed in.
+Note that `list` assembles composite directory entries only when chunk names
+match the configured format and treats non-conforming file names as normal
+non-chunked files.
-Properties:
+When using `norename` transactions, chunk names will additionally have a unique
+file version suffix. For example, `BIG_FILE_NAME.rclone_chunk.001_bp562k`.
-- Config: owned_by
-- Env Var: RCLONE_BOX_OWNED_BY
-- Type: string
-- Required: false
-#### --box-impersonate
+### Metadata
-Impersonate this user ID when using a service account.
+Besides data chunks chunker will by default create metadata object for
+a composite file. The object is named after the original file.
+Chunker allows user to disable metadata completely (the `none` format).
+Note that metadata is normally not created for files smaller than the
+configured chunk size. This may change in future rclone releases.
-Settng this flag allows rclone, when using a JWT service account, to
-act on behalf of another user by setting the as-user header.
+#### Simple JSON metadata format
-The user ID is the Box identifier for a user. User IDs can found for
-any user via the GET /users endpoint, which is only available to
-admins, or by calling the GET /users/me endpoint with an authenticated
-user session.
+This is the default format. It supports hash sums and chunk validation
+for composite files. Meta objects carry the following fields:
-See: https://developer.box.com/guides/authentication/jwt/as-user/
+- `ver` - version of format, currently `1`
+- `size` - total size of composite file
+- `nchunks` - number of data chunks in file
+- `md5` - MD5 hashsum of composite file (if present)
+- `sha1` - SHA1 hashsum (if present)
+- `txn` - identifies current version of the file
+There is no field for composite file name as it's simply equal to the name
+of meta object on the wrapped remote. Please refer to respective sections
+for details on hashsums and modified time handling.
-Properties:
+#### No metadata
-- Config: impersonate
-- Env Var: RCLONE_BOX_IMPERSONATE
-- Type: string
-- Required: false
+You can disable meta objects by setting the meta format option to `none`.
+In this mode chunker will scan directory for all files that follow
+configured chunk name format, group them by detecting chunks with the same
+base name and show group names as virtual composite files.
+This method is more prone to missing chunk errors (especially missing
+last chunk) than format with metadata enabled.
-#### --box-encoding
-The encoding for the backend.
+### Hashsums
-See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info.
+Chunker supports hashsums only when a compatible metadata is present.
+Hence, if you choose metadata format of `none`, chunker will report hashsum
+as `UNSUPPORTED`.
-Properties:
+Please note that by default metadata is stored only for composite files.
+If a file is smaller than configured chunk size, chunker will transparently
+redirect hash requests to wrapped remote, so support depends on that.
+You will see the empty string as a hashsum of requested type for small
+files if the wrapped remote doesn't support it.
-- Config: encoding
-- Env Var: RCLONE_BOX_ENCODING
-- Type: MultiEncoder
-- Default: Slash,BackSlash,Del,Ctl,RightSpace,InvalidUtf8,Dot
+Many storage backends support MD5 and SHA1 hash types, so does chunker.
+With chunker you can choose one or another but not both.
+MD5 is set by default as the most supported type.
+Since chunker keeps hashes for composite files and falls back to the
+wrapped remote hash for non-chunked ones, we advise you to choose the same
+hash type as supported by wrapped remote so that your file listings
+look coherent.
+If your storage backend does not support MD5 or SHA1 but you need consistent
+file hashing, configure chunker with `md5all` or `sha1all`. These two modes
+guarantee given hash for all files. If wrapped remote doesn't support it,
+chunker will then add metadata to all files, even small. However, this can
+double the amount of small files in storage and incur additional service charges.
+You can even use chunker to force md5/sha1 support in any other remote
+at expense of sidecar meta objects by setting e.g. `hash_type=sha1all`
+to force hashsums and `chunk_size=1P` to effectively disable chunking.
+Normally, when a file is copied to chunker controlled remote, chunker
+will ask the file source for compatible file hash and revert to on-the-fly
+calculation if none is found. This involves some CPU overhead but provides
+a guarantee that given hashsum is available. Also, chunker will reject
+a server-side copy or move operation if source and destination hashsum
+types are different resulting in the extra network bandwidth, too.
+In some rare cases this may be undesired, so chunker provides two optional
+choices: `sha1quick` and `md5quick`. If the source does not support primary
+hash type and the quick mode is enabled, chunker will try to fall back to
+the secondary type. This will save CPU and bandwidth but can result in empty
+hashsums at destination. Beware of consequences: the `sync` command will
+revert (sometimes silently) to time/size comparison if compatible hashsums
+between source and target are not found.
-## Limitations
-Note that Box is case insensitive so you can't have a file called
-"Hello.doc" and one called "hello.doc".
+### Modification times
-Box file names can't have the `\` character in. rclone maps this to
-and from an identical looking unicode equivalent `ï¼¼` (U+FF3C Fullwidth
-Reverse Solidus).
+Chunker stores modification times using the wrapped remote so support
+depends on that. For a small non-chunked file the chunker overlay simply
+manipulates modification time of the wrapped remote file.
+For a composite file with metadata chunker will get and set
+modification time of the metadata object on the wrapped remote.
+If file is chunked but metadata format is `none` then chunker will
+use modification time of the first data chunk.
-Box only supports filenames up to 255 characters in length.
-Box has [API rate limits](https://developer.box.com/guides/api-calls/permissions-and-errors/rate-limits/) that sometimes reduce the speed of rclone.
+### Migrations
-`rclone about` is not supported by the Box backend. Backends without
-this capability cannot determine free space for an rclone mount or
-use policy `mfs` (most free space) as a member of an rclone union
-remote.
+The idiomatic way to migrate to a different chunk size, hash type, transaction
+style or chunk naming scheme is to:
-See [List of backends that do not support rclone about](https://rclone.org/overview/#optional-features) and [rclone about](https://rclone.org/commands/rclone_about/)
+- Collect all your chunked files under a directory and have your
+ chunker remote point to it.
+- Create another directory (most probably on the same cloud storage)
+ and configure a new remote with desired metadata format,
+ hash type, chunk naming etc.
+- Now run `rclone sync --interactive oldchunks: newchunks:` and all your data
+ will be transparently converted in transfer.
+ This may take some time, yet chunker will try server-side
+ copy if possible.
+- After checking data integrity you may remove configuration section
+ of the old remote.
-## Get your own Box App ID
+If rclone gets killed during a long operation on a big composite file,
+hidden temporary chunks may stay in the directory. They will not be
+shown by the `list` command but will eat up your account quota.
+Please note that the `deletefile` command deletes only active
+chunks of a file. As a workaround, you can use remote of the wrapped
+file system to see them.
+An easy way to get rid of hidden garbage is to copy littered directory
+somewhere using the chunker remote and purge the original directory.
+The `copy` command will copy only active chunks while the `purge` will
+remove everything including garbage.
-Here is how to create your own Box App ID for rclone:
-1. Go to the [Box Developer Console](https://app.box.com/developers/console)
-and login, then click `My Apps` on the sidebar. Click `Create New App`
-and select `Custom App`.
+### Caveats and Limitations
-2. In the first screen on the box that pops up, you can pretty much enter
-whatever you want. The `App Name` can be whatever. For `Purpose` choose
-automation to avoid having to fill out anything else. Click `Next`.
+Chunker requires wrapped remote to support server-side `move` (or `copy` +
+`delete`) operations, otherwise it will explicitly refuse to start.
+This is because it internally renames temporary chunk files to their final
+names when an operation completes successfully.
-3. In the second screen of the creation screen, select
-`User Authentication (OAuth 2.0)`. Then click `Create App`.
+Chunker encodes chunk number in file name, so with default `name_format`
+setting it adds 17 characters. Also chunker adds 7 characters of temporary
+suffix during operations. Many file systems limit base file name without path
+by 255 characters. Using rclone's crypt remote as a base file system limits
+file name by 143 characters. Thus, maximum name length is 231 for most files
+and 119 for chunker-over-crypt. A user in need can change name format to
+e.g. `*.rcc##` and save 10 characters (provided at most 99 chunks per file).
-4. You should now be on the `Configuration` tab of your new app. If not,
-click on it at the top of the webpage. Copy down `Client ID`
-and `Client Secret`, you'll need those for rclone.
+Note that a move implemented using the copy-and-delete method may incur
+double charging with some cloud storage providers.
-5. Under "OAuth 2.0 Redirect URI", add `http://127.0.0.1:53682/`
+Chunker will not automatically rename existing chunks when you run
+`rclone config` on a live remote and change the chunk name format.
+Beware that in result of this some files which have been treated as chunks
+before the change can pop up in directory listings as normal files
+and vice versa. The same warning holds for the chunk size.
+If you desperately need to change critical chunking settings, you should
+run data migration as described above.
-6. For `Application Scopes`, select `Read all files and folders stored in Box`
-and `Write all files and folders stored in box` (assuming you want to do both).
-Leave others unchecked. Click `Save Changes` at the top right.
+If wrapped remote is case insensitive, the chunker overlay will inherit
+that property (so you can't have a file called "Hello.doc" and "hello.doc"
+in the same directory).
-# Cache
+Chunker included in rclone releases up to `v1.54` can sometimes fail to
+detect metadata produced by recent versions of rclone. We recommend users
+to keep rclone up-to-date to avoid data corruption.
-The `cache` remote wraps another existing remote and stores file structure
-and its data for long running tasks like `rclone mount`.
+Changing `transactions` is dangerous and requires explicit migration.
-## Status
-The cache backend code is working but it currently doesn't
-have a maintainer so there are [outstanding bugs](https://github.com/rclone/rclone/issues?q=is%3Aopen+is%3Aissue+label%3Abug+label%3A%22Remote%3A+Cache%22) which aren't getting fixed.
+### Standard options
-The cache backend is due to be phased out in favour of the VFS caching
-layer eventually which is more tightly integrated into rclone.
+Here are the Standard options specific to chunker (Transparently chunk/split large files).
-Until this happens we recommend only using the cache backend if you
-find you can't work without it. There are many docs online describing
-the use of the cache backend to minimize API hits and by-and-large
-these are out of date and the cache backend isn't needed in those
-scenarios any more.
+#### --chunker-remote
-## Configuration
+Remote to chunk/unchunk.
-To get started you just need to have an existing remote which can be configured
-with `cache`.
+Normally should contain a ':' and a path, e.g. "myremote:path/to/dir",
+"myremote:bucket" or maybe "myremote:" (not recommended).
-Here is an example of how to make a remote called `test-cache`. First run:
+Properties:
- rclone config
+- Config: remote
+- Env Var: RCLONE_CHUNKER_REMOTE
+- Type: string
+- Required: true
-This will guide you through an interactive setup process:
-
-No remotes found, make a new one? n) New remote r) Rename remote c) Copy remote s) Set configuration password q) Quit config n/r/c/s/q> n name> test-cache Type of storage to configure. Choose a number from below, or type in your own value [snip] XX / Cache a remote  "cache" [snip] Storage> cache Remote to cache. Normally should contain a ':' and a path, e.g. "myremote:path/to/dir", "myremote:bucket" or maybe "myremote:" (not recommended). remote> local:/test Optional: The URL of the Plex server plex_url> http://127.0.0.1:32400 Optional: The username of the Plex user plex_username> dummyusername Optional: The password of the Plex user y) Yes type in my own password g) Generate random password n) No leave this optional password blank y/g/n> y Enter the password: password: Confirm the password: password: The size of a chunk. Lower value good for slow connections but can affect seamless reading. Default: 5M Choose a number from below, or type in your own value 1 / 1 MiB  "1M" 2 / 5 MiB  "5M" 3 / 10 MiB  "10M" chunk_size> 2 How much time should object info (file size, file hashes, etc.) be stored in cache. Use a very high value if you don't plan on changing the source FS from outside the cache. Accepted units are: "s", "m", "h". Default: 5m Choose a number from below, or type in your own value 1 / 1 hour  "1h" 2 / 24 hours  "24h" 3 / 24 hours  "48h" info_age> 2 The maximum size of stored chunks. When the storage grows beyond this size, the oldest chunks will be deleted. Default: 10G Choose a number from below, or type in your own value 1 / 500 MiB  "500M" 2 / 1 GiB  "1G" 3 / 10 GiB  "10G" chunk_total_size> 3 Remote config -------------------- [test-cache] remote = local:/test plex_url = http://127.0.0.1:32400 plex_username = dummyusername plex_password = *** ENCRYPTED *** chunk_size = 5M info_age = 48h chunk_total_size = 10G
-
-You can then use it like this,
+#### --chunker-chunk-size
-List directories in top level of your drive
+Files larger than chunk size will be split in chunks.
- rclone lsd test-cache:
+Properties:
-List all the files in your drive
+- Config: chunk_size
+- Env Var: RCLONE_CHUNKER_CHUNK_SIZE
+- Type: SizeSuffix
+- Default: 2Gi
- rclone ls test-cache:
+#### --chunker-hash-type
-To start a cached mount
+Choose how chunker handles hash sums.
- rclone mount --allow-other test-cache: /var/tmp/test-cache
+All modes but "none" require metadata.
-### Write Features ###
+Properties:
-### Offline uploading ###
+- Config: hash_type
+- Env Var: RCLONE_CHUNKER_HASH_TYPE
+- Type: string
+- Default: "md5"
+- Examples:
+ - "none"
+ - Pass any hash supported by wrapped remote for non-chunked files.
+ - Return nothing otherwise.
+ - "md5"
+ - MD5 for composite files.
+ - "sha1"
+ - SHA1 for composite files.
+ - "md5all"
+ - MD5 for all files.
+ - "sha1all"
+ - SHA1 for all files.
+ - "md5quick"
+ - Copying a file to chunker will request MD5 from the source.
+ - Falling back to SHA1 if unsupported.
+ - "sha1quick"
+ - Similar to "md5quick" but prefers SHA1 over MD5.
-In an effort to make writing through cache more reliable, the backend
-now supports this feature which can be activated by specifying a
-`cache-tmp-upload-path`.
+### Advanced options
-A files goes through these states when using this feature:
+Here are the Advanced options specific to chunker (Transparently chunk/split large files).
-1. An upload is started (usually by copying a file on the cache remote)
-2. When the copy to the temporary location is complete the file is part
-of the cached remote and looks and behaves like any other file (reading included)
-3. After `cache-tmp-wait-time` passes and the file is next in line, `rclone move`
-is used to move the file to the cloud provider
-4. Reading the file still works during the upload but most modifications on it will be prohibited
-5. Once the move is complete the file is unlocked for modifications as it
-becomes as any other regular file
-6. If the file is being read through `cache` when it's actually
-deleted from the temporary path then `cache` will simply swap the source
-to the cloud provider without interrupting the reading (small blip can happen though)
+#### --chunker-name-format
-Files are uploaded in sequence and only one file is uploaded at a time.
-Uploads will be stored in a queue and be processed based on the order they were added.
-The queue and the temporary storage is persistent across restarts but
-can be cleared on startup with the `--cache-db-purge` flag.
+String format of chunk file names.
-### Write Support ###
+The two placeholders are: base file name (*) and chunk number (#...).
+There must be one and only one asterisk and one or more consecutive hash characters.
+If chunk number has less digits than the number of hashes, it is left-padded by zeros.
+If there are more digits in the number, they are left as is.
+Possible chunk files are ignored if their name does not match given format.
-Writes are supported through `cache`.
-One caveat is that a mounted cache remote does not add any retry or fallback
-mechanism to the upload operation. This will depend on the implementation
-of the wrapped remote. Consider using `Offline uploading` for reliable writes.
+Properties:
-One special case is covered with `cache-writes` which will cache the file
-data at the same time as the upload when it is enabled making it available
-from the cache store immediately once the upload is finished.
+- Config: name_format
+- Env Var: RCLONE_CHUNKER_NAME_FORMAT
+- Type: string
+- Default: "*.rclone_chunk.###"
-### Read Features ###
+#### --chunker-start-from
-#### Multiple connections ####
+Minimum valid chunk number. Usually 0 or 1.
-To counter the high latency between a local PC where rclone is running
-and cloud providers, the cache remote can split multiple requests to the
-cloud provider for smaller file chunks and combines them together locally
-where they can be available almost immediately before the reader usually
-needs them.
+By default chunk numbers start from 1.
-This is similar to buffering when media files are played online. Rclone
-will stay around the current marker but always try its best to stay ahead
-and prepare the data before.
+Properties:
-#### Plex Integration ####
+- Config: start_from
+- Env Var: RCLONE_CHUNKER_START_FROM
+- Type: int
+- Default: 1
-There is a direct integration with Plex which allows cache to detect during reading
-if the file is in playback or not. This helps cache to adapt how it queries
-the cloud provider depending on what is needed for.
+#### --chunker-meta-format
-Scans will have a minimum amount of workers (1) while in a confirmed playback cache
-will deploy the configured number of workers.
+Format of the metadata object or "none".
-This integration opens the doorway to additional performance improvements
-which will be explored in the near future.
+By default "simplejson".
+Metadata is a small JSON file named after the composite file.
-**Note:** If Plex options are not configured, `cache` will function with its
-configured options without adapting any of its settings.
+Properties:
-How to enable? Run `rclone config` and add all the Plex options (endpoint, username
-and password) in your remote and it will be automatically enabled.
+- Config: meta_format
+- Env Var: RCLONE_CHUNKER_META_FORMAT
+- Type: string
+- Default: "simplejson"
+- Examples:
+ - "none"
+ - Do not use metadata files at all.
+ - Requires hash type "none".
+ - "simplejson"
+ - Simple JSON supports hash sums and chunk validation.
+ -
+ - It has the following fields: ver, size, nchunks, md5, sha1.
-Affected settings:
-- `cache-workers`: _Configured value_ during confirmed playback or _1_ all the other times
+#### --chunker-fail-hard
-##### Certificate Validation #####
+Choose how chunker should handle files with missing or invalid chunks.
-When the Plex server is configured to only accept secure connections, it is
-possible to use `.plex.direct` URLs to ensure certificate validation succeeds.
-These URLs are used by Plex internally to connect to the Plex server securely.
+Properties:
-The format for these URLs is the following:
+- Config: fail_hard
+- Env Var: RCLONE_CHUNKER_FAIL_HARD
+- Type: bool
+- Default: false
+- Examples:
+ - "true"
+ - Report errors and abort current command.
+ - "false"
+ - Warn user, skip incomplete file and proceed.
-`https://ip-with-dots-replaced.server-hash.plex.direct:32400/`
+#### --chunker-transactions
-The `ip-with-dots-replaced` part can be any IPv4 address, where the dots
-have been replaced with dashes, e.g. `127.0.0.1` becomes `127-0-0-1`.
+Choose how chunker should handle temporary files during transactions.
-To get the `server-hash` part, the easiest way is to visit
+Properties:
-https://plex.tv/api/resources?includeHttps=1&X-Plex-Token=your-plex-token
+- Config: transactions
+- Env Var: RCLONE_CHUNKER_TRANSACTIONS
+- Type: string
+- Default: "rename"
+- Examples:
+ - "rename"
+ - Rename temporary files after a successful transaction.
+ - "norename"
+ - Leave temporary file names and write transaction ID to metadata file.
+ - Metadata is required for no rename transactions (meta format cannot be "none").
+ - If you are using norename transactions you should be careful not to downgrade Rclone
+ - as older versions of Rclone don't support this transaction style and will misinterpret
+ - files manipulated by norename transactions.
+ - This method is EXPERIMENTAL, don't use on production systems.
+ - "auto"
+ - Rename or norename will be used depending on capabilities of the backend.
+ - If meta format is set to "none", rename transactions will always be used.
+ - This method is EXPERIMENTAL, don't use on production systems.
-This page will list all the available Plex servers for your account
-with at least one `.plex.direct` link for each. Copy one URL and replace
-the IP address with the desired address. This can be used as the
-`plex_url` value.
-### Known issues ###
-#### Mount and --dir-cache-time ####
+# Citrix ShareFile
---dir-cache-time controls the first layer of directory caching which works at the mount layer.
-Being an independent caching mechanism from the `cache` backend, it will manage its own entries
-based on the configured time.
+[Citrix ShareFile](https://sharefile.com) is a secure file sharing and transfer service aimed as business.
-To avoid getting in a scenario where dir cache has obsolete data and cache would have the correct
-one, try to set `--dir-cache-time` to a lower time than `--cache-info-age`. Default values are
-already configured in this way.
+## Configuration
-#### Windows support - Experimental ####
+The initial setup for Citrix ShareFile involves getting a token from
+Citrix ShareFile which you can in your browser. `rclone config` walks you
+through it.
-There are a couple of issues with Windows `mount` functionality that still require some investigations.
-It should be considered as experimental thus far as fixes come in for this OS.
+Here is an example of how to make a remote called `remote`. First run:
-Most of the issues seem to be related to the difference between filesystems
-on Linux flavors and Windows as cache is heavily dependent on them.
+ rclone config
-Any reports or feedback on how cache behaves on this OS is greatly appreciated.
-
-- https://github.com/rclone/rclone/issues/1935
-- https://github.com/rclone/rclone/issues/1907
-- https://github.com/rclone/rclone/issues/1834
+This will guide you through an interactive setup process:
+
+No remotes found, make a new one? n) New remote s) Set configuration password q) Quit config n/s/q> n name> remote Type of storage to configure. Enter a string value. Press Enter for the default (""). Choose a number from below, or type in your own value XX / Citrix Sharefile  "sharefile" Storage> sharefile ** See help for sharefile backend at: https://rclone.org/sharefile/ **
+ID of the root folder
+Leave blank to access "Personal Folders". You can use one of the standard values here or any folder ID (long hex number ID). Enter a string value. Press Enter for the default (""). Choose a number from below, or type in your own value 1 / Access the Personal Folders. (Default) Â "" 2 / Access the Favorites folder. Â "favorites" 3 / Access all the shared folders. Â "allshared" 4 / Access all the individual connectors. Â "connectors" 5 / Access the home, favorites, and shared folders as well as the connectors. Â "top" root_folder_id> Edit advanced config? (y/n) y) Yes n) No y/n> n Remote config Use web browser to automatically authenticate rclone with remote? * Say Y if the machine running rclone has a web browser you can use * Say N if running rclone on a (remote) machine without web browser access If not sure try Y. If Y failed, try N. y) Yes n) No y/n> y If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth?state=XXX Log in and authorize rclone for access Waiting for code... Got code -------------------- [remote] type = sharefile endpoint = https://XXX.sharefile.com token = {"access_token":"XXX","token_type":"bearer","refresh_token":"XXX","expiry":"2019-09-30T19:41:45.878561877+01:00"} -------------------- y) Yes this is OK e) Edit this remote d) Delete this remote y/e/d> y
+
+See the [remote setup docs](https://rclone.org/remote_setup/) for how to set it up on a
+machine with no Internet browser available.
-#### Risk of throttling ####
+Note that rclone runs a webserver on your local machine to collect the
+token as returned from Citrix ShareFile. This only runs from the moment it opens
+your browser to the moment you get back the verification code. This
+is on `http://127.0.0.1:53682/` and this it may require you to unblock
+it temporarily if you are running a host firewall.
-Future iterations of the cache backend will make use of the pooling functionality
-of the cloud provider to synchronize and at the same time make writing through it
-more tolerant to failures.
+Once configured you can then use `rclone` like this,
-There are a couple of enhancements in track to add these but in the meantime
-there is a valid concern that the expiring cache listings can lead to cloud provider
-throttles or bans due to repeated queries on it for very large mounts.
+List directories in top level of your ShareFile
-Some recommendations:
-- don't use a very small interval for entry information (`--cache-info-age`)
-- while writes aren't yet optimised, you can still write through `cache` which gives you the advantage
-of adding the file in the cache at the same time if configured to do so.
+ rclone lsd remote:
-Future enhancements:
+List all the files in your ShareFile
-- https://github.com/rclone/rclone/issues/1937
-- https://github.com/rclone/rclone/issues/1936
+ rclone ls remote:
-#### cache and crypt ####
+To copy a local directory to an ShareFile directory called backup
-One common scenario is to keep your data encrypted in the cloud provider
-using the `crypt` remote. `crypt` uses a similar technique to wrap around
-an existing remote and handles this translation in a seamless way.
+ rclone copy /home/source remote:backup
-There is an issue with wrapping the remotes in this order:
-**cloud remote** -> **crypt** -> **cache**
+Paths may be as deep as required, e.g. `remote:directory/subdirectory`.
-During testing, I experienced a lot of bans with the remotes in this order.
-I suspect it might be related to how crypt opens files on the cloud provider
-which makes it think we're downloading the full file instead of small chunks.
-Organizing the remotes in this order yields better results:
-**cloud remote** -> **cache** -> **crypt**
+### Modification times and hashes
-#### absolute remote paths ####
+ShareFile allows modification times to be set on objects accurate to 1
+second. These will be used to detect whether objects need syncing or
+not.
-`cache` can not differentiate between relative and absolute paths for the wrapped remote.
-Any path given in the `remote` config setting and on the command line will be passed to
-the wrapped remote as is, but for storing the chunks on disk the path will be made
-relative by removing any leading `/` character.
+ShareFile supports MD5 type hashes, so you can use the `--checksum`
+flag.
-This behavior is irrelevant for most backend types, but there are backends where a leading `/`
-changes the effective directory, e.g. in the `sftp` backend paths starting with a `/` are
-relative to the root of the SSH server and paths without are relative to the user home directory.
-As a result `sftp:bin` and `sftp:/bin` will share the same cache folder, even if they represent
-a different directory on the SSH server.
+### Transfers
-### Cache and Remote Control (--rc) ###
-Cache supports the new `--rc` mode in rclone and can be remote controlled through the following end points:
-By default, the listener is disabled if you do not add the flag.
+For files above 128 MiB rclone will use a chunked transfer. Rclone will
+upload up to `--transfers` chunks at the same time (shared among all
+the multipart uploads). Chunks are buffered in memory and are
+normally 64 MiB so increasing `--transfers` will increase memory use.
-### rc cache/expire
-Purge a remote from the cache backend. Supports either a directory or a file.
-It supports both encrypted and unencrypted file names if cache is wrapped by crypt.
+### Restricted filename characters
-Params:
- - **remote** = path to remote **(required)**
- - **withData** = true/false to delete cached data (chunks) as well _(optional, false by default)_
+In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters)
+the following characters are also replaced:
+| Character | Value | Replacement |
+| --------- |:-----:|:-----------:|
+| \\ | 0x5C | ï¼¼ |
+| * | 0x2A | ＊ |
+| < | 0x3C | ＜ |
+| > | 0x3E | ＞ |
+| ? | 0x3F | ？ |
+| : | 0x3A | ： |
+| \| | 0x7C | ｜ |
+| " | 0x22 | ＂ |
-### Standard options
+File names can also not start or end with the following characters.
+These only get replaced if they are the first or last character in the
+name:
-Here are the Standard options specific to cache (Cache a remote).
+| Character | Value | Replacement |
+| --------- |:-----:|:-----------:|
+| SP | 0x20 | â |
+| . | 0x2E | ． |
-#### --cache-remote
+Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8),
+as they can't be used in JSON strings.
-Remote to cache.
-Normally should contain a ':' and a path, e.g. "myremote:path/to/dir",
-"myremote:bucket" or maybe "myremote:" (not recommended).
+### Standard options
-Properties:
+Here are the Standard options specific to sharefile (Citrix Sharefile).
-- Config: remote
-- Env Var: RCLONE_CACHE_REMOTE
-- Type: string
-- Required: true
+#### --sharefile-client-id
-#### --cache-plex-url
+OAuth Client Id.
-The URL of the Plex server.
+Leave blank normally.
Properties:
-- Config: plex_url
-- Env Var: RCLONE_CACHE_PLEX_URL
+- Config: client_id
+- Env Var: RCLONE_SHAREFILE_CLIENT_ID
- Type: string
- Required: false
-#### --cache-plex-username
+#### --sharefile-client-secret
-The username of the Plex user.
+OAuth Client Secret.
+
+Leave blank normally.
Properties:
-- Config: plex_username
-- Env Var: RCLONE_CACHE_PLEX_USERNAME
+- Config: client_secret
+- Env Var: RCLONE_SHAREFILE_CLIENT_SECRET
- Type: string
- Required: false
-#### --cache-plex-password
+#### --sharefile-root-folder-id
-The password of the Plex user.
+ID of the root folder.
-**NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/).
+Leave blank to access "Personal Folders". You can use one of the
+standard values here or any folder ID (long hex number ID).
Properties:
-- Config: plex_password
-- Env Var: RCLONE_CACHE_PLEX_PASSWORD
+- Config: root_folder_id
+- Env Var: RCLONE_SHAREFILE_ROOT_FOLDER_ID
- Type: string
- Required: false
+- Examples:
+ - ""
+ - Access the Personal Folders (default).
+ - "favorites"
+ - Access the Favorites folder.
+ - "allshared"
+ - Access all the shared folders.
+ - "connectors"
+ - Access all the individual connectors.
+ - "top"
+ - Access the home, favorites, and shared folders as well as the connectors.
-#### --cache-chunk-size
-
-The size of a chunk (partial file data).
-
-Use lower numbers for slower connections. If the chunk size is
-changed, any downloaded chunks will be invalid and cache-chunk-path
-will need to be cleared or unexpected EOF errors will occur.
-
-Properties:
+### Advanced options
-- Config: chunk_size
-- Env Var: RCLONE_CACHE_CHUNK_SIZE
-- Type: SizeSuffix
-- Default: 5Mi
-- Examples:
- - "1M"
- - 1 MiB
- - "5M"
- - 5 MiB
- - "10M"
- - 10 MiB
+Here are the Advanced options specific to sharefile (Citrix Sharefile).
-#### --cache-info-age
+#### --sharefile-token
-How long to cache file structure information (directory listings, file size, times, etc.).
-If all write operations are done through the cache then you can safely make
-this value very large as the cache store will also be updated in real time.
+OAuth Access Token as a JSON blob.
Properties:
-- Config: info_age
-- Env Var: RCLONE_CACHE_INFO_AGE
-- Type: Duration
-- Default: 6h0m0s
-- Examples:
- - "1h"
- - 1 hour
- - "24h"
- - 24 hours
- - "48h"
- - 48 hours
+- Config: token
+- Env Var: RCLONE_SHAREFILE_TOKEN
+- Type: string
+- Required: false
-#### --cache-chunk-total-size
+#### --sharefile-auth-url
-The total size that the chunks can take up on the local disk.
+Auth server URL.
-If the cache exceeds this value then it will start to delete the
-oldest chunks until it goes under this value.
+Leave blank to use the provider defaults.
Properties:
-- Config: chunk_total_size
-- Env Var: RCLONE_CACHE_CHUNK_TOTAL_SIZE
-- Type: SizeSuffix
-- Default: 10Gi
-- Examples:
- - "500M"
- - 500 MiB
- - "1G"
- - 1 GiB
- - "10G"
- - 10 GiB
-
-### Advanced options
+- Config: auth_url
+- Env Var: RCLONE_SHAREFILE_AUTH_URL
+- Type: string
+- Required: false
-Here are the Advanced options specific to cache (Cache a remote).
+#### --sharefile-token-url
-#### --cache-plex-token
+Token server url.
-The plex token for authentication - auto set normally.
+Leave blank to use the provider defaults.
Properties:
-- Config: plex_token
-- Env Var: RCLONE_CACHE_PLEX_TOKEN
+- Config: token_url
+- Env Var: RCLONE_SHAREFILE_TOKEN_URL
- Type: string
- Required: false
-#### --cache-plex-insecure
+#### --sharefile-upload-cutoff
-Skip all certificate verification when connecting to the Plex server.
+Cutoff for switching to multipart upload.
Properties:
-- Config: plex_insecure
-- Env Var: RCLONE_CACHE_PLEX_INSECURE
-- Type: string
-- Required: false
+- Config: upload_cutoff
+- Env Var: RCLONE_SHAREFILE_UPLOAD_CUTOFF
+- Type: SizeSuffix
+- Default: 128Mi
-#### --cache-db-path
+#### --sharefile-chunk-size
-Directory to store file structure metadata DB.
+Upload chunk size.
-The remote name is used as the DB file name.
+Must a power of 2 >= 256k.
+
+Making this larger will improve performance, but note that each chunk
+is buffered in memory one per transfer.
+
+Reducing this will reduce memory usage but decrease performance.
Properties:
-- Config: db_path
-- Env Var: RCLONE_CACHE_DB_PATH
-- Type: string
-- Default: "$HOME/.cache/rclone/cache-backend"
+- Config: chunk_size
+- Env Var: RCLONE_SHAREFILE_CHUNK_SIZE
+- Type: SizeSuffix
+- Default: 64Mi
-#### --cache-chunk-path
+#### --sharefile-endpoint
-Directory to cache chunk files.
+Endpoint for API calls.
-Path to where partial file data (chunks) are stored locally. The remote
-name is appended to the final path.
+This is usually auto discovered as part of the oauth process, but can
+be set manually to something like: https://XXX.sharefile.com
-This config follows the "--cache-db-path". If you specify a custom
-location for "--cache-db-path" and don't specify one for "--cache-chunk-path"
-then "--cache-chunk-path" will use the same path as "--cache-db-path".
Properties:
-- Config: chunk_path
-- Env Var: RCLONE_CACHE_CHUNK_PATH
+- Config: endpoint
+- Env Var: RCLONE_SHAREFILE_ENDPOINT
- Type: string
-- Default: "$HOME/.cache/rclone/cache-backend"
+- Required: false
-#### --cache-db-purge
+#### --sharefile-encoding
-Clear all the cached data for this remote on start.
+The encoding for the backend.
+
+See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info.
Properties:
-- Config: db_purge
-- Env Var: RCLONE_CACHE_DB_PURGE
-- Type: bool
-- Default: false
+- Config: encoding
+- Env Var: RCLONE_SHAREFILE_ENCODING
+- Type: Encoding
+- Default: Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,LeftSpace,LeftPeriod,RightSpace,RightPeriod,InvalidUtf8,Dot
-#### --cache-chunk-clean-interval
-How often should the cache perform cleanups of the chunk storage.
+## Limitations
-The default value should be ok for most people. If you find that the
-cache goes over "cache-chunk-total-size" too often then try to lower
-this value to force it to perform cleanups more often.
+Note that ShareFile is case insensitive so you can't have a file called
+"Hello.doc" and one called "hello.doc".
-Properties:
+ShareFile only supports filenames up to 256 characters in length.
-- Config: chunk_clean_interval
-- Env Var: RCLONE_CACHE_CHUNK_CLEAN_INTERVAL
-- Type: Duration
-- Default: 1m0s
+`rclone about` is not supported by the Citrix ShareFile backend. Backends without
+this capability cannot determine free space for an rclone mount or
+use policy `mfs` (most free space) as a member of an rclone union
+remote.
-#### --cache-read-retries
+See [List of backends that do not support rclone about](https://rclone.org/overview/#optional-features) and [rclone about](https://rclone.org/commands/rclone_about/)
-How many times to retry a read from a cache storage.
+# Crypt
-Since reading from a cache stream is independent from downloading file
-data, readers can get to a point where there's no more data in the
-cache. Most of the times this can indicate a connectivity issue if
-cache isn't able to provide file data anymore.
+Rclone `crypt` remotes encrypt and decrypt other remotes.
-For really slow connections, increase this to a point where the stream is
-able to provide data but your experience will be very stuttering.
+A remote of type `crypt` does not access a [storage system](https://rclone.org/overview/)
+directly, but instead wraps another remote, which in turn accesses
+the storage system. This is similar to how [alias](https://rclone.org/alias/),
+[union](https://rclone.org/union/), [chunker](https://rclone.org/chunker/)
+and a few others work. It makes the usage very flexible, as you can
+add a layer, in this case an encryption layer, on top of any other
+backend, even in multiple layers. Rclone's functionality
+can be used as with any other remote, for example you can
+[mount](https://rclone.org/commands/rclone_mount/) a crypt remote.
-Properties:
+Accessing a storage system through a crypt remote realizes client-side
+encryption, which makes it safe to keep your data in a location you do
+not trust will not get compromised.
+When working against the `crypt` remote, rclone will automatically
+encrypt (before uploading) and decrypt (after downloading) on your local
+system as needed on the fly, leaving the data encrypted at rest in the
+wrapped remote. If you access the storage system using an application
+other than rclone, or access the wrapped remote directly using rclone,
+there will not be any encryption/decryption: Downloading existing content
+will just give you the encrypted (scrambled) format, and anything you
+upload will *not* become encrypted.
-- Config: read_retries
-- Env Var: RCLONE_CACHE_READ_RETRIES
-- Type: int
-- Default: 10
+The encryption is a secret-key encryption (also called symmetric key encryption)
+algorithm, where a password (or pass phrase) is used to generate real encryption key.
+The password can be supplied by user, or you may chose to let rclone
+generate one. It will be stored in the configuration file, in a lightly obscured form.
+If you are in an environment where you are not able to keep your configuration
+secured, you should add
+[configuration encryption](https://rclone.org/docs/#configuration-encryption)
+as protection. As long as you have this configuration file, you will be able to
+decrypt your data. Without the configuration file, as long as you remember
+the password (or keep it in a safe place), you can re-create the configuration
+and gain access to the existing data. You may also configure a corresponding
+remote in a different installation to access the same data.
+See below for guidance to [changing password](#changing-password).
-#### --cache-workers
+Encryption uses [cryptographic salt](https://en.wikipedia.org/wiki/Salt_(cryptography)),
+to permute the encryption key so that the same string may be encrypted in
+different ways. When configuring the crypt remote it is optional to enter a salt,
+or to let rclone generate a unique salt. If omitted, rclone uses a built-in unique string.
+Normally in cryptography, the salt is stored together with the encrypted content,
+and do not have to be memorized by the user. This is not the case in rclone,
+because rclone does not store any additional information on the remotes. Use of
+custom salt is effectively a second password that must be memorized.
-How many workers should run in parallel to download chunks.
+[File content](#file-encryption) encryption is performed using
+[NaCl SecretBox](https://godoc.org/golang.org/x/crypto/nacl/secretbox),
+based on XSalsa20 cipher and Poly1305 for integrity.
+[Names](#name-encryption) (file- and directory names) are also encrypted
+by default, but this has some implications and is therefore
+possible to be turned off.
-Higher values will mean more parallel processing (better CPU needed)
-and more concurrent requests on the cloud provider. This impacts
-several aspects like the cloud provider API limits, more stress on the
-hardware that rclone runs on but it also means that streams will be
-more fluid and data will be available much more faster to readers.
+## Configuration
-**Note**: If the optional Plex integration is enabled then this
-setting will adapt to the type of reading performed and the value
-specified here will be used as a maximum number of workers to use.
+Here is an example of how to make a remote called `secret`.
-Properties:
+To use `crypt`, first set up the underlying remote. Follow the
+`rclone config` instructions for the specific backend.
-- Config: workers
-- Env Var: RCLONE_CACHE_WORKERS
-- Type: int
-- Default: 4
+Before configuring the crypt remote, check the underlying remote is
+working. In this example the underlying remote is called `remote`.
+We will configure a path `path` within this remote to contain the
+encrypted content. Anything inside `remote:path` will be encrypted
+and anything outside will not.
-#### --cache-chunk-no-memory
+Configure `crypt` using `rclone config`. In this example the `crypt`
+remote is called `secret`, to differentiate it from the underlying
+`remote`.
-Disable the in-memory cache for storing chunks during streaming.
+When you are done you can use the crypt remote named `secret` just
+as you would with any other remote, e.g. `rclone copy D:\docs secret:\docs`,
+and rclone will encrypt and decrypt as needed on the fly.
+If you access the wrapped remote `remote:path` directly you will bypass
+the encryption, and anything you read will be in encrypted form, and
+anything you write will be unencrypted. To avoid issues it is best to
+configure a dedicated path for encrypted content, and access it
+exclusively through a crypt remote.
+
+No remotes found, make a new one? n) New remote s) Set configuration password q) Quit config n/s/q> n name> secret Type of storage to configure. Enter a string value. Press Enter for the default (""). Choose a number from below, or type in your own value [snip] XX / Encrypt/Decrypt a remote  "crypt" [snip] Storage> crypt ** See help for crypt backend at: https://rclone.org/crypt/ **
+Remote to encrypt/decrypt. Normally should contain a ':' and a path, eg "myremote:path/to/dir", "myremote:bucket" or maybe "myremote:" (not recommended). Enter a string value. Press Enter for the default (""). remote> remote:path How to encrypt the filenames. Enter a string value. Press Enter for the default ("standard"). Choose a number from below, or type in your own value. / Encrypt the filenames. 1 | See the docs for the details. Â "standard" 2 / Very simple filename obfuscation. Â "obfuscate" / Don't encrypt the file names. 3 | Adds a ".bin" extension only. Â "off" filename_encryption> Option to either encrypt directory names or leave them intact.
+NB If filename_encryption is "off" then this option will do nothing. Enter a boolean value (true or false). Press Enter for the default ("true"). Choose a number from below, or type in your own value 1 / Encrypt directory names. Â "true" 2 / Don't encrypt directory names, leave them intact. Â "false" directory_name_encryption> Password or pass phrase for encryption. y) Yes type in my own password g) Generate random password y/g> y Enter the password: password: Confirm the password: password: Password or pass phrase for salt. Optional but recommended. Should be different to the previous password. y) Yes type in my own password g) Generate random password n) No leave this optional password blank (default) y/g/n> g Password strength in bits. 64 is just about memorable 128 is secure 1024 is the maximum Bits> 128 Your password is: JAsJvRcgR-_veXNfy_sGmQ Use this password? Please note that an obscured version of this password (and not the password itself) will be stored under your configuration file, so keep this generated password in a safe place. y) Yes (default) n) No y/n> Edit advanced config? (y/n) y) Yes n) No (default) y/n> Remote config -------------------- [secret] type = crypt remote = remote:path password = *** ENCRYPTED password2 = ENCRYPTED *** -------------------- y) Yes this is OK (default) e) Edit this remote d) Delete this remote y/e/d>
+
+**Important** The crypt password stored in `rclone.conf` is lightly
+obscured. That only protects it from cursory inspection. It is not
+secure unless [configuration encryption](https://rclone.org/docs/#configuration-encryption) of `rclone.conf` is specified.
-By default, cache will keep file data during streaming in RAM as well
-to provide it to readers as fast as possible.
+A long passphrase is recommended, or `rclone config` can generate a
+random one.
-This transient data is evicted as soon as it is read and the number of
-chunks stored doesn't exceed the number of workers. However, depending
-on other settings like "cache-chunk-size" and "cache-workers" this footprint
-can increase if there are parallel streams too (multiple files being read
-at the same time).
+The obscured password is created using AES-CTR with a static key. The
+salt is stored verbatim at the beginning of the obscured password. This
+static key is shared between all versions of rclone.
-If the hardware permits it, use this feature to provide an overall better
-performance during streaming but it can also be disabled if RAM is not
-available on the local machine.
+If you reconfigure rclone with the same passwords/passphrases
+elsewhere it will be compatible, but the obscured version will be different
+due to the different salt.
-Properties:
+Rclone does not encrypt
-- Config: chunk_no_memory
-- Env Var: RCLONE_CACHE_CHUNK_NO_MEMORY
-- Type: bool
-- Default: false
+ * file length - this can be calculated within 16 bytes
+ * modification time - used for syncing
-#### --cache-rps
+### Specifying the remote
-Limits the number of requests per second to the source FS (-1 to disable).
+When configuring the remote to encrypt/decrypt, you may specify any
+string that rclone accepts as a source/destination of other commands.
-This setting places a hard limit on the number of requests per second
-that cache will be doing to the cloud provider remote and try to
-respect that value by setting waits between reads.
+The primary use case is to specify the path into an already configured
+remote (e.g. `remote:path/to/dir` or `remote:bucket`), such that
+data in a remote untrusted location can be stored encrypted.
-If you find that you're getting banned or limited on the cloud
-provider through cache and know that a smaller number of requests per
-second will allow you to work with it then you can use this setting
-for that.
+You may also specify a local filesystem path, such as
+`/path/to/dir` on Linux, `C:\path\to\dir` on Windows. By creating
+a crypt remote pointing to such a local filesystem path, you can
+use rclone as a utility for pure local file encryption, for example
+to keep encrypted files on a removable USB drive.
-A good balance of all the other settings should make this setting
-useless but it is available to set for more special cases.
+**Note**: A string which do not contain a `:` will by rclone be treated
+as a relative path in the local filesystem. For example, if you enter
+the name `remote` without the trailing `:`, it will be treated as
+a subdirectory of the current directory with name "remote".
-**NOTE**: This will limit the number of requests during streams but
-other API calls to the cloud provider like directory listings will
-still pass.
+If a path `remote:path/to/dir` is specified, rclone stores encrypted
+files in `path/to/dir` on the remote. With file name encryption, files
+saved to `secret:subdir/subfile` are stored in the unencrypted path
+`path/to/dir` but the `subdir/subpath` element is encrypted.
-Properties:
+The path you specify does not have to exist, rclone will create
+it when needed.
-- Config: rps
-- Env Var: RCLONE_CACHE_RPS
-- Type: int
-- Default: -1
+If you intend to use the wrapped remote both directly for keeping
+unencrypted content, as well as through a crypt remote for encrypted
+content, it is recommended to point the crypt remote to a separate
+directory within the wrapped remote. If you use a bucket-based storage
+system (e.g. Swift, S3, Google Compute Storage, B2) it is generally
+advisable to wrap the crypt remote around a specific bucket (`s3:bucket`).
+If wrapping around the entire root of the storage (`s3:`), and use the
+optional file name encryption, rclone will encrypt the bucket name.
-#### --cache-writes
+### Changing password
-Cache file data on writes through the FS.
+Should the password, or the configuration file containing a lightly obscured
+form of the password, be compromised, you need to re-encrypt your data with
+a new password. Since rclone uses secret-key encryption, where the encryption
+key is generated directly from the password kept on the client, it is not
+possible to change the password/key of already encrypted content. Just changing
+the password configured for an existing crypt remote means you will no longer
+able to decrypt any of the previously encrypted content. The only possibility
+is to re-upload everything via a crypt remote configured with your new password.
-If you need to read files immediately after you upload them through
-cache you can enable this flag to have their data stored in the
-cache store at the same time during upload.
+Depending on the size of your data, your bandwidth, storage quota etc, there are
+different approaches you can take:
+- If you have everything in a different location, for example on your local system,
+you could remove all of the prior encrypted files, change the password for your
+configured crypt remote (or delete and re-create the crypt configuration),
+and then re-upload everything from the alternative location.
+- If you have enough space on the storage system you can create a new crypt
+remote pointing to a separate directory on the same backend, and then use
+rclone to copy everything from the original crypt remote to the new,
+effectively decrypting everything on the fly using the old password and
+re-encrypting using the new password. When done, delete the original crypt
+remote directory and finally the rclone crypt configuration with the old password.
+All data will be streamed from the storage system and back, so you will
+get half the bandwidth and be charged twice if you have upload and download quota
+on the storage system.
-Properties:
+**Note**: A security problem related to the random password generator
+was fixed in rclone version 1.53.3 (released 2020-11-19). Passwords generated
+by rclone config in version 1.49.0 (released 2019-08-26) to 1.53.2
+(released 2020-10-26) are not considered secure and should be changed.
+If you made up your own password, or used rclone version older than 1.49.0 or
+newer than 1.53.2 to generate it, you are *not* affected by this issue.
+See [issue #4783](https://github.com/rclone/rclone/issues/4783) for more
+details, and a tool you can use to check if you are affected.
-- Config: writes
-- Env Var: RCLONE_CACHE_WRITES
-- Type: bool
-- Default: false
+### Example
-#### --cache-tmp-upload-path
+Create the following file structure using "standard" file name
+encryption.
+
+plaintext/ ├── file0.txt ├── file1.txt └── subdir ├── file2.txt ├── file3.txt └── subsubdir └── file4.txt
+
+Copy these to the remote, and list them
+
+$ rclone -q copy plaintext secret: $ rclone -q ls secret: 7 file1.txt 6 file0.txt 8 subdir/file2.txt 10 subdir/subsubdir/file4.txt 9 subdir/file3.txt
+
+The crypt remote looks like
+
+$ rclone -q ls remote:path 55 hagjclgavj2mbiqm6u6cnjjqcg 54 v05749mltvv1tf4onltun46gls 57 86vhrsv86mpbtd3a0akjuqslj8/dlj7fkq4kdq72emafg7a7s41uo 58 86vhrsv86mpbtd3a0akjuqslj8/7uu829995du6o42n32otfhjqp4/b9pausrfansjth5ob3jkdqd4lc 56 86vhrsv86mpbtd3a0akjuqslj8/8njh1sk437gttmep3p70g81aps
+
+The directory structure is preserved
+
+$ rclone -q ls secret:subdir 8 file2.txt 9 file3.txt 10 subsubdir/file4.txt
+
+Without file name encryption `.bin` extensions are added to underlying
+names. This prevents the cloud provider attempting to interpret file
+content.
+
+$ rclone -q ls remote:path 54 file0.txt.bin 57 subdir/file3.txt.bin 56 subdir/file2.txt.bin 58 subdir/subsubdir/file4.txt.bin 55 file1.txt.bin
+
+### File name encryption modes
-Directory to keep temporary files until they are uploaded.
+Off
-This is the path where cache will use as a temporary storage for new
-files that need to be uploaded to the cloud provider.
+ * doesn't hide file names or directory structure
+ * allows for longer file names (~246 characters)
+ * can use sub paths and copy single files
-Specifying a value will enable this feature. Without it, it is
-completely disabled and files will be uploaded directly to the cloud
-provider
+Standard
-Properties:
+ * file names encrypted
+ * file names can't be as long (~143 characters)
+ * can use sub paths and copy single files
+ * directory structure visible
+ * identical files names will have identical uploaded names
+ * can use shortcuts to shorten the directory recursion
-- Config: tmp_upload_path
-- Env Var: RCLONE_CACHE_TMP_UPLOAD_PATH
-- Type: string
-- Required: false
+Obfuscation
-#### --cache-tmp-wait-time
+This is a simple "rotate" of the filename, with each file having a rot
+distance based on the filename. Rclone stores the distance at the
+beginning of the filename. A file called "hello" may become "53.jgnnq".
-How long should files be stored in local cache before being uploaded.
+Obfuscation is not a strong encryption of filenames, but hinders
+automated scanning tools picking up on filename patterns. It is an
+intermediate between "off" and "standard" which allows for longer path
+segment names.
-This is the duration that a file must wait in the temporary location
-_cache-tmp-upload-path_ before it is selected for upload.
+There is a possibility with some unicode based filenames that the
+obfuscation is weak and may map lower case characters to upper case
+equivalents.
-Note that only one file is uploaded at a time and it can take longer
-to start the upload if a queue formed for this purpose.
+Obfuscation cannot be relied upon for strong protection.
-Properties:
+ * file names very lightly obfuscated
+ * file names can be longer than standard encryption
+ * can use sub paths and copy single files
+ * directory structure visible
+ * identical files names will have identical uploaded names
-- Config: tmp_wait_time
-- Env Var: RCLONE_CACHE_TMP_WAIT_TIME
-- Type: Duration
-- Default: 15s
+Cloud storage systems have limits on file name length and
+total path length which rclone is more likely to breach using
+"Standard" file name encryption. Where file names are less than 156
+characters in length issues should not be encountered, irrespective of
+cloud storage provider.
-#### --cache-db-wait-time
+An experimental advanced option `filename_encoding` is now provided to
+address this problem to a certain degree.
+For cloud storage systems with case sensitive file names (e.g. Google Drive),
+`base64` can be used to reduce file name length.
+For cloud storage systems using UTF-16 to store file names internally
+(e.g. OneDrive, Dropbox, Box), `base32768` can be used to drastically reduce
+file name length.
-How long to wait for the DB to be available - 0 is unlimited.
+An alternative, future rclone file name encryption mode may tolerate
+backend provider path length limits.
-Only one process can have the DB open at any one time, so rclone waits
-for this duration for the DB to become available before it gives an
-error.
+### Directory name encryption
-If you set it to 0 then it will wait forever.
+Crypt offers the option of encrypting dir names or leaving them intact.
+There are two options:
-Properties:
+True
-- Config: db_wait_time
-- Env Var: RCLONE_CACHE_DB_WAIT_TIME
-- Type: Duration
-- Default: 1s
+Encrypts the whole file path including directory names
+Example:
+`1/12/123.txt` is encrypted to
+`p0e52nreeaj0a5ea7s64m4j72s/l42g6771hnv3an9cgc8cr2n1ng/qgm4avr35m5loi1th53ato71v0`
-## Backend commands
+False
-Here are the commands specific to the cache backend.
+Only encrypts file names, skips directory names
+Example:
+`1/12/123.txt` is encrypted to
+`1/12/qgm4avr35m5loi1th53ato71v0`
-Run them with
- rclone backend COMMAND remote:
+### Modification times and hashes
-The help below will explain what arguments each command takes.
+Crypt stores modification times using the underlying remote so support
+depends on that.
-See the [backend](https://rclone.org/commands/rclone_backend/) command for more
-info on how to pass options and arguments.
+Hashes are not stored for crypt. However the data integrity is
+protected by an extremely strong crypto authenticator.
-These can be run on a running backend using the rc command
-[backend/command](https://rclone.org/rc/#backend-command).
+Use the `rclone cryptcheck` command to check the
+integrity of an encrypted remote instead of `rclone check` which can't
+check the checksums properly.
-### stats
-Print stats on the cache backend in JSON format.
+### Standard options
- rclone backend stats remote: [options] [<arguments>+]
+Here are the Standard options specific to crypt (Encrypt/Decrypt a remote).
+#### --crypt-remote
+Remote to encrypt/decrypt.
-# Chunker
+Normally should contain a ':' and a path, e.g. "myremote:path/to/dir",
+"myremote:bucket" or maybe "myremote:" (not recommended).
-The `chunker` overlay transparently splits large files into smaller chunks
-during upload to wrapped remote and transparently assembles them back
-when the file is downloaded. This allows to effectively overcome size limits
-imposed by storage providers.
+Properties:
-## Configuration
+- Config: remote
+- Env Var: RCLONE_CRYPT_REMOTE
+- Type: string
+- Required: true
-To use it, first set up the underlying remote following the configuration
-instructions for that remote. You can also use a local pathname instead of
-a remote.
+#### --crypt-filename-encryption
-First check your chosen remote is working - we'll call it `remote:path` here.
-Note that anything inside `remote:path` will be chunked and anything outside
-won't. This means that if you are using a bucket-based remote (e.g. S3, B2, swift)
-then you should probably put the bucket in the remote `s3:bucket`.
+How to encrypt the filenames.
-Now configure `chunker` using `rclone config`. We will call this one `overlay`
-to separate it from the `remote` itself.
-
-No remotes found, make a new one? n) New remote s) Set configuration password q) Quit config n/s/q> n name> overlay Type of storage to configure. Choose a number from below, or type in your own value [snip] XX / Transparently chunk/split large files  "chunker" [snip] Storage> chunker Remote to chunk/unchunk. Normally should contain a ':' and a path, e.g. "myremote:path/to/dir", "myremote:bucket" or maybe "myremote:" (not recommended). Enter a string value. Press Enter for the default (""). remote> remote:path Files larger than chunk size will be split in chunks. Enter a size with suffix K,M,G,T. Press Enter for the default ("2G"). chunk_size> 100M Choose how chunker handles hash sums. All modes but "none" require metadata. Enter a string value. Press Enter for the default ("md5"). Choose a number from below, or type in your own value 1 / Pass any hash supported by wrapped remote for non-chunked files, return nothing otherwise  "none" 2 / MD5 for composite files  "md5" 3 / SHA1 for composite files  "sha1" 4 / MD5 for all files  "md5all" 5 / SHA1 for all files  "sha1all" 6 / Copying a file to chunker will request MD5 from the source falling back to SHA1 if unsupported  "md5quick" 7 / Similar to "md5quick" but prefers SHA1 over MD5  "sha1quick" hash_type> md5 Edit advanced config? (y/n) y) Yes n) No y/n> n Remote config -------------------- [overlay] type = chunker remote = remote:bucket chunk_size = 100M hash_type = md5 -------------------- y) Yes this is OK e) Edit this remote d) Delete this remote y/e/d> y
-
-### Specifying the remote
+Properties:
+
+- Config: filename_encryption
+- Env Var: RCLONE_CRYPT_FILENAME_ENCRYPTION
+- Type: string
+- Default: "standard"
+- Examples:
+ - "standard"
+ - Encrypt the filenames.
+ - See the docs for the details.
+ - "obfuscate"
+ - Very simple filename obfuscation.
+ - "off"
+ - Don't encrypt the file names.
+ - Adds a ".bin", or "suffix" extension only.
+
+#### --crypt-directory-name-encryption
+
+Option to either encrypt directory names or leave them intact.
+
+NB If filename_encryption is "off" then this option will do nothing.
+
+Properties:
+
+- Config: directory_name_encryption
+- Env Var: RCLONE_CRYPT_DIRECTORY_NAME_ENCRYPTION
+- Type: bool
+- Default: true
+- Examples:
+ - "true"
+ - Encrypt directory names.
+ - "false"
+ - Don't encrypt directory names, leave them intact.
-In normal use, make sure the remote has a `:` in. If you specify the remote
-without a `:` then rclone will use a local directory of that name.
-So if you use a remote of `/path/to/secret/files` then rclone will
-chunk stuff in that directory. If you use a remote of `name` then rclone
-will put files in a directory called `name` in the current directory.
+#### --crypt-password
+Password or pass phrase for encryption.
-### Chunking
+**NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/).
-When rclone starts a file upload, chunker checks the file size. If it
-doesn't exceed the configured chunk size, chunker will just pass the file
-to the wrapped remote (however, see caveat below). If a file is large, chunker will transparently cut
-data in pieces with temporary names and stream them one by one, on the fly.
-Each data chunk will contain the specified number of bytes, except for the
-last one which may have less data. If file size is unknown in advance
-(this is called a streaming upload), chunker will internally create
-a temporary copy, record its size and repeat the above process.
+Properties:
-When upload completes, temporary chunk files are finally renamed.
-This scheme guarantees that operations can be run in parallel and look
-from outside as atomic.
-A similar method with hidden temporary chunks is used for other operations
-(copy/move/rename, etc.). If an operation fails, hidden chunks are normally
-destroyed, and the target composite file stays intact.
+- Config: password
+- Env Var: RCLONE_CRYPT_PASSWORD
+- Type: string
+- Required: true
-When a composite file download is requested, chunker transparently
-assembles it by concatenating data chunks in order. As the split is trivial
-one could even manually concatenate data chunks together to obtain the
-original content.
+#### --crypt-password2
-When the `list` rclone command scans a directory on wrapped remote,
-the potential chunk files are accounted for, grouped and assembled into
-composite directory entries. Any temporary chunks are hidden.
+Password or pass phrase for salt.
-List and other commands can sometimes come across composite files with
-missing or invalid chunks, e.g. shadowed by like-named directory or
-another file. This usually means that wrapped file system has been directly
-tampered with or damaged. If chunker detects a missing chunk it will
-by default print warning, skip the whole incomplete group of chunks but
-proceed with current command.
-You can set the `--chunker-fail-hard` flag to have commands abort with
-error message in such cases.
+Optional but recommended.
+Should be different to the previous password.
-**Caveat**: As it is now, chunker will always create a temporary file in the
-backend and then rename it, even if the file is below the chunk threshold.
-This will result in unnecessary API calls and can severely restrict throughput
-when handling transfers primarily composed of small files on some backends (e.g. Box).
-A workaround to this issue is to use chunker only for files above the chunk threshold
-via `--min-size` and then perform a separate call without chunker on the remaining
-files.
+**NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/).
+Properties:
-#### Chunk names
+- Config: password2
+- Env Var: RCLONE_CRYPT_PASSWORD2
+- Type: string
+- Required: false
-The default chunk name format is `*.rclone_chunk.###`, hence by default
-chunk names are `BIG_FILE_NAME.rclone_chunk.001`,
-`BIG_FILE_NAME.rclone_chunk.002` etc. You can configure another name format
-using the `name_format` configuration file option. The format uses asterisk
-`*` as a placeholder for the base file name and one or more consecutive
-hash characters `#` as a placeholder for sequential chunk number.
-There must be one and only one asterisk. The number of consecutive hash
-characters defines the minimum length of a string representing a chunk number.
-If decimal chunk number has less digits than the number of hashes, it is
-left-padded by zeros. If the decimal string is longer, it is left intact.
-By default numbering starts from 1 but there is another option that allows
-user to start from 0, e.g. for compatibility with legacy software.
+### Advanced options
-For example, if name format is `big_*-##.part` and original file name is
-`data.txt` and numbering starts from 0, then the first chunk will be named
-`big_data.txt-00.part`, the 99th chunk will be `big_data.txt-98.part`
-and the 302nd chunk will become `big_data.txt-301.part`.
+Here are the Advanced options specific to crypt (Encrypt/Decrypt a remote).
-Note that `list` assembles composite directory entries only when chunk names
-match the configured format and treats non-conforming file names as normal
-non-chunked files.
+#### --crypt-server-side-across-configs
-When using `norename` transactions, chunk names will additionally have a unique
-file version suffix. For example, `BIG_FILE_NAME.rclone_chunk.001_bp562k`.
+Deprecated: use --server-side-across-configs instead.
+Allow server-side operations (e.g. copy) to work across different crypt configs.
-### Metadata
+Normally this option is not what you want, but if you have two crypts
+pointing to the same backend you can use it.
-Besides data chunks chunker will by default create metadata object for
-a composite file. The object is named after the original file.
-Chunker allows user to disable metadata completely (the `none` format).
-Note that metadata is normally not created for files smaller than the
-configured chunk size. This may change in future rclone releases.
+This can be used, for example, to change file name encryption type
+without re-uploading all the data. Just make two crypt backends
+pointing to two different directories with the single changed
+parameter and use rclone move to move the files between the crypt
+remotes.
-#### Simple JSON metadata format
+Properties:
-This is the default format. It supports hash sums and chunk validation
-for composite files. Meta objects carry the following fields:
+- Config: server_side_across_configs
+- Env Var: RCLONE_CRYPT_SERVER_SIDE_ACROSS_CONFIGS
+- Type: bool
+- Default: false
-- `ver` - version of format, currently `1`
-- `size` - total size of composite file
-- `nchunks` - number of data chunks in file
-- `md5` - MD5 hashsum of composite file (if present)
-- `sha1` - SHA1 hashsum (if present)
-- `txn` - identifies current version of the file
+#### --crypt-show-mapping
-There is no field for composite file name as it's simply equal to the name
-of meta object on the wrapped remote. Please refer to respective sections
-for details on hashsums and modified time handling.
+For all files listed show how the names encrypt.
-#### No metadata
+If this flag is set then for each file that the remote is asked to
+list, it will log (at level INFO) a line stating the decrypted file
+name and the encrypted file name.
-You can disable meta objects by setting the meta format option to `none`.
-In this mode chunker will scan directory for all files that follow
-configured chunk name format, group them by detecting chunks with the same
-base name and show group names as virtual composite files.
-This method is more prone to missing chunk errors (especially missing
-last chunk) than format with metadata enabled.
+This is so you can work out which encrypted names are which decrypted
+names just in case you need to do something with the encrypted file
+names, or for debugging purposes.
+Properties:
-### Hashsums
+- Config: show_mapping
+- Env Var: RCLONE_CRYPT_SHOW_MAPPING
+- Type: bool
+- Default: false
-Chunker supports hashsums only when a compatible metadata is present.
-Hence, if you choose metadata format of `none`, chunker will report hashsum
-as `UNSUPPORTED`.
+#### --crypt-no-data-encryption
-Please note that by default metadata is stored only for composite files.
-If a file is smaller than configured chunk size, chunker will transparently
-redirect hash requests to wrapped remote, so support depends on that.
-You will see the empty string as a hashsum of requested type for small
-files if the wrapped remote doesn't support it.
+Option to either encrypt file data or leave it unencrypted.
-Many storage backends support MD5 and SHA1 hash types, so does chunker.
-With chunker you can choose one or another but not both.
-MD5 is set by default as the most supported type.
-Since chunker keeps hashes for composite files and falls back to the
-wrapped remote hash for non-chunked ones, we advise you to choose the same
-hash type as supported by wrapped remote so that your file listings
-look coherent.
+Properties:
-If your storage backend does not support MD5 or SHA1 but you need consistent
-file hashing, configure chunker with `md5all` or `sha1all`. These two modes
-guarantee given hash for all files. If wrapped remote doesn't support it,
-chunker will then add metadata to all files, even small. However, this can
-double the amount of small files in storage and incur additional service charges.
-You can even use chunker to force md5/sha1 support in any other remote
-at expense of sidecar meta objects by setting e.g. `hash_type=sha1all`
-to force hashsums and `chunk_size=1P` to effectively disable chunking.
+- Config: no_data_encryption
+- Env Var: RCLONE_CRYPT_NO_DATA_ENCRYPTION
+- Type: bool
+- Default: false
+- Examples:
+ - "true"
+ - Don't encrypt file data, leave it unencrypted.
+ - "false"
+ - Encrypt file data.
-Normally, when a file is copied to chunker controlled remote, chunker
-will ask the file source for compatible file hash and revert to on-the-fly
-calculation if none is found. This involves some CPU overhead but provides
-a guarantee that given hashsum is available. Also, chunker will reject
-a server-side copy or move operation if source and destination hashsum
-types are different resulting in the extra network bandwidth, too.
-In some rare cases this may be undesired, so chunker provides two optional
-choices: `sha1quick` and `md5quick`. If the source does not support primary
-hash type and the quick mode is enabled, chunker will try to fall back to
-the secondary type. This will save CPU and bandwidth but can result in empty
-hashsums at destination. Beware of consequences: the `sync` command will
-revert (sometimes silently) to time/size comparison if compatible hashsums
-between source and target are not found.
+#### --crypt-pass-bad-blocks
+If set this will pass bad blocks through as all 0.
-### Modified time
+This should not be set in normal operation, it should only be set if
+trying to recover an encrypted file with errors and it is desired to
+recover as much of the file as possible.
-Chunker stores modification times using the wrapped remote so support
-depends on that. For a small non-chunked file the chunker overlay simply
-manipulates modification time of the wrapped remote file.
-For a composite file with metadata chunker will get and set
-modification time of the metadata object on the wrapped remote.
-If file is chunked but metadata format is `none` then chunker will
-use modification time of the first data chunk.
+Properties:
+- Config: pass_bad_blocks
+- Env Var: RCLONE_CRYPT_PASS_BAD_BLOCKS
+- Type: bool
+- Default: false
-### Migrations
+#### --crypt-filename-encoding
-The idiomatic way to migrate to a different chunk size, hash type, transaction
-style or chunk naming scheme is to:
+How to encode the encrypted filename to text string.
-- Collect all your chunked files under a directory and have your
- chunker remote point to it.
-- Create another directory (most probably on the same cloud storage)
- and configure a new remote with desired metadata format,
- hash type, chunk naming etc.
-- Now run `rclone sync --interactive oldchunks: newchunks:` and all your data
- will be transparently converted in transfer.
- This may take some time, yet chunker will try server-side
- copy if possible.
-- After checking data integrity you may remove configuration section
- of the old remote.
+This option could help with shortening the encrypted filename. The
+suitable option would depend on the way your remote count the filename
+length and if it's case sensitive.
-If rclone gets killed during a long operation on a big composite file,
-hidden temporary chunks may stay in the directory. They will not be
-shown by the `list` command but will eat up your account quota.
-Please note that the `deletefile` command deletes only active
-chunks of a file. As a workaround, you can use remote of the wrapped
-file system to see them.
-An easy way to get rid of hidden garbage is to copy littered directory
-somewhere using the chunker remote and purge the original directory.
-The `copy` command will copy only active chunks while the `purge` will
-remove everything including garbage.
+Properties:
+- Config: filename_encoding
+- Env Var: RCLONE_CRYPT_FILENAME_ENCODING
+- Type: string
+- Default: "base32"
+- Examples:
+ - "base32"
+ - Encode using base32. Suitable for all remote.
+ - "base64"
+ - Encode using base64. Suitable for case sensitive remote.
+ - "base32768"
+ - Encode using base32768. Suitable if your remote counts UTF-16 or
+ - Unicode codepoint instead of UTF-8 byte length. (Eg. Onedrive, Dropbox)
-### Caveats and Limitations
+#### --crypt-suffix
-Chunker requires wrapped remote to support server-side `move` (or `copy` +
-`delete`) operations, otherwise it will explicitly refuse to start.
-This is because it internally renames temporary chunk files to their final
-names when an operation completes successfully.
+If this is set it will override the default suffix of ".bin".
-Chunker encodes chunk number in file name, so with default `name_format`
-setting it adds 17 characters. Also chunker adds 7 characters of temporary
-suffix during operations. Many file systems limit base file name without path
-by 255 characters. Using rclone's crypt remote as a base file system limits
-file name by 143 characters. Thus, maximum name length is 231 for most files
-and 119 for chunker-over-crypt. A user in need can change name format to
-e.g. `*.rcc##` and save 10 characters (provided at most 99 chunks per file).
+Setting suffix to "none" will result in an empty suffix. This may be useful
+when the path length is critical.
-Note that a move implemented using the copy-and-delete method may incur
-double charging with some cloud storage providers.
+Properties:
-Chunker will not automatically rename existing chunks when you run
-`rclone config` on a live remote and change the chunk name format.
-Beware that in result of this some files which have been treated as chunks
-before the change can pop up in directory listings as normal files
-and vice versa. The same warning holds for the chunk size.
-If you desperately need to change critical chunking settings, you should
-run data migration as described above.
+- Config: suffix
+- Env Var: RCLONE_CRYPT_SUFFIX
+- Type: string
+- Default: ".bin"
-If wrapped remote is case insensitive, the chunker overlay will inherit
-that property (so you can't have a file called "Hello.doc" and "hello.doc"
-in the same directory).
+### Metadata
-Chunker included in rclone releases up to `v1.54` can sometimes fail to
-detect metadata produced by recent versions of rclone. We recommend users
-to keep rclone up-to-date to avoid data corruption.
+Any metadata supported by the underlying remote is read and written.
-Changing `transactions` is dangerous and requires explicit migration.
+See the [metadata](https://rclone.org/docs/#metadata) docs for more info.
+## Backend commands
-### Standard options
+Here are the commands specific to the crypt backend.
-Here are the Standard options specific to chunker (Transparently chunk/split large files).
+Run them with
-#### --chunker-remote
+ rclone backend COMMAND remote:
-Remote to chunk/unchunk.
+The help below will explain what arguments each command takes.
-Normally should contain a ':' and a path, e.g. "myremote:path/to/dir",
-"myremote:bucket" or maybe "myremote:" (not recommended).
+See the [backend](https://rclone.org/commands/rclone_backend/) command for more
+info on how to pass options and arguments.
-Properties:
+These can be run on a running backend using the rc command
+[backend/command](https://rclone.org/rc/#backend-command).
-- Config: remote
-- Env Var: RCLONE_CHUNKER_REMOTE
-- Type: string
-- Required: true
+### encode
-#### --chunker-chunk-size
+Encode the given filename(s)
-Files larger than chunk size will be split in chunks.
+ rclone backend encode remote: [options] [<arguments>+]
-Properties:
+This encodes the filenames given as arguments returning a list of
+strings of the encoded results.
-- Config: chunk_size
-- Env Var: RCLONE_CHUNKER_CHUNK_SIZE
-- Type: SizeSuffix
-- Default: 2Gi
+Usage Example:
-#### --chunker-hash-type
+ rclone backend encode crypt: file1 [file2...]
+ rclone rc backend/command command=encode fs=crypt: file1 [file2...]
-Choose how chunker handles hash sums.
-All modes but "none" require metadata.
+### decode
-Properties:
+Decode the given filename(s)
-- Config: hash_type
-- Env Var: RCLONE_CHUNKER_HASH_TYPE
-- Type: string
-- Default: "md5"
-- Examples:
- - "none"
- - Pass any hash supported by wrapped remote for non-chunked files.
- - Return nothing otherwise.
- - "md5"
- - MD5 for composite files.
- - "sha1"
- - SHA1 for composite files.
- - "md5all"
- - MD5 for all files.
- - "sha1all"
- - SHA1 for all files.
- - "md5quick"
- - Copying a file to chunker will request MD5 from the source.
- - Falling back to SHA1 if unsupported.
- - "sha1quick"
- - Similar to "md5quick" but prefers SHA1 over MD5.
+ rclone backend decode remote: [options] [<arguments>+]
-### Advanced options
+This decodes the filenames given as arguments returning a list of
+strings of the decoded results. It will return an error if any of the
+inputs are invalid.
-Here are the Advanced options specific to chunker (Transparently chunk/split large files).
+Usage Example:
-#### --chunker-name-format
+ rclone backend decode crypt: encryptedfile1 [encryptedfile2...]
+ rclone rc backend/command command=decode fs=crypt: encryptedfile1 [encryptedfile2...]
-String format of chunk file names.
-The two placeholders are: base file name (*) and chunk number (#...).
-There must be one and only one asterisk and one or more consecutive hash characters.
-If chunk number has less digits than the number of hashes, it is left-padded by zeros.
-If there are more digits in the number, they are left as is.
-Possible chunk files are ignored if their name does not match given format.
-Properties:
-- Config: name_format
-- Env Var: RCLONE_CHUNKER_NAME_FORMAT
-- Type: string
-- Default: "*.rclone_chunk.###"
+## Backing up an encrypted remote
-#### --chunker-start-from
+If you wish to backup an encrypted remote, it is recommended that you use
+`rclone sync` on the encrypted files, and make sure the passwords are
+the same in the new encrypted remote.
-Minimum valid chunk number. Usually 0 or 1.
+This will have the following advantages
-By default chunk numbers start from 1.
+ * `rclone sync` will check the checksums while copying
+ * you can use `rclone check` between the encrypted remotes
+ * you don't decrypt and encrypt unnecessarily
-Properties:
+For example, let's say you have your original remote at `remote:` with
+the encrypted version at `eremote:` with path `remote:crypt`. You
+would then set up the new remote `remote2:` and then the encrypted
+version `eremote2:` with path `remote2:crypt` using the same passwords
+as `eremote:`.
-- Config: start_from
-- Env Var: RCLONE_CHUNKER_START_FROM
-- Type: int
-- Default: 1
+To sync the two remotes you would do
-#### --chunker-meta-format
+ rclone sync --interactive remote:crypt remote2:crypt
-Format of the metadata object or "none".
+And to check the integrity you would do
-By default "simplejson".
-Metadata is a small JSON file named after the composite file.
+ rclone check remote:crypt remote2:crypt
-Properties:
+## File formats
-- Config: meta_format
-- Env Var: RCLONE_CHUNKER_META_FORMAT
-- Type: string
-- Default: "simplejson"
-- Examples:
- - "none"
- - Do not use metadata files at all.
- - Requires hash type "none".
- - "simplejson"
- - Simple JSON supports hash sums and chunk validation.
- -
- - It has the following fields: ver, size, nchunks, md5, sha1.
+### File encryption
-#### --chunker-fail-hard
+Files are encrypted 1:1 source file to destination object. The file
+has a header and is divided into chunks.
-Choose how chunker should handle files with missing or invalid chunks.
+#### Header
-Properties:
+ * 8 bytes magic string `RCLONE\x00\x00`
+ * 24 bytes Nonce (IV)
-- Config: fail_hard
-- Env Var: RCLONE_CHUNKER_FAIL_HARD
-- Type: bool
-- Default: false
-- Examples:
- - "true"
- - Report errors and abort current command.
- - "false"
- - Warn user, skip incomplete file and proceed.
+The initial nonce is generated from the operating systems crypto
+strong random number generator. The nonce is incremented for each
+chunk read making sure each nonce is unique for each block written.
+The chance of a nonce being reused is minuscule. If you wrote an
+exabyte of data (10¹⸠bytes) you would have a probability of
+approximately 2×10â»Â³Â² of re-using a nonce.
-#### --chunker-transactions
+#### Chunk
-Choose how chunker should handle temporary files during transactions.
+Each chunk will contain 64 KiB of data, except for the last one which
+may have less data. The data chunk is in standard NaCl SecretBox
+format. SecretBox uses XSalsa20 and Poly1305 to encrypt and
+authenticate messages.
-Properties:
+Each chunk contains:
-- Config: transactions
-- Env Var: RCLONE_CHUNKER_TRANSACTIONS
-- Type: string
-- Default: "rename"
-- Examples:
- - "rename"
- - Rename temporary files after a successful transaction.
- - "norename"
- - Leave temporary file names and write transaction ID to metadata file.
- - Metadata is required for no rename transactions (meta format cannot be "none").
- - If you are using norename transactions you should be careful not to downgrade Rclone
- - as older versions of Rclone don't support this transaction style and will misinterpret
- - files manipulated by norename transactions.
- - This method is EXPERIMENTAL, don't use on production systems.
- - "auto"
- - Rename or norename will be used depending on capabilities of the backend.
- - If meta format is set to "none", rename transactions will always be used.
- - This method is EXPERIMENTAL, don't use on production systems.
+ * 16 Bytes of Poly1305 authenticator
+ * 1 - 65536 bytes XSalsa20 encrypted data
+64k chunk size was chosen as the best performing chunk size (the
+authenticator takes too much time below this and the performance drops
+off due to cache effects above this). Note that these chunks are
+buffered in memory so they can't be too big.
+This uses a 32 byte (256 bit key) key derived from the user password.
-# Citrix ShareFile
+#### Examples
-[Citrix ShareFile](https://sharefile.com) is a secure file sharing and transfer service aimed as business.
+1 byte file will encrypt to
-## Configuration
+ * 32 bytes header
+ * 17 bytes data chunk
-The initial setup for Citrix ShareFile involves getting a token from
-Citrix ShareFile which you can in your browser. `rclone config` walks you
-through it.
+49 bytes total
-Here is an example of how to make a remote called `remote`. First run:
+1 MiB (1048576 bytes) file will encrypt to
- rclone config
+ * 32 bytes header
+ * 16 chunks of 65568 bytes
-This will guide you through an interactive setup process:
-
-No remotes found, make a new one? n) New remote s) Set configuration password q) Quit config n/s/q> n name> remote Type of storage to configure. Enter a string value. Press Enter for the default (""). Choose a number from below, or type in your own value XX / Citrix Sharefile  "sharefile" Storage> sharefile ** See help for sharefile backend at: https://rclone.org/sharefile/ **
-ID of the root folder
-Leave blank to access "Personal Folders". You can use one of the standard values here or any folder ID (long hex number ID). Enter a string value. Press Enter for the default (""). Choose a number from below, or type in your own value 1 / Access the Personal Folders. (Default) Â "" 2 / Access the Favorites folder. Â "favorites" 3 / Access all the shared folders. Â "allshared" 4 / Access all the individual connectors. Â "connectors" 5 / Access the home, favorites, and shared folders as well as the connectors. Â "top" root_folder_id> Edit advanced config? (y/n) y) Yes n) No y/n> n Remote config Use web browser to automatically authenticate rclone with remote? * Say Y if the machine running rclone has a web browser you can use * Say N if running rclone on a (remote) machine without web browser access If not sure try Y. If Y failed, try N. y) Yes n) No y/n> y If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth?state=XXX Log in and authorize rclone for access Waiting for code... Got code -------------------- [remote] type = sharefile endpoint = https://XXX.sharefile.com token = {"access_token":"XXX","token_type":"bearer","refresh_token":"XXX","expiry":"2019-09-30T19:41:45.878561877+01:00"} -------------------- y) Yes this is OK e) Edit this remote d) Delete this remote y/e/d> y
-
-See the [remote setup docs](https://rclone.org/remote_setup/) for how to set it up on a
-machine with no Internet browser available.
+1049120 bytes total (a 0.05% overhead). This is the overhead for big
+files.
-Note that rclone runs a webserver on your local machine to collect the
-token as returned from Citrix ShareFile. This only runs from the moment it opens
-your browser to the moment you get back the verification code. This
-is on `http://127.0.0.1:53682/` and this it may require you to unblock
-it temporarily if you are running a host firewall.
+### Name encryption
-Once configured you can then use `rclone` like this,
+File names are encrypted segment by segment - the path is broken up
+into `/` separated strings and these are encrypted individually.
-List directories in top level of your ShareFile
+File segments are padded using PKCS#7 to a multiple of 16 bytes
+before encryption.
+
+They are then encrypted with EME using AES with 256 bit key. EME
+(ECB-Mix-ECB) is a wide-block encryption mode presented in the 2003
+paper "A Parallelizable Enciphering Mode" by Halevi and Rogaway.
- rclone lsd remote:
+This makes for deterministic encryption which is what we want - the
+same filename must encrypt to the same thing otherwise we can't find
+it on the cloud storage system.
-List all the files in your ShareFile
+This means that
- rclone ls remote:
+ * filenames with the same name will encrypt the same
+ * filenames which start the same won't have a common prefix
-To copy a local directory to an ShareFile directory called backup
+This uses a 32 byte key (256 bits) and a 16 byte (128 bits) IV both of
+which are derived from the user password.
- rclone copy /home/source remote:backup
+After encryption they are written out using a modified version of
+standard `base32` encoding as described in RFC4648. The standard
+encoding is modified in two ways:
-Paths may be as deep as required, e.g. `remote:directory/subdirectory`.
+ * it becomes lower case (no-one likes upper case filenames!)
+ * we strip the padding character `=`
-### Modified time and hashes
+`base32` is used rather than the more efficient `base64` so rclone can be
+used on case insensitive remotes (e.g. Windows, Amazon Drive).
-ShareFile allows modification times to be set on objects accurate to 1
-second. These will be used to detect whether objects need syncing or
-not.
+### Key derivation
-ShareFile supports MD5 type hashes, so you can use the `--checksum`
-flag.
+Rclone uses `scrypt` with parameters `N=16384, r=8, p=1` with an
+optional user supplied salt (password2) to derive the 32+32+16 = 80
+bytes of key material required. If the user doesn't supply a salt
+then rclone uses an internal one.
-### Transfers
+`scrypt` makes it impractical to mount a dictionary attack on rclone
+encrypted data. For full protection against this you should always use
+a salt.
-For files above 128 MiB rclone will use a chunked transfer. Rclone will
-upload up to `--transfers` chunks at the same time (shared among all
-the multipart uploads). Chunks are buffered in memory and are
-normally 64 MiB so increasing `--transfers` will increase memory use.
+## SEE ALSO
-### Restricted filename characters
+* [rclone cryptdecode](https://rclone.org/commands/rclone_cryptdecode/) - Show forward/reverse mapping of encrypted filenames
-In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters)
-the following characters are also replaced:
+# Compress
-| Character | Value | Replacement |
-| --------- |:-----:|:-----------:|
-| \\ | 0x5C | ï¼¼ |
-| * | 0x2A | ＊ |
-| < | 0x3C | ＜ |
-| > | 0x3E | ＞ |
-| ? | 0x3F | ？ |
-| : | 0x3A | ： |
-| \| | 0x7C | ｜ |
-| " | 0x22 | ＂ |
+## Warning
-File names can also not start or end with the following characters.
-These only get replaced if they are the first or last character in the
-name:
+This remote is currently **experimental**. Things may break and data may be lost. Anything you do with this remote is
+at your own risk. Please understand the risks associated with using experimental code and don't use this remote in
+critical applications.
-| Character | Value | Replacement |
-| --------- |:-----:|:-----------:|
-| SP | 0x20 | â |
-| . | 0x2E | ． |
+The `Compress` remote adds compression to another remote. It is best used with remotes containing
+many large compressible files.
-Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8),
-as they can't be used in JSON strings.
+## Configuration
+To use this remote, all you need to do is specify another remote and a compression mode to use:
+
+Current remotes:
+Name Type ==== ==== remote_to_press sometype
+
+- Edit existing remote $ rclone config
+- New remote
+- Delete remote
+- Rename remote
+- Copy remote
+- Set configuration password
+- Quit config e/n/d/r/c/s/q> n name> compress ... 8 / Compress a remote  "compress" ... Storage> compress ** See help for compress backend at: https://rclone.org/compress/ **
+
+Remote to compress. Enter a string value. Press Enter for the default (""). remote> remote_to_press:subdir Compression mode. Enter a string value. Press Enter for the default ("gzip"). Choose a number from below, or type in your own value 1 / Gzip compression balanced for speed and compression strength. Â "gzip" compression_mode> gzip Edit advanced config? (y/n) y) Yes n) No (default) y/n> n Remote config -------------------- [compress] type = compress remote = remote_to_press:subdir compression_mode = gzip -------------------- y) Yes this is OK (default) e) Edit this remote d) Delete this remote y/e/d> y
+
+### Compression Modes
-### Standard options
+Currently only gzip compression is supported. It provides a decent balance between speed and size and is well
+supported by other applications. Compression strength can further be configured via an advanced setting where 0 is no
+compression and 9 is strongest compression.
-Here are the Standard options specific to sharefile (Citrix Sharefile).
+### File types
-#### --sharefile-client-id
+If you open a remote wrapped by compress, you will see that there are many files with an extension corresponding to
+the compression algorithm you chose. These files are standard files that can be opened by various archive programs,
+but they have some hidden metadata that allows them to be used by rclone.
+While you may download and decompress these files at will, do **not** manually delete or rename files. Files without
+correct metadata files will not be recognized by rclone.
-OAuth Client Id.
+### File names
-Leave blank normally.
+The compressed files will be named `*.###########.gz` where `*` is the base file and the `#` part is base64 encoded
+size of the uncompressed file. The file names should not be changed by anything other than the rclone compression backend.
-Properties:
-- Config: client_id
-- Env Var: RCLONE_SHAREFILE_CLIENT_ID
-- Type: string
-- Required: false
+### Standard options
-#### --sharefile-client-secret
+Here are the Standard options specific to compress (Compress a remote).
-OAuth Client Secret.
+#### --compress-remote
-Leave blank normally.
+Remote to compress.
Properties:
-- Config: client_secret
-- Env Var: RCLONE_SHAREFILE_CLIENT_SECRET
+- Config: remote
+- Env Var: RCLONE_COMPRESS_REMOTE
- Type: string
-- Required: false
-
-#### --sharefile-root-folder-id
+- Required: true
-ID of the root folder.
+#### --compress-mode
-Leave blank to access "Personal Folders". You can use one of the
-standard values here or any folder ID (long hex number ID).
+Compression mode.
Properties:
-- Config: root_folder_id
-- Env Var: RCLONE_SHAREFILE_ROOT_FOLDER_ID
+- Config: mode
+- Env Var: RCLONE_COMPRESS_MODE
- Type: string
-- Required: false
+- Default: "gzip"
- Examples:
- - ""
- - Access the Personal Folders (default).
- - "favorites"
- - Access the Favorites folder.
- - "allshared"
- - Access all the shared folders.
- - "connectors"
- - Access all the individual connectors.
- - "top"
- - Access the home, favorites, and shared folders as well as the connectors.
+ - "gzip"
+ - Standard gzip compression with fastest parameters.
### Advanced options
-Here are the Advanced options specific to sharefile (Citrix Sharefile).
+Here are the Advanced options specific to compress (Compress a remote).
-#### --sharefile-token
+#### --compress-level
-OAuth Access Token as a JSON blob.
+GZIP compression level (-2 to 9).
+
+Generally -1 (default, equivalent to 5) is recommended.
+Levels 1 to 9 increase compression at the cost of speed. Going past 6
+generally offers very little return.
+
+Level -2 uses Huffman encoding only. Only use if you know what you
+are doing.
+Level 0 turns off compression.
Properties:
-- Config: token
-- Env Var: RCLONE_SHAREFILE_TOKEN
-- Type: string
-- Required: false
+- Config: level
+- Env Var: RCLONE_COMPRESS_LEVEL
+- Type: int
+- Default: -1
-#### --sharefile-auth-url
+#### --compress-ram-cache-limit
-Auth server URL.
+Some remotes don't allow the upload of files with unknown size.
+In this case the compressed file will need to be cached to determine
+it's size.
-Leave blank to use the provider defaults.
+Files smaller than this limit will be cached in RAM, files larger than
+this limit will be cached on disk.
Properties:
-- Config: auth_url
-- Env Var: RCLONE_SHAREFILE_AUTH_URL
-- Type: string
-- Required: false
+- Config: ram_cache_limit
+- Env Var: RCLONE_COMPRESS_RAM_CACHE_LIMIT
+- Type: SizeSuffix
+- Default: 20Mi
-#### --sharefile-token-url
+### Metadata
-Token server url.
+Any metadata supported by the underlying remote is read and written.
-Leave blank to use the provider defaults.
+See the [metadata](https://rclone.org/docs/#metadata) docs for more info.
-Properties:
-- Config: token_url
-- Env Var: RCLONE_SHAREFILE_TOKEN_URL
-- Type: string
-- Required: false
-#### --sharefile-upload-cutoff
+# Combine
-Cutoff for switching to multipart upload.
+The `combine` backend joins remotes together into a single directory
+tree.
-Properties:
+For example you might have a remote for images on one provider:
+
+$ rclone tree s3:imagesbucket / ├── image1.jpg └── image2.jpg
+
+And a remote for files on another:
+
+$ rclone tree drive:important/files / ├── file1.txt └── file2.txt
+
+The `combine` backend can join these together into a synthetic
+directory structure like this:
+
+$ rclone tree combined: / ├── files │ ├── file1.txt │ └── file2.txt └── images ├── image1.jpg └── image2.jpg
+
+You'd do this by specifying an `upstreams` parameter in the config
+like this
-- Config: upload_cutoff
-- Env Var: RCLONE_SHAREFILE_UPLOAD_CUTOFF
-- Type: SizeSuffix
-- Default: 128Mi
+ upstreams = images=s3:imagesbucket files=drive:important/files
-#### --sharefile-chunk-size
+During the initial setup with `rclone config` you will specify the
+upstreams remotes as a space separated list. The upstream remotes can
+either be a local paths or other remotes.
-Upload chunk size.
+## Configuration
-Must a power of 2 >= 256k.
+Here is an example of how to make a combine called `remote` for the
+example above. First run:
-Making this larger will improve performance, but note that each chunk
-is buffered in memory one per transfer.
+ rclone config
-Reducing this will reduce memory usage but decrease performance.
+This will guide you through an interactive setup process:
+
+No remotes found, make a new one? n) New remote s) Set configuration password q) Quit config n/s/q> n name> remote Option Storage. Type of storage to configure. Choose a number from below, or type in your own value. ... XX / Combine several remotes into one  (combine) ... Storage> combine Option upstreams. Upstreams for combining These should be in the form dir=remote:path dir2=remote2:path Where before the = is specified the root directory and after is the remote to put there. Embedded spaces can be added using quotes "dir=remote:path with space" "dir2=remote2:path with space" Enter a fs.SpaceSepList value. upstreams> images=s3:imagesbucket files=drive:important/files -------------------- [remote] type = combine upstreams = images=s3:imagesbucket files=drive:important/files -------------------- y) Yes this is OK (default) e) Edit this remote d) Delete this remote y/e/d> y
+
+### Configuring for Google Drive Shared Drives
-Properties:
+Rclone has a convenience feature for making a combine backend for all
+the shared drives you have access to.
-- Config: chunk_size
-- Env Var: RCLONE_SHAREFILE_CHUNK_SIZE
-- Type: SizeSuffix
-- Default: 64Mi
+Assuming your main (non shared drive) Google drive remote is called
+`drive:` you would run
-#### --sharefile-endpoint
+ rclone backend -o config drives drive:
-Endpoint for API calls.
+This would produce something like this:
-This is usually auto discovered as part of the oauth process, but can
-be set manually to something like: https://XXX.sharefile.com
+ [My Drive]
+ type = alias
+ remote = drive,team_drive=0ABCDEF-01234567890,root_folder_id=:
+ [Test Drive]
+ type = alias
+ remote = drive,team_drive=0ABCDEFabcdefghijkl,root_folder_id=:
-Properties:
+ [AllDrives]
+ type = combine
+ upstreams = "My Drive=My Drive:" "Test Drive=Test Drive:"
-- Config: endpoint
-- Env Var: RCLONE_SHAREFILE_ENDPOINT
-- Type: string
-- Required: false
+If you then add that config to your config file (find it with `rclone
+config file`) then you can access all the shared drives in one place
+with the `AllDrives:` remote.
-#### --sharefile-encoding
+See [the Google Drive docs](https://rclone.org/drive/#drives) for full info.
-The encoding for the backend.
-See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info.
+### Standard options
-Properties:
+Here are the Standard options specific to combine (Combine several remotes into one).
-- Config: encoding
-- Env Var: RCLONE_SHAREFILE_ENCODING
-- Type: MultiEncoder
-- Default: Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,LeftSpace,LeftPeriod,RightSpace,RightPeriod,InvalidUtf8,Dot
+#### --combine-upstreams
+Upstreams for combining
-## Limitations
+These should be in the form
-Note that ShareFile is case insensitive so you can't have a file called
-"Hello.doc" and one called "hello.doc".
+ dir=remote:path dir2=remote2:path
-ShareFile only supports filenames up to 256 characters in length.
+Where before the = is specified the root directory and after is the remote to
+put there.
-`rclone about` is not supported by the Citrix ShareFile backend. Backends without
-this capability cannot determine free space for an rclone mount or
-use policy `mfs` (most free space) as a member of an rclone union
-remote.
+Embedded spaces can be added using quotes
-See [List of backends that do not support rclone about](https://rclone.org/overview/#optional-features) and [rclone about](https://rclone.org/commands/rclone_about/)
+ "dir=remote:path with space" "dir2=remote2:path with space"
-# Crypt
-Rclone `crypt` remotes encrypt and decrypt other remotes.
-A remote of type `crypt` does not access a [storage system](https://rclone.org/overview/)
-directly, but instead wraps another remote, which in turn accesses
-the storage system. This is similar to how [alias](https://rclone.org/alias/),
-[union](https://rclone.org/union/), [chunker](https://rclone.org/chunker/)
-and a few others work. It makes the usage very flexible, as you can
-add a layer, in this case an encryption layer, on top of any other
-backend, even in multiple layers. Rclone's functionality
-can be used as with any other remote, for example you can
-[mount](https://rclone.org/commands/rclone_mount/) a crypt remote.
+Properties:
-Accessing a storage system through a crypt remote realizes client-side
-encryption, which makes it safe to keep your data in a location you do
-not trust will not get compromised.
-When working against the `crypt` remote, rclone will automatically
-encrypt (before uploading) and decrypt (after downloading) on your local
-system as needed on the fly, leaving the data encrypted at rest in the
-wrapped remote. If you access the storage system using an application
-other than rclone, or access the wrapped remote directly using rclone,
-there will not be any encryption/decryption: Downloading existing content
-will just give you the encrypted (scrambled) format, and anything you
-upload will *not* become encrypted.
+- Config: upstreams
+- Env Var: RCLONE_COMBINE_UPSTREAMS
+- Type: SpaceSepList
+- Default:
-The encryption is a secret-key encryption (also called symmetric key encryption)
-algorithm, where a password (or pass phrase) is used to generate real encryption key.
-The password can be supplied by user, or you may chose to let rclone
-generate one. It will be stored in the configuration file, in a lightly obscured form.
-If you are in an environment where you are not able to keep your configuration
-secured, you should add
-[configuration encryption](https://rclone.org/docs/#configuration-encryption)
-as protection. As long as you have this configuration file, you will be able to
-decrypt your data. Without the configuration file, as long as you remember
-the password (or keep it in a safe place), you can re-create the configuration
-and gain access to the existing data. You may also configure a corresponding
-remote in a different installation to access the same data.
-See below for guidance to [changing password](#changing-password).
+### Metadata
-Encryption uses [cryptographic salt](https://en.wikipedia.org/wiki/Salt_(cryptography)),
-to permute the encryption key so that the same string may be encrypted in
-different ways. When configuring the crypt remote it is optional to enter a salt,
-or to let rclone generate a unique salt. If omitted, rclone uses a built-in unique string.
-Normally in cryptography, the salt is stored together with the encrypted content,
-and do not have to be memorized by the user. This is not the case in rclone,
-because rclone does not store any additional information on the remotes. Use of
-custom salt is effectively a second password that must be memorized.
+Any metadata supported by the underlying remote is read and written.
-[File content](#file-encryption) encryption is performed using
-[NaCl SecretBox](https://godoc.org/golang.org/x/crypto/nacl/secretbox),
-based on XSalsa20 cipher and Poly1305 for integrity.
-[Names](#name-encryption) (file- and directory names) are also encrypted
-by default, but this has some implications and is therefore
-possible to be turned off.
+See the [metadata](https://rclone.org/docs/#metadata) docs for more info.
-## Configuration
-Here is an example of how to make a remote called `secret`.
-To use `crypt`, first set up the underlying remote. Follow the
-`rclone config` instructions for the specific backend.
+# Dropbox
-Before configuring the crypt remote, check the underlying remote is
-working. In this example the underlying remote is called `remote`.
-We will configure a path `path` within this remote to contain the
-encrypted content. Anything inside `remote:path` will be encrypted
-and anything outside will not.
+Paths are specified as `remote:path`
-Configure `crypt` using `rclone config`. In this example the `crypt`
-remote is called `secret`, to differentiate it from the underlying
-`remote`.
+Dropbox paths may be as deep as required, e.g.
+`remote:directory/subdirectory`.
-When you are done you can use the crypt remote named `secret` just
-as you would with any other remote, e.g. `rclone copy D:\docs secret:\docs`,
-and rclone will encrypt and decrypt as needed on the fly.
-If you access the wrapped remote `remote:path` directly you will bypass
-the encryption, and anything you read will be in encrypted form, and
-anything you write will be unencrypted. To avoid issues it is best to
-configure a dedicated path for encrypted content, and access it
-exclusively through a crypt remote.
+## Configuration
+
+The initial setup for dropbox involves getting a token from Dropbox
+which you need to do in your browser. `rclone config` walks you
+through it.
+
+Here is an example of how to make a remote called `remote`. First run:
+
+ rclone config
+
+This will guide you through an interactive setup process:
-No remotes found, make a new one? n) New remote s) Set configuration password q) Quit config n/s/q> n name> secret Type of storage to configure. Enter a string value. Press Enter for the default (""). Choose a number from below, or type in your own value [snip] XX / Encrypt/Decrypt a remote  "crypt" [snip] Storage> crypt ** See help for crypt backend at: https://rclone.org/crypt/ **
-Remote to encrypt/decrypt. Normally should contain a ':' and a path, eg "myremote:path/to/dir", "myremote:bucket" or maybe "myremote:" (not recommended). Enter a string value. Press Enter for the default (""). remote> remote:path How to encrypt the filenames. Enter a string value. Press Enter for the default ("standard"). Choose a number from below, or type in your own value. / Encrypt the filenames. 1 | See the docs for the details. Â "standard" 2 / Very simple filename obfuscation. Â "obfuscate" / Don't encrypt the file names. 3 | Adds a ".bin" extension only. Â "off" filename_encryption> Option to either encrypt directory names or leave them intact.
-NB If filename_encryption is "off" then this option will do nothing. Enter a boolean value (true or false). Press Enter for the default ("true"). Choose a number from below, or type in your own value 1 / Encrypt directory names. Â "true" 2 / Don't encrypt directory names, leave them intact. Â "false" directory_name_encryption> Password or pass phrase for encryption. y) Yes type in my own password g) Generate random password y/g> y Enter the password: password: Confirm the password: password: Password or pass phrase for salt. Optional but recommended. Should be different to the previous password. y) Yes type in my own password g) Generate random password n) No leave this optional password blank (default) y/g/n> g Password strength in bits. 64 is just about memorable 128 is secure 1024 is the maximum Bits> 128 Your password is: JAsJvRcgR-_veXNfy_sGmQ Use this password? Please note that an obscured version of this password (and not the password itself) will be stored under your configuration file, so keep this generated password in a safe place. y) Yes (default) n) No y/n> Edit advanced config? (y/n) y) Yes n) No (default) y/n> Remote config -------------------- [secret] type = crypt remote = remote:path password = *** ENCRYPTED password2 = ENCRYPTED *** -------------------- y) Yes this is OK (default) e) Edit this remote d) Delete this remote y/e/d>
+
+- New remote
+- Delete remote
+- Quit config e/n/d/q> n name> remote Type of storage to configure. Choose a number from below, or type in your own value [snip] XX / Dropbox  "dropbox" [snip] Storage> dropbox Dropbox App Key - leave blank normally. app_key> Dropbox App Secret - leave blank normally. app_secret> Remote config Please visit: https://www.dropbox.com/1/oauth2/authorize?client_id=XXXXXXXXXXXXXXX&response_type=code Enter the code: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX_XXXXXXXXXX -------------------- [remote] app_key = app_secret = token = XXXXXXXXXXXXXXXXXXXXXXXXXXXXX_XXXX_XXXXXXXXXXXXXXXXXXXXXXXXXXXXX --------------------
+- Yes this is OK
+- Edit this remote
+- Delete this remote y/e/d> y
+
-**Important** The crypt password stored in `rclone.conf` is lightly
-obscured. That only protects it from cursory inspection. It is not
-secure unless [configuration encryption](https://rclone.org/docs/#configuration-encryption) of `rclone.conf` is specified.
+See the [remote setup docs](https://rclone.org/remote_setup/) for how to set it up on a
+machine with no Internet browser available.
-A long passphrase is recommended, or `rclone config` can generate a
-random one.
+Note that rclone runs a webserver on your local machine to collect the
+token as returned from Dropbox. This only
+runs from the moment it opens your browser to the moment you get back
+the verification code. This is on `http://127.0.0.1:53682/` and it
+may require you to unblock it temporarily if you are running a host
+firewall, or use manual mode.
-The obscured password is created using AES-CTR with a static key. The
-salt is stored verbatim at the beginning of the obscured password. This
-static key is shared between all versions of rclone.
+You can then use it like this,
-If you reconfigure rclone with the same passwords/passphrases
-elsewhere it will be compatible, but the obscured version will be different
-due to the different salt.
+List directories in top level of your dropbox
-Rclone does not encrypt
+ rclone lsd remote:
- * file length - this can be calculated within 16 bytes
- * modification time - used for syncing
+List all the files in your dropbox
-### Specifying the remote
+ rclone ls remote:
-When configuring the remote to encrypt/decrypt, you may specify any
-string that rclone accepts as a source/destination of other commands.
+To copy a local directory to a dropbox directory called backup
-The primary use case is to specify the path into an already configured
-remote (e.g. `remote:path/to/dir` or `remote:bucket`), such that
-data in a remote untrusted location can be stored encrypted.
+ rclone copy /home/source remote:backup
-You may also specify a local filesystem path, such as
-`/path/to/dir` on Linux, `C:\path\to\dir` on Windows. By creating
-a crypt remote pointing to such a local filesystem path, you can
-use rclone as a utility for pure local file encryption, for example
-to keep encrypted files on a removable USB drive.
+### Dropbox for business
-**Note**: A string which do not contain a `:` will by rclone be treated
-as a relative path in the local filesystem. For example, if you enter
-the name `remote` without the trailing `:`, it will be treated as
-a subdirectory of the current directory with name "remote".
+Rclone supports Dropbox for business and Team Folders.
-If a path `remote:path/to/dir` is specified, rclone stores encrypted
-files in `path/to/dir` on the remote. With file name encryption, files
-saved to `secret:subdir/subfile` are stored in the unencrypted path
-`path/to/dir` but the `subdir/subpath` element is encrypted.
+When using Dropbox for business `remote:` and `remote:path/to/file`
+will refer to your personal folder.
-The path you specify does not have to exist, rclone will create
-it when needed.
+If you wish to see Team Folders you must use a leading `/` in the
+path, so `rclone lsd remote:/` will refer to the root and show you all
+Team Folders and your User Folder.
-If you intend to use the wrapped remote both directly for keeping
-unencrypted content, as well as through a crypt remote for encrypted
-content, it is recommended to point the crypt remote to a separate
-directory within the wrapped remote. If you use a bucket-based storage
-system (e.g. Swift, S3, Google Compute Storage, B2) it is generally
-advisable to wrap the crypt remote around a specific bucket (`s3:bucket`).
-If wrapping around the entire root of the storage (`s3:`), and use the
-optional file name encryption, rclone will encrypt the bucket name.
+You can then use team folders like this `remote:/TeamFolder` and
+`remote:/TeamFolder/path/to/file`.
-### Changing password
+A leading `/` for a Dropbox personal account will do nothing, but it
+will take an extra HTTP transaction so it should be avoided.
-Should the password, or the configuration file containing a lightly obscured
-form of the password, be compromised, you need to re-encrypt your data with
-a new password. Since rclone uses secret-key encryption, where the encryption
-key is generated directly from the password kept on the client, it is not
-possible to change the password/key of already encrypted content. Just changing
-the password configured for an existing crypt remote means you will no longer
-able to decrypt any of the previously encrypted content. The only possibility
-is to re-upload everything via a crypt remote configured with your new password.
+### Modification times and hashes
-Depending on the size of your data, your bandwidth, storage quota etc, there are
-different approaches you can take:
-- If you have everything in a different location, for example on your local system,
-you could remove all of the prior encrypted files, change the password for your
-configured crypt remote (or delete and re-create the crypt configuration),
-and then re-upload everything from the alternative location.
-- If you have enough space on the storage system you can create a new crypt
-remote pointing to a separate directory on the same backend, and then use
-rclone to copy everything from the original crypt remote to the new,
-effectively decrypting everything on the fly using the old password and
-re-encrypting using the new password. When done, delete the original crypt
-remote directory and finally the rclone crypt configuration with the old password.
-All data will be streamed from the storage system and back, so you will
-get half the bandwidth and be charged twice if you have upload and download quota
-on the storage system.
+Dropbox supports modified times, but the only way to set a
+modification time is to re-upload the file.
-**Note**: A security problem related to the random password generator
-was fixed in rclone version 1.53.3 (released 2020-11-19). Passwords generated
-by rclone config in version 1.49.0 (released 2019-08-26) to 1.53.2
-(released 2020-10-26) are not considered secure and should be changed.
-If you made up your own password, or used rclone version older than 1.49.0 or
-newer than 1.53.2 to generate it, you are *not* affected by this issue.
-See [issue #4783](https://github.com/rclone/rclone/issues/4783) for more
-details, and a tool you can use to check if you are affected.
+This means that if you uploaded your data with an older version of
+rclone which didn't support the v2 API and modified times, rclone will
+decide to upload all your old data to fix the modification times. If
+you don't want this to happen use `--size-only` or `--checksum` flag
+to stop it.
-### Example
+Dropbox supports [its own hash
+type](https://www.dropbox.com/developers/reference/content-hash) which
+is checked for all transfers.
-Create the following file structure using "standard" file name
-encryption.
-
-plaintext/ ├── file0.txt ├── file1.txt └── subdir ├── file2.txt ├── file3.txt └── subsubdir └── file4.txt
-
-Copy these to the remote, and list them
-
-$ rclone -q copy plaintext secret: $ rclone -q ls secret: 7 file1.txt 6 file0.txt 8 subdir/file2.txt 10 subdir/subsubdir/file4.txt 9 subdir/file3.txt
-
-The crypt remote looks like
-
-$ rclone -q ls remote:path 55 hagjclgavj2mbiqm6u6cnjjqcg 54 v05749mltvv1tf4onltun46gls 57 86vhrsv86mpbtd3a0akjuqslj8/dlj7fkq4kdq72emafg7a7s41uo 58 86vhrsv86mpbtd3a0akjuqslj8/7uu829995du6o42n32otfhjqp4/b9pausrfansjth5ob3jkdqd4lc 56 86vhrsv86mpbtd3a0akjuqslj8/8njh1sk437gttmep3p70g81aps
-
-The directory structure is preserved
-
-$ rclone -q ls secret:subdir 8 file2.txt 9 file3.txt 10 subsubdir/file4.txt
-
-Without file name encryption `.bin` extensions are added to underlying
-names. This prevents the cloud provider attempting to interpret file
-content.
-
-$ rclone -q ls remote:path 54 file0.txt.bin 57 subdir/file3.txt.bin 56 subdir/file2.txt.bin 58 subdir/subsubdir/file4.txt.bin 55 file1.txt.bin
-
-### File name encryption modes
+### Restricted filename characters
-Off
+| Character | Value | Replacement |
+| --------- |:-----:|:-----------:|
+| NUL | 0x00 | †|
+| / | 0x2F | ï¼ |
+| DEL | 0x7F | â¡ |
+| \ | 0x5C | ï¼¼ |
- * doesn't hide file names or directory structure
- * allows for longer file names (~246 characters)
- * can use sub paths and copy single files
+File names can also not end with the following characters.
+These only get replaced if they are the last character in the name:
-Standard
+| Character | Value | Replacement |
+| --------- |:-----:|:-----------:|
+| SP | 0x20 | â |
- * file names encrypted
- * file names can't be as long (~143 characters)
- * can use sub paths and copy single files
- * directory structure visible
- * identical files names will have identical uploaded names
- * can use shortcuts to shorten the directory recursion
+Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8),
+as they can't be used in JSON strings.
-Obfuscation
+### Batch mode uploads {#batch-mode}
-This is a simple "rotate" of the filename, with each file having a rot
-distance based on the filename. Rclone stores the distance at the
-beginning of the filename. A file called "hello" may become "53.jgnnq".
+Using batch mode uploads is very important for performance when using
+the Dropbox API. See [the dropbox performance guide](https://developers.dropbox.com/dbx-performance-guide)
+for more info.
-Obfuscation is not a strong encryption of filenames, but hinders
-automated scanning tools picking up on filename patterns. It is an
-intermediate between "off" and "standard" which allows for longer path
-segment names.
+There are 3 modes rclone can use for uploads.
-There is a possibility with some unicode based filenames that the
-obfuscation is weak and may map lower case characters to upper case
-equivalents.
+#### --dropbox-batch-mode off
-Obfuscation cannot be relied upon for strong protection.
+In this mode rclone will not use upload batching. This was the default
+before rclone v1.55. It has the disadvantage that it is very likely to
+encounter `too_many_requests` errors like this
- * file names very lightly obfuscated
- * file names can be longer than standard encryption
- * can use sub paths and copy single files
- * directory structure visible
- * identical files names will have identical uploaded names
+ NOTICE: too_many_requests/.: Too many requests or write operations. Trying again in 15 seconds.
-Cloud storage systems have limits on file name length and
-total path length which rclone is more likely to breach using
-"Standard" file name encryption. Where file names are less than 156
-characters in length issues should not be encountered, irrespective of
-cloud storage provider.
+When rclone receives these it has to wait for 15s or sometimes 300s
+before continuing which really slows down transfers.
-An experimental advanced option `filename_encoding` is now provided to
-address this problem to a certain degree.
-For cloud storage systems with case sensitive file names (e.g. Google Drive),
-`base64` can be used to reduce file name length.
-For cloud storage systems using UTF-16 to store file names internally
-(e.g. OneDrive, Dropbox, Box), `base32768` can be used to drastically reduce
-file name length.
+This will happen especially if `--transfers` is large, so this mode
+isn't recommended except for compatibility or investigating problems.
-An alternative, future rclone file name encryption mode may tolerate
-backend provider path length limits.
+#### --dropbox-batch-mode sync
-### Directory name encryption
+In this mode rclone will batch up uploads to the size specified by
+`--dropbox-batch-size` and commit them together.
-Crypt offers the option of encrypting dir names or leaving them intact.
-There are two options:
+Using this mode means you can use a much higher `--transfers`
+parameter (32 or 64 works fine) without receiving `too_many_requests`
+errors.
-True
+This mode ensures full data integrity.
-Encrypts the whole file path including directory names
-Example:
-`1/12/123.txt` is encrypted to
-`p0e52nreeaj0a5ea7s64m4j72s/l42g6771hnv3an9cgc8cr2n1ng/qgm4avr35m5loi1th53ato71v0`
+Note that there may be a pause when quitting rclone while rclone
+finishes up the last batch using this mode.
-False
+#### --dropbox-batch-mode async
-Only encrypts file names, skips directory names
-Example:
-`1/12/123.txt` is encrypted to
-`1/12/qgm4avr35m5loi1th53ato71v0`
+In this mode rclone will batch up uploads to the size specified by
+`--dropbox-batch-size` and commit them together.
+However it will not wait for the status of the batch to be returned to
+the caller. This means rclone can use a much bigger batch size (much
+bigger than `--transfers`), at the cost of not being able to check the
+status of the upload.
-### Modified time and hashes
+This provides the maximum possible upload speed especially with lots
+of small files, however rclone can't check the file got uploaded
+properly using this mode.
-Crypt stores modification times using the underlying remote so support
-depends on that.
+If you are using this mode then using "rclone check" after the
+transfer completes is recommended. Or you could do an initial transfer
+with `--dropbox-batch-mode async` then do a final transfer with
+`--dropbox-batch-mode sync` (the default).
-Hashes are not stored for crypt. However the data integrity is
-protected by an extremely strong crypto authenticator.
+Note that there may be a pause when quitting rclone while rclone
+finishes up the last batch using this mode.
-Use the `rclone cryptcheck` command to check the
-integrity of an encrypted remote instead of `rclone check` which can't
-check the checksums properly.
### Standard options
-Here are the Standard options specific to crypt (Encrypt/Decrypt a remote).
+Here are the Standard options specific to dropbox (Dropbox).
-#### --crypt-remote
+#### --dropbox-client-id
-Remote to encrypt/decrypt.
+OAuth Client Id.
-Normally should contain a ':' and a path, e.g. "myremote:path/to/dir",
-"myremote:bucket" or maybe "myremote:" (not recommended).
+Leave blank normally.
Properties:
-- Config: remote
-- Env Var: RCLONE_CRYPT_REMOTE
+- Config: client_id
+- Env Var: RCLONE_DROPBOX_CLIENT_ID
- Type: string
-- Required: true
+- Required: false
-#### --crypt-filename-encryption
+#### --dropbox-client-secret
-How to encrypt the filenames.
+OAuth Client Secret.
+
+Leave blank normally.
Properties:
-- Config: filename_encryption
-- Env Var: RCLONE_CRYPT_FILENAME_ENCRYPTION
+- Config: client_secret
+- Env Var: RCLONE_DROPBOX_CLIENT_SECRET
- Type: string
-- Default: "standard"
-- Examples:
- - "standard"
- - Encrypt the filenames.
- - See the docs for the details.
- - "obfuscate"
- - Very simple filename obfuscation.
- - "off"
- - Don't encrypt the file names.
- - Adds a ".bin", or "suffix" extension only.
+- Required: false
-#### --crypt-directory-name-encryption
+### Advanced options
-Option to either encrypt directory names or leave them intact.
+Here are the Advanced options specific to dropbox (Dropbox).
-NB If filename_encryption is "off" then this option will do nothing.
+#### --dropbox-token
+
+OAuth Access Token as a JSON blob.
Properties:
-- Config: directory_name_encryption
-- Env Var: RCLONE_CRYPT_DIRECTORY_NAME_ENCRYPTION
-- Type: bool
-- Default: true
-- Examples:
- - "true"
- - Encrypt directory names.
- - "false"
- - Don't encrypt directory names, leave them intact.
+- Config: token
+- Env Var: RCLONE_DROPBOX_TOKEN
+- Type: string
+- Required: false
-#### --crypt-password
+#### --dropbox-auth-url
-Password or pass phrase for encryption.
+Auth server URL.
-**NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/).
+Leave blank to use the provider defaults.
Properties:
-- Config: password
-- Env Var: RCLONE_CRYPT_PASSWORD
+- Config: auth_url
+- Env Var: RCLONE_DROPBOX_AUTH_URL
- Type: string
-- Required: true
-
-#### --crypt-password2
+- Required: false
-Password or pass phrase for salt.
+#### --dropbox-token-url
-Optional but recommended.
-Should be different to the previous password.
+Token server url.
-**NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/).
+Leave blank to use the provider defaults.
Properties:
-- Config: password2
-- Env Var: RCLONE_CRYPT_PASSWORD2
+- Config: token_url
+- Env Var: RCLONE_DROPBOX_TOKEN_URL
- Type: string
- Required: false
-### Advanced options
+#### --dropbox-chunk-size
-Here are the Advanced options specific to crypt (Encrypt/Decrypt a remote).
+Upload chunk size (< 150Mi).
-#### --crypt-server-side-across-configs
+Any files larger than this will be uploaded in chunks of this size.
-Deprecated: use --server-side-across-configs instead.
+Note that chunks are buffered in memory (one at a time) so rclone can
+deal with retries. Setting this larger will increase the speed
+slightly (at most 10% for 128 MiB in tests) at the cost of using more
+memory. It can be set smaller if you are tight on memory.
-Allow server-side operations (e.g. copy) to work across different crypt configs.
+Properties:
-Normally this option is not what you want, but if you have two crypts
-pointing to the same backend you can use it.
+- Config: chunk_size
+- Env Var: RCLONE_DROPBOX_CHUNK_SIZE
+- Type: SizeSuffix
+- Default: 48Mi
-This can be used, for example, to change file name encryption type
-without re-uploading all the data. Just make two crypt backends
-pointing to two different directories with the single changed
-parameter and use rclone move to move the files between the crypt
-remotes.
+#### --dropbox-impersonate
-Properties:
+Impersonate this user when using a business account.
-- Config: server_side_across_configs
-- Env Var: RCLONE_CRYPT_SERVER_SIDE_ACROSS_CONFIGS
-- Type: bool
-- Default: false
+Note that if you want to use impersonate, you should make sure this
+flag is set when running "rclone config" as this will cause rclone to
+request the "members.read" scope which it won't normally. This is
+needed to lookup a members email address into the internal ID that
+dropbox uses in the API.
-#### --crypt-show-mapping
+Using the "members.read" scope will require a Dropbox Team Admin
+to approve during the OAuth flow.
-For all files listed show how the names encrypt.
+You will have to use your own App (setting your own client_id and
+client_secret) to use this option as currently rclone's default set of
+permissions doesn't include "members.read". This can be added once
+v1.55 or later is in use everywhere.
-If this flag is set then for each file that the remote is asked to
-list, it will log (at level INFO) a line stating the decrypted file
-name and the encrypted file name.
-This is so you can work out which encrypted names are which decrypted
-names just in case you need to do something with the encrypted file
-names, or for debugging purposes.
+Properties:
+
+- Config: impersonate
+- Env Var: RCLONE_DROPBOX_IMPERSONATE
+- Type: string
+- Required: false
+
+#### --dropbox-shared-files
+
+Instructs rclone to work on individual shared files.
+
+In this mode rclone's features are extremely limited - only list (ls, lsl, etc.)
+operations and read operations (e.g. downloading) are supported in this mode.
+All other operations will be disabled.
Properties:
-- Config: show_mapping
-- Env Var: RCLONE_CRYPT_SHOW_MAPPING
+- Config: shared_files
+- Env Var: RCLONE_DROPBOX_SHARED_FILES
- Type: bool
- Default: false
-#### --crypt-no-data-encryption
+#### --dropbox-shared-folders
-Option to either encrypt file data or leave it unencrypted.
+Instructs rclone to work on shared folders.
+
+When this flag is used with no path only the List operation is supported and
+all available shared folders will be listed. If you specify a path the first part
+will be interpreted as the name of shared folder. Rclone will then try to mount this
+shared to the root namespace. On success shared folder rclone proceeds normally.
+The shared folder is now pretty much a normal folder and all normal operations
+are supported.
+
+Note that we don't unmount the shared folder afterwards so the
+--dropbox-shared-folders can be omitted after the first use of a particular
+shared folder.
Properties:
-- Config: no_data_encryption
-- Env Var: RCLONE_CRYPT_NO_DATA_ENCRYPTION
+- Config: shared_folders
+- Env Var: RCLONE_DROPBOX_SHARED_FOLDERS
- Type: bool
- Default: false
-- Examples:
- - "true"
- - Don't encrypt file data, leave it unencrypted.
- - "false"
- - Encrypt file data.
-#### --crypt-pass-bad-blocks
+#### --dropbox-pacer-min-sleep
-If set this will pass bad blocks through as all 0.
+Minimum time to sleep between API calls.
-This should not be set in normal operation, it should only be set if
-trying to recover an encrypted file with errors and it is desired to
-recover as much of the file as possible.
+Properties:
+
+- Config: pacer_min_sleep
+- Env Var: RCLONE_DROPBOX_PACER_MIN_SLEEP
+- Type: Duration
+- Default: 10ms
+
+#### --dropbox-encoding
+
+The encoding for the backend.
+
+See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info.
Properties:
-- Config: pass_bad_blocks
-- Env Var: RCLONE_CRYPT_PASS_BAD_BLOCKS
-- Type: bool
-- Default: false
+- Config: encoding
+- Env Var: RCLONE_DROPBOX_ENCODING
+- Type: Encoding
+- Default: Slash,BackSlash,Del,RightSpace,InvalidUtf8,Dot
-#### --crypt-filename-encoding
+#### --dropbox-batch-mode
-How to encode the encrypted filename to text string.
+Upload file batching sync|async|off.
+
+This sets the batch mode used by rclone.
+
+For full info see [the main docs](https://rclone.org/dropbox/#batch-mode)
+
+This has 3 possible values
+
+- off - no batching
+- sync - batch uploads and check completion (default)
+- async - batch upload and don't check completion
+
+Rclone will close any outstanding batches when it exits which may make
+a delay on quit.
-This option could help with shortening the encrypted filename. The
-suitable option would depend on the way your remote count the filename
-length and if it's case sensitive.
Properties:
-- Config: filename_encoding
-- Env Var: RCLONE_CRYPT_FILENAME_ENCODING
+- Config: batch_mode
+- Env Var: RCLONE_DROPBOX_BATCH_MODE
- Type: string
-- Default: "base32"
-- Examples:
- - "base32"
- - Encode using base32. Suitable for all remote.
- - "base64"
- - Encode using base64. Suitable for case sensitive remote.
- - "base32768"
- - Encode using base32768. Suitable if your remote counts UTF-16 or
- - Unicode codepoint instead of UTF-8 byte length. (Eg. Onedrive, Dropbox)
+- Default: "sync"
-#### --crypt-suffix
+#### --dropbox-batch-size
-If this is set it will override the default suffix of ".bin".
+Max number of files in upload batch.
+
+This sets the batch size of files to upload. It has to be less than 1000.
+
+By default this is 0 which means rclone which calculate the batch size
+depending on the setting of batch_mode.
+
+- batch_mode: async - default batch_size is 100
+- batch_mode: sync - default batch_size is the same as --transfers
+- batch_mode: off - not in use
+
+Rclone will close any outstanding batches when it exits which may make
+a delay on quit.
+
+Setting this is a great idea if you are uploading lots of small files
+as it will make them a lot quicker. You can use --transfers 32 to
+maximise throughput.
-Setting suffix to "none" will result in an empty suffix. This may be useful
-when the path length is critical.
Properties:
-- Config: suffix
-- Env Var: RCLONE_CRYPT_SUFFIX
-- Type: string
-- Default: ".bin"
+- Config: batch_size
+- Env Var: RCLONE_DROPBOX_BATCH_SIZE
+- Type: int
+- Default: 0
-### Metadata
+#### --dropbox-batch-timeout
-Any metadata supported by the underlying remote is read and written.
+Max time to allow an idle upload batch before uploading.
-See the [metadata](https://rclone.org/docs/#metadata) docs for more info.
+If an upload batch is idle for more than this long then it will be
+uploaded.
-## Backend commands
+The default for this is 0 which means rclone will choose a sensible
+default based on the batch_mode in use.
-Here are the commands specific to the crypt backend.
+- batch_mode: async - default batch_timeout is 10s
+- batch_mode: sync - default batch_timeout is 500ms
+- batch_mode: off - not in use
-Run them with
- rclone backend COMMAND remote:
+Properties:
-The help below will explain what arguments each command takes.
+- Config: batch_timeout
+- Env Var: RCLONE_DROPBOX_BATCH_TIMEOUT
+- Type: Duration
+- Default: 0s
-See the [backend](https://rclone.org/commands/rclone_backend/) command for more
-info on how to pass options and arguments.
+#### --dropbox-batch-commit-timeout
-These can be run on a running backend using the rc command
-[backend/command](https://rclone.org/rc/#backend-command).
+Max time to wait for a batch to finish committing
-### encode
+Properties:
-Encode the given filename(s)
+- Config: batch_commit_timeout
+- Env Var: RCLONE_DROPBOX_BATCH_COMMIT_TIMEOUT
+- Type: Duration
+- Default: 10m0s
- rclone backend encode remote: [options] [<arguments>+]
-This encodes the filenames given as arguments returning a list of
-strings of the encoded results.
-Usage Example:
+## Limitations
- rclone backend encode crypt: file1 [file2...]
- rclone rc backend/command command=encode fs=crypt: file1 [file2...]
+Note that Dropbox is case insensitive so you can't have a file called
+"Hello.doc" and one called "hello.doc".
+There are some file names such as `thumbs.db` which Dropbox can't
+store. There is a full list of them in the ["Ignored Files" section
+of this document](https://www.dropbox.com/en/help/145). Rclone will
+issue an error message `File name disallowed - not uploading` if it
+attempts to upload one of those file names, but the sync won't fail.
-### decode
+Some errors may occur if you try to sync copyright-protected files
+because Dropbox has its own [copyright detector](https://techcrunch.com/2014/03/30/how-dropbox-knows-when-youre-sharing-copyrighted-stuff-without-actually-looking-at-your-stuff/) that
+prevents this sort of file being downloaded. This will return the error `ERROR :
+/path/to/your/file: Failed to copy: failed to open source object:
+path/restricted_content/.`
-Decode the given filename(s)
+If you have more than 10,000 files in a directory then `rclone purge
+dropbox:dir` will return the error `Failed to purge: There are too
+many files involved in this operation`. As a work-around do an
+`rclone delete dropbox:dir` followed by an `rclone rmdir dropbox:dir`.
- rclone backend decode remote: [options] [<arguments>+]
+When using `rclone link` you'll need to set `--expire` if using a
+non-personal account otherwise the visibility may not be correct.
+(Note that `--expire` isn't supported on personal accounts). See the
+[forum discussion](https://forum.rclone.org/t/rclone-link-dropbox-permissions/23211) and the
+[dropbox SDK issue](https://github.com/dropbox/dropbox-sdk-go-unofficial/issues/75).
-This decodes the filenames given as arguments returning a list of
-strings of the decoded results. It will return an error if any of the
-inputs are invalid.
+## Get your own Dropbox App ID
-Usage Example:
+When you use rclone with Dropbox in its default configuration you are using rclone's App ID. This is shared between all the rclone users.
- rclone backend decode crypt: encryptedfile1 [encryptedfile2...]
- rclone rc backend/command command=decode fs=crypt: encryptedfile1 [encryptedfile2...]
+Here is how to create your own Dropbox App ID for rclone:
+
+1. Log into the [Dropbox App console](https://www.dropbox.com/developers/apps/create) with your Dropbox Account (It need not
+to be the same account as the Dropbox you want to access)
+
+2. Choose an API => Usually this should be `Dropbox API`
+
+3. Choose the type of access you want to use => `Full Dropbox` or `App Folder`. If you want to use Team Folders, `Full Dropbox` is required ([see here](https://www.dropboxforum.com/t5/Dropbox-API-Support-Feedback/How-to-create-team-folder-inside-my-app-s-folder/m-p/601005/highlight/true#M27911)).
+
+4. Name your App. The app name is global, so you can't use `rclone` for example
+
+5. Click the button `Create App`
+
+6. Switch to the `Permissions` tab. Enable at least the following permissions: `account_info.read`, `files.metadata.write`, `files.content.write`, `files.content.read`, `sharing.write`. The `files.metadata.read` and `sharing.read` checkboxes will be marked too. Click `Submit`
+
+7. Switch to the `Settings` tab. Fill `OAuth2 - Redirect URIs` as `http://localhost:53682/` and click on `Add`
+
+8. Find the `App key` and `App secret` values on the `Settings` tab. Use these values in rclone config to add a new remote or edit an existing remote. The `App key` setting corresponds to `client_id` in rclone config, the `App secret` corresponds to `client_secret`
+
+# Enterprise File Fabric
+
+This backend supports [Storage Made Easy's Enterprise File
+Fabricâ„¢](https://storagemadeeasy.com/about/) which provides a software
+solution to integrate and unify File and Object Storage accessible
+through a global file system.
+
+## Configuration
+
+The initial setup for the Enterprise File Fabric backend involves
+getting a token from the Enterprise File Fabric which you need to
+do in your browser. `rclone config` walks you through it.
+Here is an example of how to make a remote called `remote`. First run:
+ rclone config
+This will guide you through an interactive setup process:
+
+No remotes found, make a new one? n) New remote s) Set configuration password q) Quit config n/s/q> n name> remote Type of storage to configure. Enter a string value. Press Enter for the default (""). Choose a number from below, or type in your own value [snip] XX / Enterprise File Fabric  "filefabric" [snip] Storage> filefabric ** See help for filefabric backend at: https://rclone.org/filefabric/ **
+URL of the Enterprise File Fabric to connect to Enter a string value. Press Enter for the default (""). Choose a number from below, or type in your own value 1 / Storage Made Easy US  "https://storagemadeeasy.com" 2 / Storage Made Easy EU  "https://eu.storagemadeeasy.com" 3 / Connect to your Enterprise File Fabric  "https://yourfabric.smestorage.com" url> https://yourfabric.smestorage.com/ ID of the root folder Leave blank normally.
+Fill in to make rclone start with directory of a given ID.
+Enter a string value. Press Enter for the default (""). root_folder_id> Permanent Authentication Token
+A Permanent Authentication Token can be created in the Enterprise File Fabric, on the users Dashboard under Security, there is an entry you'll see called "My Authentication Tokens". Click the Manage button to create one.
+These tokens are normally valid for several years.
+For more info see: https://docs.storagemadeeasy.com/organisationcloud/api-tokens
+Enter a string value. Press Enter for the default (""). permanent_token> xxxxxxxxxxxxxxx-xxxxxxxxxxxxxxxx Edit advanced config? (y/n) y) Yes n) No (default) y/n> n Remote config -------------------- [remote] type = filefabric url = https://yourfabric.smestorage.com/ permanent_token = xxxxxxxxxxxxxxx-xxxxxxxxxxxxxxxx -------------------- y) Yes this is OK (default) e) Edit this remote d) Delete this remote y/e/d> y
+
+Once configured you can then use `rclone` like this,
-## Backing up an encrypted remote
+List directories in top level of your Enterprise File Fabric
-If you wish to backup an encrypted remote, it is recommended that you use
-`rclone sync` on the encrypted files, and make sure the passwords are
-the same in the new encrypted remote.
+ rclone lsd remote:
-This will have the following advantages
+List all the files in your Enterprise File Fabric
- * `rclone sync` will check the checksums while copying
- * you can use `rclone check` between the encrypted remotes
- * you don't decrypt and encrypt unnecessarily
+ rclone ls remote:
-For example, let's say you have your original remote at `remote:` with
-the encrypted version at `eremote:` with path `remote:crypt`. You
-would then set up the new remote `remote2:` and then the encrypted
-version `eremote2:` with path `remote2:crypt` using the same passwords
-as `eremote:`.
+To copy a local directory to an Enterprise File Fabric directory called backup
-To sync the two remotes you would do
+ rclone copy /home/source remote:backup
- rclone sync --interactive remote:crypt remote2:crypt
+### Modification times and hashes
-And to check the integrity you would do
+The Enterprise File Fabric allows modification times to be set on
+files accurate to 1 second. These will be used to detect whether
+objects need syncing or not.
- rclone check remote:crypt remote2:crypt
+The Enterprise File Fabric does not support any data hashes at this time.
-## File formats
+### Restricted filename characters
-### File encryption
+The [default restricted characters set](https://rclone.org/overview/#restricted-characters)
+will be replaced.
-Files are encrypted 1:1 source file to destination object. The file
-has a header and is divided into chunks.
+Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8),
+as they can't be used in JSON strings.
-#### Header
+### Empty files
- * 8 bytes magic string `RCLONE\x00\x00`
- * 24 bytes Nonce (IV)
+Empty files aren't supported by the Enterprise File Fabric. Rclone will therefore
+upload an empty file as a single space with a mime type of
+`application/vnd.rclone.empty.file` and files with that mime type are
+treated as empty.
-The initial nonce is generated from the operating systems crypto
-strong random number generator. The nonce is incremented for each
-chunk read making sure each nonce is unique for each block written.
-The chance of a nonce being re-used is minuscule. If you wrote an
-exabyte of data (10¹⸠bytes) you would have a probability of
-approximately 2×10â»Â³Â² of re-using a nonce.
+### Root folder ID ###
-#### Chunk
+You can set the `root_folder_id` for rclone. This is the directory
+(identified by its `Folder ID`) that rclone considers to be the root
+of your Enterprise File Fabric.
-Each chunk will contain 64 KiB of data, except for the last one which
-may have less data. The data chunk is in standard NaCl SecretBox
-format. SecretBox uses XSalsa20 and Poly1305 to encrypt and
-authenticate messages.
+Normally you will leave this blank and rclone will determine the
+correct root to use itself.
-Each chunk contains:
+However you can set this to restrict rclone to a specific folder
+hierarchy.
- * 16 Bytes of Poly1305 authenticator
- * 1 - 65536 bytes XSalsa20 encrypted data
+In order to do this you will have to find the `Folder ID` of the
+directory you wish rclone to display. These aren't displayed in the
+web interface, but you can use `rclone lsf` to find them, for example
+
+$ rclone lsf --dirs-only -Fip --csv filefabric: 120673758,Burnt PDFs/ 120673759,My Quick Uploads/ 120673755,My Syncs/ 120673756,My backups/ 120673757,My contacts/ 120673761,S3 Storage/
+
+The ID for "S3 Storage" would be `120673761`.
-64k chunk size was chosen as the best performing chunk size (the
-authenticator takes too much time below this and the performance drops
-off due to cache effects above this). Note that these chunks are
-buffered in memory so they can't be too big.
-This uses a 32 byte (256 bit key) key derived from the user password.
+### Standard options
-#### Examples
+Here are the Standard options specific to filefabric (Enterprise File Fabric).
-1 byte file will encrypt to
+#### --filefabric-url
- * 32 bytes header
- * 17 bytes data chunk
+URL of the Enterprise File Fabric to connect to.
-49 bytes total
+Properties:
-1 MiB (1048576 bytes) file will encrypt to
+- Config: url
+- Env Var: RCLONE_FILEFABRIC_URL
+- Type: string
+- Required: true
+- Examples:
+ - "https://storagemadeeasy.com"
+ - Storage Made Easy US
+ - "https://eu.storagemadeeasy.com"
+ - Storage Made Easy EU
+ - "https://yourfabric.smestorage.com"
+ - Connect to your Enterprise File Fabric
- * 32 bytes header
- * 16 chunks of 65568 bytes
+#### --filefabric-root-folder-id
-1049120 bytes total (a 0.05% overhead). This is the overhead for big
-files.
+ID of the root folder.
-### Name encryption
+Leave blank normally.
-File names are encrypted segment by segment - the path is broken up
-into `/` separated strings and these are encrypted individually.
+Fill in to make rclone start with directory of a given ID.
-File segments are padded using PKCS#7 to a multiple of 16 bytes
-before encryption.
-They are then encrypted with EME using AES with 256 bit key. EME
-(ECB-Mix-ECB) is a wide-block encryption mode presented in the 2003
-paper "A Parallelizable Enciphering Mode" by Halevi and Rogaway.
+Properties:
-This makes for deterministic encryption which is what we want - the
-same filename must encrypt to the same thing otherwise we can't find
-it on the cloud storage system.
+- Config: root_folder_id
+- Env Var: RCLONE_FILEFABRIC_ROOT_FOLDER_ID
+- Type: string
+- Required: false
-This means that
+#### --filefabric-permanent-token
- * filenames with the same name will encrypt the same
- * filenames which start the same won't have a common prefix
+Permanent Authentication Token.
-This uses a 32 byte key (256 bits) and a 16 byte (128 bits) IV both of
-which are derived from the user password.
+A Permanent Authentication Token can be created in the Enterprise File
+Fabric, on the users Dashboard under Security, there is an entry
+you'll see called "My Authentication Tokens". Click the Manage button
+to create one.
-After encryption they are written out using a modified version of
-standard `base32` encoding as described in RFC4648. The standard
-encoding is modified in two ways:
+These tokens are normally valid for several years.
- * it becomes lower case (no-one likes upper case filenames!)
- * we strip the padding character `=`
+For more info see: https://docs.storagemadeeasy.com/organisationcloud/api-tokens
-`base32` is used rather than the more efficient `base64` so rclone can be
-used on case insensitive remotes (e.g. Windows, Amazon Drive).
-### Key derivation
+Properties:
-Rclone uses `scrypt` with parameters `N=16384, r=8, p=1` with an
-optional user supplied salt (password2) to derive the 32+32+16 = 80
-bytes of key material required. If the user doesn't supply a salt
-then rclone uses an internal one.
+- Config: permanent_token
+- Env Var: RCLONE_FILEFABRIC_PERMANENT_TOKEN
+- Type: string
+- Required: false
-`scrypt` makes it impractical to mount a dictionary attack on rclone
-encrypted data. For full protection against this you should always use
-a salt.
+### Advanced options
-## SEE ALSO
+Here are the Advanced options specific to filefabric (Enterprise File Fabric).
-* [rclone cryptdecode](https://rclone.org/commands/rclone_cryptdecode/) - Show forward/reverse mapping of encrypted filenames
+#### --filefabric-token
-# Compress
+Session Token.
-## Warning
+This is a session token which rclone caches in the config file. It is
+usually valid for 1 hour.
-This remote is currently **experimental**. Things may break and data may be lost. Anything you do with this remote is
-at your own risk. Please understand the risks associated with using experimental code and don't use this remote in
-critical applications.
+Don't set this value - rclone will set it automatically.
-The `Compress` remote adds compression to another remote. It is best used with remotes containing
-many large compressible files.
-## Configuration
+Properties:
-To use this remote, all you need to do is specify another remote and a compression mode to use:
-
-Current remotes:
-Name Type ==== ==== remote_to_press sometype
-
-- Edit existing remote $ rclone config
-- New remote
-- Delete remote
-- Rename remote
-- Copy remote
-- Set configuration password
-- Quit config e/n/d/r/c/s/q> n name> compress ... 8 / Compress a remote  "compress" ... Storage> compress ** See help for compress backend at: https://rclone.org/compress/ **
-
-Remote to compress. Enter a string value. Press Enter for the default (""). remote> remote_to_press:subdir Compression mode. Enter a string value. Press Enter for the default ("gzip"). Choose a number from below, or type in your own value 1 / Gzip compression balanced for speed and compression strength. Â "gzip" compression_mode> gzip Edit advanced config? (y/n) y) Yes n) No (default) y/n> n Remote config -------------------- [compress] type = compress remote = remote_to_press:subdir compression_mode = gzip -------------------- y) Yes this is OK (default) e) Edit this remote d) Delete this remote y/e/d> y
-
-### Compression Modes
+- Config: token
+- Env Var: RCLONE_FILEFABRIC_TOKEN
+- Type: string
+- Required: false
-Currently only gzip compression is supported. It provides a decent balance between speed and size and is well
-supported by other applications. Compression strength can further be configured via an advanced setting where 0 is no
-compression and 9 is strongest compression.
+#### --filefabric-token-expiry
-### File types
+Token expiry time.
-If you open a remote wrapped by compress, you will see that there are many files with an extension corresponding to
-the compression algorithm you chose. These files are standard files that can be opened by various archive programs,
-but they have some hidden metadata that allows them to be used by rclone.
-While you may download and decompress these files at will, do **not** manually delete or rename files. Files without
-correct metadata files will not be recognized by rclone.
+Don't set this value - rclone will set it automatically.
-### File names
-The compressed files will be named `*.###########.gz` where `*` is the base file and the `#` part is base64 encoded
-size of the uncompressed file. The file names should not be changed by anything other than the rclone compression backend.
+Properties:
+- Config: token_expiry
+- Env Var: RCLONE_FILEFABRIC_TOKEN_EXPIRY
+- Type: string
+- Required: false
-### Standard options
+#### --filefabric-version
-Here are the Standard options specific to compress (Compress a remote).
+Version read from the file fabric.
-#### --compress-remote
+Don't set this value - rclone will set it automatically.
-Remote to compress.
Properties:
-- Config: remote
-- Env Var: RCLONE_COMPRESS_REMOTE
+- Config: version
+- Env Var: RCLONE_FILEFABRIC_VERSION
- Type: string
-- Required: true
+- Required: false
-#### --compress-mode
+#### --filefabric-encoding
-Compression mode.
+The encoding for the backend.
+
+See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info.
Properties:
-- Config: mode
-- Env Var: RCLONE_COMPRESS_MODE
-- Type: string
-- Default: "gzip"
-- Examples:
- - "gzip"
- - Standard gzip compression with fastest parameters.
+- Config: encoding
+- Env Var: RCLONE_FILEFABRIC_ENCODING
+- Type: Encoding
+- Default: Slash,Del,Ctl,InvalidUtf8,Dot
-### Advanced options
-Here are the Advanced options specific to compress (Compress a remote).
-#### --compress-level
+# FTP
-GZIP compression level (-2 to 9).
+FTP is the File Transfer Protocol. Rclone FTP support is provided using the
+[github.com/jlaffaye/ftp](https://godoc.org/github.com/jlaffaye/ftp)
+package.
-Generally -1 (default, equivalent to 5) is recommended.
-Levels 1 to 9 increase compression at the cost of speed. Going past 6
-generally offers very little return.
+[Limitations of Rclone's FTP backend](#limitations)
-Level -2 uses Huffman encoding only. Only use if you know what you
-are doing.
-Level 0 turns off compression.
+Paths are specified as `remote:path`. If the path does not begin with
+a `/` it is relative to the home directory of the user. An empty path
+`remote:` refers to the user's home directory.
-Properties:
+## Configuration
-- Config: level
-- Env Var: RCLONE_COMPRESS_LEVEL
-- Type: int
-- Default: -1
+To create an FTP configuration named `remote`, run
-#### --compress-ram-cache-limit
+ rclone config
-Some remotes don't allow the upload of files with unknown size.
-In this case the compressed file will need to be cached to determine
-it's size.
+Rclone config guides you through an interactive setup process. A minimal
+rclone FTP remote definition only requires host, username and password.
+For an anonymous FTP server, see [below](#anonymous-ftp).
+
+No remotes found, make a new one? n) New remote r) Rename remote c) Copy remote s) Set configuration password q) Quit config n/r/c/s/q> n name> remote Type of storage to configure. Enter a string value. Press Enter for the default (""). Choose a number from below, or type in your own value [snip] XX / FTP Â "ftp" [snip] Storage> ftp ** See help for ftp backend at: https://rclone.org/ftp/ **
+FTP host to connect to Enter a string value. Press Enter for the default (""). Choose a number from below, or type in your own value 1 / Connect to ftp.example.com  "ftp.example.com" host> ftp.example.com FTP username Enter a string value. Press Enter for the default ("$USER"). user> FTP port number Enter a signed integer. Press Enter for the default (21). port> FTP password y) Yes type in my own password g) Generate random password y/g> y Enter the password: password: Confirm the password: password: Use FTP over TLS (Implicit) Enter a boolean value (true or false). Press Enter for the default ("false"). tls> Use FTP over TLS (Explicit) Enter a boolean value (true or false). Press Enter for the default ("false"). explicit_tls> Remote config -------------------- [remote] type = ftp host = ftp.example.com pass = *** ENCRYPTED *** -------------------- y) Yes this is OK e) Edit this remote d) Delete this remote y/e/d> y
+
+To see all directories in the home directory of `remote`
-Files smaller than this limit will be cached in RAM, files larger than
-this limit will be cached on disk.
+ rclone lsd remote:
-Properties:
+Make a new directory
-- Config: ram_cache_limit
-- Env Var: RCLONE_COMPRESS_RAM_CACHE_LIMIT
-- Type: SizeSuffix
-- Default: 20Mi
+ rclone mkdir remote:path/to/directory
-### Metadata
+List the contents of a directory
-Any metadata supported by the underlying remote is read and written.
+ rclone ls remote:path/to/directory
-See the [metadata](https://rclone.org/docs/#metadata) docs for more info.
+Sync `/home/local/directory` to the remote directory, deleting any
+excess files in the directory.
+ rclone sync --interactive /home/local/directory remote:directory
+### Anonymous FTP
-# Combine
+When connecting to a FTP server that allows anonymous login, you can use the
+special "anonymous" username. Traditionally, this user account accepts any
+string as a password, although it is common to use either the password
+"anonymous" or "guest". Some servers require the use of a valid e-mail
+address as password.
-The `combine` backend joins remotes together into a single directory
-tree.
+Using [on-the-fly](#backend-path-to-dir) or
+[connection string](https://rclone.org/docs/#connection-strings) remotes makes it easy to access
+such servers, without requiring any configuration in advance. The following
+are examples of that:
-For example you might have a remote for images on one provider:
-
-$ rclone tree s3:imagesbucket / ├── image1.jpg └── image2.jpg
-
-And a remote for files on another:
-
-$ rclone tree drive:important/files / ├── file1.txt └── file2.txt
-
-The `combine` backend can join these together into a synthetic
-directory structure like this:
-
-$ rclone tree combined: / ├── files │ ├── file1.txt │ └── file2.txt └── images ├── image1.jpg └── image2.jpg
-
-You'd do this by specifying an `upstreams` parameter in the config
-like this
+ rclone lsf :ftp: --ftp-host=speedtest.tele2.net --ftp-user=anonymous --ftp-pass=$(rclone obscure dummy)
+ rclone lsf :ftp,host=speedtest.tele2.net,user=anonymous,pass=$(rclone obscure dummy):
- upstreams = images=s3:imagesbucket files=drive:important/files
+The above examples work in Linux shells and in PowerShell, but not Windows
+Command Prompt. They execute the [rclone obscure](https://rclone.org/commands/rclone_obscure/)
+command to create a password string in the format required by the
+[pass](#ftp-pass) option. The following examples are exactly the same, except use
+an already obscured string representation of the same password "dummy", and
+therefore works even in Windows Command Prompt:
-During the initial setup with `rclone config` you will specify the
-upstreams remotes as a space separated list. The upstream remotes can
-either be a local paths or other remotes.
+ rclone lsf :ftp: --ftp-host=speedtest.tele2.net --ftp-user=anonymous --ftp-pass=IXs2wc8OJOz7SYLBk47Ji1rHTmxM
+ rclone lsf :ftp,host=speedtest.tele2.net,user=anonymous,pass=IXs2wc8OJOz7SYLBk47Ji1rHTmxM:
-## Configuration
+### Implicit TLS
-Here is an example of how to make a combine called `remote` for the
-example above. First run:
+Rlone FTP supports implicit FTP over TLS servers (FTPS). This has to
+be enabled in the FTP backend config for the remote, or with
+[`--ftp-tls`](#ftp-tls). The default FTPS port is `990`, not `21` and
+can be set with [`--ftp-port`](#ftp-port).
- rclone config
+### Restricted filename characters
-This will guide you through an interactive setup process:
-
-No remotes found, make a new one? n) New remote s) Set configuration password q) Quit config n/s/q> n name> remote Option Storage. Type of storage to configure. Choose a number from below, or type in your own value. ... XX / Combine several remotes into one  (combine) ... Storage> combine Option upstreams. Upstreams for combining These should be in the form dir=remote:path dir2=remote2:path Where before the = is specified the root directory and after is the remote to put there. Embedded spaces can be added using quotes "dir=remote:path with space" "dir2=remote2:path with space" Enter a fs.SpaceSepList value. upstreams> images=s3:imagesbucket files=drive:important/files -------------------- [remote] type = combine upstreams = images=s3:imagesbucket files=drive:important/files -------------------- y) Yes this is OK (default) e) Edit this remote d) Delete this remote y/e/d> y
-
-### Configuring for Google Drive Shared Drives
+In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters)
+the following characters are also replaced:
-Rclone has a convenience feature for making a combine backend for all
-the shared drives you have access to.
+File names cannot end with the following characters. Replacement is
+limited to the last character in a file name:
-Assuming your main (non shared drive) Google drive remote is called
-`drive:` you would run
+| Character | Value | Replacement |
+| --------- |:-----:|:-----------:|
+| SP | 0x20 | â |
- rclone backend -o config drives drive:
+Not all FTP servers can have all characters in file names, for example:
-This would produce something like this:
+| FTP Server| Forbidden characters |
+| --------- |:--------------------:|
+| proftpd | `*` |
+| pureftpd | `\ [ ]` |
- [My Drive]
- type = alias
- remote = drive,team_drive=0ABCDEF-01234567890,root_folder_id=:
+This backend's interactive configuration wizard provides a selection of
+sensible encoding settings for major FTP servers: ProFTPd, PureFTPd, VsFTPd.
+Just hit a selection number when prompted.
- [Test Drive]
- type = alias
- remote = drive,team_drive=0ABCDEFabcdefghijkl,root_folder_id=:
- [AllDrives]
- type = combine
- upstreams = "My Drive=My Drive:" "Test Drive=Test Drive:"
+### Standard options
-If you then add that config to your config file (find it with `rclone
-config file`) then you can access all the shared drives in one place
-with the `AllDrives:` remote.
+Here are the Standard options specific to ftp (FTP).
-See [the Google Drive docs](https://rclone.org/drive/#drives) for full info.
+#### --ftp-host
+FTP host to connect to.
-### Standard options
+E.g. "ftp.example.com".
-Here are the Standard options specific to combine (Combine several remotes into one).
+Properties:
-#### --combine-upstreams
+- Config: host
+- Env Var: RCLONE_FTP_HOST
+- Type: string
+- Required: true
-Upstreams for combining
+#### --ftp-user
-These should be in the form
+FTP username.
- dir=remote:path dir2=remote2:path
+Properties:
-Where before the = is specified the root directory and after is the remote to
-put there.
+- Config: user
+- Env Var: RCLONE_FTP_USER
+- Type: string
+- Default: "$USER"
-Embedded spaces can be added using quotes
+#### --ftp-port
- "dir=remote:path with space" "dir2=remote2:path with space"
+FTP port number.
+
+Properties:
+
+- Config: port
+- Env Var: RCLONE_FTP_PORT
+- Type: int
+- Default: 21
+
+#### --ftp-pass
+FTP password.
+**NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/).
Properties:
-- Config: upstreams
-- Env Var: RCLONE_COMBINE_UPSTREAMS
-- Type: SpaceSepList
-- Default:
+- Config: pass
+- Env Var: RCLONE_FTP_PASS
+- Type: string
+- Required: false
-### Metadata
+#### --ftp-tls
-Any metadata supported by the underlying remote is read and written.
+Use Implicit FTPS (FTP over TLS).
-See the [metadata](https://rclone.org/docs/#metadata) docs for more info.
+When using implicit FTP over TLS the client connects using TLS
+right from the start which breaks compatibility with
+non-TLS-aware servers. This is usually served over port 990 rather
+than port 21. Cannot be used in combination with explicit FTPS.
+Properties:
+- Config: tls
+- Env Var: RCLONE_FTP_TLS
+- Type: bool
+- Default: false
-# Dropbox
+#### --ftp-explicit-tls
-Paths are specified as `remote:path`
+Use Explicit FTPS (FTP over TLS).
-Dropbox paths may be as deep as required, e.g.
-`remote:directory/subdirectory`.
+When using explicit FTP over TLS the client explicitly requests
+security from the server in order to upgrade a plain text connection
+to an encrypted one. Cannot be used in combination with implicit FTPS.
-## Configuration
+Properties:
-The initial setup for dropbox involves getting a token from Dropbox
-which you need to do in your browser. `rclone config` walks you
-through it.
+- Config: explicit_tls
+- Env Var: RCLONE_FTP_EXPLICIT_TLS
+- Type: bool
+- Default: false
-Here is an example of how to make a remote called `remote`. First run:
+### Advanced options
- rclone config
+Here are the Advanced options specific to ftp (FTP).
-This will guide you through an interactive setup process:
-
-
-- New remote
-- Delete remote
-- Quit config e/n/d/q> n name> remote Type of storage to configure. Choose a number from below, or type in your own value [snip] XX / Dropbox  "dropbox" [snip] Storage> dropbox Dropbox App Key - leave blank normally. app_key> Dropbox App Secret - leave blank normally. app_secret> Remote config Please visit: https://www.dropbox.com/1/oauth2/authorize?client_id=XXXXXXXXXXXXXXX&response_type=code Enter the code: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX_XXXXXXXXXX -------------------- [remote] app_key = app_secret = token = XXXXXXXXXXXXXXXXXXXXXXXXXXXXX_XXXX_XXXXXXXXXXXXXXXXXXXXXXXXXXXXX --------------------
-- Yes this is OK
-- Edit this remote
-- Delete this remote y/e/d> y
-
-
-See the [remote setup docs](https://rclone.org/remote_setup/) for how to set it up on a
-machine with no Internet browser available.
+#### --ftp-concurrency
-Note that rclone runs a webserver on your local machine to collect the
-token as returned from Dropbox. This only
-runs from the moment it opens your browser to the moment you get back
-the verification code. This is on `http://127.0.0.1:53682/` and it
-may require you to unblock it temporarily if you are running a host
-firewall, or use manual mode.
+Maximum number of FTP simultaneous connections, 0 for unlimited.
+
+Note that setting this is very likely to cause deadlocks so it should
+be used with care.
-You can then use it like this,
+If you are doing a sync or copy then make sure concurrency is one more
+than the sum of `--transfers` and `--checkers`.
-List directories in top level of your dropbox
+If you use `--check-first` then it just needs to be one more than the
+maximum of `--checkers` and `--transfers`.
- rclone lsd remote:
+So for `concurrency 3` you'd use `--checkers 2 --transfers 2
+--check-first` or `--checkers 1 --transfers 1`.
-List all the files in your dropbox
- rclone ls remote:
-To copy a local directory to a dropbox directory called backup
+Properties:
- rclone copy /home/source remote:backup
+- Config: concurrency
+- Env Var: RCLONE_FTP_CONCURRENCY
+- Type: int
+- Default: 0
-### Dropbox for business
+#### --ftp-no-check-certificate
-Rclone supports Dropbox for business and Team Folders.
+Do not verify the TLS certificate of the server.
-When using Dropbox for business `remote:` and `remote:path/to/file`
-will refer to your personal folder.
+Properties:
-If you wish to see Team Folders you must use a leading `/` in the
-path, so `rclone lsd remote:/` will refer to the root and show you all
-Team Folders and your User Folder.
+- Config: no_check_certificate
+- Env Var: RCLONE_FTP_NO_CHECK_CERTIFICATE
+- Type: bool
+- Default: false
-You can then use team folders like this `remote:/TeamFolder` and
-`remote:/TeamFolder/path/to/file`.
+#### --ftp-disable-epsv
-A leading `/` for a Dropbox personal account will do nothing, but it
-will take an extra HTTP transaction so it should be avoided.
+Disable using EPSV even if server advertises support.
-### Modified time and Hashes
+Properties:
-Dropbox supports modified times, but the only way to set a
-modification time is to re-upload the file.
+- Config: disable_epsv
+- Env Var: RCLONE_FTP_DISABLE_EPSV
+- Type: bool
+- Default: false
-This means that if you uploaded your data with an older version of
-rclone which didn't support the v2 API and modified times, rclone will
-decide to upload all your old data to fix the modification times. If
-you don't want this to happen use `--size-only` or `--checksum` flag
-to stop it.
+#### --ftp-disable-mlsd
-Dropbox supports [its own hash
-type](https://www.dropbox.com/developers/reference/content-hash) which
-is checked for all transfers.
+Disable using MLSD even if server advertises support.
-### Restricted filename characters
+Properties:
-| Character | Value | Replacement |
-| --------- |:-----:|:-----------:|
-| NUL | 0x00 | †|
-| / | 0x2F | ï¼ |
-| DEL | 0x7F | â¡ |
-| \ | 0x5C | ï¼¼ |
+- Config: disable_mlsd
+- Env Var: RCLONE_FTP_DISABLE_MLSD
+- Type: bool
+- Default: false
-File names can also not end with the following characters.
-These only get replaced if they are the last character in the name:
+#### --ftp-disable-utf8
-| Character | Value | Replacement |
-| --------- |:-----:|:-----------:|
-| SP | 0x20 | â |
+Disable using UTF-8 even if server advertises support.
-Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8),
-as they can't be used in JSON strings.
+Properties:
-### Batch mode uploads {#batch-mode}
+- Config: disable_utf8
+- Env Var: RCLONE_FTP_DISABLE_UTF8
+- Type: bool
+- Default: false
-Using batch mode uploads is very important for performance when using
-the Dropbox API. See [the dropbox performance guide](https://developers.dropbox.com/dbx-performance-guide)
-for more info.
+#### --ftp-writing-mdtm
-There are 3 modes rclone can use for uploads.
+Use MDTM to set modification time (VsFtpd quirk)
-#### --dropbox-batch-mode off
+Properties:
-In this mode rclone will not use upload batching. This was the default
-before rclone v1.55. It has the disadvantage that it is very likely to
-encounter `too_many_requests` errors like this
+- Config: writing_mdtm
+- Env Var: RCLONE_FTP_WRITING_MDTM
+- Type: bool
+- Default: false
- NOTICE: too_many_requests/.: Too many requests or write operations. Trying again in 15 seconds.
+#### --ftp-force-list-hidden
-When rclone receives these it has to wait for 15s or sometimes 300s
-before continuing which really slows down transfers.
+Use LIST -a to force listing of hidden files and folders. This will disable the use of MLSD.
-This will happen especially if `--transfers` is large, so this mode
-isn't recommended except for compatibility or investigating problems.
+Properties:
-#### --dropbox-batch-mode sync
+- Config: force_list_hidden
+- Env Var: RCLONE_FTP_FORCE_LIST_HIDDEN
+- Type: bool
+- Default: false
-In this mode rclone will batch up uploads to the size specified by
-`--dropbox-batch-size` and commit them together.
+#### --ftp-idle-timeout
-Using this mode means you can use a much higher `--transfers`
-parameter (32 or 64 works fine) without receiving `too_many_requests`
-errors.
+Max time before closing idle connections.
-This mode ensures full data integrity.
+If no connections have been returned to the connection pool in the time
+given, rclone will empty the connection pool.
-Note that there may be a pause when quitting rclone while rclone
-finishes up the last batch using this mode.
+Set to 0 to keep connections indefinitely.
-#### --dropbox-batch-mode async
-In this mode rclone will batch up uploads to the size specified by
-`--dropbox-batch-size` and commit them together.
+Properties:
-However it will not wait for the status of the batch to be returned to
-the caller. This means rclone can use a much bigger batch size (much
-bigger than `--transfers`), at the cost of not being able to check the
-status of the upload.
+- Config: idle_timeout
+- Env Var: RCLONE_FTP_IDLE_TIMEOUT
+- Type: Duration
+- Default: 1m0s
-This provides the maximum possible upload speed especially with lots
-of small files, however rclone can't check the file got uploaded
-properly using this mode.
+#### --ftp-close-timeout
-If you are using this mode then using "rclone check" after the
-transfer completes is recommended. Or you could do an initial transfer
-with `--dropbox-batch-mode async` then do a final transfer with
-`--dropbox-batch-mode sync` (the default).
+Maximum time to wait for a response to close.
-Note that there may be a pause when quitting rclone while rclone
-finishes up the last batch using this mode.
+Properties:
+- Config: close_timeout
+- Env Var: RCLONE_FTP_CLOSE_TIMEOUT
+- Type: Duration
+- Default: 1m0s
+#### --ftp-tls-cache-size
-### Standard options
+Size of TLS session cache for all control and data connections.
-Here are the Standard options specific to dropbox (Dropbox).
+TLS cache allows to resume TLS sessions and reuse PSK between connections.
+Increase if default size is not enough resulting in TLS resumption errors.
+Enabled by default. Use 0 to disable.
-#### --dropbox-client-id
+Properties:
-OAuth Client Id.
+- Config: tls_cache_size
+- Env Var: RCLONE_FTP_TLS_CACHE_SIZE
+- Type: int
+- Default: 32
-Leave blank normally.
+#### --ftp-disable-tls13
-Properties:
+Disable TLS 1.3 (workaround for FTP servers with buggy TLS)
-- Config: client_id
-- Env Var: RCLONE_DROPBOX_CLIENT_ID
-- Type: string
-- Required: false
+Properties:
-#### --dropbox-client-secret
+- Config: disable_tls13
+- Env Var: RCLONE_FTP_DISABLE_TLS13
+- Type: bool
+- Default: false
-OAuth Client Secret.
+#### --ftp-shut-timeout
-Leave blank normally.
+Maximum time to wait for data connection closing status.
Properties:
-- Config: client_secret
-- Env Var: RCLONE_DROPBOX_CLIENT_SECRET
-- Type: string
-- Required: false
+- Config: shut_timeout
+- Env Var: RCLONE_FTP_SHUT_TIMEOUT
+- Type: Duration
+- Default: 1m0s
-### Advanced options
+#### --ftp-ask-password
-Here are the Advanced options specific to dropbox (Dropbox).
+Allow asking for FTP password when needed.
-#### --dropbox-token
+If this is set and no password is supplied then rclone will ask for a password
-OAuth Access Token as a JSON blob.
Properties:
-- Config: token
-- Env Var: RCLONE_DROPBOX_TOKEN
-- Type: string
-- Required: false
-
-#### --dropbox-auth-url
+- Config: ask_password
+- Env Var: RCLONE_FTP_ASK_PASSWORD
+- Type: bool
+- Default: false
-Auth server URL.
+#### --ftp-socks-proxy
-Leave blank to use the provider defaults.
+Socks 5 proxy host.
+
+ Supports the format user:pass@host:port, user@host:port, host:port.
+
+ Example:
+
+ myUser:myPass@localhost:9005
+
Properties:
-- Config: auth_url
-- Env Var: RCLONE_DROPBOX_AUTH_URL
+- Config: socks_proxy
+- Env Var: RCLONE_FTP_SOCKS_PROXY
- Type: string
- Required: false
-#### --dropbox-token-url
+#### --ftp-encoding
-Token server url.
+The encoding for the backend.
-Leave blank to use the provider defaults.
+See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info.
Properties:
-- Config: token_url
-- Env Var: RCLONE_DROPBOX_TOKEN_URL
-- Type: string
-- Required: false
+- Config: encoding
+- Env Var: RCLONE_FTP_ENCODING
+- Type: Encoding
+- Default: Slash,Del,Ctl,RightSpace,Dot
+- Examples:
+ - "Asterisk,Ctl,Dot,Slash"
+ - ProFTPd can't handle '*' in file names
+ - "BackSlash,Ctl,Del,Dot,RightSpace,Slash,SquareBracket"
+ - PureFTPd can't handle '[]' or '*' in file names
+ - "Ctl,LeftPeriod,Slash"
+ - VsFTPd can't handle file names starting with dot
-#### --dropbox-chunk-size
-Upload chunk size (< 150Mi).
-Any files larger than this will be uploaded in chunks of this size.
+## Limitations
-Note that chunks are buffered in memory (one at a time) so rclone can
-deal with retries. Setting this larger will increase the speed
-slightly (at most 10% for 128 MiB in tests) at the cost of using more
-memory. It can be set smaller if you are tight on memory.
+FTP servers acting as rclone remotes must support `passive` mode.
+The mode cannot be configured as `passive` is the only supported one.
+Rclone's FTP implementation is not compatible with `active` mode
+as [the library it uses doesn't support it](https://github.com/jlaffaye/ftp/issues/29).
+This will likely never be supported due to security concerns.
-Properties:
+Rclone's FTP backend does not support any checksums but can compare
+file sizes.
-- Config: chunk_size
-- Env Var: RCLONE_DROPBOX_CHUNK_SIZE
-- Type: SizeSuffix
-- Default: 48Mi
+`rclone about` is not supported by the FTP backend. Backends without
+this capability cannot determine free space for an rclone mount or
+use policy `mfs` (most free space) as a member of an rclone union
+remote.
-#### --dropbox-impersonate
+See [List of backends that do not support rclone about](https://rclone.org/overview/#optional-features) and [rclone about](https://rclone.org/commands/rclone_about/)
-Impersonate this user when using a business account.
+The implementation of : `--dump headers`,
+`--dump bodies`, `--dump auth` for debugging isn't the same as
+for rclone HTTP based backends - it has less fine grained control.
-Note that if you want to use impersonate, you should make sure this
-flag is set when running "rclone config" as this will cause rclone to
-request the "members.read" scope which it won't normally. This is
-needed to lookup a members email address into the internal ID that
-dropbox uses in the API.
+`--timeout` isn't supported (but `--contimeout` is).
-Using the "members.read" scope will require a Dropbox Team Admin
-to approve during the OAuth flow.
+`--bind` isn't supported.
-You will have to use your own App (setting your own client_id and
-client_secret) to use this option as currently rclone's default set of
-permissions doesn't include "members.read". This can be added once
-v1.55 or later is in use everywhere.
+Rclone's FTP backend could support server-side move but does not
+at present.
+The `ftp_proxy` environment variable is not currently supported.
-Properties:
+### Modification times
-- Config: impersonate
-- Env Var: RCLONE_DROPBOX_IMPERSONATE
-- Type: string
-- Required: false
+File modification time (timestamps) is supported to 1 second resolution
+for major FTP servers: ProFTPd, PureFTPd, VsFTPd, and FileZilla FTP server.
+The `VsFTPd` server has non-standard implementation of time related protocol
+commands and needs a special configuration setting: `writing_mdtm = true`.
-#### --dropbox-shared-files
+Support for precise file time with other FTP servers varies depending on what
+protocol extensions they advertise. If all the `MLSD`, `MDTM` and `MFTM`
+extensions are present, rclone will use them together to provide precise time.
+Otherwise the times you see on the FTP server through rclone are those of the
+last file upload.
-Instructs rclone to work on individual shared files.
+You can use the following command to check whether rclone can use precise time
+with your FTP server: `rclone backend features your_ftp_remote:` (the trailing
+colon is important). Look for the number in the line tagged by `Precision`
+designating the remote time precision expressed as nanoseconds. A value of
+`1000000000` means that file time precision of 1 second is available.
+A value of `3153600000000000000` (or another large number) means "unsupported".
-In this mode rclone's features are extremely limited - only list (ls, lsl, etc.)
-operations and read operations (e.g. downloading) are supported in this mode.
-All other operations will be disabled.
+# Google Cloud Storage
-Properties:
+Paths are specified as `remote:bucket` (or `remote:` for the `lsd`
+command.) You may put subdirectories in too, e.g. `remote:bucket/path/to/dir`.
-- Config: shared_files
-- Env Var: RCLONE_DROPBOX_SHARED_FILES
-- Type: bool
-- Default: false
+## Configuration
-#### --dropbox-shared-folders
+The initial setup for google cloud storage involves getting a token from Google Cloud Storage
+which you need to do in your browser. `rclone config` walks you
+through it.
-Instructs rclone to work on shared folders.
-
-When this flag is used with no path only the List operation is supported and
-all available shared folders will be listed. If you specify a path the first part
-will be interpreted as the name of shared folder. Rclone will then try to mount this
-shared to the root namespace. On success shared folder rclone proceeds normally.
-The shared folder is now pretty much a normal folder and all normal operations
-are supported.
+Here is an example of how to make a remote called `remote`. First run:
-Note that we don't unmount the shared folder afterwards so the
---dropbox-shared-folders can be omitted after the first use of a particular
-shared folder.
+ rclone config
-Properties:
+This will guide you through an interactive setup process:
+
+
+- New remote
+- Delete remote
+- Quit config e/n/d/q> n name> remote Type of storage to configure. Choose a number from below, or type in your own value [snip] XX / Google Cloud Storage (this is not Google Drive)  "google cloud storage" [snip] Storage> google cloud storage Google Application Client Id - leave blank normally. client_id> Google Application Client Secret - leave blank normally. client_secret> Project number optional - needed only for list/create/delete buckets - see your developer console. project_number> 12345678 Service Account Credentials JSON file path - needed only if you want use SA instead of interactive login. service_account_file> Access Control List for new objects. Choose a number from below, or type in your own value 1 / Object owner gets OWNER access, and all Authenticated Users get READER access.  "authenticatedRead" 2 / Object owner gets OWNER access, and project team owners get OWNER access.  "bucketOwnerFullControl" 3 / Object owner gets OWNER access, and project team owners get READER access.  "bucketOwnerRead" 4 / Object owner gets OWNER access [default if left blank].  "private" 5 / Object owner gets OWNER access, and project team members get access according to their roles.  "projectPrivate" 6 / Object owner gets OWNER access, and all Users get READER access.  "publicRead" object_acl> 4 Access Control List for new buckets. Choose a number from below, or type in your own value 1 / Project team owners get OWNER access, and all Authenticated Users get READER access.  "authenticatedRead" 2 / Project team owners get OWNER access [default if left blank].  "private" 3 / Project team members get access according to their roles.  "projectPrivate" 4 / Project team owners get OWNER access, and all Users get READER access.  "publicRead" 5 / Project team owners get OWNER access, and all Users get WRITER access.  "publicReadWrite" bucket_acl> 2 Location for the newly created buckets. Choose a number from below, or type in your own value 1 / Empty for default location (US).  "" 2 / Multi-regional location for Asia.  "asia" 3 / Multi-regional location for Europe.  "eu" 4 / Multi-regional location for United States.  "us" 5 / Taiwan.  "asia-east1" 6 / Tokyo.  "asia-northeast1" 7 / Singapore.  "asia-southeast1" 8 / Sydney.  "australia-southeast1" 9 / Belgium.  "europe-west1" 10 / London.  "europe-west2" 11 / Iowa.  "us-central1" 12 / South Carolina.  "us-east1" 13 / Northern Virginia.  "us-east4" 14 / Oregon.  "us-west1" location> 12 The storage class to use when storing objects in Google Cloud Storage. Choose a number from below, or type in your own value 1 / Default  "" 2 / Multi-regional storage class  "MULTI_REGIONAL" 3 / Regional storage class  "REGIONAL" 4 / Nearline storage class  "NEARLINE" 5 / Coldline storage class  "COLDLINE" 6 / Durable reduced availability storage class  "DURABLE_REDUCED_AVAILABILITY" storage_class> 5 Remote config Use web browser to automatically authenticate rclone with remote?
+
+
+- Say Y if the machine running rclone has a web browser you can use
+- Say N if running rclone on a (remote) machine without web browser access If not sure try Y. If Y failed, try N.
+
+
+- Yes
+- No y/n> y If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth Log in and authorize rclone for access Waiting for code... Got code -------------------- [remote] type = google cloud storage client_id = client_secret = token = {"AccessToken":"xxxx.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","RefreshToken":"x/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx_xxxxxxxxx","Expiry":"2014-07-17T20:49:14.929208288+01:00","Extra":null} project_number = 12345678 object_acl = private bucket_acl = private --------------------
+- Yes this is OK
+- Edit this remote
+- Delete this remote y/e/d> y
+
+
+See the [remote setup docs](https://rclone.org/remote_setup/) for how to set it up on a
+machine with no Internet browser available.
-- Config: shared_folders
-- Env Var: RCLONE_DROPBOX_SHARED_FOLDERS
-- Type: bool
-- Default: false
+Note that rclone runs a webserver on your local machine to collect the
+token as returned from Google if using web browser to automatically
+authenticate. This only
+runs from the moment it opens your browser to the moment you get back
+the verification code. This is on `http://127.0.0.1:53682/` and this
+it may require you to unblock it temporarily if you are running a host
+firewall, or use manual mode.
-#### --dropbox-batch-mode
+This remote is called `remote` and can now be used like this
-Upload file batching sync|async|off.
+See all the buckets in your project
-This sets the batch mode used by rclone.
+ rclone lsd remote:
-For full info see [the main docs](https://rclone.org/dropbox/#batch-mode)
+Make a new bucket
-This has 3 possible values
+ rclone mkdir remote:bucket
-- off - no batching
-- sync - batch uploads and check completion (default)
-- async - batch upload and don't check completion
+List the contents of a bucket
-Rclone will close any outstanding batches when it exits which may make
-a delay on quit.
+ rclone ls remote:bucket
+Sync `/home/local/directory` to the remote bucket, deleting any excess
+files in the bucket.
-Properties:
+ rclone sync --interactive /home/local/directory remote:bucket
-- Config: batch_mode
-- Env Var: RCLONE_DROPBOX_BATCH_MODE
-- Type: string
-- Default: "sync"
+### Service Account support
-#### --dropbox-batch-size
+You can set up rclone with Google Cloud Storage in an unattended mode,
+i.e. not tied to a specific end-user Google account. This is useful
+when you want to synchronise files onto machines that don't have
+actively logged-in users, for example build machines.
-Max number of files in upload batch.
+To get credentials for Google Cloud Platform
+[IAM Service Accounts](https://cloud.google.com/iam/docs/service-accounts),
+please head to the
+[Service Account](https://console.cloud.google.com/permissions/serviceaccounts)
+section of the Google Developer Console. Service Accounts behave just
+like normal `User` permissions in
+[Google Cloud Storage ACLs](https://cloud.google.com/storage/docs/access-control),
+so you can limit their access (e.g. make them read only). After
+creating an account, a JSON file containing the Service Account's
+credentials will be downloaded onto your machines. These credentials
+are what rclone will use for authentication.
-This sets the batch size of files to upload. It has to be less than 1000.
+To use a Service Account instead of OAuth2 token flow, enter the path
+to your Service Account credentials at the `service_account_file`
+prompt and rclone won't use the browser based authentication
+flow. If you'd rather stuff the contents of the credentials file into
+the rclone config file, you can set `service_account_credentials` with
+the actual contents of the file instead, or set the equivalent
+environment variable.
-By default this is 0 which means rclone which calculate the batch size
-depending on the setting of batch_mode.
+### Anonymous Access
-- batch_mode: async - default batch_size is 100
-- batch_mode: sync - default batch_size is the same as --transfers
-- batch_mode: off - not in use
+For downloads of objects that permit public access you can configure rclone
+to use anonymous access by setting `anonymous` to `true`.
+With unauthorized access you can't write or create files but only read or list
+those buckets and objects that have public read access.
-Rclone will close any outstanding batches when it exits which may make
-a delay on quit.
+### Application Default Credentials
-Setting this is a great idea if you are uploading lots of small files
-as it will make them a lot quicker. You can use --transfers 32 to
-maximise throughput.
+If no other source of credentials is provided, rclone will fall back
+to
+[Application Default Credentials](https://cloud.google.com/video-intelligence/docs/common/auth#authenticating_with_application_default_credentials)
+this is useful both when you already have configured authentication
+for your developer account, or in production when running on a google
+compute host. Note that if running in docker, you may need to run
+additional commands on your google compute machine -
+[see this page](https://cloud.google.com/container-registry/docs/advanced-authentication#gcloud_as_a_docker_credential_helper).
+Note that in the case application default credentials are used, there
+is no need to explicitly configure a project number.
-Properties:
+### --fast-list
-- Config: batch_size
-- Env Var: RCLONE_DROPBOX_BATCH_SIZE
-- Type: int
-- Default: 0
+This remote supports `--fast-list` which allows you to use fewer
+transactions in exchange for more memory. See the [rclone
+docs](https://rclone.org/docs/#fast-list) for more details.
-#### --dropbox-batch-timeout
+### Custom upload headers
-Max time to allow an idle upload batch before uploading.
+You can set custom upload headers with the `--header-upload`
+flag. Google Cloud Storage supports the headers as described in the
+[working with metadata documentation](https://cloud.google.com/storage/docs/gsutil/addlhelp/WorkingWithObjectMetadata)
-If an upload batch is idle for more than this long then it will be
-uploaded.
+- Cache-Control
+- Content-Disposition
+- Content-Encoding
+- Content-Language
+- Content-Type
+- X-Goog-Storage-Class
+- X-Goog-Meta-
-The default for this is 0 which means rclone will choose a sensible
-default based on the batch_mode in use.
+Eg `--header-upload "Content-Type text/potato"`
-- batch_mode: async - default batch_timeout is 10s
-- batch_mode: sync - default batch_timeout is 500ms
-- batch_mode: off - not in use
+Note that the last of these is for setting custom metadata in the form
+`--header-upload "x-goog-meta-key: value"`
+
+### Modification times
+
+Google Cloud Storage stores md5sum natively.
+Google's [gsutil](https://cloud.google.com/storage/docs/gsutil) tool stores modification time
+with one-second precision as `goog-reserved-file-mtime` in file metadata.
+
+To ensure compatibility with gsutil, rclone stores modification time in 2 separate metadata entries.
+`mtime` uses RFC3339 format with one-nanosecond precision.
+`goog-reserved-file-mtime` uses Unix timestamp format with one-second precision.
+To get modification time from object metadata, rclone reads the metadata in the following order: `mtime`, `goog-reserved-file-mtime`, object updated time.
+
+Note that rclone's default modify window is 1ns.
+Files uploaded by gsutil only contain timestamps with one-second precision.
+If you use rclone to sync files previously uploaded by gsutil,
+rclone will attempt to update modification time for all these files.
+To avoid these possibly unnecessary updates, use `--modify-window 1s`.
+
+### Restricted filename characters
+
+| Character | Value | Replacement |
+| --------- |:-----:|:-----------:|
+| NUL | 0x00 | †|
+| LF | 0x0A | ⊠|
+| CR | 0x0D | â |
+| / | 0x2F | ï¼ |
+
+Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8),
+as they can't be used in JSON strings.
-Properties:
+### Standard options
-- Config: batch_timeout
-- Env Var: RCLONE_DROPBOX_BATCH_TIMEOUT
-- Type: Duration
-- Default: 0s
+Here are the Standard options specific to google cloud storage (Google Cloud Storage (this is not Google Drive)).
-#### --dropbox-batch-commit-timeout
+#### --gcs-client-id
-Max time to wait for a batch to finish committing
+OAuth Client Id.
+
+Leave blank normally.
Properties:
-- Config: batch_commit_timeout
-- Env Var: RCLONE_DROPBOX_BATCH_COMMIT_TIMEOUT
-- Type: Duration
-- Default: 10m0s
+- Config: client_id
+- Env Var: RCLONE_GCS_CLIENT_ID
+- Type: string
+- Required: false
-#### --dropbox-pacer-min-sleep
+#### --gcs-client-secret
-Minimum time to sleep between API calls.
+OAuth Client Secret.
+
+Leave blank normally.
Properties:
-- Config: pacer_min_sleep
-- Env Var: RCLONE_DROPBOX_PACER_MIN_SLEEP
-- Type: Duration
-- Default: 10ms
+- Config: client_secret
+- Env Var: RCLONE_GCS_CLIENT_SECRET
+- Type: string
+- Required: false
-#### --dropbox-encoding
+#### --gcs-project-number
-The encoding for the backend.
+Project number.
-See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info.
+Optional - needed only for list/create/delete buckets - see your developer console.
Properties:
-- Config: encoding
-- Env Var: RCLONE_DROPBOX_ENCODING
-- Type: MultiEncoder
-- Default: Slash,BackSlash,Del,RightSpace,InvalidUtf8,Dot
-
+- Config: project_number
+- Env Var: RCLONE_GCS_PROJECT_NUMBER
+- Type: string
+- Required: false
+#### --gcs-user-project
-## Limitations
+User project.
-Note that Dropbox is case insensitive so you can't have a file called
-"Hello.doc" and one called "hello.doc".
+Optional - needed only for requester pays.
-There are some file names such as `thumbs.db` which Dropbox can't
-store. There is a full list of them in the ["Ignored Files" section
-of this document](https://www.dropbox.com/en/help/145). Rclone will
-issue an error message `File name disallowed - not uploading` if it
-attempts to upload one of those file names, but the sync won't fail.
+Properties:
-Some errors may occur if you try to sync copyright-protected files
-because Dropbox has its own [copyright detector](https://techcrunch.com/2014/03/30/how-dropbox-knows-when-youre-sharing-copyrighted-stuff-without-actually-looking-at-your-stuff/) that
-prevents this sort of file being downloaded. This will return the error `ERROR :
-/path/to/your/file: Failed to copy: failed to open source object:
-path/restricted_content/.`
+- Config: user_project
+- Env Var: RCLONE_GCS_USER_PROJECT
+- Type: string
+- Required: false
-If you have more than 10,000 files in a directory then `rclone purge
-dropbox:dir` will return the error `Failed to purge: There are too
-many files involved in this operation`. As a work-around do an
-`rclone delete dropbox:dir` followed by an `rclone rmdir dropbox:dir`.
+#### --gcs-service-account-file
-When using `rclone link` you'll need to set `--expire` if using a
-non-personal account otherwise the visibility may not be correct.
-(Note that `--expire` isn't supported on personal accounts). See the
-[forum discussion](https://forum.rclone.org/t/rclone-link-dropbox-permissions/23211) and the
-[dropbox SDK issue](https://github.com/dropbox/dropbox-sdk-go-unofficial/issues/75).
+Service Account Credentials JSON file path.
-## Get your own Dropbox App ID
+Leave blank normally.
+Needed only if you want use SA instead of interactive login.
-When you use rclone with Dropbox in its default configuration you are using rclone's App ID. This is shared between all the rclone users.
+Leading `~` will be expanded in the file name as will environment variables such as `${RCLONE_CONFIG_DIR}`.
-Here is how to create your own Dropbox App ID for rclone:
+Properties:
-1. Log into the [Dropbox App console](https://www.dropbox.com/developers/apps/create) with your Dropbox Account (It need not
-to be the same account as the Dropbox you want to access)
+- Config: service_account_file
+- Env Var: RCLONE_GCS_SERVICE_ACCOUNT_FILE
+- Type: string
+- Required: false
-2. Choose an API => Usually this should be `Dropbox API`
+#### --gcs-service-account-credentials
-3. Choose the type of access you want to use => `Full Dropbox` or `App Folder`. If you want to use Team Folders, `Full Dropbox` is required ([see here](https://www.dropboxforum.com/t5/Dropbox-API-Support-Feedback/How-to-create-team-folder-inside-my-app-s-folder/m-p/601005/highlight/true#M27911)).
+Service Account Credentials JSON blob.
-4. Name your App. The app name is global, so you can't use `rclone` for example
+Leave blank normally.
+Needed only if you want use SA instead of interactive login.
-5. Click the button `Create App`
+Properties:
-6. Switch to the `Permissions` tab. Enable at least the following permissions: `account_info.read`, `files.metadata.write`, `files.content.write`, `files.content.read`, `sharing.write`. The `files.metadata.read` and `sharing.read` checkboxes will be marked too. Click `Submit`
+- Config: service_account_credentials
+- Env Var: RCLONE_GCS_SERVICE_ACCOUNT_CREDENTIALS
+- Type: string
+- Required: false
-7. Switch to the `Settings` tab. Fill `OAuth2 - Redirect URIs` as `http://localhost:53682/` and click on `Add`
+#### --gcs-anonymous
-8. Find the `App key` and `App secret` values on the `Settings` tab. Use these values in rclone config to add a new remote or edit an existing remote. The `App key` setting corresponds to `client_id` in rclone config, the `App secret` corresponds to `client_secret`
+Access public buckets and objects without credentials.
-# Enterprise File Fabric
+Set to 'true' if you just want to download files and don't configure credentials.
-This backend supports [Storage Made Easy's Enterprise File
-Fabricâ„¢](https://storagemadeeasy.com/about/) which provides a software
-solution to integrate and unify File and Object Storage accessible
-through a global file system.
+Properties:
-## Configuration
+- Config: anonymous
+- Env Var: RCLONE_GCS_ANONYMOUS
+- Type: bool
+- Default: false
-The initial setup for the Enterprise File Fabric backend involves
-getting a token from the Enterprise File Fabric which you need to
-do in your browser. `rclone config` walks you through it.
+#### --gcs-object-acl
-Here is an example of how to make a remote called `remote`. First run:
+Access Control List for new objects.
- rclone config
+Properties:
-This will guide you through an interactive setup process:
-
-No remotes found, make a new one? n) New remote s) Set configuration password q) Quit config n/s/q> n name> remote Type of storage to configure. Enter a string value. Press Enter for the default (""). Choose a number from below, or type in your own value [snip] XX / Enterprise File Fabric  "filefabric" [snip] Storage> filefabric ** See help for filefabric backend at: https://rclone.org/filefabric/ **
-URL of the Enterprise File Fabric to connect to Enter a string value. Press Enter for the default (""). Choose a number from below, or type in your own value 1 / Storage Made Easy US  "https://storagemadeeasy.com" 2 / Storage Made Easy EU  "https://eu.storagemadeeasy.com" 3 / Connect to your Enterprise File Fabric  "https://yourfabric.smestorage.com" url> https://yourfabric.smestorage.com/ ID of the root folder Leave blank normally.
-Fill in to make rclone start with directory of a given ID.
-Enter a string value. Press Enter for the default (""). root_folder_id> Permanent Authentication Token
-A Permanent Authentication Token can be created in the Enterprise File Fabric, on the users Dashboard under Security, there is an entry you'll see called "My Authentication Tokens". Click the Manage button to create one.
-These tokens are normally valid for several years.
-For more info see: https://docs.storagemadeeasy.com/organisationcloud/api-tokens
-Enter a string value. Press Enter for the default (""). permanent_token> xxxxxxxxxxxxxxx-xxxxxxxxxxxxxxxx Edit advanced config? (y/n) y) Yes n) No (default) y/n> n Remote config -------------------- [remote] type = filefabric url = https://yourfabric.smestorage.com/ permanent_token = xxxxxxxxxxxxxxx-xxxxxxxxxxxxxxxx -------------------- y) Yes this is OK (default) e) Edit this remote d) Delete this remote y/e/d> y
-
-Once configured you can then use `rclone` like this,
+- Config: object_acl
+- Env Var: RCLONE_GCS_OBJECT_ACL
+- Type: string
+- Required: false
+- Examples:
+ - "authenticatedRead"
+ - Object owner gets OWNER access.
+ - All Authenticated Users get READER access.
+ - "bucketOwnerFullControl"
+ - Object owner gets OWNER access.
+ - Project team owners get OWNER access.
+ - "bucketOwnerRead"
+ - Object owner gets OWNER access.
+ - Project team owners get READER access.
+ - "private"
+ - Object owner gets OWNER access.
+ - Default if left blank.
+ - "projectPrivate"
+ - Object owner gets OWNER access.
+ - Project team members get access according to their roles.
+ - "publicRead"
+ - Object owner gets OWNER access.
+ - All Users get READER access.
-List directories in top level of your Enterprise File Fabric
+#### --gcs-bucket-acl
- rclone lsd remote:
+Access Control List for new buckets.
-List all the files in your Enterprise File Fabric
+Properties:
- rclone ls remote:
+- Config: bucket_acl
+- Env Var: RCLONE_GCS_BUCKET_ACL
+- Type: string
+- Required: false
+- Examples:
+ - "authenticatedRead"
+ - Project team owners get OWNER access.
+ - All Authenticated Users get READER access.
+ - "private"
+ - Project team owners get OWNER access.
+ - Default if left blank.
+ - "projectPrivate"
+ - Project team members get access according to their roles.
+ - "publicRead"
+ - Project team owners get OWNER access.
+ - All Users get READER access.
+ - "publicReadWrite"
+ - Project team owners get OWNER access.
+ - All Users get WRITER access.
-To copy a local directory to an Enterprise File Fabric directory called backup
+#### --gcs-bucket-policy-only
- rclone copy /home/source remote:backup
+Access checks should use bucket-level IAM policies.
-### Modified time and hashes
+If you want to upload objects to a bucket with Bucket Policy Only set
+then you will need to set this.
-The Enterprise File Fabric allows modification times to be set on
-files accurate to 1 second. These will be used to detect whether
-objects need syncing or not.
+When it is set, rclone:
-The Enterprise File Fabric does not support any data hashes at this time.
+- ignores ACLs set on buckets
+- ignores ACLs set on objects
+- creates buckets with Bucket Policy Only set
-### Restricted filename characters
+Docs: https://cloud.google.com/storage/docs/bucket-policy-only
-The [default restricted characters set](https://rclone.org/overview/#restricted-characters)
-will be replaced.
-Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8),
-as they can't be used in JSON strings.
+Properties:
-### Empty files
+- Config: bucket_policy_only
+- Env Var: RCLONE_GCS_BUCKET_POLICY_ONLY
+- Type: bool
+- Default: false
-Empty files aren't supported by the Enterprise File Fabric. Rclone will therefore
-upload an empty file as a single space with a mime type of
-`application/vnd.rclone.empty.file` and files with that mime type are
-treated as empty.
+#### --gcs-location
-### Root folder ID ###
+Location for the newly created buckets.
-You can set the `root_folder_id` for rclone. This is the directory
-(identified by its `Folder ID`) that rclone considers to be the root
-of your Enterprise File Fabric.
+Properties:
-Normally you will leave this blank and rclone will determine the
-correct root to use itself.
+- Config: location
+- Env Var: RCLONE_GCS_LOCATION
+- Type: string
+- Required: false
+- Examples:
+ - ""
+ - Empty for default location (US)
+ - "asia"
+ - Multi-regional location for Asia
+ - "eu"
+ - Multi-regional location for Europe
+ - "us"
+ - Multi-regional location for United States
+ - "asia-east1"
+ - Taiwan
+ - "asia-east2"
+ - Hong Kong
+ - "asia-northeast1"
+ - Tokyo
+ - "asia-northeast2"
+ - Osaka
+ - "asia-northeast3"
+ - Seoul
+ - "asia-south1"
+ - Mumbai
+ - "asia-south2"
+ - Delhi
+ - "asia-southeast1"
+ - Singapore
+ - "asia-southeast2"
+ - Jakarta
+ - "australia-southeast1"
+ - Sydney
+ - "australia-southeast2"
+ - Melbourne
+ - "europe-north1"
+ - Finland
+ - "europe-west1"
+ - Belgium
+ - "europe-west2"
+ - London
+ - "europe-west3"
+ - Frankfurt
+ - "europe-west4"
+ - Netherlands
+ - "europe-west6"
+ - Zürich
+ - "europe-central2"
+ - Warsaw
+ - "us-central1"
+ - Iowa
+ - "us-east1"
+ - South Carolina
+ - "us-east4"
+ - Northern Virginia
+ - "us-west1"
+ - Oregon
+ - "us-west2"
+ - California
+ - "us-west3"
+ - Salt Lake City
+ - "us-west4"
+ - Las Vegas
+ - "northamerica-northeast1"
+ - Montréal
+ - "northamerica-northeast2"
+ - Toronto
+ - "southamerica-east1"
+ - São Paulo
+ - "southamerica-west1"
+ - Santiago
+ - "asia1"
+ - Dual region: asia-northeast1 and asia-northeast2.
+ - "eur4"
+ - Dual region: europe-north1 and europe-west4.
+ - "nam4"
+ - Dual region: us-central1 and us-east1.
-However you can set this to restrict rclone to a specific folder
-hierarchy.
+#### --gcs-storage-class
-In order to do this you will have to find the `Folder ID` of the
-directory you wish rclone to display. These aren't displayed in the
-web interface, but you can use `rclone lsf` to find them, for example
-
-$ rclone lsf --dirs-only -Fip --csv filefabric: 120673758,Burnt PDFs/ 120673759,My Quick Uploads/ 120673755,My Syncs/ 120673756,My backups/ 120673757,My contacts/ 120673761,S3 Storage/
-
-The ID for "S3 Storage" would be `120673761`.
+The storage class to use when storing objects in Google Cloud Storage.
+Properties:
-### Standard options
+- Config: storage_class
+- Env Var: RCLONE_GCS_STORAGE_CLASS
+- Type: string
+- Required: false
+- Examples:
+ - ""
+ - Default
+ - "MULTI_REGIONAL"
+ - Multi-regional storage class
+ - "REGIONAL"
+ - Regional storage class
+ - "NEARLINE"
+ - Nearline storage class
+ - "COLDLINE"
+ - Coldline storage class
+ - "ARCHIVE"
+ - Archive storage class
+ - "DURABLE_REDUCED_AVAILABILITY"
+ - Durable reduced availability storage class
-Here are the Standard options specific to filefabric (Enterprise File Fabric).
+#### --gcs-env-auth
-#### --filefabric-url
+Get GCP IAM credentials from runtime (environment variables or instance meta data if no env vars).
-URL of the Enterprise File Fabric to connect to.
+Only applies if service_account_file and service_account_credentials is blank.
Properties:
-- Config: url
-- Env Var: RCLONE_FILEFABRIC_URL
-- Type: string
-- Required: true
+- Config: env_auth
+- Env Var: RCLONE_GCS_ENV_AUTH
+- Type: bool
+- Default: false
- Examples:
- - "https://storagemadeeasy.com"
- - Storage Made Easy US
- - "https://eu.storagemadeeasy.com"
- - Storage Made Easy EU
- - "https://yourfabric.smestorage.com"
- - Connect to your Enterprise File Fabric
-
-#### --filefabric-root-folder-id
+ - "false"
+ - Enter credentials in the next step.
+ - "true"
+ - Get GCP IAM credentials from the environment (env vars or IAM).
-ID of the root folder.
+### Advanced options
-Leave blank normally.
+Here are the Advanced options specific to google cloud storage (Google Cloud Storage (this is not Google Drive)).
-Fill in to make rclone start with directory of a given ID.
+#### --gcs-token
+OAuth Access Token as a JSON blob.
Properties:
-- Config: root_folder_id
-- Env Var: RCLONE_FILEFABRIC_ROOT_FOLDER_ID
+- Config: token
+- Env Var: RCLONE_GCS_TOKEN
- Type: string
- Required: false
-#### --filefabric-permanent-token
+#### --gcs-auth-url
-Permanent Authentication Token.
+Auth server URL.
-A Permanent Authentication Token can be created in the Enterprise File
-Fabric, on the users Dashboard under Security, there is an entry
-you'll see called "My Authentication Tokens". Click the Manage button
-to create one.
+Leave blank to use the provider defaults.
-These tokens are normally valid for several years.
+Properties:
-For more info see: https://docs.storagemadeeasy.com/organisationcloud/api-tokens
+- Config: auth_url
+- Env Var: RCLONE_GCS_AUTH_URL
+- Type: string
+- Required: false
+
+#### --gcs-token-url
+Token server url.
+
+Leave blank to use the provider defaults.
Properties:
-- Config: permanent_token
-- Env Var: RCLONE_FILEFABRIC_PERMANENT_TOKEN
+- Config: token_url
+- Env Var: RCLONE_GCS_TOKEN_URL
- Type: string
- Required: false
-### Advanced options
+#### --gcs-directory-markers
-Here are the Advanced options specific to filefabric (Enterprise File Fabric).
+Upload an empty object with a trailing slash when a new directory is created
-#### --filefabric-token
+Empty folders are unsupported for bucket based remotes, this option creates an empty
+object ending with "/", to persist the folder.
-Session Token.
-This is a session token which rclone caches in the config file. It is
-usually valid for 1 hour.
+Properties:
-Don't set this value - rclone will set it automatically.
+- Config: directory_markers
+- Env Var: RCLONE_GCS_DIRECTORY_MARKERS
+- Type: bool
+- Default: false
+
+#### --gcs-no-check-bucket
+
+If set, don't attempt to check the bucket exists or create it.
+
+This can be useful when trying to minimise the number of transactions
+rclone does if you know the bucket exists already.
Properties:
-- Config: token
-- Env Var: RCLONE_FILEFABRIC_TOKEN
-- Type: string
-- Required: false
+- Config: no_check_bucket
+- Env Var: RCLONE_GCS_NO_CHECK_BUCKET
+- Type: bool
+- Default: false
-#### --filefabric-token-expiry
+#### --gcs-decompress
-Token expiry time.
+If set this will decompress gzip encoded objects.
-Don't set this value - rclone will set it automatically.
+It is possible to upload objects to GCS with "Content-Encoding: gzip"
+set. Normally rclone will download these files as compressed objects.
+If this flag is set then rclone will decompress these files with
+"Content-Encoding: gzip" as they are received. This means that rclone
+can't check the size and hash but the file contents will be decompressed.
-Properties:
-- Config: token_expiry
-- Env Var: RCLONE_FILEFABRIC_TOKEN_EXPIRY
-- Type: string
-- Required: false
+Properties:
-#### --filefabric-version
+- Config: decompress
+- Env Var: RCLONE_GCS_DECOMPRESS
+- Type: bool
+- Default: false
-Version read from the file fabric.
+#### --gcs-endpoint
-Don't set this value - rclone will set it automatically.
+Endpoint for the service.
+Leave blank normally.
Properties:
-- Config: version
-- Env Var: RCLONE_FILEFABRIC_VERSION
+- Config: endpoint
+- Env Var: RCLONE_GCS_ENDPOINT
- Type: string
- Required: false
-#### --filefabric-encoding
+#### --gcs-encoding
The encoding for the backend.
@@ -22312,3084 +22223,3149 @@ Synology C2 Object Storage
Properties:
- Config: encoding
-- Env Var: RCLONE_FILEFABRIC_ENCODING
-- Type: MultiEncoder
-- Default: Slash,Del,Ctl,InvalidUtf8,Dot
+- Env Var: RCLONE_GCS_ENCODING
+- Type: Encoding
+- Default: Slash,CrLf,InvalidUtf8,Dot
-# FTP
+## Limitations
-FTP is the File Transfer Protocol. Rclone FTP support is provided using the
-[github.com/jlaffaye/ftp](https://godoc.org/github.com/jlaffaye/ftp)
-package.
+`rclone about` is not supported by the Google Cloud Storage backend. Backends without
+this capability cannot determine free space for an rclone mount or
+use policy `mfs` (most free space) as a member of an rclone union
+remote.
+
+See [List of backends that do not support rclone about](https://rclone.org/overview/#optional-features) and [rclone about](https://rclone.org/commands/rclone_about/)
+
+# Google Drive
+
+Paths are specified as `drive:path`
+
+Drive paths may be as deep as required, e.g. `drive:directory/subdirectory`.
+
+## Configuration
+
+The initial setup for drive involves getting a token from Google drive
+which you need to do in your browser. `rclone config` walks you
+through it.
+
+Here is an example of how to make a remote called `remote`. First run:
+
+ rclone config
+
+This will guide you through an interactive setup process:
+
+No remotes found, make a new one? n) New remote r) Rename remote c) Copy remote s) Set configuration password q) Quit config n/r/c/s/q> n name> remote Type of storage to configure. Choose a number from below, or type in your own value [snip] XX / Google Drive  "drive" [snip] Storage> drive Google Application Client Id - leave blank normally. client_id> Google Application Client Secret - leave blank normally. client_secret> Scope that rclone should use when requesting access from drive. Choose a number from below, or type in your own value 1 / Full access all files, excluding Application Data Folder.  "drive" 2 / Read-only access to file metadata and file contents.  "drive.readonly" / Access to files created by rclone only. 3 | These are visible in the drive website. | File authorization is revoked when the user deauthorizes the app.  "drive.file" / Allows read and write access to the Application Data folder. 4 | This is not visible in the drive website.  "drive.appfolder" / Allows read-only access to file metadata but 5 | does not allow any access to read or download file content.  "drive.metadata.readonly" scope> 1 Service Account Credentials JSON file path - needed only if you want use SA instead of interactive login. service_account_file> Remote config Use web browser to automatically authenticate rclone with remote? * Say Y if the machine running rclone has a web browser you can use * Say N if running rclone on a (remote) machine without web browser access If not sure try Y. If Y failed, try N. y) Yes n) No y/n> y If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth Log in and authorize rclone for access Waiting for code... Got code Configure this as a Shared Drive (Team Drive)? y) Yes n) No y/n> n -------------------- [remote] client_id = client_secret = scope = drive root_folder_id = service_account_file = token = {"access_token":"XXX","token_type":"Bearer","refresh_token":"XXX","expiry":"2014-03-16T13:57:58.955387075Z"} -------------------- y) Yes this is OK e) Edit this remote d) Delete this remote y/e/d> y
+
+See the [remote setup docs](https://rclone.org/remote_setup/) for how to set it up on a
+machine with no Internet browser available.
+
+Note that rclone runs a webserver on your local machine to collect the
+token as returned from Google if using web browser to automatically
+authenticate. This only
+runs from the moment it opens your browser to the moment you get back
+the verification code. This is on `http://127.0.0.1:53682/` and it
+may require you to unblock it temporarily if you are running a host
+firewall, or use manual mode.
-[Limitations of Rclone's FTP backend](#limitations)
+You can then use it like this,
-Paths are specified as `remote:path`. If the path does not begin with
-a `/` it is relative to the home directory of the user. An empty path
-`remote:` refers to the user's home directory.
+List directories in top level of your drive
-## Configuration
+ rclone lsd remote:
-To create an FTP configuration named `remote`, run
+List all the files in your drive
- rclone config
+ rclone ls remote:
-Rclone config guides you through an interactive setup process. A minimal
-rclone FTP remote definition only requires host, username and password.
-For an anonymous FTP server, see [below](#anonymous-ftp).
-
-No remotes found, make a new one? n) New remote r) Rename remote c) Copy remote s) Set configuration password q) Quit config n/r/c/s/q> n name> remote Type of storage to configure. Enter a string value. Press Enter for the default (""). Choose a number from below, or type in your own value [snip] XX / FTP Â "ftp" [snip] Storage> ftp ** See help for ftp backend at: https://rclone.org/ftp/ **
-FTP host to connect to Enter a string value. Press Enter for the default (""). Choose a number from below, or type in your own value 1 / Connect to ftp.example.com  "ftp.example.com" host> ftp.example.com FTP username Enter a string value. Press Enter for the default ("$USER"). user> FTP port number Enter a signed integer. Press Enter for the default (21). port> FTP password y) Yes type in my own password g) Generate random password y/g> y Enter the password: password: Confirm the password: password: Use FTP over TLS (Implicit) Enter a boolean value (true or false). Press Enter for the default ("false"). tls> Use FTP over TLS (Explicit) Enter a boolean value (true or false). Press Enter for the default ("false"). explicit_tls> Remote config -------------------- [remote] type = ftp host = ftp.example.com pass = *** ENCRYPTED *** -------------------- y) Yes this is OK e) Edit this remote d) Delete this remote y/e/d> y
-
-To see all directories in the home directory of `remote`
+To copy a local directory to a drive directory called backup
- rclone lsd remote:
+ rclone copy /home/source remote:backup
-Make a new directory
+### Scopes
- rclone mkdir remote:path/to/directory
+Rclone allows you to select which scope you would like for rclone to
+use. This changes what type of token is granted to rclone. [The
+scopes are defined
+here](https://developers.google.com/drive/v3/web/about-auth).
-List the contents of a directory
+A comma-separated list is allowed e.g. `drive.readonly,drive.file`.
- rclone ls remote:path/to/directory
+The scope are
-Sync `/home/local/directory` to the remote directory, deleting any
-excess files in the directory.
+#### drive
- rclone sync --interactive /home/local/directory remote:directory
+This is the default scope and allows full access to all files, except
+for the Application Data Folder (see below).
-### Anonymous FTP
+Choose this one if you aren't sure.
-When connecting to a FTP server that allows anonymous login, you can use the
-special "anonymous" username. Traditionally, this user account accepts any
-string as a password, although it is common to use either the password
-"anonymous" or "guest". Some servers require the use of a valid e-mail
-address as password.
+#### drive.readonly
-Using [on-the-fly](#backend-path-to-dir) or
-[connection string](https://rclone.org/docs/#connection-strings) remotes makes it easy to access
-such servers, without requiring any configuration in advance. The following
-are examples of that:
+This allows read only access to all files. Files may be listed and
+downloaded but not uploaded, renamed or deleted.
- rclone lsf :ftp: --ftp-host=speedtest.tele2.net --ftp-user=anonymous --ftp-pass=$(rclone obscure dummy)
- rclone lsf :ftp,host=speedtest.tele2.net,user=anonymous,pass=$(rclone obscure dummy):
+#### drive.file
-The above examples work in Linux shells and in PowerShell, but not Windows
-Command Prompt. They execute the [rclone obscure](https://rclone.org/commands/rclone_obscure/)
-command to create a password string in the format required by the
-[pass](#ftp-pass) option. The following examples are exactly the same, except use
-an already obscured string representation of the same password "dummy", and
-therefore works even in Windows Command Prompt:
+With this scope rclone can read/view/modify only those files and
+folders it creates.
- rclone lsf :ftp: --ftp-host=speedtest.tele2.net --ftp-user=anonymous --ftp-pass=IXs2wc8OJOz7SYLBk47Ji1rHTmxM
- rclone lsf :ftp,host=speedtest.tele2.net,user=anonymous,pass=IXs2wc8OJOz7SYLBk47Ji1rHTmxM:
+So if you uploaded files to drive via the web interface (or any other
+means) they will not be visible to rclone.
-### Implicit TLS
+This can be useful if you are using rclone to backup data and you want
+to be sure confidential data on your drive is not visible to rclone.
-Rlone FTP supports implicit FTP over TLS servers (FTPS). This has to
-be enabled in the FTP backend config for the remote, or with
-[`--ftp-tls`](#ftp-tls). The default FTPS port is `990`, not `21` and
-can be set with [`--ftp-port`](#ftp-port).
+Files created with this scope are visible in the web interface.
-### Restricted filename characters
+#### drive.appfolder
-In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters)
-the following characters are also replaced:
+This gives rclone its own private area to store files. Rclone will
+not be able to see any other files on your drive and you won't be able
+to see rclone's files from the web interface either.
-File names cannot end with the following characters. Replacement is
-limited to the last character in a file name:
+#### drive.metadata.readonly
-| Character | Value | Replacement |
-| --------- |:-----:|:-----------:|
-| SP | 0x20 | â |
+This allows read only access to file names only. It does not allow
+rclone to download or upload data, or rename or delete files or
+directories.
-Not all FTP servers can have all characters in file names, for example:
+### Root folder ID
-| FTP Server| Forbidden characters |
-| --------- |:--------------------:|
-| proftpd | `*` |
-| pureftpd | `\ [ ]` |
+This option has been moved to the advanced section. You can set the `root_folder_id` for rclone. This is the directory
+(identified by its `Folder ID`) that rclone considers to be the root
+of your drive.
-This backend's interactive configuration wizard provides a selection of
-sensible encoding settings for major FTP servers: ProFTPd, PureFTPd, VsFTPd.
-Just hit a selection number when prompted.
+Normally you will leave this blank and rclone will determine the
+correct root to use itself.
+However you can set this to restrict rclone to a specific folder
+hierarchy or to access data within the "Computers" tab on the drive
+web interface (where files from Google's Backup and Sync desktop
+program go).
-### Standard options
+In order to do this you will have to find the `Folder ID` of the
+directory you wish rclone to display. This will be the last segment
+of the URL when you open the relevant folder in the drive web
+interface.
-Here are the Standard options specific to ftp (FTP).
+So if the folder you want rclone to use has a URL which looks like
+`https://drive.google.com/drive/folders/1XyfxxxxxxxxxxxxxxxxxxxxxxxxxKHCh`
+in the browser, then you use `1XyfxxxxxxxxxxxxxxxxxxxxxxxxxKHCh` as
+the `root_folder_id` in the config.
-#### --ftp-host
+**NB** folders under the "Computers" tab seem to be read only (drive
+gives a 500 error) when using rclone.
-FTP host to connect to.
+There doesn't appear to be an API to discover the folder IDs of the
+"Computers" tab - please contact us if you know otherwise!
-E.g. "ftp.example.com".
+Note also that rclone can't access any data under the "Backups" tab on
+the google drive web interface yet.
-Properties:
+### Service Account support
-- Config: host
-- Env Var: RCLONE_FTP_HOST
-- Type: string
-- Required: true
+You can set up rclone with Google Drive in an unattended mode,
+i.e. not tied to a specific end-user Google account. This is useful
+when you want to synchronise files onto machines that don't have
+actively logged-in users, for example build machines.
-#### --ftp-user
+To use a Service Account instead of OAuth2 token flow, enter the path
+to your Service Account credentials at the `service_account_file`
+prompt during `rclone config` and rclone won't use the browser based
+authentication flow. If you'd rather stuff the contents of the
+credentials file into the rclone config file, you can set
+`service_account_credentials` with the actual contents of the file
+instead, or set the equivalent environment variable.
-FTP username.
+#### Use case - Google Apps/G-suite account and individual Drive
-Properties:
+Let's say that you are the administrator of a Google Apps (old) or
+G-suite account.
+The goal is to store data on an individual's Drive account, who IS
+a member of the domain.
+We'll call the domain **example.com**, and the user
+**foo@example.com**.
-- Config: user
-- Env Var: RCLONE_FTP_USER
-- Type: string
-- Default: "$USER"
+There's a few steps we need to go through to accomplish this:
-#### --ftp-port
+##### 1. Create a service account for example.com
+ - To create a service account and obtain its credentials, go to the
+[Google Developer Console](https://console.developers.google.com).
+ - You must have a project - create one if you don't.
+ - Then go to "IAM & admin" -> "Service Accounts".
+ - Use the "Create Service Account" button. Fill in "Service account name"
+and "Service account ID" with something that identifies your client.
+ - Select "Create And Continue". Step 2 and 3 are optional.
+ - These credentials are what rclone will use for authentication.
+If you ever need to remove access, press the "Delete service
+account key" button.
-FTP port number.
+##### 2. Allowing API access to example.com Google Drive
+ - Go to example.com's admin console
+ - Go into "Security" (or use the search bar)
+ - Select "Show more" and then "Advanced settings"
+ - Select "Manage API client access" in the "Authentication" section
+ - In the "Client Name" field enter the service account's
+"Client ID" - this can be found in the Developer Console under
+"IAM & Admin" -> "Service Accounts", then "View Client ID" for
+the newly created service account.
+It is a ~21 character numerical string.
+ - In the next field, "One or More API Scopes", enter
+`https://www.googleapis.com/auth/drive`
+to grant access to Google Drive specifically.
-Properties:
+##### 3. Configure rclone, assuming a new install
+
+rclone config
+n/s/q> n # New name>gdrive # Gdrive is an example name Storage> # Select the number shown for Google Drive client_id> # Can be left blank client_secret> # Can be left blank scope> # Select your scope, 1 for example root_folder_id> # Can be left blank service_account_file> /home/foo/myJSONfile.json # This is where the JSON file goes! y/n> # Auto config, n
+
+##### 4. Verify that it's working
+ - `rclone -v --drive-impersonate foo@example.com lsf gdrive:backup`
+ - The arguments do:
+ - `-v` - verbose logging
+ - `--drive-impersonate foo@example.com` - this is what does
+the magic, pretending to be user foo.
+ - `lsf` - list files in a parsing friendly way
+ - `gdrive:backup` - use the remote called gdrive, work in
+the folder named backup.
-- Config: port
-- Env Var: RCLONE_FTP_PORT
-- Type: int
-- Default: 21
+Note: in case you configured a specific root folder on gdrive and rclone is unable to access the contents of that folder when using `--drive-impersonate`, do this instead:
+ - in the gdrive web interface, share your root folder with the user/email of the new Service Account you created/selected at step #1
+ - use rclone without specifying the `--drive-impersonate` option, like this:
+ `rclone -v lsf gdrive:backup`
-#### --ftp-pass
-FTP password.
+### Shared drives (team drives)
-**NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/).
+If you want to configure the remote to point to a Google Shared Drive
+(previously known as Team Drives) then answer `y` to the question
+`Configure this as a Shared Drive (Team Drive)?`.
-Properties:
+This will fetch the list of Shared Drives from google and allow you to
+configure which one you want to use. You can also type in a Shared
+Drive ID if you prefer.
-- Config: pass
-- Env Var: RCLONE_FTP_PASS
-- Type: string
-- Required: false
+For example:
+
+Configure this as a Shared Drive (Team Drive)? y) Yes n) No y/n> y Fetching Shared Drive list... Choose a number from below, or type in your own value 1 / Rclone Test  "xxxxxxxxxxxxxxxxxxxx" 2 / Rclone Test 2  "yyyyyyyyyyyyyyyyyyyy" 3 / Rclone Test 3  "zzzzzzzzzzzzzzzzzzzz" Enter a Shared Drive ID> 1 -------------------- [remote] client_id = client_secret = token = {"AccessToken":"xxxx.x.xxxxx_xxxxxxxxxxx_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","RefreshToken":"1/xxxxxxxxxxxxxxxx_xxxxxxxxxxxxxxxxxxxxxxxxxx","Expiry":"2014-03-16T13:57:58.955387075Z","Extra":null} team_drive = xxxxxxxxxxxxxxxxxxxx -------------------- y) Yes this is OK e) Edit this remote d) Delete this remote y/e/d> y
+
+### --fast-list
-#### --ftp-tls
+This remote supports `--fast-list` which allows you to use fewer
+transactions in exchange for more memory. See the [rclone
+docs](https://rclone.org/docs/#fast-list) for more details.
-Use Implicit FTPS (FTP over TLS).
+It does this by combining multiple `list` calls into a single API request.
-When using implicit FTP over TLS the client connects using TLS
-right from the start which breaks compatibility with
-non-TLS-aware servers. This is usually served over port 990 rather
-than port 21. Cannot be used in combination with explicit FTPS.
+This works by combining many `'%s' in parents` filters into one expression.
+To list the contents of directories a, b and c, the following requests will be send by the regular `List` function:
+trashed=false and 'a' in parents trashed=false and 'b' in parents trashed=false and 'c' in parents
+These can now be combined into a single request:
+trashed=false and ('a' in parents or 'b' in parents or 'c' in parents)
+
+The implementation of `ListR` will put up to 50 `parents` filters into one request.
+It will use the `--checkers` value to specify the number of requests to run in parallel.
-Properties:
+In tests, these batch requests were up to 20x faster than the regular method.
+Running the following command against different sized folders gives:
+rclone lsjson -vv -R --checkers=6 gdrive:folder
+
+small folder (220 directories, 700 files):
-- Config: tls
-- Env Var: RCLONE_FTP_TLS
-- Type: bool
-- Default: false
+- without `--fast-list`: 38s
+- with `--fast-list`: 10s
-#### --ftp-explicit-tls
+large folder (10600 directories, 39000 files):
-Use Explicit FTPS (FTP over TLS).
+- without `--fast-list`: 22:05 min
+- with `--fast-list`: 58s
-When using explicit FTP over TLS the client explicitly requests
-security from the server in order to upgrade a plain text connection
-to an encrypted one. Cannot be used in combination with implicit FTPS.
+### Modification times and hashes
-Properties:
+Google drive stores modification times accurate to 1 ms.
-- Config: explicit_tls
-- Env Var: RCLONE_FTP_EXPLICIT_TLS
-- Type: bool
-- Default: false
+Hash algorithms MD5, SHA1 and SHA256 are supported. Note, however,
+that a small fraction of files uploaded may not have SHA1 or SHA256
+hashes especially if they were uploaded before 2018.
-### Advanced options
+### Restricted filename characters
-Here are the Advanced options specific to ftp (FTP).
+Only Invalid UTF-8 bytes will be [replaced](https://rclone.org/overview/#invalid-utf8),
+as they can't be used in JSON strings.
-#### --ftp-concurrency
+In contrast to other backends, `/` can also be used in names and `.`
+or `..` are valid names.
-Maximum number of FTP simultaneous connections, 0 for unlimited.
+### Revisions
-Note that setting this is very likely to cause deadlocks so it should
-be used with care.
+Google drive stores revisions of files. When you upload a change to
+an existing file to google drive using rclone it will create a new
+revision of that file.
-If you are doing a sync or copy then make sure concurrency is one more
-than the sum of `--transfers` and `--checkers`.
+Revisions follow the standard google policy which at time of writing
+was
-If you use `--check-first` then it just needs to be one more than the
-maximum of `--checkers` and `--transfers`.
+ * They are deleted after 30 days or 100 revisions (whatever comes first).
+ * They do not count towards a user storage quota.
-So for `concurrency 3` you'd use `--checkers 2 --transfers 2
---check-first` or `--checkers 1 --transfers 1`.
+### Deleting files
+By default rclone will send all files to the trash when deleting
+files. If deleting them permanently is required then use the
+`--drive-use-trash=false` flag, or set the equivalent environment
+variable.
+### Shortcuts
-Properties:
+In March 2020 Google introduced a new feature in Google Drive called
+[drive shortcuts](https://support.google.com/drive/answer/9700156)
+([API](https://developers.google.com/drive/api/v3/shortcuts)). These
+will (by September 2020) [replace the ability for files or folders to
+be in multiple folders at once](https://cloud.google.com/blog/products/g-suite/simplifying-google-drives-folder-structure-and-sharing-models).
-- Config: concurrency
-- Env Var: RCLONE_FTP_CONCURRENCY
-- Type: int
-- Default: 0
+Shortcuts are files that link to other files on Google Drive somewhat
+like a symlink in unix, except they point to the underlying file data
+(e.g. the inode in unix terms) so they don't break if the source is
+renamed or moved about.
-#### --ftp-no-check-certificate
+By default rclone treats these as follows.
-Do not verify the TLS certificate of the server.
+For shortcuts pointing to files:
-Properties:
+- When listing a file shortcut appears as the destination file.
+- When downloading the contents of the destination file is downloaded.
+- When updating shortcut file with a non shortcut file, the shortcut is removed then a new file is uploaded in place of the shortcut.
+- When server-side moving (renaming) the shortcut is renamed, not the destination file.
+- When server-side copying the shortcut is copied, not the contents of the shortcut. (unless `--drive-copy-shortcut-content` is in use in which case the contents of the shortcut gets copied).
+- When deleting the shortcut is deleted not the linked file.
+- When setting the modification time, the modification time of the linked file will be set.
-- Config: no_check_certificate
-- Env Var: RCLONE_FTP_NO_CHECK_CERTIFICATE
-- Type: bool
-- Default: false
+For shortcuts pointing to folders:
-#### --ftp-disable-epsv
+- When listing the shortcut appears as a folder and that folder will contain the contents of the linked folder appear (including any sub folders)
+- When downloading the contents of the linked folder and sub contents are downloaded
+- When uploading to a shortcut folder the file will be placed in the linked folder
+- When server-side moving (renaming) the shortcut is renamed, not the destination folder
+- When server-side copying the contents of the linked folder is copied, not the shortcut.
+- When deleting with `rclone rmdir` or `rclone purge` the shortcut is deleted not the linked folder.
+- **NB** When deleting with `rclone remove` or `rclone mount` the contents of the linked folder will be deleted.
-Disable using EPSV even if server advertises support.
+The [rclone backend](https://rclone.org/commands/rclone_backend/) command can be used to create shortcuts.
-Properties:
+Shortcuts can be completely ignored with the `--drive-skip-shortcuts` flag
+or the corresponding `skip_shortcuts` configuration setting.
-- Config: disable_epsv
-- Env Var: RCLONE_FTP_DISABLE_EPSV
-- Type: bool
-- Default: false
+### Emptying trash
-#### --ftp-disable-mlsd
+If you wish to empty your trash you can use the `rclone cleanup remote:`
+command which will permanently delete all your trashed files. This command
+does not take any path arguments.
-Disable using MLSD even if server advertises support.
+Note that Google Drive takes some time (minutes to days) to empty the
+trash even though the command returns within a few seconds. No output
+is echoed, so there will be no confirmation even using -v or -vv.
-Properties:
+### Quota information
-- Config: disable_mlsd
-- Env Var: RCLONE_FTP_DISABLE_MLSD
-- Type: bool
-- Default: false
+To view your current quota you can use the `rclone about remote:`
+command which will display your usage limit (quota), the usage in Google
+Drive, the size of all files in the Trash and the space used by other
+Google services such as Gmail. This command does not take any path
+arguments.
-#### --ftp-disable-utf8
+#### Import/Export of google documents
-Disable using UTF-8 even if server advertises support.
+Google documents can be exported from and uploaded to Google Drive.
-Properties:
+When rclone downloads a Google doc it chooses a format to download
+depending upon the `--drive-export-formats` setting.
+By default the export formats are `docx,xlsx,pptx,svg` which are a
+sensible default for an editable document.
-- Config: disable_utf8
-- Env Var: RCLONE_FTP_DISABLE_UTF8
-- Type: bool
-- Default: false
+When choosing a format, rclone runs down the list provided in order
+and chooses the first file format the doc can be exported as from the
+list. If the file can't be exported to a format on the formats list,
+then rclone will choose a format from the default list.
-#### --ftp-writing-mdtm
+If you prefer an archive copy then you might use `--drive-export-formats
+pdf`, or if you prefer openoffice/libreoffice formats you might use
+`--drive-export-formats ods,odt,odp`.
-Use MDTM to set modification time (VsFtpd quirk)
+Note that rclone adds the extension to the google doc, so if it is
+called `My Spreadsheet` on google docs, it will be exported as `My
+Spreadsheet.xlsx` or `My Spreadsheet.pdf` etc.
-Properties:
+When importing files into Google Drive, rclone will convert all
+files with an extension in `--drive-import-formats` to their
+associated document type.
+rclone will not convert any files by default, since the conversion
+is lossy process.
-- Config: writing_mdtm
-- Env Var: RCLONE_FTP_WRITING_MDTM
-- Type: bool
-- Default: false
+The conversion must result in a file with the same extension when
+the `--drive-export-formats` rules are applied to the uploaded document.
-#### --ftp-force-list-hidden
+Here are some examples for allowed and prohibited conversions.
-Use LIST -a to force listing of hidden files and folders. This will disable the use of MLSD.
+| export-formats | import-formats | Upload Ext | Document Ext | Allowed |
+| -------------- | -------------- | ---------- | ------------ | ------- |
+| odt | odt | odt | odt | Yes |
+| odt | docx,odt | odt | odt | Yes |
+| | docx | docx | docx | Yes |
+| | odt | odt | docx | No |
+| odt,docx | docx,odt | docx | odt | No |
+| docx,odt | docx,odt | docx | docx | Yes |
+| docx,odt | docx,odt | odt | docx | No |
-Properties:
+This limitation can be disabled by specifying `--drive-allow-import-name-change`.
+When using this flag, rclone can convert multiple files types resulting
+in the same document type at once, e.g. with `--drive-import-formats docx,odt,txt`,
+all files having these extension would result in a document represented as a docx file.
+This brings the additional risk of overwriting a document, if multiple files
+have the same stem. Many rclone operations will not handle this name change
+in any way. They assume an equal name when copying files and might copy the
+file again or delete them when the name changes.
-- Config: force_list_hidden
-- Env Var: RCLONE_FTP_FORCE_LIST_HIDDEN
-- Type: bool
-- Default: false
+Here are the possible export extensions with their corresponding mime types.
+Most of these can also be used for importing, but there more that are not
+listed here. Some of these additional ones might only be available when
+the operating system provides the correct MIME type entries.
-#### --ftp-idle-timeout
+This list can be changed by Google Drive at any time and might not
+represent the currently available conversions.
-Max time before closing idle connections.
+| Extension | Mime Type | Description |
+| --------- |-----------| ------------|
+| bmp | image/bmp | Windows Bitmap format |
+| csv | text/csv | Standard CSV format for Spreadsheets |
+| doc | application/msword | Classic Word file |
+| docx | application/vnd.openxmlformats-officedocument.wordprocessingml.document | Microsoft Office Document |
+| epub | application/epub+zip | E-book format |
+| html | text/html | An HTML Document |
+| jpg | image/jpeg | A JPEG Image File |
+| json | application/vnd.google-apps.script+json | JSON Text Format for Google Apps scripts |
+| odp | application/vnd.oasis.opendocument.presentation | Openoffice Presentation |
+| ods | application/vnd.oasis.opendocument.spreadsheet | Openoffice Spreadsheet |
+| ods | application/x-vnd.oasis.opendocument.spreadsheet | Openoffice Spreadsheet |
+| odt | application/vnd.oasis.opendocument.text | Openoffice Document |
+| pdf | application/pdf | Adobe PDF Format |
+| pjpeg | image/pjpeg | Progressive JPEG Image |
+| png | image/png | PNG Image Format|
+| pptx | application/vnd.openxmlformats-officedocument.presentationml.presentation | Microsoft Office Powerpoint |
+| rtf | application/rtf | Rich Text Format |
+| svg | image/svg+xml | Scalable Vector Graphics Format |
+| tsv | text/tab-separated-values | Standard TSV format for spreadsheets |
+| txt | text/plain | Plain Text |
+| wmf | application/x-msmetafile | Windows Meta File |
+| xls | application/vnd.ms-excel | Classic Excel file |
+| xlsx | application/vnd.openxmlformats-officedocument.spreadsheetml.sheet | Microsoft Office Spreadsheet |
+| zip | application/zip | A ZIP file of HTML, Images CSS |
-If no connections have been returned to the connection pool in the time
-given, rclone will empty the connection pool.
+Google documents can also be exported as link files. These files will
+open a browser window for the Google Docs website of that document
+when opened. The link file extension has to be specified as a
+`--drive-export-formats` parameter. They will match all available
+Google Documents.
-Set to 0 to keep connections indefinitely.
+| Extension | Description | OS Support |
+| --------- | ----------- | ---------- |
+| desktop | freedesktop.org specified desktop entry | Linux |
+| link.html | An HTML Document with a redirect | All |
+| url | INI style link file | macOS, Windows |
+| webloc | macOS specific XML format | macOS |
-Properties:
+### Standard options
-- Config: idle_timeout
-- Env Var: RCLONE_FTP_IDLE_TIMEOUT
-- Type: Duration
-- Default: 1m0s
+Here are the Standard options specific to drive (Google Drive).
-#### --ftp-close-timeout
+#### --drive-client-id
-Maximum time to wait for a response to close.
+Google Application Client Id
+Setting your own is recommended.
+See https://rclone.org/drive/#making-your-own-client-id for how to create your own.
+If you leave this blank, it will use an internal key which is low performance.
Properties:
-- Config: close_timeout
-- Env Var: RCLONE_FTP_CLOSE_TIMEOUT
-- Type: Duration
-- Default: 1m0s
-
-#### --ftp-tls-cache-size
+- Config: client_id
+- Env Var: RCLONE_DRIVE_CLIENT_ID
+- Type: string
+- Required: false
-Size of TLS session cache for all control and data connections.
+#### --drive-client-secret
-TLS cache allows to resume TLS sessions and reuse PSK between connections.
-Increase if default size is not enough resulting in TLS resumption errors.
-Enabled by default. Use 0 to disable.
+OAuth Client Secret.
+
+Leave blank normally.
Properties:
-- Config: tls_cache_size
-- Env Var: RCLONE_FTP_TLS_CACHE_SIZE
-- Type: int
-- Default: 32
+- Config: client_secret
+- Env Var: RCLONE_DRIVE_CLIENT_SECRET
+- Type: string
+- Required: false
-#### --ftp-disable-tls13
+#### --drive-scope
-Disable TLS 1.3 (workaround for FTP servers with buggy TLS)
+Comma separated list of scopes that rclone should use when requesting access from drive.
Properties:
-- Config: disable_tls13
-- Env Var: RCLONE_FTP_DISABLE_TLS13
-- Type: bool
-- Default: false
+- Config: scope
+- Env Var: RCLONE_DRIVE_SCOPE
+- Type: string
+- Required: false
+- Examples:
+ - "drive"
+ - Full access all files, excluding Application Data Folder.
+ - "drive.readonly"
+ - Read-only access to file metadata and file contents.
+ - "drive.file"
+ - Access to files created by rclone only.
+ - These are visible in the drive website.
+ - File authorization is revoked when the user deauthorizes the app.
+ - "drive.appfolder"
+ - Allows read and write access to the Application Data folder.
+ - This is not visible in the drive website.
+ - "drive.metadata.readonly"
+ - Allows read-only access to file metadata but
+ - does not allow any access to read or download file content.
-#### --ftp-shut-timeout
+#### --drive-service-account-file
-Maximum time to wait for data connection closing status.
+Service Account Credentials JSON file path.
-Properties:
+Leave blank normally.
+Needed only if you want use SA instead of interactive login.
-- Config: shut_timeout
-- Env Var: RCLONE_FTP_SHUT_TIMEOUT
-- Type: Duration
-- Default: 1m0s
+Leading `~` will be expanded in the file name as will environment variables such as `${RCLONE_CONFIG_DIR}`.
-#### --ftp-ask-password
+Properties:
-Allow asking for FTP password when needed.
+- Config: service_account_file
+- Env Var: RCLONE_DRIVE_SERVICE_ACCOUNT_FILE
+- Type: string
+- Required: false
-If this is set and no password is supplied then rclone will ask for a password
+#### --drive-alternate-export
+Deprecated: No longer needed.
Properties:
-- Config: ask_password
-- Env Var: RCLONE_FTP_ASK_PASSWORD
+- Config: alternate_export
+- Env Var: RCLONE_DRIVE_ALTERNATE_EXPORT
- Type: bool
- Default: false
-#### --ftp-socks-proxy
+### Advanced options
-Socks 5 proxy host.
-
- Supports the format user:pass@host:port, user@host:port, host:port.
-
- Example:
-
- myUser:myPass@localhost:9005
-
+Here are the Advanced options specific to drive (Google Drive).
+
+#### --drive-token
+
+OAuth Access Token as a JSON blob.
Properties:
-- Config: socks_proxy
-- Env Var: RCLONE_FTP_SOCKS_PROXY
+- Config: token
+- Env Var: RCLONE_DRIVE_TOKEN
- Type: string
- Required: false
-#### --ftp-encoding
+#### --drive-auth-url
-The encoding for the backend.
+Auth server URL.
-See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info.
+Leave blank to use the provider defaults.
Properties:
-- Config: encoding
-- Env Var: RCLONE_FTP_ENCODING
-- Type: MultiEncoder
-- Default: Slash,Del,Ctl,RightSpace,Dot
-- Examples:
- - "Asterisk,Ctl,Dot,Slash"
- - ProFTPd can't handle '*' in file names
- - "BackSlash,Ctl,Del,Dot,RightSpace,Slash,SquareBracket"
- - PureFTPd can't handle '[]' or '*' in file names
- - "Ctl,LeftPeriod,Slash"
- - VsFTPd can't handle file names starting with dot
+- Config: auth_url
+- Env Var: RCLONE_DRIVE_AUTH_URL
+- Type: string
+- Required: false
+#### --drive-token-url
+Token server url.
-## Limitations
+Leave blank to use the provider defaults.
-FTP servers acting as rclone remotes must support `passive` mode.
-The mode cannot be configured as `passive` is the only supported one.
-Rclone's FTP implementation is not compatible with `active` mode
-as [the library it uses doesn't support it](https://github.com/jlaffaye/ftp/issues/29).
-This will likely never be supported due to security concerns.
+Properties:
-Rclone's FTP backend does not support any checksums but can compare
-file sizes.
+- Config: token_url
+- Env Var: RCLONE_DRIVE_TOKEN_URL
+- Type: string
+- Required: false
-`rclone about` is not supported by the FTP backend. Backends without
-this capability cannot determine free space for an rclone mount or
-use policy `mfs` (most free space) as a member of an rclone union
-remote.
+#### --drive-root-folder-id
-See [List of backends that do not support rclone about](https://rclone.org/overview/#optional-features) and [rclone about](https://rclone.org/commands/rclone_about/)
+ID of the root folder.
+Leave blank normally.
-The implementation of : `--dump headers`,
-`--dump bodies`, `--dump auth` for debugging isn't the same as
-for rclone HTTP based backends - it has less fine grained control.
+Fill in to access "Computers" folders (see docs), or for rclone to use
+a non root folder as its starting point.
-`--timeout` isn't supported (but `--contimeout` is).
-`--bind` isn't supported.
+Properties:
-Rclone's FTP backend could support server-side move but does not
-at present.
+- Config: root_folder_id
+- Env Var: RCLONE_DRIVE_ROOT_FOLDER_ID
+- Type: string
+- Required: false
-The `ftp_proxy` environment variable is not currently supported.
+#### --drive-service-account-credentials
-#### Modified time
+Service Account Credentials JSON blob.
-File modification time (timestamps) is supported to 1 second resolution
-for major FTP servers: ProFTPd, PureFTPd, VsFTPd, and FileZilla FTP server.
-The `VsFTPd` server has non-standard implementation of time related protocol
-commands and needs a special configuration setting: `writing_mdtm = true`.
+Leave blank normally.
+Needed only if you want use SA instead of interactive login.
-Support for precise file time with other FTP servers varies depending on what
-protocol extensions they advertise. If all the `MLSD`, `MDTM` and `MFTM`
-extensions are present, rclone will use them together to provide precise time.
-Otherwise the times you see on the FTP server through rclone are those of the
-last file upload.
+Properties:
-You can use the following command to check whether rclone can use precise time
-with your FTP server: `rclone backend features your_ftp_remote:` (the trailing
-colon is important). Look for the number in the line tagged by `Precision`
-designating the remote time precision expressed as nanoseconds. A value of
-`1000000000` means that file time precision of 1 second is available.
-A value of `3153600000000000000` (or another large number) means "unsupported".
+- Config: service_account_credentials
+- Env Var: RCLONE_DRIVE_SERVICE_ACCOUNT_CREDENTIALS
+- Type: string
+- Required: false
-# Google Cloud Storage
+#### --drive-team-drive
-Paths are specified as `remote:bucket` (or `remote:` for the `lsd`
-command.) You may put subdirectories in too, e.g. `remote:bucket/path/to/dir`.
+ID of the Shared Drive (Team Drive).
-## Configuration
+Properties:
-The initial setup for google cloud storage involves getting a token from Google Cloud Storage
-which you need to do in your browser. `rclone config` walks you
-through it.
+- Config: team_drive
+- Env Var: RCLONE_DRIVE_TEAM_DRIVE
+- Type: string
+- Required: false
-Here is an example of how to make a remote called `remote`. First run:
+#### --drive-auth-owner-only
- rclone config
+Only consider files owned by the authenticated user.
-This will guide you through an interactive setup process:
-
-
-- New remote
-- Delete remote
-- Quit config e/n/d/q> n name> remote Type of storage to configure. Choose a number from below, or type in your own value [snip] XX / Google Cloud Storage (this is not Google Drive)  "google cloud storage" [snip] Storage> google cloud storage Google Application Client Id - leave blank normally. client_id> Google Application Client Secret - leave blank normally. client_secret> Project number optional - needed only for list/create/delete buckets - see your developer console. project_number> 12345678 Service Account Credentials JSON file path - needed only if you want use SA instead of interactive login. service_account_file> Access Control List for new objects. Choose a number from below, or type in your own value 1 / Object owner gets OWNER access, and all Authenticated Users get READER access.  "authenticatedRead" 2 / Object owner gets OWNER access, and project team owners get OWNER access.  "bucketOwnerFullControl" 3 / Object owner gets OWNER access, and project team owners get READER access.  "bucketOwnerRead" 4 / Object owner gets OWNER access [default if left blank].  "private" 5 / Object owner gets OWNER access, and project team members get access according to their roles.  "projectPrivate" 6 / Object owner gets OWNER access, and all Users get READER access.  "publicRead" object_acl> 4 Access Control List for new buckets. Choose a number from below, or type in your own value 1 / Project team owners get OWNER access, and all Authenticated Users get READER access.  "authenticatedRead" 2 / Project team owners get OWNER access [default if left blank].  "private" 3 / Project team members get access according to their roles.  "projectPrivate" 4 / Project team owners get OWNER access, and all Users get READER access.  "publicRead" 5 / Project team owners get OWNER access, and all Users get WRITER access.  "publicReadWrite" bucket_acl> 2 Location for the newly created buckets. Choose a number from below, or type in your own value 1 / Empty for default location (US).  "" 2 / Multi-regional location for Asia.  "asia" 3 / Multi-regional location for Europe.  "eu" 4 / Multi-regional location for United States.  "us" 5 / Taiwan.  "asia-east1" 6 / Tokyo.  "asia-northeast1" 7 / Singapore.  "asia-southeast1" 8 / Sydney.  "australia-southeast1" 9 / Belgium.  "europe-west1" 10 / London.  "europe-west2" 11 / Iowa.  "us-central1" 12 / South Carolina.  "us-east1" 13 / Northern Virginia.  "us-east4" 14 / Oregon.  "us-west1" location> 12 The storage class to use when storing objects in Google Cloud Storage. Choose a number from below, or type in your own value 1 / Default  "" 2 / Multi-regional storage class  "MULTI_REGIONAL" 3 / Regional storage class  "REGIONAL" 4 / Nearline storage class  "NEARLINE" 5 / Coldline storage class  "COLDLINE" 6 / Durable reduced availability storage class  "DURABLE_REDUCED_AVAILABILITY" storage_class> 5 Remote config Use web browser to automatically authenticate rclone with remote?
-
-
-- Say Y if the machine running rclone has a web browser you can use
-- Say N if running rclone on a (remote) machine without web browser access If not sure try Y. If Y failed, try N.
-
-
-- Yes
-- No y/n> y If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth Log in and authorize rclone for access Waiting for code... Got code -------------------- [remote] type = google cloud storage client_id = client_secret = token = {"AccessToken":"xxxx.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","RefreshToken":"x/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx_xxxxxxxxx","Expiry":"2014-07-17T20:49:14.929208288+01:00","Extra":null} project_number = 12345678 object_acl = private bucket_acl = private --------------------
-- Yes this is OK
-- Edit this remote
-- Delete this remote y/e/d> y
-
-
-See the [remote setup docs](https://rclone.org/remote_setup/) for how to set it up on a
-machine with no Internet browser available.
+Properties:
-Note that rclone runs a webserver on your local machine to collect the
-token as returned from Google if using web browser to automatically
-authenticate. This only
-runs from the moment it opens your browser to the moment you get back
-the verification code. This is on `http://127.0.0.1:53682/` and this
-it may require you to unblock it temporarily if you are running a host
-firewall, or use manual mode.
+- Config: auth_owner_only
+- Env Var: RCLONE_DRIVE_AUTH_OWNER_ONLY
+- Type: bool
+- Default: false
-This remote is called `remote` and can now be used like this
+#### --drive-use-trash
-See all the buckets in your project
+Send files to the trash instead of deleting permanently.
- rclone lsd remote:
+Defaults to true, namely sending files to the trash.
+Use `--drive-use-trash=false` to delete files permanently instead.
-Make a new bucket
+Properties:
- rclone mkdir remote:bucket
+- Config: use_trash
+- Env Var: RCLONE_DRIVE_USE_TRASH
+- Type: bool
+- Default: true
-List the contents of a bucket
+#### --drive-copy-shortcut-content
- rclone ls remote:bucket
+Server side copy contents of shortcuts instead of the shortcut.
-Sync `/home/local/directory` to the remote bucket, deleting any excess
-files in the bucket.
+When doing server side copies, normally rclone will copy shortcuts as
+shortcuts.
- rclone sync --interactive /home/local/directory remote:bucket
+If this flag is used then rclone will copy the contents of shortcuts
+rather than shortcuts themselves when doing server side copies.
-### Service Account support
+Properties:
-You can set up rclone with Google Cloud Storage in an unattended mode,
-i.e. not tied to a specific end-user Google account. This is useful
-when you want to synchronise files onto machines that don't have
-actively logged-in users, for example build machines.
+- Config: copy_shortcut_content
+- Env Var: RCLONE_DRIVE_COPY_SHORTCUT_CONTENT
+- Type: bool
+- Default: false
-To get credentials for Google Cloud Platform
-[IAM Service Accounts](https://cloud.google.com/iam/docs/service-accounts),
-please head to the
-[Service Account](https://console.cloud.google.com/permissions/serviceaccounts)
-section of the Google Developer Console. Service Accounts behave just
-like normal `User` permissions in
-[Google Cloud Storage ACLs](https://cloud.google.com/storage/docs/access-control),
-so you can limit their access (e.g. make them read only). After
-creating an account, a JSON file containing the Service Account's
-credentials will be downloaded onto your machines. These credentials
-are what rclone will use for authentication.
+#### --drive-skip-gdocs
-To use a Service Account instead of OAuth2 token flow, enter the path
-to your Service Account credentials at the `service_account_file`
-prompt and rclone won't use the browser based authentication
-flow. If you'd rather stuff the contents of the credentials file into
-the rclone config file, you can set `service_account_credentials` with
-the actual contents of the file instead, or set the equivalent
-environment variable.
+Skip google documents in all listings.
-### Anonymous Access
+If given, gdocs practically become invisible to rclone.
-For downloads of objects that permit public access you can configure rclone
-to use anonymous access by setting `anonymous` to `true`.
-With unauthorized access you can't write or create files but only read or list
-those buckets and objects that have public read access.
+Properties:
-### Application Default Credentials
+- Config: skip_gdocs
+- Env Var: RCLONE_DRIVE_SKIP_GDOCS
+- Type: bool
+- Default: false
-If no other source of credentials is provided, rclone will fall back
-to
-[Application Default Credentials](https://cloud.google.com/video-intelligence/docs/common/auth#authenticating_with_application_default_credentials)
-this is useful both when you already have configured authentication
-for your developer account, or in production when running on a google
-compute host. Note that if running in docker, you may need to run
-additional commands on your google compute machine -
-[see this page](https://cloud.google.com/container-registry/docs/advanced-authentication#gcloud_as_a_docker_credential_helper).
+#### --drive-show-all-gdocs
-Note that in the case application default credentials are used, there
-is no need to explicitly configure a project number.
+Show all Google Docs including non-exportable ones in listings.
-### --fast-list
+If you try a server side copy on a Google Form without this flag, you
+will get this error:
-This remote supports `--fast-list` which allows you to use fewer
-transactions in exchange for more memory. See the [rclone
-docs](https://rclone.org/docs/#fast-list) for more details.
+ No export formats found for "application/vnd.google-apps.form"
-### Custom upload headers
+However adding this flag will allow the form to be server side copied.
-You can set custom upload headers with the `--header-upload`
-flag. Google Cloud Storage supports the headers as described in the
-[working with metadata documentation](https://cloud.google.com/storage/docs/gsutil/addlhelp/WorkingWithObjectMetadata)
+Note that rclone doesn't add extensions to the Google Docs file names
+in this mode.
-- Cache-Control
-- Content-Disposition
-- Content-Encoding
-- Content-Language
-- Content-Type
-- X-Goog-Storage-Class
-- X-Goog-Meta-
+Do **not** use this flag when trying to download Google Docs - rclone
+will fail to download them.
-Eg `--header-upload "Content-Type text/potato"`
-Note that the last of these is for setting custom metadata in the form
-`--header-upload "x-goog-meta-key: value"`
+Properties:
-### Modification time
+- Config: show_all_gdocs
+- Env Var: RCLONE_DRIVE_SHOW_ALL_GDOCS
+- Type: bool
+- Default: false
-Google Cloud Storage stores md5sum natively.
-Google's [gsutil](https://cloud.google.com/storage/docs/gsutil) tool stores modification time
-with one-second precision as `goog-reserved-file-mtime` in file metadata.
+#### --drive-skip-checksum-gphotos
-To ensure compatibility with gsutil, rclone stores modification time in 2 separate metadata entries.
-`mtime` uses RFC3339 format with one-nanosecond precision.
-`goog-reserved-file-mtime` uses Unix timestamp format with one-second precision.
-To get modification time from object metadata, rclone reads the metadata in the following order: `mtime`, `goog-reserved-file-mtime`, object updated time.
+Skip checksums on Google photos and videos only.
-Note that rclone's default modify window is 1ns.
-Files uploaded by gsutil only contain timestamps with one-second precision.
-If you use rclone to sync files previously uploaded by gsutil,
-rclone will attempt to update modification time for all these files.
-To avoid these possibly unnecessary updates, use `--modify-window 1s`.
+Use this if you get checksum errors when transferring Google photos or
+videos.
-### Restricted filename characters
+Setting this flag will cause Google photos and videos to return a
+blank checksums.
-| Character | Value | Replacement |
-| --------- |:-----:|:-----------:|
-| NUL | 0x00 | †|
-| LF | 0x0A | ⊠|
-| CR | 0x0D | â |
-| / | 0x2F | ï¼ |
+Google photos are identified by being in the "photos" space.
-Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8),
-as they can't be used in JSON strings.
+Corrupted checksums are caused by Google modifying the image/video but
+not updating the checksum.
+Properties:
-### Standard options
+- Config: skip_checksum_gphotos
+- Env Var: RCLONE_DRIVE_SKIP_CHECKSUM_GPHOTOS
+- Type: bool
+- Default: false
-Here are the Standard options specific to google cloud storage (Google Cloud Storage (this is not Google Drive)).
+#### --drive-shared-with-me
-#### --gcs-client-id
+Only show files that are shared with me.
-OAuth Client Id.
+Instructs rclone to operate on your "Shared with me" folder (where
+Google Drive lets you access the files and folders others have shared
+with you).
-Leave blank normally.
+This works both with the "list" (lsd, lsl, etc.) and the "copy"
+commands (copy, sync, etc.), and with all other commands too.
Properties:
-- Config: client_id
-- Env Var: RCLONE_GCS_CLIENT_ID
-- Type: string
-- Required: false
+- Config: shared_with_me
+- Env Var: RCLONE_DRIVE_SHARED_WITH_ME
+- Type: bool
+- Default: false
-#### --gcs-client-secret
+#### --drive-trashed-only
-OAuth Client Secret.
+Only show files that are in the trash.
-Leave blank normally.
+This will show trashed files in their original directory structure.
Properties:
-- Config: client_secret
-- Env Var: RCLONE_GCS_CLIENT_SECRET
-- Type: string
-- Required: false
-
-#### --gcs-project-number
+- Config: trashed_only
+- Env Var: RCLONE_DRIVE_TRASHED_ONLY
+- Type: bool
+- Default: false
-Project number.
+#### --drive-starred-only
-Optional - needed only for list/create/delete buckets - see your developer console.
+Only show files that are starred.
Properties:
-- Config: project_number
-- Env Var: RCLONE_GCS_PROJECT_NUMBER
-- Type: string
-- Required: false
-
-#### --gcs-user-project
+- Config: starred_only
+- Env Var: RCLONE_DRIVE_STARRED_ONLY
+- Type: bool
+- Default: false
-User project.
+#### --drive-formats
-Optional - needed only for requester pays.
+Deprecated: See export_formats.
Properties:
-- Config: user_project
-- Env Var: RCLONE_GCS_USER_PROJECT
+- Config: formats
+- Env Var: RCLONE_DRIVE_FORMATS
- Type: string
- Required: false
-#### --gcs-service-account-file
-
-Service Account Credentials JSON file path.
-
-Leave blank normally.
-Needed only if you want use SA instead of interactive login.
+#### --drive-export-formats
-Leading `~` will be expanded in the file name as will environment variables such as `${RCLONE_CONFIG_DIR}`.
+Comma separated list of preferred formats for downloading Google docs.
Properties:
-- Config: service_account_file
-- Env Var: RCLONE_GCS_SERVICE_ACCOUNT_FILE
+- Config: export_formats
+- Env Var: RCLONE_DRIVE_EXPORT_FORMATS
- Type: string
-- Required: false
-
-#### --gcs-service-account-credentials
+- Default: "docx,xlsx,pptx,svg"
-Service Account Credentials JSON blob.
+#### --drive-import-formats
-Leave blank normally.
-Needed only if you want use SA instead of interactive login.
+Comma separated list of preferred formats for uploading Google docs.
Properties:
-- Config: service_account_credentials
-- Env Var: RCLONE_GCS_SERVICE_ACCOUNT_CREDENTIALS
+- Config: import_formats
+- Env Var: RCLONE_DRIVE_IMPORT_FORMATS
- Type: string
- Required: false
-#### --gcs-anonymous
+#### --drive-allow-import-name-change
-Access public buckets and objects without credentials.
+Allow the filetype to change when uploading Google docs.
-Set to 'true' if you just want to download files and don't configure credentials.
+E.g. file.doc to file.docx. This will confuse sync and reupload every time.
Properties:
-- Config: anonymous
-- Env Var: RCLONE_GCS_ANONYMOUS
+- Config: allow_import_name_change
+- Env Var: RCLONE_DRIVE_ALLOW_IMPORT_NAME_CHANGE
- Type: bool
- Default: false
-#### --gcs-object-acl
+#### --drive-use-created-date
-Access Control List for new objects.
+Use file created date instead of modified date.
-Properties:
+Useful when downloading data and you want the creation date used in
+place of the last modified date.
-- Config: object_acl
-- Env Var: RCLONE_GCS_OBJECT_ACL
-- Type: string
-- Required: false
-- Examples:
- - "authenticatedRead"
- - Object owner gets OWNER access.
- - All Authenticated Users get READER access.
- - "bucketOwnerFullControl"
- - Object owner gets OWNER access.
- - Project team owners get OWNER access.
- - "bucketOwnerRead"
- - Object owner gets OWNER access.
- - Project team owners get READER access.
- - "private"
- - Object owner gets OWNER access.
- - Default if left blank.
- - "projectPrivate"
- - Object owner gets OWNER access.
- - Project team members get access according to their roles.
- - "publicRead"
- - Object owner gets OWNER access.
- - All Users get READER access.
+**WARNING**: This flag may have some unexpected consequences.
-#### --gcs-bucket-acl
+When uploading to your drive all files will be overwritten unless they
+haven't been modified since their creation. And the inverse will occur
+while downloading. This side effect can be avoided by using the
+"--checksum" flag.
-Access Control List for new buckets.
+This feature was implemented to retain photos capture date as recorded
+by google photos. You will first need to check the "Create a Google
+Photos folder" option in your google drive settings. You can then copy
+or move the photos locally and use the date the image was taken
+(created) set as the modification date.
Properties:
-- Config: bucket_acl
-- Env Var: RCLONE_GCS_BUCKET_ACL
-- Type: string
-- Required: false
-- Examples:
- - "authenticatedRead"
- - Project team owners get OWNER access.
- - All Authenticated Users get READER access.
- - "private"
- - Project team owners get OWNER access.
- - Default if left blank.
- - "projectPrivate"
- - Project team members get access according to their roles.
- - "publicRead"
- - Project team owners get OWNER access.
- - All Users get READER access.
- - "publicReadWrite"
- - Project team owners get OWNER access.
- - All Users get WRITER access.
-
-#### --gcs-bucket-policy-only
-
-Access checks should use bucket-level IAM policies.
-
-If you want to upload objects to a bucket with Bucket Policy Only set
-then you will need to set this.
+- Config: use_created_date
+- Env Var: RCLONE_DRIVE_USE_CREATED_DATE
+- Type: bool
+- Default: false
-When it is set, rclone:
+#### --drive-use-shared-date
-- ignores ACLs set on buckets
-- ignores ACLs set on objects
-- creates buckets with Bucket Policy Only set
+Use date file was shared instead of modified date.
-Docs: https://cloud.google.com/storage/docs/bucket-policy-only
+Note that, as with "--drive-use-created-date", this flag may have
+unexpected consequences when uploading/downloading files.
+If both this flag and "--drive-use-created-date" are set, the created
+date is used.
Properties:
-- Config: bucket_policy_only
-- Env Var: RCLONE_GCS_BUCKET_POLICY_ONLY
+- Config: use_shared_date
+- Env Var: RCLONE_DRIVE_USE_SHARED_DATE
- Type: bool
- Default: false
-#### --gcs-location
+#### --drive-list-chunk
-Location for the newly created buckets.
+Size of listing chunk 100-1000, 0 to disable.
Properties:
-- Config: location
-- Env Var: RCLONE_GCS_LOCATION
-- Type: string
-- Required: false
-- Examples:
- - ""
- - Empty for default location (US)
- - "asia"
- - Multi-regional location for Asia
- - "eu"
- - Multi-regional location for Europe
- - "us"
- - Multi-regional location for United States
- - "asia-east1"
- - Taiwan
- - "asia-east2"
- - Hong Kong
- - "asia-northeast1"
- - Tokyo
- - "asia-northeast2"
- - Osaka
- - "asia-northeast3"
- - Seoul
- - "asia-south1"
- - Mumbai
- - "asia-south2"
- - Delhi
- - "asia-southeast1"
- - Singapore
- - "asia-southeast2"
- - Jakarta
- - "australia-southeast1"
- - Sydney
- - "australia-southeast2"
- - Melbourne
- - "europe-north1"
- - Finland
- - "europe-west1"
- - Belgium
- - "europe-west2"
- - London
- - "europe-west3"
- - Frankfurt
- - "europe-west4"
- - Netherlands
- - "europe-west6"
- - Zürich
- - "europe-central2"
- - Warsaw
- - "us-central1"
- - Iowa
- - "us-east1"
- - South Carolina
- - "us-east4"
- - Northern Virginia
- - "us-west1"
- - Oregon
- - "us-west2"
- - California
- - "us-west3"
- - Salt Lake City
- - "us-west4"
- - Las Vegas
- - "northamerica-northeast1"
- - Montréal
- - "northamerica-northeast2"
- - Toronto
- - "southamerica-east1"
- - São Paulo
- - "southamerica-west1"
- - Santiago
- - "asia1"
- - Dual region: asia-northeast1 and asia-northeast2.
- - "eur4"
- - Dual region: europe-north1 and europe-west4.
- - "nam4"
- - Dual region: us-central1 and us-east1.
+- Config: list_chunk
+- Env Var: RCLONE_DRIVE_LIST_CHUNK
+- Type: int
+- Default: 1000
-#### --gcs-storage-class
+#### --drive-impersonate
-The storage class to use when storing objects in Google Cloud Storage.
+Impersonate this user when using a service account.
Properties:
-- Config: storage_class
-- Env Var: RCLONE_GCS_STORAGE_CLASS
+- Config: impersonate
+- Env Var: RCLONE_DRIVE_IMPERSONATE
- Type: string
- Required: false
-- Examples:
- - ""
- - Default
- - "MULTI_REGIONAL"
- - Multi-regional storage class
- - "REGIONAL"
- - Regional storage class
- - "NEARLINE"
- - Nearline storage class
- - "COLDLINE"
- - Coldline storage class
- - "ARCHIVE"
- - Archive storage class
- - "DURABLE_REDUCED_AVAILABILITY"
- - Durable reduced availability storage class
-#### --gcs-env-auth
-
-Get GCP IAM credentials from runtime (environment variables or instance meta data if no env vars).
+#### --drive-upload-cutoff
-Only applies if service_account_file and service_account_credentials is blank.
+Cutoff for switching to chunked upload.
Properties:
-- Config: env_auth
-- Env Var: RCLONE_GCS_ENV_AUTH
-- Type: bool
-- Default: false
-- Examples:
- - "false"
- - Enter credentials in the next step.
- - "true"
- - Get GCP IAM credentials from the environment (env vars or IAM).
+- Config: upload_cutoff
+- Env Var: RCLONE_DRIVE_UPLOAD_CUTOFF
+- Type: SizeSuffix
+- Default: 8Mi
-### Advanced options
+#### --drive-chunk-size
-Here are the Advanced options specific to google cloud storage (Google Cloud Storage (this is not Google Drive)).
+Upload chunk size.
-#### --gcs-token
+Must a power of 2 >= 256k.
-OAuth Access Token as a JSON blob.
+Making this larger will improve performance, but note that each chunk
+is buffered in memory one per transfer.
+
+Reducing this will reduce memory usage but decrease performance.
Properties:
-- Config: token
-- Env Var: RCLONE_GCS_TOKEN
-- Type: string
-- Required: false
+- Config: chunk_size
+- Env Var: RCLONE_DRIVE_CHUNK_SIZE
+- Type: SizeSuffix
+- Default: 8Mi
-#### --gcs-auth-url
+#### --drive-acknowledge-abuse
-Auth server URL.
+Set to allow files which return cannotDownloadAbusiveFile to be downloaded.
-Leave blank to use the provider defaults.
+If downloading a file returns the error "This file has been identified
+as malware or spam and cannot be downloaded" with the error code
+"cannotDownloadAbusiveFile" then supply this flag to rclone to
+indicate you acknowledge the risks of downloading the file and rclone
+will download it anyway.
-Properties:
+Note that if you are using service account it will need Manager
+permission (not Content Manager) to for this flag to work. If the SA
+does not have the right permission, Google will just ignore the flag.
-- Config: auth_url
-- Env Var: RCLONE_GCS_AUTH_URL
-- Type: string
-- Required: false
+Properties:
-#### --gcs-token-url
+- Config: acknowledge_abuse
+- Env Var: RCLONE_DRIVE_ACKNOWLEDGE_ABUSE
+- Type: bool
+- Default: false
-Token server url.
+#### --drive-keep-revision-forever
-Leave blank to use the provider defaults.
+Keep new head revision of each file forever.
Properties:
-- Config: token_url
-- Env Var: RCLONE_GCS_TOKEN_URL
-- Type: string
-- Required: false
+- Config: keep_revision_forever
+- Env Var: RCLONE_DRIVE_KEEP_REVISION_FOREVER
+- Type: bool
+- Default: false
-#### --gcs-directory-markers
+#### --drive-size-as-quota
-Upload an empty object with a trailing slash when a new directory is created
+Show sizes as storage quota usage, not actual size.
-Empty folders are unsupported for bucket based remotes, this option creates an empty
-object ending with "/", to persist the folder.
+Show the size of a file as the storage quota used. This is the
+current version plus any older versions that have been set to keep
+forever.
+
+**WARNING**: This flag may have some unexpected consequences.
+
+It is not recommended to set this flag in your config - the
+recommended usage is using the flag form --drive-size-as-quota when
+doing rclone ls/lsl/lsf/lsjson/etc only.
+If you do use this flag for syncing (not recommended) then you will
+need to use --ignore size also.
Properties:
-- Config: directory_markers
-- Env Var: RCLONE_GCS_DIRECTORY_MARKERS
+- Config: size_as_quota
+- Env Var: RCLONE_DRIVE_SIZE_AS_QUOTA
- Type: bool
- Default: false
-#### --gcs-no-check-bucket
+#### --drive-v2-download-min-size
-If set, don't attempt to check the bucket exists or create it.
+If Object's are greater, use drive v2 API to download.
-This can be useful when trying to minimise the number of transactions
-rclone does if you know the bucket exists already.
+Properties:
+
+- Config: v2_download_min_size
+- Env Var: RCLONE_DRIVE_V2_DOWNLOAD_MIN_SIZE
+- Type: SizeSuffix
+- Default: off
+
+#### --drive-pacer-min-sleep
+Minimum time to sleep between API calls.
Properties:
-- Config: no_check_bucket
-- Env Var: RCLONE_GCS_NO_CHECK_BUCKET
-- Type: bool
-- Default: false
+- Config: pacer_min_sleep
+- Env Var: RCLONE_DRIVE_PACER_MIN_SLEEP
+- Type: Duration
+- Default: 100ms
-#### --gcs-decompress
+#### --drive-pacer-burst
-If set this will decompress gzip encoded objects.
+Number of API calls to allow without sleeping.
-It is possible to upload objects to GCS with "Content-Encoding: gzip"
-set. Normally rclone will download these files as compressed objects.
+Properties:
-If this flag is set then rclone will decompress these files with
-"Content-Encoding: gzip" as they are received. This means that rclone
-can't check the size and hash but the file contents will be decompressed.
+- Config: pacer_burst
+- Env Var: RCLONE_DRIVE_PACER_BURST
+- Type: int
+- Default: 100
+#### --drive-server-side-across-configs
+
+Deprecated: use --server-side-across-configs instead.
+
+Allow server-side operations (e.g. copy) to work across different drive configs.
+
+This can be useful if you wish to do a server-side copy between two
+different Google drives. Note that this isn't enabled by default
+because it isn't easy to tell if it will work between any two
+configurations.
Properties:
-- Config: decompress
-- Env Var: RCLONE_GCS_DECOMPRESS
+- Config: server_side_across_configs
+- Env Var: RCLONE_DRIVE_SERVER_SIDE_ACROSS_CONFIGS
- Type: bool
- Default: false
-#### --gcs-endpoint
+#### --drive-disable-http2
+
+Disable drive using http2.
+
+There is currently an unsolved issue with the google drive backend and
+HTTP/2. HTTP/2 is therefore disabled by default for the drive backend
+but can be re-enabled here. When the issue is solved this flag will
+be removed.
+
+See: https://github.com/rclone/rclone/issues/3631
-Endpoint for the service.
-Leave blank normally.
Properties:
-- Config: endpoint
-- Env Var: RCLONE_GCS_ENDPOINT
-- Type: string
-- Required: false
+- Config: disable_http2
+- Env Var: RCLONE_DRIVE_DISABLE_HTTP2
+- Type: bool
+- Default: true
-#### --gcs-encoding
+#### --drive-stop-on-upload-limit
-The encoding for the backend.
+Make upload limit errors be fatal.
+
+At the time of writing it is only possible to upload 750 GiB of data to
+Google Drive a day (this is an undocumented limit). When this limit is
+reached Google Drive produces a slightly different error message. When
+this flag is set it causes these errors to be fatal. These will stop
+the in-progress sync.
+
+Note that this detection is relying on error message strings which
+Google don't document so it may break in the future.
+
+See: https://github.com/rclone/rclone/issues/3857
-See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info.
Properties:
-- Config: encoding
-- Env Var: RCLONE_GCS_ENCODING
-- Type: MultiEncoder
-- Default: Slash,CrLf,InvalidUtf8,Dot
+- Config: stop_on_upload_limit
+- Env Var: RCLONE_DRIVE_STOP_ON_UPLOAD_LIMIT
+- Type: bool
+- Default: false
+#### --drive-stop-on-download-limit
+Make download limit errors be fatal.
-## Limitations
+At the time of writing it is only possible to download 10 TiB of data from
+Google Drive a day (this is an undocumented limit). When this limit is
+reached Google Drive produces a slightly different error message. When
+this flag is set it causes these errors to be fatal. These will stop
+the in-progress sync.
-`rclone about` is not supported by the Google Cloud Storage backend. Backends without
-this capability cannot determine free space for an rclone mount or
-use policy `mfs` (most free space) as a member of an rclone union
-remote.
+Note that this detection is relying on error message strings which
+Google don't document so it may break in the future.
-See [List of backends that do not support rclone about](https://rclone.org/overview/#optional-features) and [rclone about](https://rclone.org/commands/rclone_about/)
-# Google Drive
+Properties:
-Paths are specified as `drive:path`
+- Config: stop_on_download_limit
+- Env Var: RCLONE_DRIVE_STOP_ON_DOWNLOAD_LIMIT
+- Type: bool
+- Default: false
-Drive paths may be as deep as required, e.g. `drive:directory/subdirectory`.
+#### --drive-skip-shortcuts
-## Configuration
+If set skip shortcut files.
-The initial setup for drive involves getting a token from Google drive
-which you need to do in your browser. `rclone config` walks you
-through it.
+Normally rclone dereferences shortcut files making them appear as if
+they are the original file (see [the shortcuts section](#shortcuts)).
+If this flag is set then rclone will ignore shortcut files completely.
-Here is an example of how to make a remote called `remote`. First run:
- rclone config
+Properties:
-This will guide you through an interactive setup process:
-
-No remotes found, make a new one? n) New remote r) Rename remote c) Copy remote s) Set configuration password q) Quit config n/r/c/s/q> n name> remote Type of storage to configure. Choose a number from below, or type in your own value [snip] XX / Google Drive  "drive" [snip] Storage> drive Google Application Client Id - leave blank normally. client_id> Google Application Client Secret - leave blank normally. client_secret> Scope that rclone should use when requesting access from drive. Choose a number from below, or type in your own value 1 / Full access all files, excluding Application Data Folder.  "drive" 2 / Read-only access to file metadata and file contents.  "drive.readonly" / Access to files created by rclone only. 3 | These are visible in the drive website. | File authorization is revoked when the user deauthorizes the app.  "drive.file" / Allows read and write access to the Application Data folder. 4 | This is not visible in the drive website.  "drive.appfolder" / Allows read-only access to file metadata but 5 | does not allow any access to read or download file content.  "drive.metadata.readonly" scope> 1 Service Account Credentials JSON file path - needed only if you want use SA instead of interactive login. service_account_file> Remote config Use web browser to automatically authenticate rclone with remote? * Say Y if the machine running rclone has a web browser you can use * Say N if running rclone on a (remote) machine without web browser access If not sure try Y. If Y failed, try N. y) Yes n) No y/n> y If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth Log in and authorize rclone for access Waiting for code... Got code Configure this as a Shared Drive (Team Drive)? y) Yes n) No y/n> n -------------------- [remote] client_id = client_secret = scope = drive root_folder_id = service_account_file = token = {"access_token":"XXX","token_type":"Bearer","refresh_token":"XXX","expiry":"2014-03-16T13:57:58.955387075Z"} -------------------- y) Yes this is OK e) Edit this remote d) Delete this remote y/e/d> y
-
-See the [remote setup docs](https://rclone.org/remote_setup/) for how to set it up on a
-machine with no Internet browser available.
+- Config: skip_shortcuts
+- Env Var: RCLONE_DRIVE_SKIP_SHORTCUTS
+- Type: bool
+- Default: false
-Note that rclone runs a webserver on your local machine to collect the
-token as returned from Google if using web browser to automatically
-authenticate. This only
-runs from the moment it opens your browser to the moment you get back
-the verification code. This is on `http://127.0.0.1:53682/` and it
-may require you to unblock it temporarily if you are running a host
-firewall, or use manual mode.
+#### --drive-skip-dangling-shortcuts
-You can then use it like this,
+If set skip dangling shortcut files.
-List directories in top level of your drive
+If this is set then rclone will not show any dangling shortcuts in listings.
- rclone lsd remote:
-List all the files in your drive
+Properties:
- rclone ls remote:
+- Config: skip_dangling_shortcuts
+- Env Var: RCLONE_DRIVE_SKIP_DANGLING_SHORTCUTS
+- Type: bool
+- Default: false
-To copy a local directory to a drive directory called backup
+#### --drive-resource-key
- rclone copy /home/source remote:backup
+Resource key for accessing a link-shared file.
-### Scopes
+If you need to access files shared with a link like this
-Rclone allows you to select which scope you would like for rclone to
-use. This changes what type of token is granted to rclone. [The
-scopes are defined
-here](https://developers.google.com/drive/v3/web/about-auth).
+ https://drive.google.com/drive/folders/XXX?resourcekey=YYY&usp=sharing
-The scope are
+Then you will need to use the first part "XXX" as the "root_folder_id"
+and the second part "YYY" as the "resource_key" otherwise you will get
+404 not found errors when trying to access the directory.
-#### drive
+See: https://developers.google.com/drive/api/guides/resource-keys
-This is the default scope and allows full access to all files, except
-for the Application Data Folder (see below).
+This resource key requirement only applies to a subset of old files.
-Choose this one if you aren't sure.
+Note also that opening the folder once in the web interface (with the
+user you've authenticated rclone with) seems to be enough so that the
+resource key is not needed.
-#### drive.readonly
-This allows read only access to all files. Files may be listed and
-downloaded but not uploaded, renamed or deleted.
+Properties:
-#### drive.file
+- Config: resource_key
+- Env Var: RCLONE_DRIVE_RESOURCE_KEY
+- Type: string
+- Required: false
-With this scope rclone can read/view/modify only those files and
-folders it creates.
+#### --drive-fast-list-bug-fix
-So if you uploaded files to drive via the web interface (or any other
-means) they will not be visible to rclone.
+Work around a bug in Google Drive listing.
-This can be useful if you are using rclone to backup data and you want
-to be sure confidential data on your drive is not visible to rclone.
+Normally rclone will work around a bug in Google Drive when using
+--fast-list (ListR) where the search "(A in parents) or (B in
+parents)" returns nothing sometimes. See #3114, #4289 and
+https://issuetracker.google.com/issues/149522397
-Files created with this scope are visible in the web interface.
+Rclone detects this by finding no items in more than one directory
+when listing and retries them as lists of individual directories.
-#### drive.appfolder
+This means that if you have a lot of empty directories rclone will end
+up listing them all individually and this can take many more API
+calls.
-This gives rclone its own private area to store files. Rclone will
-not be able to see any other files on your drive and you won't be able
-to see rclone's files from the web interface either.
+This flag allows the work-around to be disabled. This is **not**
+recommended in normal use - only if you have a particular case you are
+having trouble with like many empty directories.
-#### drive.metadata.readonly
-This allows read only access to file names only. It does not allow
-rclone to download or upload data, or rename or delete files or
-directories.
+Properties:
-### Root folder ID
+- Config: fast_list_bug_fix
+- Env Var: RCLONE_DRIVE_FAST_LIST_BUG_FIX
+- Type: bool
+- Default: true
-This option has been moved to the advanced section. You can set the `root_folder_id` for rclone. This is the directory
-(identified by its `Folder ID`) that rclone considers to be the root
-of your drive.
+#### --drive-metadata-owner
-Normally you will leave this blank and rclone will determine the
-correct root to use itself.
+Control whether owner should be read or written in metadata.
-However you can set this to restrict rclone to a specific folder
-hierarchy or to access data within the "Computers" tab on the drive
-web interface (where files from Google's Backup and Sync desktop
-program go).
+Owner is a standard part of the file metadata so is easy to read. But it
+isn't always desirable to set the owner from the metadata.
-In order to do this you will have to find the `Folder ID` of the
-directory you wish rclone to display. This will be the last segment
-of the URL when you open the relevant folder in the drive web
-interface.
+Note that you can't set the owner on Shared Drives, and that setting
+ownership will generate an email to the new owner (this can't be
+disabled), and you can't transfer ownership to someone outside your
+organization.
-So if the folder you want rclone to use has a URL which looks like
-`https://drive.google.com/drive/folders/1XyfxxxxxxxxxxxxxxxxxxxxxxxxxKHCh`
-in the browser, then you use `1XyfxxxxxxxxxxxxxxxxxxxxxxxxxKHCh` as
-the `root_folder_id` in the config.
-**NB** folders under the "Computers" tab seem to be read only (drive
-gives a 500 error) when using rclone.
+Properties:
-There doesn't appear to be an API to discover the folder IDs of the
-"Computers" tab - please contact us if you know otherwise!
+- Config: metadata_owner
+- Env Var: RCLONE_DRIVE_METADATA_OWNER
+- Type: Bits
+- Default: read
+- Examples:
+ - "off"
+ - Do not read or write the value
+ - "read"
+ - Read the value only
+ - "write"
+ - Write the value only
+ - "read,write"
+ - Read and Write the value.
-Note also that rclone can't access any data under the "Backups" tab on
-the google drive web interface yet.
+#### --drive-metadata-permissions
-### Service Account support
+Control whether permissions should be read or written in metadata.
-You can set up rclone with Google Drive in an unattended mode,
-i.e. not tied to a specific end-user Google account. This is useful
-when you want to synchronise files onto machines that don't have
-actively logged-in users, for example build machines.
+Reading permissions metadata from files can be done quickly, but it
+isn't always desirable to set the permissions from the metadata.
-To use a Service Account instead of OAuth2 token flow, enter the path
-to your Service Account credentials at the `service_account_file`
-prompt during `rclone config` and rclone won't use the browser based
-authentication flow. If you'd rather stuff the contents of the
-credentials file into the rclone config file, you can set
-`service_account_credentials` with the actual contents of the file
-instead, or set the equivalent environment variable.
+Note that rclone drops any inherited permissions on Shared Drives and
+any owner permission on My Drives as these are duplicated in the owner
+metadata.
-#### Use case - Google Apps/G-suite account and individual Drive
-Let's say that you are the administrator of a Google Apps (old) or
-G-suite account.
-The goal is to store data on an individual's Drive account, who IS
-a member of the domain.
-We'll call the domain **example.com**, and the user
-**foo@example.com**.
+Properties:
-There's a few steps we need to go through to accomplish this:
+- Config: metadata_permissions
+- Env Var: RCLONE_DRIVE_METADATA_PERMISSIONS
+- Type: Bits
+- Default: off
+- Examples:
+ - "off"
+ - Do not read or write the value
+ - "read"
+ - Read the value only
+ - "write"
+ - Write the value only
+ - "read,write"
+ - Read and Write the value.
-##### 1. Create a service account for example.com
- - To create a service account and obtain its credentials, go to the
-[Google Developer Console](https://console.developers.google.com).
- - You must have a project - create one if you don't.
- - Then go to "IAM & admin" -> "Service Accounts".
- - Use the "Create Service Account" button. Fill in "Service account name"
-and "Service account ID" with something that identifies your client.
- - Select "Create And Continue". Step 2 and 3 are optional.
- - These credentials are what rclone will use for authentication.
-If you ever need to remove access, press the "Delete service
-account key" button.
+#### --drive-metadata-labels
+
+Control whether labels should be read or written in metadata.
+
+Reading labels metadata from files takes an extra API transaction and
+will slow down listings. It isn't always desirable to set the labels
+from the metadata.
+
+The format of labels is documented in the drive API documentation at
+https://developers.google.com/drive/api/reference/rest/v3/Label -
+rclone just provides a JSON dump of this format.
-##### 2. Allowing API access to example.com Google Drive
- - Go to example.com's admin console
- - Go into "Security" (or use the search bar)
- - Select "Show more" and then "Advanced settings"
- - Select "Manage API client access" in the "Authentication" section
- - In the "Client Name" field enter the service account's
-"Client ID" - this can be found in the Developer Console under
-"IAM & Admin" -> "Service Accounts", then "View Client ID" for
-the newly created service account.
-It is a ~21 character numerical string.
- - In the next field, "One or More API Scopes", enter
-`https://www.googleapis.com/auth/drive`
-to grant access to Google Drive specifically.
+When setting labels, the label and fields must already exist - rclone
+will not create them. This means that if you are transferring labels
+from two different accounts you will have to create the labels in
+advance and use the metadata mapper to translate the IDs between the
+two accounts.
-##### 3. Configure rclone, assuming a new install
-
-rclone config
-n/s/q> n # New name>gdrive # Gdrive is an example name Storage> # Select the number shown for Google Drive client_id> # Can be left blank client_secret> # Can be left blank scope> # Select your scope, 1 for example root_folder_id> # Can be left blank service_account_file> /home/foo/myJSONfile.json # This is where the JSON file goes! y/n> # Auto config, n
-
-##### 4. Verify that it's working
- - `rclone -v --drive-impersonate foo@example.com lsf gdrive:backup`
- - The arguments do:
- - `-v` - verbose logging
- - `--drive-impersonate foo@example.com` - this is what does
-the magic, pretending to be user foo.
- - `lsf` - list files in a parsing friendly way
- - `gdrive:backup` - use the remote called gdrive, work in
-the folder named backup.
-Note: in case you configured a specific root folder on gdrive and rclone is unable to access the contents of that folder when using `--drive-impersonate`, do this instead:
- - in the gdrive web interface, share your root folder with the user/email of the new Service Account you created/selected at step #1
- - use rclone without specifying the `--drive-impersonate` option, like this:
- `rclone -v lsf gdrive:backup`
+Properties:
+- Config: metadata_labels
+- Env Var: RCLONE_DRIVE_METADATA_LABELS
+- Type: Bits
+- Default: off
+- Examples:
+ - "off"
+ - Do not read or write the value
+ - "read"
+ - Read the value only
+ - "write"
+ - Write the value only
+ - "read,write"
+ - Read and Write the value.
-### Shared drives (team drives)
+#### --drive-encoding
-If you want to configure the remote to point to a Google Shared Drive
-(previously known as Team Drives) then answer `y` to the question
-`Configure this as a Shared Drive (Team Drive)?`.
+The encoding for the backend.
-This will fetch the list of Shared Drives from google and allow you to
-configure which one you want to use. You can also type in a Shared
-Drive ID if you prefer.
+See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info.
-For example:
-
-Configure this as a Shared Drive (Team Drive)? y) Yes n) No y/n> y Fetching Shared Drive list... Choose a number from below, or type in your own value 1 / Rclone Test  "xxxxxxxxxxxxxxxxxxxx" 2 / Rclone Test 2  "yyyyyyyyyyyyyyyyyyyy" 3 / Rclone Test 3  "zzzzzzzzzzzzzzzzzzzz" Enter a Shared Drive ID> 1 -------------------- [remote] client_id = client_secret = token = {"AccessToken":"xxxx.x.xxxxx_xxxxxxxxxxx_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","RefreshToken":"1/xxxxxxxxxxxxxxxx_xxxxxxxxxxxxxxxxxxxxxxxxxx","Expiry":"2014-03-16T13:57:58.955387075Z","Extra":null} team_drive = xxxxxxxxxxxxxxxxxxxx -------------------- y) Yes this is OK e) Edit this remote d) Delete this remote y/e/d> y
-
-### --fast-list
+Properties:
-This remote supports `--fast-list` which allows you to use fewer
-transactions in exchange for more memory. See the [rclone
-docs](https://rclone.org/docs/#fast-list) for more details.
+- Config: encoding
+- Env Var: RCLONE_DRIVE_ENCODING
+- Type: Encoding
+- Default: InvalidUtf8
-It does this by combining multiple `list` calls into a single API request.
+#### --drive-env-auth
-This works by combining many `'%s' in parents` filters into one expression.
-To list the contents of directories a, b and c, the following requests will be send by the regular `List` function:
-trashed=false and 'a' in parents trashed=false and 'b' in parents trashed=false and 'c' in parents
-These can now be combined into a single request:
-trashed=false and ('a' in parents or 'b' in parents or 'c' in parents)
-
-The implementation of `ListR` will put up to 50 `parents` filters into one request.
-It will use the `--checkers` value to specify the number of requests to run in parallel.
+Get IAM credentials from runtime (environment variables or instance meta data if no env vars).
-In tests, these batch requests were up to 20x faster than the regular method.
-Running the following command against different sized folders gives:
-rclone lsjson -vv -R --checkers=6 gdrive:folder
-
-small folder (220 directories, 700 files):
+Only applies if service_account_file and service_account_credentials is blank.
-- without `--fast-list`: 38s
-- with `--fast-list`: 10s
+Properties:
-large folder (10600 directories, 39000 files):
+- Config: env_auth
+- Env Var: RCLONE_DRIVE_ENV_AUTH
+- Type: bool
+- Default: false
+- Examples:
+ - "false"
+ - Enter credentials in the next step.
+ - "true"
+ - Get GCP IAM credentials from the environment (env vars or IAM).
-- without `--fast-list`: 22:05 min
-- with `--fast-list`: 58s
+### Metadata
-### Modified time
+User metadata is stored in the properties field of the drive object.
-Google drive stores modification times accurate to 1 ms.
+Here are the possible system metadata items for the drive backend.
-### Restricted filename characters
+| Name | Help | Type | Example | Read Only |
+|------|------|------|---------|-----------|
+| btime | Time of file birth (creation) with mS accuracy. Note that this is only writable on fresh uploads - it can't be written for updates. | RFC 3339 | 2006-01-02T15:04:05.999Z07:00 | N |
+| content-type | The MIME type of the file. | string | text/plain | N |
+| copy-requires-writer-permission | Whether the options to copy, print, or download this file, should be disabled for readers and commenters. | boolean | true | N |
+| description | A short description of the file. | string | Contract for signing | N |
+| folder-color-rgb | The color for a folder or a shortcut to a folder as an RGB hex string. | string | 881133 | N |
+| labels | Labels attached to this file in a JSON dump of Googled drive format. Enable with --drive-metadata-labels. | JSON | [] | N |
+| mtime | Time of last modification with mS accuracy. | RFC 3339 | 2006-01-02T15:04:05.999Z07:00 | N |
+| owner | The owner of the file. Usually an email address. Enable with --drive-metadata-owner. | string | user@example.com | N |
+| permissions | Permissions in a JSON dump of Google drive format. On shared drives these will only be present if they aren't inherited. Enable with --drive-metadata-permissions. | JSON | {} | N |
+| starred | Whether the user has starred the file. | boolean | false | N |
+| viewed-by-me | Whether the file has been viewed by this user. | boolean | true | **Y** |
+| writers-can-share | Whether users with only writer permission can modify the file's permissions. Not populated for items in shared drives. | boolean | false | N |
-Only Invalid UTF-8 bytes will be [replaced](https://rclone.org/overview/#invalid-utf8),
-as they can't be used in JSON strings.
+See the [metadata](https://rclone.org/docs/#metadata) docs for more info.
-In contrast to other backends, `/` can also be used in names and `.`
-or `..` are valid names.
+## Backend commands
-### Revisions
+Here are the commands specific to the drive backend.
-Google drive stores revisions of files. When you upload a change to
-an existing file to google drive using rclone it will create a new
-revision of that file.
+Run them with
-Revisions follow the standard google policy which at time of writing
-was
+ rclone backend COMMAND remote:
- * They are deleted after 30 days or 100 revisions (whatever comes first).
- * They do not count towards a user storage quota.
+The help below will explain what arguments each command takes.
-### Deleting files
+See the [backend](https://rclone.org/commands/rclone_backend/) command for more
+info on how to pass options and arguments.
-By default rclone will send all files to the trash when deleting
-files. If deleting them permanently is required then use the
-`--drive-use-trash=false` flag, or set the equivalent environment
-variable.
+These can be run on a running backend using the rc command
+[backend/command](https://rclone.org/rc/#backend-command).
-### Shortcuts
+### get
-In March 2020 Google introduced a new feature in Google Drive called
-[drive shortcuts](https://support.google.com/drive/answer/9700156)
-([API](https://developers.google.com/drive/api/v3/shortcuts)). These
-will (by September 2020) [replace the ability for files or folders to
-be in multiple folders at once](https://cloud.google.com/blog/products/g-suite/simplifying-google-drives-folder-structure-and-sharing-models).
+Get command for fetching the drive config parameters
-Shortcuts are files that link to other files on Google Drive somewhat
-like a symlink in unix, except they point to the underlying file data
-(e.g. the inode in unix terms) so they don't break if the source is
-renamed or moved about.
+ rclone backend get remote: [options] [<arguments>+]
-By default rclone treats these as follows.
+This is a get command which will be used to fetch the various drive config parameters
-For shortcuts pointing to files:
+Usage Examples:
-- When listing a file shortcut appears as the destination file.
-- When downloading the contents of the destination file is downloaded.
-- When updating shortcut file with a non shortcut file, the shortcut is removed then a new file is uploaded in place of the shortcut.
-- When server-side moving (renaming) the shortcut is renamed, not the destination file.
-- When server-side copying the shortcut is copied, not the contents of the shortcut. (unless `--drive-copy-shortcut-content` is in use in which case the contents of the shortcut gets copied).
-- When deleting the shortcut is deleted not the linked file.
-- When setting the modification time, the modification time of the linked file will be set.
+ rclone backend get drive: [-o service_account_file] [-o chunk_size]
+ rclone rc backend/command command=get fs=drive: [-o service_account_file] [-o chunk_size]
-For shortcuts pointing to folders:
-- When listing the shortcut appears as a folder and that folder will contain the contents of the linked folder appear (including any sub folders)
-- When downloading the contents of the linked folder and sub contents are downloaded
-- When uploading to a shortcut folder the file will be placed in the linked folder
-- When server-side moving (renaming) the shortcut is renamed, not the destination folder
-- When server-side copying the contents of the linked folder is copied, not the shortcut.
-- When deleting with `rclone rmdir` or `rclone purge` the shortcut is deleted not the linked folder.
-- **NB** When deleting with `rclone remove` or `rclone mount` the contents of the linked folder will be deleted.
+Options:
-The [rclone backend](https://rclone.org/commands/rclone_backend/) command can be used to create shortcuts.
+- "chunk_size": show the current upload chunk size
+- "service_account_file": show the current service account file
-Shortcuts can be completely ignored with the `--drive-skip-shortcuts` flag
-or the corresponding `skip_shortcuts` configuration setting.
+### set
-### Emptying trash
+Set command for updating the drive config parameters
-If you wish to empty your trash you can use the `rclone cleanup remote:`
-command which will permanently delete all your trashed files. This command
-does not take any path arguments.
+ rclone backend set remote: [options] [<arguments>+]
-Note that Google Drive takes some time (minutes to days) to empty the
-trash even though the command returns within a few seconds. No output
-is echoed, so there will be no confirmation even using -v or -vv.
+This is a set command which will be used to update the various drive config parameters
-### Quota information
+Usage Examples:
-To view your current quota you can use the `rclone about remote:`
-command which will display your usage limit (quota), the usage in Google
-Drive, the size of all files in the Trash and the space used by other
-Google services such as Gmail. This command does not take any path
-arguments.
+ rclone backend set drive: [-o service_account_file=sa.json] [-o chunk_size=67108864]
+ rclone rc backend/command command=set fs=drive: [-o service_account_file=sa.json] [-o chunk_size=67108864]
-#### Import/Export of google documents
-Google documents can be exported from and uploaded to Google Drive.
+Options:
-When rclone downloads a Google doc it chooses a format to download
-depending upon the `--drive-export-formats` setting.
-By default the export formats are `docx,xlsx,pptx,svg` which are a
-sensible default for an editable document.
+- "chunk_size": update the current upload chunk size
+- "service_account_file": update the current service account file
-When choosing a format, rclone runs down the list provided in order
-and chooses the first file format the doc can be exported as from the
-list. If the file can't be exported to a format on the formats list,
-then rclone will choose a format from the default list.
+### shortcut
-If you prefer an archive copy then you might use `--drive-export-formats
-pdf`, or if you prefer openoffice/libreoffice formats you might use
-`--drive-export-formats ods,odt,odp`.
+Create shortcuts from files or directories
-Note that rclone adds the extension to the google doc, so if it is
-called `My Spreadsheet` on google docs, it will be exported as `My
-Spreadsheet.xlsx` or `My Spreadsheet.pdf` etc.
+ rclone backend shortcut remote: [options] [<arguments>+]
-When importing files into Google Drive, rclone will convert all
-files with an extension in `--drive-import-formats` to their
-associated document type.
-rclone will not convert any files by default, since the conversion
-is lossy process.
+This command creates shortcuts from files or directories.
-The conversion must result in a file with the same extension when
-the `--drive-export-formats` rules are applied to the uploaded document.
+Usage:
-Here are some examples for allowed and prohibited conversions.
+ rclone backend shortcut drive: source_item destination_shortcut
+ rclone backend shortcut drive: source_item -o target=drive2: destination_shortcut
-| export-formats | import-formats | Upload Ext | Document Ext | Allowed |
-| -------------- | -------------- | ---------- | ------------ | ------- |
-| odt | odt | odt | odt | Yes |
-| odt | docx,odt | odt | odt | Yes |
-| | docx | docx | docx | Yes |
-| | odt | odt | docx | No |
-| odt,docx | docx,odt | docx | odt | No |
-| docx,odt | docx,odt | docx | docx | Yes |
-| docx,odt | docx,odt | odt | docx | No |
+In the first example this creates a shortcut from the "source_item"
+which can be a file or a directory to the "destination_shortcut". The
+"source_item" and the "destination_shortcut" should be relative paths
+from "drive:"
-This limitation can be disabled by specifying `--drive-allow-import-name-change`.
-When using this flag, rclone can convert multiple files types resulting
-in the same document type at once, e.g. with `--drive-import-formats docx,odt,txt`,
-all files having these extension would result in a document represented as a docx file.
-This brings the additional risk of overwriting a document, if multiple files
-have the same stem. Many rclone operations will not handle this name change
-in any way. They assume an equal name when copying files and might copy the
-file again or delete them when the name changes.
+In the second example this creates a shortcut from the "source_item"
+relative to "drive:" to the "destination_shortcut" relative to
+"drive2:". This may fail with a permission error if the user
+authenticated with "drive2:" can't read files from "drive:".
-Here are the possible export extensions with their corresponding mime types.
-Most of these can also be used for importing, but there more that are not
-listed here. Some of these additional ones might only be available when
-the operating system provides the correct MIME type entries.
-This list can be changed by Google Drive at any time and might not
-represent the currently available conversions.
+Options:
-| Extension | Mime Type | Description |
-| --------- |-----------| ------------|
-| bmp | image/bmp | Windows Bitmap format |
-| csv | text/csv | Standard CSV format for Spreadsheets |
-| doc | application/msword | Classic Word file |
-| docx | application/vnd.openxmlformats-officedocument.wordprocessingml.document | Microsoft Office Document |
-| epub | application/epub+zip | E-book format |
-| html | text/html | An HTML Document |
-| jpg | image/jpeg | A JPEG Image File |
-| json | application/vnd.google-apps.script+json | JSON Text Format for Google Apps scripts |
-| odp | application/vnd.oasis.opendocument.presentation | Openoffice Presentation |
-| ods | application/vnd.oasis.opendocument.spreadsheet | Openoffice Spreadsheet |
-| ods | application/x-vnd.oasis.opendocument.spreadsheet | Openoffice Spreadsheet |
-| odt | application/vnd.oasis.opendocument.text | Openoffice Document |
-| pdf | application/pdf | Adobe PDF Format |
-| pjpeg | image/pjpeg | Progressive JPEG Image |
-| png | image/png | PNG Image Format|
-| pptx | application/vnd.openxmlformats-officedocument.presentationml.presentation | Microsoft Office Powerpoint |
-| rtf | application/rtf | Rich Text Format |
-| svg | image/svg+xml | Scalable Vector Graphics Format |
-| tsv | text/tab-separated-values | Standard TSV format for spreadsheets |
-| txt | text/plain | Plain Text |
-| wmf | application/x-msmetafile | Windows Meta File |
-| xls | application/vnd.ms-excel | Classic Excel file |
-| xlsx | application/vnd.openxmlformats-officedocument.spreadsheetml.sheet | Microsoft Office Spreadsheet |
-| zip | application/zip | A ZIP file of HTML, Images CSS |
+- "target": optional target remote for the shortcut destination
-Google documents can also be exported as link files. These files will
-open a browser window for the Google Docs website of that document
-when opened. The link file extension has to be specified as a
-`--drive-export-formats` parameter. They will match all available
-Google Documents.
+### drives
-| Extension | Description | OS Support |
-| --------- | ----------- | ---------- |
-| desktop | freedesktop.org specified desktop entry | Linux |
-| link.html | An HTML Document with a redirect | All |
-| url | INI style link file | macOS, Windows |
-| webloc | macOS specific XML format | macOS |
+List the Shared Drives available to this account
+ rclone backend drives remote: [options] [<arguments>+]
-### Standard options
+This command lists the Shared Drives (Team Drives) available to this
+account.
-Here are the Standard options specific to drive (Google Drive).
+Usage:
-#### --drive-client-id
+ rclone backend [-o config] drives drive:
-Google Application Client Id
-Setting your own is recommended.
-See https://rclone.org/drive/#making-your-own-client-id for how to create your own.
-If you leave this blank, it will use an internal key which is low performance.
+This will return a JSON list of objects like this
-Properties:
+ [
+ {
+ "id": "0ABCDEF-01234567890",
+ "kind": "drive#teamDrive",
+ "name": "My Drive"
+ },
+ {
+ "id": "0ABCDEFabcdefghijkl",
+ "kind": "drive#teamDrive",
+ "name": "Test Drive"
+ }
+ ]
-- Config: client_id
-- Env Var: RCLONE_DRIVE_CLIENT_ID
-- Type: string
-- Required: false
+With the -o config parameter it will output the list in a format
+suitable for adding to a config file to make aliases for all the
+drives found and a combined drive.
-#### --drive-client-secret
+ [My Drive]
+ type = alias
+ remote = drive,team_drive=0ABCDEF-01234567890,root_folder_id=:
-OAuth Client Secret.
+ [Test Drive]
+ type = alias
+ remote = drive,team_drive=0ABCDEFabcdefghijkl,root_folder_id=:
-Leave blank normally.
+ [AllDrives]
+ type = combine
+ upstreams = "My Drive=My Drive:" "Test Drive=Test Drive:"
-Properties:
+Adding this to the rclone config file will cause those team drives to
+be accessible with the aliases shown. Any illegal characters will be
+substituted with "_" and duplicate names will have numbers suffixed.
+It will also add a remote called AllDrives which shows all the shared
+drives combined into one directory tree.
-- Config: client_secret
-- Env Var: RCLONE_DRIVE_CLIENT_SECRET
-- Type: string
-- Required: false
-#### --drive-scope
+### untrash
-Scope that rclone should use when requesting access from drive.
+Untrash files and directories
-Properties:
+ rclone backend untrash remote: [options] [<arguments>+]
-- Config: scope
-- Env Var: RCLONE_DRIVE_SCOPE
-- Type: string
-- Required: false
-- Examples:
- - "drive"
- - Full access all files, excluding Application Data Folder.
- - "drive.readonly"
- - Read-only access to file metadata and file contents.
- - "drive.file"
- - Access to files created by rclone only.
- - These are visible in the drive website.
- - File authorization is revoked when the user deauthorizes the app.
- - "drive.appfolder"
- - Allows read and write access to the Application Data folder.
- - This is not visible in the drive website.
- - "drive.metadata.readonly"
- - Allows read-only access to file metadata but
- - does not allow any access to read or download file content.
+This command untrashes all the files and directories in the directory
+passed in recursively.
-#### --drive-service-account-file
+Usage:
-Service Account Credentials JSON file path.
+This takes an optional directory to trash which make this easier to
+use via the API.
-Leave blank normally.
-Needed only if you want use SA instead of interactive login.
+ rclone backend untrash drive:directory
+ rclone backend --interactive untrash drive:directory subdir
-Leading `~` will be expanded in the file name as will environment variables such as `${RCLONE_CONFIG_DIR}`.
+Use the --interactive/-i or --dry-run flag to see what would be restored before restoring it.
-Properties:
+Result:
-- Config: service_account_file
-- Env Var: RCLONE_DRIVE_SERVICE_ACCOUNT_FILE
-- Type: string
-- Required: false
+ {
+ "Untrashed": 17,
+ "Errors": 0
+ }
-#### --drive-alternate-export
-Deprecated: No longer needed.
+### copyid
-Properties:
+Copy files by ID
-- Config: alternate_export
-- Env Var: RCLONE_DRIVE_ALTERNATE_EXPORT
-- Type: bool
-- Default: false
+ rclone backend copyid remote: [options] [<arguments>+]
-### Advanced options
+This command copies files by ID
-Here are the Advanced options specific to drive (Google Drive).
+Usage:
-#### --drive-token
+ rclone backend copyid drive: ID path
+ rclone backend copyid drive: ID1 path1 ID2 path2
-OAuth Access Token as a JSON blob.
+It copies the drive file with ID given to the path (an rclone path which
+will be passed internally to rclone copyto). The ID and path pairs can be
+repeated.
-Properties:
+The path should end with a / to indicate copy the file as named to
+this directory. If it doesn't end with a / then the last path
+component will be used as the file name.
-- Config: token
-- Env Var: RCLONE_DRIVE_TOKEN
-- Type: string
-- Required: false
+If the destination is a drive backend then server-side copying will be
+attempted if possible.
-#### --drive-auth-url
+Use the --interactive/-i or --dry-run flag to see what would be copied before copying.
-Auth server URL.
-Leave blank to use the provider defaults.
+### exportformats
-Properties:
+Dump the export formats for debug purposes
-- Config: auth_url
-- Env Var: RCLONE_DRIVE_AUTH_URL
-- Type: string
-- Required: false
+ rclone backend exportformats remote: [options] [<arguments>+]
-#### --drive-token-url
+### importformats
-Token server url.
+Dump the import formats for debug purposes
-Leave blank to use the provider defaults.
+ rclone backend importformats remote: [options] [<arguments>+]
-Properties:
-- Config: token_url
-- Env Var: RCLONE_DRIVE_TOKEN_URL
-- Type: string
-- Required: false
-#### --drive-root-folder-id
+## Limitations
-ID of the root folder.
-Leave blank normally.
+Drive has quite a lot of rate limiting. This causes rclone to be
+limited to transferring about 2 files per second only. Individual
+files may be transferred much faster at 100s of MiB/s but lots of
+small files can take a long time.
-Fill in to access "Computers" folders (see docs), or for rclone to use
-a non root folder as its starting point.
+Server side copies are also subject to a separate rate limit. If you
+see User rate limit exceeded errors, wait at least 24 hours and retry.
+You can disable server-side copies with `--disable copy` to download
+and upload the files if you prefer.
+### Limitations of Google Docs
-Properties:
+Google docs will appear as size -1 in `rclone ls`, `rclone ncdu` etc,
+and as size 0 in anything which uses the VFS layer, e.g. `rclone mount`
+and `rclone serve`. When calculating directory totals, e.g. in
+`rclone size` and `rclone ncdu`, they will be counted in as empty
+files.
-- Config: root_folder_id
-- Env Var: RCLONE_DRIVE_ROOT_FOLDER_ID
-- Type: string
-- Required: false
+This is because rclone can't find out the size of the Google docs
+without downloading them.
-#### --drive-service-account-credentials
+Google docs will transfer correctly with `rclone sync`, `rclone copy`
+etc as rclone knows to ignore the size when doing the transfer.
-Service Account Credentials JSON blob.
+However an unfortunate consequence of this is that you may not be able
+to download Google docs using `rclone mount`. If it doesn't work you
+will get a 0 sized file. If you try again the doc may gain its
+correct size and be downloadable. Whether it will work on not depends
+on the application accessing the mount and the OS you are running -
+experiment to find out if it does work for you!
-Leave blank normally.
-Needed only if you want use SA instead of interactive login.
+### Duplicated files
-Properties:
+Sometimes, for no reason I've been able to track down, drive will
+duplicate a file that rclone uploads. Drive unlike all the other
+remotes can have duplicated files.
-- Config: service_account_credentials
-- Env Var: RCLONE_DRIVE_SERVICE_ACCOUNT_CREDENTIALS
-- Type: string
-- Required: false
+Duplicated files cause problems with the syncing and you will see
+messages in the log about duplicates.
-#### --drive-team-drive
+Use `rclone dedupe` to fix duplicated files.
+
+Note that this isn't just a problem with rclone, even Google Photos on
+Android duplicates files on drive sometimes.
+
+### Rclone appears to be re-copying files it shouldn't
+
+The most likely cause of this is the duplicated file issue above - run
+`rclone dedupe` and check your logs for duplicate object or directory
+messages.
+
+This can also be caused by a delay/caching on google drive's end when
+comparing directory listings. Specifically with team drives used in
+combination with --fast-list. Files that were uploaded recently may
+not appear on the directory list sent to rclone when using --fast-list.
+
+Waiting a moderate period of time between attempts (estimated to be
+approximately 1 hour) and/or not using --fast-list both seem to be
+effective in preventing the problem.
-ID of the Shared Drive (Team Drive).
+### SHA1 or SHA256 hashes may be missing
-Properties:
+All files have MD5 hashes, but a small fraction of files uploaded may
+not have SHA1 or SHA256 hashes especially if they were uploaded before 2018.
-- Config: team_drive
-- Env Var: RCLONE_DRIVE_TEAM_DRIVE
-- Type: string
-- Required: false
+## Making your own client_id
-#### --drive-auth-owner-only
+When you use rclone with Google drive in its default configuration you
+are using rclone's client_id. This is shared between all the rclone
+users. There is a global rate limit on the number of queries per
+second that each client_id can do set by Google. rclone already has a
+high quota and I will continue to make sure it is high enough by
+contacting Google.
-Only consider files owned by the authenticated user.
+It is strongly recommended to use your own client ID as the default rclone ID is heavily used. If you have multiple services running, it is recommended to use an API key for each service. The default Google quota is 10 transactions per second so it is recommended to stay under that number as if you use more than that, it will cause rclone to rate limit and make things slower.
-Properties:
+Here is how to create your own Google Drive client ID for rclone:
-- Config: auth_owner_only
-- Env Var: RCLONE_DRIVE_AUTH_OWNER_ONLY
-- Type: bool
-- Default: false
+1. Log into the [Google API
+Console](https://console.developers.google.com/) with your Google
+account. It doesn't matter what Google account you use. (It need not
+be the same account as the Google Drive you want to access)
-#### --drive-use-trash
+2. Select a project or create a new project.
-Send files to the trash instead of deleting permanently.
+3. Under "ENABLE APIS AND SERVICES" search for "Drive", and enable the
+"Google Drive API".
-Defaults to true, namely sending files to the trash.
-Use `--drive-use-trash=false` to delete files permanently instead.
+4. Click "Credentials" in the left-side panel (not "Create
+credentials", which opens the wizard).
-Properties:
+5. If you already configured an "Oauth Consent Screen", then skip
+to the next step; if not, click on "CONFIGURE CONSENT SCREEN" button
+(near the top right corner of the right panel), then select "External"
+and click on "CREATE"; on the next screen, enter an "Application name"
+("rclone" is OK); enter "User Support Email" (your own email is OK);
+enter "Developer Contact Email" (your own email is OK); then click on
+"Save" (all other data is optional). You will also have to add some scopes,
+including `.../auth/docs` and `.../auth/drive` in order to be able to edit,
+create and delete files with RClone. You may also want to include the
+`../auth/drive.metadata.readonly` scope. After adding scopes, click
+"Save and continue" to add test users. Be sure to add your own account to
+the test users. Once you've added yourself as a test user and saved the
+changes, click again on "Credentials" on the left panel to go back to
+the "Credentials" screen.
-- Config: use_trash
-- Env Var: RCLONE_DRIVE_USE_TRASH
-- Type: bool
-- Default: true
+ (PS: if you are a GSuite user, you could also select "Internal" instead
+of "External" above, but this will restrict API use to Google Workspace
+users in your organisation).
-#### --drive-copy-shortcut-content
+6. Click on the "+ CREATE CREDENTIALS" button at the top of the screen,
+then select "OAuth client ID".
-Server side copy contents of shortcuts instead of the shortcut.
+7. Choose an application type of "Desktop app" and click "Create". (the default name is fine)
-When doing server side copies, normally rclone will copy shortcuts as
-shortcuts.
+8. It will show you a client ID and client secret. Make a note of these.
+
+ (If you selected "External" at Step 5 continue to Step 9.
+ If you chose "Internal" you don't need to publish and can skip straight to
+ Step 10 but your destination drive must be part of the same Google Workspace.)
-If this flag is used then rclone will copy the contents of shortcuts
-rather than shortcuts themselves when doing server side copies.
+9. Go to "Oauth consent screen" and then click "PUBLISH APP" button and confirm.
+ You will also want to add yourself as a test user.
-Properties:
+10. Provide the noted client ID and client secret to rclone.
-- Config: copy_shortcut_content
-- Env Var: RCLONE_DRIVE_COPY_SHORTCUT_CONTENT
-- Type: bool
-- Default: false
+Be aware that, due to the "enhanced security" recently introduced by
+Google, you are theoretically expected to "submit your app for verification"
+and then wait a few weeks(!) for their response; in practice, you can go right
+ahead and use the client ID and client secret with rclone, the only issue will
+be a very scary confirmation screen shown when you connect via your browser
+for rclone to be able to get its token-id (but as this only happens during
+the remote configuration, it's not such a big deal). Keeping the application in
+"Testing" will work as well, but the limitation is that any grants will expire
+after a week, which can be annoying to refresh constantly. If, for whatever
+reason, a short grant time is not a problem, then keeping the application in
+testing mode would also be sufficient.
-#### --drive-skip-gdocs
+(Thanks to @balazer on github for these instructions.)
-Skip google documents in all listings.
+Sometimes, creation of an OAuth consent in Google API Console fails due to an error message
+“The request failed because changes to one of the field of the resource is not supportedâ€.
+As a convenient workaround, the necessary Google Drive API key can be created on the
+[Python Quickstart](https://developers.google.com/drive/api/v3/quickstart/python) page.
+Just push the Enable the Drive API button to receive the Client ID and Secret.
+Note that it will automatically create a new project in the API Console.
-If given, gdocs practically become invisible to rclone.
+# Google Photos
-Properties:
+The rclone backend for [Google Photos](https://www.google.com/photos/about/) is
+a specialized backend for transferring photos and videos to and from
+Google Photos.
-- Config: skip_gdocs
-- Env Var: RCLONE_DRIVE_SKIP_GDOCS
-- Type: bool
-- Default: false
+**NB** The Google Photos API which rclone uses has quite a few
+limitations, so please read the [limitations section](#limitations)
+carefully to make sure it is suitable for your use.
-#### --drive-skip-checksum-gphotos
+## Configuration
-Skip MD5 checksum on Google photos and videos only.
+The initial setup for google cloud storage involves getting a token from Google Photos
+which you need to do in your browser. `rclone config` walks you
+through it.
-Use this if you get checksum errors when transferring Google photos or
-videos.
+Here is an example of how to make a remote called `remote`. First run:
-Setting this flag will cause Google photos and videos to return a
-blank MD5 checksum.
+ rclone config
-Google photos are identified by being in the "photos" space.
+This will guide you through an interactive setup process:
+
+No remotes found, make a new one? n) New remote s) Set configuration password q) Quit config n/s/q> n name> remote Type of storage to configure. Enter a string value. Press Enter for the default (""). Choose a number from below, or type in your own value [snip] XX / Google Photos  "google photos" [snip] Storage> google photos ** See help for google photos backend at: https://rclone.org/googlephotos/ **
+Google Application Client Id Leave blank normally. Enter a string value. Press Enter for the default (""). client_id> Google Application Client Secret Leave blank normally. Enter a string value. Press Enter for the default (""). client_secret> Set to make the Google Photos backend read only.
+If you choose read only then rclone will only request read only access to your photos, otherwise rclone will request full access. Enter a boolean value (true or false). Press Enter for the default ("false"). read_only> Edit advanced config? (y/n) y) Yes n) No y/n> n Remote config Use web browser to automatically authenticate rclone with remote? * Say Y if the machine running rclone has a web browser you can use * Say N if running rclone on a (remote) machine without web browser access If not sure try Y. If Y failed, try N. y) Yes n) No y/n> y If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth Log in and authorize rclone for access Waiting for code... Got code
+*** IMPORTANT: All media items uploaded to Google Photos with rclone *** are stored in full resolution at original quality. These uploads *** will count towards storage in your Google Account.
+
+
+
+
+
+[remote] type = google photos token = {"access_token":"XXX","token_type":"Bearer","refresh_token":"XXX","expiry":"2019-06-28T17:38:04.644930156+01:00"}
+
+
+
+
+y) Yes this is OK e) Edit this remote d) Delete this remote y/e/d> y ```
+
+
+See the remote setup docs for how to set it up on a machine with no Internet browser available.
+
+
+Note that rclone runs a webserver on your local machine to collect the token as returned from Google if using web browser to automatically authenticate. This only runs from the moment it opens your browser to the moment you get back the verification code. This is on http://127.0.0.1:53682/ and this may require you to unblock it temporarily if you are running a host firewall, or use manual mode.
+
+
+This remote is called remote and can now be used like this
+
+
+See all the albums in your photos
+
+
+rclone lsd remote:album
+
+
+Make a new album
+
+
+rclone mkdir remote:album/newAlbum
+
+
+List the contents of an album
+
+
+rclone ls remote:album/newAlbum
+
+
+Sync /home/local/images to the Google Photos, removing any excess files in the album.
+
+
+rclone sync --interactive /home/local/image remote:album/newAlbum
+
+
+### Layout
+
+
+As Google Photos is not a general purpose cloud storage system, the backend is laid out to help you navigate it.
+
+
+The directories under media show different ways of categorizing the media. Each file will appear multiple times. So if you want to make a backup of your google photos you might choose to backup remote:media/by-month. (NB remote:media/by-day is rather slow at the moment so avoid for syncing.)
+
+
+Note that all your photos and videos will appear somewhere under media, but they may not appear under album unless you've put them into albums.
+
+
+/ - upload - file1.jpg - file2.jpg - ... - media - all - file1.jpg - file2.jpg - ... - by-year - 2000 - file1.jpg - ... - 2001 - file2.jpg - ... - ... - by-month - 2000 - 2000-01 - file1.jpg - ... - 2000-02 - file2.jpg - ... - ... - by-day - 2000 - 2000-01-01 - file1.jpg - ... - 2000-01-02 - file2.jpg - ... - ... - album - album name - album name/sub - shared-album - album name - album name/sub - feature - favorites - file1.jpg - file2.jpg
+
+There are two writable parts of the tree, the upload directory and sub directories of the album directory.
+
+
+The upload directory is for uploading files you don't want to put into albums. This will be empty to start with and will contain the files you've uploaded for one rclone session only, becoming empty again when you restart rclone. The use case for this would be if you have a load of files you just want to once off dump into Google Photos. For repeated syncing, uploading to album will work better.
+
+
+Directories within the album directory are also writeable and you may create new directories (albums) under album. If you copy files with a directory hierarchy in there then rclone will create albums with the / character in them. For example if you do
+
+
+rclone copy /path/to/images remote:album/images
+
+
+and the images directory contains
+
+
+images - file1.jpg dir file2.jpg dir2 dir3 file3.jpg
+
+Then rclone will create the following albums with the following files in
+
+
+- images - file1.jpg - images/dir - file2.jpg - images/dir2/dir3 - file3.jpg
+
+
+This means that you can use the album path pretty much like a normal filesystem and it is a good target for repeated syncing.
+
+
+The shared-album directory shows albums shared with you or by you. This is similar to the Sharing tab in the Google Photos web interface.
+
+
+### Standard options
+
+
+Here are the Standard options specific to google photos (Google Photos).
+
+
+#### --gphotos-client-id
+
+
+OAuth Client Id.
+
+
+Leave blank normally.
+
+
+Properties:
+
+
+- Config: client_id - Env Var: RCLONE_GPHOTOS_CLIENT_ID - Type: string - Required: false
+
+
+#### --gphotos-client-secret
+
+
+OAuth Client Secret.
+
+
+Leave blank normally.
+
+
+Properties:
+
+
+- Config: client_secret - Env Var: RCLONE_GPHOTOS_CLIENT_SECRET - Type: string - Required: false
+
+
+#### --gphotos-read-only
+
+
+Set to make the Google Photos backend read only.
+
+
+If you choose read only then rclone will only request read only access to your photos, otherwise rclone will request full access.
+
+
+Properties:
+
+
+- Config: read_only - Env Var: RCLONE_GPHOTOS_READ_ONLY - Type: bool - Default: false
+
+
+### Advanced options
+
+
+Here are the Advanced options specific to google photos (Google Photos).
+
+
+#### --gphotos-token
+
+
+OAuth Access Token as a JSON blob.
+
+
+Properties:
+
+
+- Config: token - Env Var: RCLONE_GPHOTOS_TOKEN - Type: string - Required: false
+
+
+#### --gphotos-auth-url
+
+
+Auth server URL.
+
+
+Leave blank to use the provider defaults.
+
+
+Properties:
+
+
+- Config: auth_url - Env Var: RCLONE_GPHOTOS_AUTH_URL - Type: string - Required: false
+
+
+#### --gphotos-token-url
+
+
+Token server url.
+
+
+Leave blank to use the provider defaults.
+
+
+Properties:
+
+
+- Config: token_url - Env Var: RCLONE_GPHOTOS_TOKEN_URL - Type: string - Required: false
+
+
+#### --gphotos-read-size
+
+
+Set to read the size of media items.
+
+
+Normally rclone does not read the size of media items since this takes another transaction. This isn't necessary for syncing. However rclone mount needs to know the size of files in advance of reading them, so setting this flag when using rclone mount is recommended if you want to read the media.
+
+
+Properties:
+
+
+- Config: read_size - Env Var: RCLONE_GPHOTOS_READ_SIZE - Type: bool - Default: false
+
+
+#### --gphotos-start-year
+
+
+Year limits the photos to be downloaded to those which are uploaded after the given year.
+
+
+Properties:
+
+
+- Config: start_year - Env Var: RCLONE_GPHOTOS_START_YEAR - Type: int - Default: 2000
+
+
+#### --gphotos-include-archived
+
+
+Also view and download archived media.
+
+
+By default, rclone does not request archived media. Thus, when syncing, archived media is not visible in directory listings or transferred.
+
+
+Note that media in albums is always visible and synced, no matter their archive status.
+
+
+With this flag, archived media are always visible in directory listings and transferred.
+
+
+Without this flag, archived media will not be visible in directory listings and won't be transferred.
+
+
+Properties:
+
+
+- Config: include_archived - Env Var: RCLONE_GPHOTOS_INCLUDE_ARCHIVED - Type: bool - Default: false
+
+
+#### --gphotos-encoding
+
+
+The encoding for the backend.
+
+
+See the encoding section in the overview for more info.
+
+
+Properties:
+
+
+- Config: encoding - Env Var: RCLONE_GPHOTOS_ENCODING - Type: Encoding - Default: Slash,CrLf,InvalidUtf8,Dot
+
+
+#### --gphotos-batch-mode
+
+
+Upload file batching sync|async|off.
+
+
+This sets the batch mode used by rclone.
+
+
+This has 3 possible values
+
+
+- off - no batching - sync - batch uploads and check completion (default) - async - batch upload and don't check completion
+
+
+Rclone will close any outstanding batches when it exits which may make a delay on quit.
+
+
+Properties:
+
+
+- Config: batch_mode - Env Var: RCLONE_GPHOTOS_BATCH_MODE - Type: string - Default: "sync"
+
+
+#### --gphotos-batch-size
+
+
+Max number of files in upload batch.
+
+
+This sets the batch size of files to upload. It has to be less than 50.
+
+
+By default this is 0 which means rclone which calculate the batch size depending on the setting of batch_mode.
+
+
+- batch_mode: async - default batch_size is 50 - batch_mode: sync - default batch_size is the same as --transfers - batch_mode: off - not in use
+
+
+Rclone will close any outstanding batches when it exits which may make a delay on quit.
+
+
+Setting this is a great idea if you are uploading lots of small files as it will make them a lot quicker. You can use --transfers 32 to maximise throughput.
+
+
+Properties:
+
+
+- Config: batch_size - Env Var: RCLONE_GPHOTOS_BATCH_SIZE - Type: int - Default: 0
+
+
+#### --gphotos-batch-timeout
+
+
+Max time to allow an idle upload batch before uploading.
+
+
+If an upload batch is idle for more than this long then it will be uploaded.
+
+
+The default for this is 0 which means rclone will choose a sensible default based on the batch_mode in use.
+
+
+- batch_mode: async - default batch_timeout is 10s - batch_mode: sync - default batch_timeout is 1s - batch_mode: off - not in use
+
+
+Properties:
+
+
+- Config: batch_timeout - Env Var: RCLONE_GPHOTOS_BATCH_TIMEOUT - Type: Duration - Default: 0s
+
+
+#### --gphotos-batch-commit-timeout
+
+
+Max time to wait for a batch to finish committing
+
+
+Properties:
+
+
+- Config: batch_commit_timeout - Env Var: RCLONE_GPHOTOS_BATCH_COMMIT_TIMEOUT - Type: Duration - Default: 10m0s
+
+
+## Limitations
+
+
+Only images and videos can be uploaded. If you attempt to upload non videos or images or formats that Google Photos doesn't understand, rclone will upload the file, then Google Photos will give an error when it is put turned into a media item.
+
+
+Note that all media items uploaded to Google Photos through the API are stored in full resolution at "original quality" and will count towards your storage quota in your Google Account. The API does not offer a way to upload in "high quality" mode..
+
+
+rclone about is not supported by the Google Photos backend. Backends without this capability cannot determine free space for an rclone mount or use policy mfs (most free space) as a member of an rclone union remote.
+
+See List of backends that do not support rclone about See rclone about
+
+
+### Downloading Images
+
+
+When Images are downloaded this strips EXIF location (according to the docs and my tests). This is a limitation of the Google Photos API and is covered by bug #112096115.
+
+
+The current google API does not allow photos to be downloaded at original resolution. This is very important if you are, for example, relying on "Google Photos" as a backup of your photos. You will not be able to use rclone to redownload original images. You could use 'google takeout' to recover the original photos as a last resort
+
+
+### Downloading Videos
+
+
+When videos are downloaded they are downloaded in a really compressed version of the video compared to downloading it via the Google Photos web interface. This is covered by bug #113672044.
+
+
+### Duplicates
+
+
+If a file name is duplicated in a directory then rclone will add the file ID into its name. So two files called file.jpg would then appear as file {123456}.jpg and file {ABCDEF}.jpg (the actual IDs are a lot longer alas!).
+
+
+If you upload the same image (with the same binary data) twice then Google Photos will deduplicate it. However it will retain the filename from the first upload which may confuse rclone. For example if you uploaded an image to upload then uploaded the same image to album/my_album the filename of the image in album/my_album will be what it was uploaded with initially, not what you uploaded it with to album. In practise this shouldn't cause too many problems.
+
+
+### Modification times
+
+
+The date shown of media in Google Photos is the creation date as determined by the EXIF information, or the upload date if that is not known.
+
+
+This is not changeable by rclone and is not the modification date of the media on local disk. This means that rclone cannot use the dates from Google Photos for syncing purposes.
+
+
+### Size
+
+
+The Google Photos API does not return the size of media. This means that when syncing to Google Photos, rclone can only do a file existence check.
+
+
+It is possible to read the size of the media, but this needs an extra HTTP HEAD request per media item so is very slow and uses up a lot of transactions. This can be enabled with the --gphotos-read-size option or the read_size = true config parameter.
+
+
+If you want to use the backend with rclone mount you may need to enable this flag (depending on your OS and application using the photos) otherwise you may not be able to read media off the mount. You'll need to experiment to see if it works for you without the flag.
+
+
+### Albums
+
+
+Rclone can only upload files to albums it created. This is a limitation of the Google Photos API.
+
+
+Rclone can remove files it uploaded from albums it created only.
+
+
+### Deleting files
+
+
+Rclone can remove files from albums it created, but note that the Google Photos API does not allow media to be deleted permanently so this media will still remain. See bug #109759781.
+
+
+Rclone cannot delete files anywhere except under album.
+
+
+### Deleting albums
+
+
+The Google Photos API does not support deleting albums - see bug #135714733.
+
+
+# Hasher
+
+
+Hasher is a special overlay backend to create remotes which handle checksums for other remotes. It's main functions include: - Emulate hash types unimplemented by backends - Cache checksums to help with slow hashing of large local or (S)FTP files - Warm up checksum cache from external SUM files
+
+
+## Getting started
+
+
+To use Hasher, first set up the underlying remote following the configuration instructions for that remote. You can also use a local pathname instead of a remote. Check that your base remote is working.
+
+
+Let's call the base remote myRemote:path here. Note that anything inside myRemote:path will be handled by hasher and anything outside won't. This means that if you are using a bucket based remote (S3, B2, Swift) then you should put the bucket in the remote s3:bucket.
+
+
+Now proceed to interactive or manual configuration.
+
+
+### Interactive configuration
+
+
+Run rclone config: ``` No remotes found, make a new one? n) New remote s) Set configuration password q) Quit config n/s/q> n name> Hasher1 Type of storage to configure. Choose a number from below, or type in your own value [snip] XX / Handle checksums for other remotes  "hasher" [snip] Storage> hasher Remote to cache checksums for, like myremote:mypath. Enter a string value. Press Enter for the default (""). remote> myRemote:path Comma separated list of supported checksum types. Enter a string value. Press Enter for the default ("md5,sha1"). hashsums> md5 Maximum time to keep checksums in cache. 0 = no cache, off = cache forever. max_age> off Edit advanced config? (y/n) y) Yes n) No y/n> n Remote config
+
+
+
+[Hasher1] type = hasher remote = myRemote:path hashsums = md5 max_age = off -------------------- y) Yes this is OK e) Edit this remote d) Delete this remote y/e/d> y
+
+### Manual configuration
-Corrupted checksums are caused by Google modifying the image/video but
-not updating the checksum.
+Run `rclone config path` to see the path of current active config file,
+usually `YOURHOME/.config/rclone/rclone.conf`.
+Open it in your favorite text editor, find section for the base remote
+and create new section for hasher like in the following examples:
+
+[Hasher1] type = hasher remote = myRemote:path hashes = md5 max_age = off
+[Hasher2] type = hasher remote = /local/path hashes = dropbox,sha1 max_age = 24h
+
+Hasher takes basically the following parameters:
+- `remote` is required,
+- `hashes` is a comma separated list of supported checksums
+ (by default `md5,sha1`),
+- `max_age` - maximum time to keep a checksum value in the cache,
+ `0` will disable caching completely,
+ `off` will cache "forever" (that is until the files get changed).
-Properties:
+Make sure the `remote` has `:` (colon) in. If you specify the remote without
+a colon then rclone will use a local directory of that name. So if you use
+a remote of `/local/path` then rclone will handle hashes for that directory.
+If you use `remote = name` literally then rclone will put files
+**in a directory called `name` located under current directory**.
-- Config: skip_checksum_gphotos
-- Env Var: RCLONE_DRIVE_SKIP_CHECKSUM_GPHOTOS
-- Type: bool
-- Default: false
+## Usage
-#### --drive-shared-with-me
+### Basic operations
-Only show files that are shared with me.
+Now you can use it as `Hasher2:subdir/file` instead of base remote.
+Hasher will transparently update cache with new checksums when a file
+is fully read or overwritten, like:
+rclone copy External:path/file Hasher:dest/path
+rclone cat Hasher:path/to/file > /dev/null
+
+The way to refresh **all** cached checksums (even unsupported by the base backend)
+for a subtree is to **re-download** all files in the subtree. For example,
+use `hashsum --download` using **any** supported hashsum on the command line
+(we just care to re-read):
+rclone hashsum MD5 --download Hasher:path/to/subtree > /dev/null
+rclone backend dump Hasher:path/to/subtree
+
+You can print or drop hashsum cache using custom backend commands:
+rclone backend dump Hasher:dir/subdir
+rclone backend drop Hasher:
+
+### Pre-Seed from a SUM File
-Instructs rclone to operate on your "Shared with me" folder (where
-Google Drive lets you access the files and folders others have shared
-with you).
+Hasher supports two backend commands: generic SUM file `import` and faster
+but less consistent `stickyimport`.
+
+rclone backend import Hasher:dir/subdir SHA1 /path/to/SHA1SUM [--checkers 4]
+
+Instead of SHA1 it can be any hash supported by the remote. The last argument
+can point to either a local or an `other-remote:path` text file in SUM format.
+The command will parse the SUM file, then walk down the path given by the
+first argument, snapshot current fingerprints and fill in the cache entries
+correspondingly.
+- Paths in the SUM file are treated as relative to `hasher:dir/subdir`.
+- The command will **not** check that supplied values are correct.
+ You **must know** what you are doing.
+- This is a one-time action. The SUM file will not get "attached" to the
+ remote. Cache entries can still be overwritten later, should the object's
+ fingerprint change.
+- The tree walk can take long depending on the tree size. You can increase
+ `--checkers` to make it faster. Or use `stickyimport` if you don't care
+ about fingerprints and consistency.
+
+rclone backend stickyimport hasher:path/to/data sha1 remote:/path/to/sum.sha1
+
+`stickyimport` is similar to `import` but works much faster because it
+does not need to stat existing files and skips initial tree walk.
+Instead of binding cache entries to file fingerprints it creates _sticky_
+entries bound to the file name alone ignoring size, modification time etc.
+Such hash entries can be replaced only by `purge`, `delete`, `backend drop`
+or by full re-read/re-write of the files.
-This works both with the "list" (lsd, lsl, etc.) and the "copy"
-commands (copy, sync, etc.), and with all other commands too.
+## Configuration reference
-Properties:
-- Config: shared_with_me
-- Env Var: RCLONE_DRIVE_SHARED_WITH_ME
-- Type: bool
-- Default: false
+### Standard options
-#### --drive-trashed-only
+Here are the Standard options specific to hasher (Better checksums for other remotes).
-Only show files that are in the trash.
+#### --hasher-remote
-This will show trashed files in their original directory structure.
+Remote to cache checksums for (e.g. myRemote:path).
Properties:
-- Config: trashed_only
-- Env Var: RCLONE_DRIVE_TRASHED_ONLY
-- Type: bool
-- Default: false
+- Config: remote
+- Env Var: RCLONE_HASHER_REMOTE
+- Type: string
+- Required: true
-#### --drive-starred-only
+#### --hasher-hashes
-Only show files that are starred.
+Comma separated list of supported checksum types.
Properties:
-- Config: starred_only
-- Env Var: RCLONE_DRIVE_STARRED_ONLY
-- Type: bool
-- Default: false
+- Config: hashes
+- Env Var: RCLONE_HASHER_HASHES
+- Type: CommaSepList
+- Default: md5,sha1
-#### --drive-formats
+#### --hasher-max-age
-Deprecated: See export_formats.
+Maximum time to keep checksums in cache (0 = no cache, off = cache forever).
Properties:
-- Config: formats
-- Env Var: RCLONE_DRIVE_FORMATS
-- Type: string
-- Required: false
-
-#### --drive-export-formats
-
-Comma separated list of preferred formats for downloading Google docs.
+- Config: max_age
+- Env Var: RCLONE_HASHER_MAX_AGE
+- Type: Duration
+- Default: off
-Properties:
+### Advanced options
-- Config: export_formats
-- Env Var: RCLONE_DRIVE_EXPORT_FORMATS
-- Type: string
-- Default: "docx,xlsx,pptx,svg"
+Here are the Advanced options specific to hasher (Better checksums for other remotes).
-#### --drive-import-formats
+#### --hasher-auto-size
-Comma separated list of preferred formats for uploading Google docs.
+Auto-update checksum for files smaller than this size (disabled by default).
Properties:
-- Config: import_formats
-- Env Var: RCLONE_DRIVE_IMPORT_FORMATS
-- Type: string
-- Required: false
+- Config: auto_size
+- Env Var: RCLONE_HASHER_AUTO_SIZE
+- Type: SizeSuffix
+- Default: 0
-#### --drive-allow-import-name-change
+### Metadata
-Allow the filetype to change when uploading Google docs.
+Any metadata supported by the underlying remote is read and written.
-E.g. file.doc to file.docx. This will confuse sync and reupload every time.
+See the [metadata](https://rclone.org/docs/#metadata) docs for more info.
-Properties:
+## Backend commands
-- Config: allow_import_name_change
-- Env Var: RCLONE_DRIVE_ALLOW_IMPORT_NAME_CHANGE
-- Type: bool
-- Default: false
+Here are the commands specific to the hasher backend.
-#### --drive-use-created-date
+Run them with
-Use file created date instead of modified date.
+ rclone backend COMMAND remote:
-Useful when downloading data and you want the creation date used in
-place of the last modified date.
+The help below will explain what arguments each command takes.
-**WARNING**: This flag may have some unexpected consequences.
+See the [backend](https://rclone.org/commands/rclone_backend/) command for more
+info on how to pass options and arguments.
-When uploading to your drive all files will be overwritten unless they
-haven't been modified since their creation. And the inverse will occur
-while downloading. This side effect can be avoided by using the
-"--checksum" flag.
+These can be run on a running backend using the rc command
+[backend/command](https://rclone.org/rc/#backend-command).
-This feature was implemented to retain photos capture date as recorded
-by google photos. You will first need to check the "Create a Google
-Photos folder" option in your google drive settings. You can then copy
-or move the photos locally and use the date the image was taken
-(created) set as the modification date.
+### drop
-Properties:
+Drop cache
-- Config: use_created_date
-- Env Var: RCLONE_DRIVE_USE_CREATED_DATE
-- Type: bool
-- Default: false
+ rclone backend drop remote: [options] [<arguments>+]
-#### --drive-use-shared-date
+Completely drop checksum cache.
+Usage Example:
+ rclone backend drop hasher:
-Use date file was shared instead of modified date.
-Note that, as with "--drive-use-created-date", this flag may have
-unexpected consequences when uploading/downloading files.
+### dump
-If both this flag and "--drive-use-created-date" are set, the created
-date is used.
+Dump the database
-Properties:
+ rclone backend dump remote: [options] [<arguments>+]
-- Config: use_shared_date
-- Env Var: RCLONE_DRIVE_USE_SHARED_DATE
-- Type: bool
-- Default: false
+Dump cache records covered by the current remote
-#### --drive-list-chunk
+### fulldump
-Size of listing chunk 100-1000, 0 to disable.
+Full dump of the database
-Properties:
+ rclone backend fulldump remote: [options] [<arguments>+]
-- Config: list_chunk
-- Env Var: RCLONE_DRIVE_LIST_CHUNK
-- Type: int
-- Default: 1000
+Dump all cache records in the database
-#### --drive-impersonate
+### import
-Impersonate this user when using a service account.
+Import a SUM file
-Properties:
+ rclone backend import remote: [options] [<arguments>+]
-- Config: impersonate
-- Env Var: RCLONE_DRIVE_IMPERSONATE
-- Type: string
-- Required: false
+Amend hash cache from a SUM file and bind checksums to files by size/time.
+Usage Example:
+ rclone backend import hasher:subdir md5 /path/to/sum.md5
-#### --drive-upload-cutoff
-Cutoff for switching to chunked upload.
+### stickyimport
-Properties:
+Perform fast import of a SUM file
-- Config: upload_cutoff
-- Env Var: RCLONE_DRIVE_UPLOAD_CUTOFF
-- Type: SizeSuffix
-- Default: 8Mi
+ rclone backend stickyimport remote: [options] [<arguments>+]
-#### --drive-chunk-size
+Fill hash cache from a SUM file without verifying file fingerprints.
+Usage Example:
+ rclone backend stickyimport hasher:subdir md5 remote:path/to/sum.md5
-Upload chunk size.
-Must a power of 2 >= 256k.
-Making this larger will improve performance, but note that each chunk
-is buffered in memory one per transfer.
-Reducing this will reduce memory usage but decrease performance.
+## Implementation details (advanced)
-Properties:
+This section explains how various rclone operations work on a hasher remote.
-- Config: chunk_size
-- Env Var: RCLONE_DRIVE_CHUNK_SIZE
-- Type: SizeSuffix
-- Default: 8Mi
+**Disclaimer. This section describes current implementation which can
+change in future rclone versions!.**
-#### --drive-acknowledge-abuse
+### Hashsum command
-Set to allow files which return cannotDownloadAbusiveFile to be downloaded.
+The `rclone hashsum` (or `md5sum` or `sha1sum`) command will:
-If downloading a file returns the error "This file has been identified
-as malware or spam and cannot be downloaded" with the error code
-"cannotDownloadAbusiveFile" then supply this flag to rclone to
-indicate you acknowledge the risks of downloading the file and rclone
-will download it anyway.
+1. if requested hash is supported by lower level, just pass it.
+2. if object size is below `auto_size` then download object and calculate
+ _requested_ hashes on the fly.
+3. if unsupported and the size is big enough, build object `fingerprint`
+ (including size, modtime if supported, first-found _other_ hash if any).
+4. if the strict match is found in cache for the requested remote, return
+ the stored hash.
+5. if remote found but fingerprint mismatched, then purge the entry and
+ proceed to step 6.
+6. if remote not found or had no requested hash type or after step 5:
+ download object, calculate all _supported_ hashes on the fly and store
+ in cache; return requested hash.
-Note that if you are using service account it will need Manager
-permission (not Content Manager) to for this flag to work. If the SA
-does not have the right permission, Google will just ignore the flag.
+### Other operations
-Properties:
+- whenever a file is uploaded or downloaded **in full**, capture the stream
+ to calculate all supported hashes on the fly and update database
+- server-side `move` will update keys of existing cache entries
+- `deletefile` will remove a single cache entry
+- `purge` will remove all cache entries under the purged path
-- Config: acknowledge_abuse
-- Env Var: RCLONE_DRIVE_ACKNOWLEDGE_ABUSE
-- Type: bool
-- Default: false
+Note that setting `max_age = 0` will disable checksum caching completely.
-#### --drive-keep-revision-forever
+If you set `max_age = off`, checksums in cache will never age, unless you
+fully rewrite or delete the file.
-Keep new head revision of each file forever.
+### Cache storage
-Properties:
+Cached checksums are stored as `bolt` database files under rclone cache
+directory, usually `~/.cache/rclone/kv/`. Databases are maintained
+one per _base_ backend, named like `BaseRemote~hasher.bolt`.
+Checksums for multiple `alias`-es into a single base backend
+will be stored in the single database. All local paths are treated as
+aliases into the `local` backend (unless encrypted or chunked) and stored
+in `~/.cache/rclone/kv/local~hasher.bolt`.
+Databases can be shared between multiple rclone processes.
-- Config: keep_revision_forever
-- Env Var: RCLONE_DRIVE_KEEP_REVISION_FOREVER
-- Type: bool
-- Default: false
+# HDFS
-#### --drive-size-as-quota
+[HDFS](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html) is a
+distributed file-system, part of the [Apache Hadoop](https://hadoop.apache.org/) framework.
-Show sizes as storage quota usage, not actual size.
+Paths are specified as `remote:` or `remote:path/to/dir`.
-Show the size of a file as the storage quota used. This is the
-current version plus any older versions that have been set to keep
-forever.
+## Configuration
-**WARNING**: This flag may have some unexpected consequences.
+Here is an example of how to make a remote called `remote`. First run:
-It is not recommended to set this flag in your config - the
-recommended usage is using the flag form --drive-size-as-quota when
-doing rclone ls/lsl/lsf/lsjson/etc only.
+ rclone config
-If you do use this flag for syncing (not recommended) then you will
-need to use --ignore size also.
+This will guide you through an interactive setup process:
+
+No remotes found, make a new one? n) New remote s) Set configuration password q) Quit config n/s/q> n name> remote Type of storage to configure. Enter a string value. Press Enter for the default (""). Choose a number from below, or type in your own value [skip] XX / Hadoop distributed file system  "hdfs" [skip] Storage> hdfs ** See help for hdfs backend at: https://rclone.org/hdfs/ **
+hadoop name node and port Enter a string value. Press Enter for the default (""). Choose a number from below, or type in your own value 1 / Connect to host namenode at port 8020  "namenode:8020" namenode> namenode.hadoop:8020 hadoop user name Enter a string value. Press Enter for the default (""). Choose a number from below, or type in your own value 1 / Connect to hdfs as root  "root" username> root Edit advanced config? (y/n) y) Yes n) No (default) y/n> n Remote config -------------------- [remote] type = hdfs namenode = namenode.hadoop:8020 username = root -------------------- y) Yes this is OK (default) e) Edit this remote d) Delete this remote y/e/d> y Current remotes:
+Name Type ==== ==== hadoop hdfs
+
+- Edit existing remote
+- New remote
+- Delete remote
+- Rename remote
+- Copy remote
+- Set configuration password
+- Quit config e/n/d/r/c/s/q> q
+
+
+This remote is called `remote` and can now be used like this
-Properties:
+See all the top level directories
-- Config: size_as_quota
-- Env Var: RCLONE_DRIVE_SIZE_AS_QUOTA
-- Type: bool
-- Default: false
+ rclone lsd remote:
-#### --drive-v2-download-min-size
+List the contents of a directory
-If Object's are greater, use drive v2 API to download.
+ rclone ls remote:directory
-Properties:
+Sync the remote `directory` to `/home/local/directory`, deleting any excess files.
-- Config: v2_download_min_size
-- Env Var: RCLONE_DRIVE_V2_DOWNLOAD_MIN_SIZE
-- Type: SizeSuffix
-- Default: off
+ rclone sync --interactive remote:directory /home/local/directory
-#### --drive-pacer-min-sleep
+### Setting up your own HDFS instance for testing
-Minimum time to sleep between API calls.
+You may start with a [manual setup](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html)
+or use the docker image from the tests:
-Properties:
+If you want to build the docker image
+
+git clone https://github.com/rclone/rclone.git cd rclone/fstest/testserver/images/test-hdfs docker build --rm -t rclone/test-hdfs .
+
+Or you can just use the latest one pushed
+
+docker run --rm --name "rclone-hdfs" -p 127.0.0.1:9866:9866 -p 127.0.0.1:8020:8020 --hostname "rclone-hdfs" rclone/test-hdfs
+
+**NB** it need few seconds to startup.
-- Config: pacer_min_sleep
-- Env Var: RCLONE_DRIVE_PACER_MIN_SLEEP
-- Type: Duration
-- Default: 100ms
+For this docker image the remote needs to be configured like this:
+
+[remote] type = hdfs namenode = 127.0.0.1:8020 username = root
+
+You can stop this image with `docker kill rclone-hdfs` (**NB** it does not use volumes, so all data
+uploaded will be lost.)
-#### --drive-pacer-burst
+### Modification times
-Number of API calls to allow without sleeping.
+Time accurate to 1 second is stored.
-Properties:
+### Checksum
-- Config: pacer_burst
-- Env Var: RCLONE_DRIVE_PACER_BURST
-- Type: int
-- Default: 100
+No checksums are implemented.
-#### --drive-server-side-across-configs
+### Usage information
-Deprecated: use --server-side-across-configs instead.
+You can use the `rclone about remote:` command which will display filesystem size and current usage.
-Allow server-side operations (e.g. copy) to work across different drive configs.
+### Restricted filename characters
-This can be useful if you wish to do a server-side copy between two
-different Google drives. Note that this isn't enabled by default
-because it isn't easy to tell if it will work between any two
-configurations.
+In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters)
+the following characters are also replaced:
-Properties:
+| Character | Value | Replacement |
+| --------- |:-----:|:-----------:|
+| : | 0x3A | ： |
-- Config: server_side_across_configs
-- Env Var: RCLONE_DRIVE_SERVER_SIDE_ACROSS_CONFIGS
-- Type: bool
-- Default: false
+Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8).
-#### --drive-disable-http2
-Disable drive using http2.
+### Standard options
-There is currently an unsolved issue with the google drive backend and
-HTTP/2. HTTP/2 is therefore disabled by default for the drive backend
-but can be re-enabled here. When the issue is solved this flag will
-be removed.
+Here are the Standard options specific to hdfs (Hadoop distributed file system).
-See: https://github.com/rclone/rclone/issues/3631
+#### --hdfs-namenode
+Hadoop name nodes and ports.
+E.g. "namenode-1:8020,namenode-2:8020,..." to connect to host namenodes at port 8020.
Properties:
-- Config: disable_http2
-- Env Var: RCLONE_DRIVE_DISABLE_HTTP2
-- Type: bool
-- Default: true
-
-#### --drive-stop-on-upload-limit
-
-Make upload limit errors be fatal.
-
-At the time of writing it is only possible to upload 750 GiB of data to
-Google Drive a day (this is an undocumented limit). When this limit is
-reached Google Drive produces a slightly different error message. When
-this flag is set it causes these errors to be fatal. These will stop
-the in-progress sync.
-
-Note that this detection is relying on error message strings which
-Google don't document so it may break in the future.
+- Config: namenode
+- Env Var: RCLONE_HDFS_NAMENODE
+- Type: CommaSepList
+- Default:
-See: https://github.com/rclone/rclone/issues/3857
+#### --hdfs-username
+Hadoop user name.
Properties:
-- Config: stop_on_upload_limit
-- Env Var: RCLONE_DRIVE_STOP_ON_UPLOAD_LIMIT
-- Type: bool
-- Default: false
+- Config: username
+- Env Var: RCLONE_HDFS_USERNAME
+- Type: string
+- Required: false
+- Examples:
+ - "root"
+ - Connect to hdfs as root.
-#### --drive-stop-on-download-limit
+### Advanced options
-Make download limit errors be fatal.
+Here are the Advanced options specific to hdfs (Hadoop distributed file system).
-At the time of writing it is only possible to download 10 TiB of data from
-Google Drive a day (this is an undocumented limit). When this limit is
-reached Google Drive produces a slightly different error message. When
-this flag is set it causes these errors to be fatal. These will stop
-the in-progress sync.
+#### --hdfs-service-principal-name
-Note that this detection is relying on error message strings which
-Google don't document so it may break in the future.
+Kerberos service principal name for the namenode.
+Enables KERBEROS authentication. Specifies the Service Principal Name
+(SERVICE/FQDN) for the namenode. E.g. \"hdfs/namenode.hadoop.docker\"
+for namenode running as service 'hdfs' with FQDN 'namenode.hadoop.docker'.
Properties:
-- Config: stop_on_download_limit
-- Env Var: RCLONE_DRIVE_STOP_ON_DOWNLOAD_LIMIT
-- Type: bool
-- Default: false
-
-#### --drive-skip-shortcuts
+- Config: service_principal_name
+- Env Var: RCLONE_HDFS_SERVICE_PRINCIPAL_NAME
+- Type: string
+- Required: false
-If set skip shortcut files.
+#### --hdfs-data-transfer-protection
-Normally rclone dereferences shortcut files making them appear as if
-they are the original file (see [the shortcuts section](#shortcuts)).
-If this flag is set then rclone will ignore shortcut files completely.
+Kerberos data transfer protection: authentication|integrity|privacy.
+Specifies whether or not authentication, data signature integrity
+checks, and wire encryption are required when communicating with
+the datanodes. Possible values are 'authentication', 'integrity'
+and 'privacy'. Used only with KERBEROS enabled.
Properties:
-- Config: skip_shortcuts
-- Env Var: RCLONE_DRIVE_SKIP_SHORTCUTS
-- Type: bool
-- Default: false
-
-#### --drive-skip-dangling-shortcuts
+- Config: data_transfer_protection
+- Env Var: RCLONE_HDFS_DATA_TRANSFER_PROTECTION
+- Type: string
+- Required: false
+- Examples:
+ - "privacy"
+ - Ensure authentication, integrity and encryption enabled.
-If set skip dangling shortcut files.
+#### --hdfs-encoding
-If this is set then rclone will not show any dangling shortcuts in listings.
+The encoding for the backend.
+See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info.
Properties:
-- Config: skip_dangling_shortcuts
-- Env Var: RCLONE_DRIVE_SKIP_DANGLING_SHORTCUTS
-- Type: bool
-- Default: false
+- Config: encoding
+- Env Var: RCLONE_HDFS_ENCODING
+- Type: Encoding
+- Default: Slash,Colon,Del,Ctl,InvalidUtf8,Dot
-#### --drive-resource-key
-Resource key for accessing a link-shared file.
-If you need to access files shared with a link like this
+## Limitations
- https://drive.google.com/drive/folders/XXX?resourcekey=YYY&usp=sharing
+- No server-side `Move` or `DirMove`.
+- Checksums not implemented.
-Then you will need to use the first part "XXX" as the "root_folder_id"
-and the second part "YYY" as the "resource_key" otherwise you will get
-404 not found errors when trying to access the directory.
+# HiDrive
-See: https://developers.google.com/drive/api/guides/resource-keys
+Paths are specified as `remote:path`
-This resource key requirement only applies to a subset of old files.
+Paths may be as deep as required, e.g. `remote:directory/subdirectory`.
-Note also that opening the folder once in the web interface (with the
-user you've authenticated rclone with) seems to be enough so that the
-resource key is not needed.
+The initial setup for hidrive involves getting a token from HiDrive
+which you need to do in your browser.
+`rclone config` walks you through it.
+## Configuration
-Properties:
+Here is an example of how to make a remote called `remote`. First run:
-- Config: resource_key
-- Env Var: RCLONE_DRIVE_RESOURCE_KEY
-- Type: string
-- Required: false
+ rclone config
-#### --drive-fast-list-bug-fix
+This will guide you through an interactive setup process:
+
+No remotes found - make a new one n) New remote s) Set configuration password q) Quit config n/s/q> n name> remote Type of storage to configure. Choose a number from below, or type in your own value [snip] XX / HiDrive  "hidrive" [snip] Storage> hidrive OAuth Client Id - Leave blank normally. client_id> OAuth Client Secret - Leave blank normally. client_secret> Access permissions that rclone should use when requesting access from HiDrive. Leave blank normally. scope_access> Edit advanced config? y/n> n Use web browser to automatically authenticate rclone with remote? * Say Y if the machine running rclone has a web browser you can use * Say N if running rclone on a (remote) machine without web browser access If not sure try Y. If Y failed, try N. y/n> y If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth?state=xxxxxxxxxxxxxxxxxxxxxx Log in and authorize rclone for access Waiting for code... Got code -------------------- [remote] type = hidrive token = {"access_token":"xxxxxxxxxxxxxxxxxxxx","token_type":"Bearer","refresh_token":"xxxxxxxxxxxxxxxxxxxxxxx","expiry":"xxxxxxxxxxxxxxxxxxxxxxx"} -------------------- y) Yes this is OK (default) e) Edit this remote d) Delete this remote y/e/d> y
+
+**You should be aware that OAuth-tokens can be used to access your account
+and hence should not be shared with other persons.**
+See the [below section](#keeping-your-tokens-safe) for more information.
-Work around a bug in Google Drive listing.
+See the [remote setup docs](https://rclone.org/remote_setup/) for how to set it up on a
+machine with no Internet browser available.
-Normally rclone will work around a bug in Google Drive when using
---fast-list (ListR) where the search "(A in parents) or (B in
-parents)" returns nothing sometimes. See #3114, #4289 and
-https://issuetracker.google.com/issues/149522397
+Note that rclone runs a webserver on your local machine to collect the
+token as returned from HiDrive. This only runs from the moment it opens
+your browser to the moment you get back the verification code.
+The webserver runs on `http://127.0.0.1:53682/`.
+If local port `53682` is protected by a firewall you may need to temporarily
+unblock the firewall to complete authorization.
-Rclone detects this by finding no items in more than one directory
-when listing and retries them as lists of individual directories.
+Once configured you can then use `rclone` like this,
-This means that if you have a lot of empty directories rclone will end
-up listing them all individually and this can take many more API
-calls.
+List directories in top level of your HiDrive root folder
-This flag allows the work-around to be disabled. This is **not**
-recommended in normal use - only if you have a particular case you are
-having trouble with like many empty directories.
+ rclone lsd remote:
+List all the files in your HiDrive filesystem
-Properties:
+ rclone ls remote:
-- Config: fast_list_bug_fix
-- Env Var: RCLONE_DRIVE_FAST_LIST_BUG_FIX
-- Type: bool
-- Default: true
+To copy a local directory to a HiDrive directory called backup
-#### --drive-encoding
+ rclone copy /home/source remote:backup
-The encoding for the backend.
+### Keeping your tokens safe
-See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info.
+Any OAuth-tokens will be stored by rclone in the remote's configuration file as unencrypted text.
+Anyone can use a valid refresh-token to access your HiDrive filesystem without knowing your password.
+Therefore you should make sure no one else can access your configuration.
-Properties:
+It is possible to encrypt rclone's configuration file.
+You can find information on securing your configuration file by viewing the [configuration encryption docs](https://rclone.org/docs/#configuration-encryption).
-- Config: encoding
-- Env Var: RCLONE_DRIVE_ENCODING
-- Type: MultiEncoder
-- Default: InvalidUtf8
+### Invalid refresh token
-#### --drive-env-auth
+As can be verified [here](https://developer.hidrive.com/basics-flows/),
+each `refresh_token` (for Native Applications) is valid for 60 days.
+If used to access HiDrivei, its validity will be automatically extended.
-Get IAM credentials from runtime (environment variables or instance meta data if no env vars).
+This means that if you
-Only applies if service_account_file and service_account_credentials is blank.
+ * Don't use the HiDrive remote for 60 days
-Properties:
+then rclone will return an error which includes a text
+that implies the refresh token is *invalid* or *expired*.
-- Config: env_auth
-- Env Var: RCLONE_DRIVE_ENV_AUTH
-- Type: bool
-- Default: false
-- Examples:
- - "false"
- - Enter credentials in the next step.
- - "true"
- - Get GCP IAM credentials from the environment (env vars or IAM).
+To fix this you will need to authorize rclone to access your HiDrive account again.
-## Backend commands
+Using
-Here are the commands specific to the drive backend.
+ rclone config reconnect remote:
-Run them with
+the process is very similar to the process of initial setup exemplified before.
- rclone backend COMMAND remote:
+### Modification times and hashes
-The help below will explain what arguments each command takes.
+HiDrive allows modification times to be set on objects accurate to 1 second.
-See the [backend](https://rclone.org/commands/rclone_backend/) command for more
-info on how to pass options and arguments.
+HiDrive supports [its own hash type](https://static.hidrive.com/dev/0001)
+which is used to verify the integrity of file contents after successful transfers.
-These can be run on a running backend using the rc command
-[backend/command](https://rclone.org/rc/#backend-command).
+### Restricted filename characters
-### get
+HiDrive cannot store files or folders that include
+`/` (0x2F) or null-bytes (0x00) in their name.
+Any other characters can be used in the names of files or folders.
+Additionally, files or folders cannot be named either of the following: `.` or `..`
-Get command for fetching the drive config parameters
+Therefore rclone will automatically replace these characters,
+if files or folders are stored or accessed with such names.
- rclone backend get remote: [options] [<arguments>+]
+You can read about how this filename encoding works in general
+[here](overview/#restricted-filenames).
-This is a get command which will be used to fetch the various drive config parameters
+Keep in mind that HiDrive only supports file or folder names
+with a length of 255 characters or less.
-Usage Examples:
+### Transfers
- rclone backend get drive: [-o service_account_file] [-o chunk_size]
- rclone rc backend/command command=get fs=drive: [-o service_account_file] [-o chunk_size]
+HiDrive limits file sizes per single request to a maximum of 2 GiB.
+To allow storage of larger files and allow for better upload performance,
+the hidrive backend will use a chunked transfer for files larger than 96 MiB.
+Rclone will upload multiple parts/chunks of the file at the same time.
+Chunks in the process of being uploaded are buffered in memory,
+so you may want to restrict this behaviour on systems with limited resources.
+You can customize this behaviour using the following options:
-Options:
+* `chunk_size`: size of file parts
+* `upload_cutoff`: files larger or equal to this in size will use a chunked transfer
+* `upload_concurrency`: number of file-parts to upload at the same time
-- "chunk_size": show the current upload chunk size
-- "service_account_file": show the current service account file
+See the below section about configuration options for more details.
-### set
+### Root folder
-Set command for updating the drive config parameters
+You can set the root folder for rclone.
+This is the directory that rclone considers to be the root of your HiDrive.
- rclone backend set remote: [options] [<arguments>+]
+Usually, you will leave this blank, and rclone will use the root of the account.
-This is a set command which will be used to update the various drive config parameters
+However, you can set this to restrict rclone to a specific folder hierarchy.
-Usage Examples:
+This works by prepending the contents of the `root_prefix` option
+to any paths accessed by rclone.
+For example, the following two ways to access the home directory are equivalent:
- rclone backend set drive: [-o service_account_file=sa.json] [-o chunk_size=67108864]
- rclone rc backend/command command=set fs=drive: [-o service_account_file=sa.json] [-o chunk_size=67108864]
+ rclone lsd --hidrive-root-prefix="/users/test/" remote:path
+ rclone lsd remote:/users/test/path
-Options:
+See the below section about configuration options for more details.
-- "chunk_size": update the current upload chunk size
-- "service_account_file": update the current service account file
+### Directory member count
-### shortcut
+By default, rclone will know the number of directory members contained in a directory.
+For example, `rclone lsd` uses this information.
-Create shortcuts from files or directories
+The acquisition of this information will result in additional time costs for HiDrive's API.
+When dealing with large directory structures, it may be desirable to circumvent this time cost,
+especially when this information is not explicitly needed.
+For this, the `disable_fetching_member_count` option can be used.
- rclone backend shortcut remote: [options] [<arguments>+]
+See the below section about configuration options for more details.
-This command creates shortcuts from files or directories.
-Usage:
+### Standard options
- rclone backend shortcut drive: source_item destination_shortcut
- rclone backend shortcut drive: source_item -o target=drive2: destination_shortcut
+Here are the Standard options specific to hidrive (HiDrive).
-In the first example this creates a shortcut from the "source_item"
-which can be a file or a directory to the "destination_shortcut". The
-"source_item" and the "destination_shortcut" should be relative paths
-from "drive:"
+#### --hidrive-client-id
-In the second example this creates a shortcut from the "source_item"
-relative to "drive:" to the "destination_shortcut" relative to
-"drive2:". This may fail with a permission error if the user
-authenticated with "drive2:" can't read files from "drive:".
+OAuth Client Id.
+Leave blank normally.
-Options:
+Properties:
-- "target": optional target remote for the shortcut destination
+- Config: client_id
+- Env Var: RCLONE_HIDRIVE_CLIENT_ID
+- Type: string
+- Required: false
-### drives
+#### --hidrive-client-secret
-List the Shared Drives available to this account
+OAuth Client Secret.
- rclone backend drives remote: [options] [<arguments>+]
+Leave blank normally.
-This command lists the Shared Drives (Team Drives) available to this
-account.
+Properties:
-Usage:
+- Config: client_secret
+- Env Var: RCLONE_HIDRIVE_CLIENT_SECRET
+- Type: string
+- Required: false
- rclone backend [-o config] drives drive:
+#### --hidrive-scope-access
-This will return a JSON list of objects like this
+Access permissions that rclone should use when requesting access from HiDrive.
- [
- {
- "id": "0ABCDEF-01234567890",
- "kind": "drive#teamDrive",
- "name": "My Drive"
- },
- {
- "id": "0ABCDEFabcdefghijkl",
- "kind": "drive#teamDrive",
- "name": "Test Drive"
- }
- ]
+Properties:
-With the -o config parameter it will output the list in a format
-suitable for adding to a config file to make aliases for all the
-drives found and a combined drive.
+- Config: scope_access
+- Env Var: RCLONE_HIDRIVE_SCOPE_ACCESS
+- Type: string
+- Default: "rw"
+- Examples:
+ - "rw"
+ - Read and write access to resources.
+ - "ro"
+ - Read-only access to resources.
- [My Drive]
- type = alias
- remote = drive,team_drive=0ABCDEF-01234567890,root_folder_id=:
+### Advanced options
- [Test Drive]
- type = alias
- remote = drive,team_drive=0ABCDEFabcdefghijkl,root_folder_id=:
+Here are the Advanced options specific to hidrive (HiDrive).
- [AllDrives]
- type = combine
- upstreams = "My Drive=My Drive:" "Test Drive=Test Drive:"
+#### --hidrive-token
-Adding this to the rclone config file will cause those team drives to
-be accessible with the aliases shown. Any illegal characters will be
-substituted with "_" and duplicate names will have numbers suffixed.
-It will also add a remote called AllDrives which shows all the shared
-drives combined into one directory tree.
+OAuth Access Token as a JSON blob.
+Properties:
-### untrash
+- Config: token
+- Env Var: RCLONE_HIDRIVE_TOKEN
+- Type: string
+- Required: false
-Untrash files and directories
+#### --hidrive-auth-url
- rclone backend untrash remote: [options] [<arguments>+]
+Auth server URL.
-This command untrashes all the files and directories in the directory
-passed in recursively.
+Leave blank to use the provider defaults.
-Usage:
+Properties:
-This takes an optional directory to trash which make this easier to
-use via the API.
+- Config: auth_url
+- Env Var: RCLONE_HIDRIVE_AUTH_URL
+- Type: string
+- Required: false
- rclone backend untrash drive:directory
- rclone backend --interactive untrash drive:directory subdir
+#### --hidrive-token-url
-Use the --interactive/-i or --dry-run flag to see what would be restored before restoring it.
+Token server url.
-Result:
+Leave blank to use the provider defaults.
- {
- "Untrashed": 17,
- "Errors": 0
- }
+Properties:
+- Config: token_url
+- Env Var: RCLONE_HIDRIVE_TOKEN_URL
+- Type: string
+- Required: false
-### copyid
+#### --hidrive-scope-role
-Copy files by ID
+User-level that rclone should use when requesting access from HiDrive.
- rclone backend copyid remote: [options] [<arguments>+]
+Properties:
-This command copies files by ID
+- Config: scope_role
+- Env Var: RCLONE_HIDRIVE_SCOPE_ROLE
+- Type: string
+- Default: "user"
+- Examples:
+ - "user"
+ - User-level access to management permissions.
+ - This will be sufficient in most cases.
+ - "admin"
+ - Extensive access to management permissions.
+ - "owner"
+ - Full access to management permissions.
-Usage:
+#### --hidrive-root-prefix
- rclone backend copyid drive: ID path
- rclone backend copyid drive: ID1 path1 ID2 path2
+The root/parent folder for all paths.
-It copies the drive file with ID given to the path (an rclone path which
-will be passed internally to rclone copyto). The ID and path pairs can be
-repeated.
+Fill in to use the specified folder as the parent for all paths given to the remote.
+This way rclone can use any folder as its starting point.
-The path should end with a / to indicate copy the file as named to
-this directory. If it doesn't end with a / then the last path
-component will be used as the file name.
+Properties:
-If the destination is a drive backend then server-side copying will be
-attempted if possible.
+- Config: root_prefix
+- Env Var: RCLONE_HIDRIVE_ROOT_PREFIX
+- Type: string
+- Default: "/"
+- Examples:
+ - "/"
+ - The topmost directory accessible by rclone.
+ - This will be equivalent with "root" if rclone uses a regular HiDrive user account.
+ - "root"
+ - The topmost directory of the HiDrive user account
+ - ""
+ - This specifies that there is no root-prefix for your paths.
+ - When using this you will always need to specify paths to this remote with a valid parent e.g. "remote:/path/to/dir" or "remote:root/path/to/dir".
-Use the --interactive/-i or --dry-run flag to see what would be copied before copying.
+#### --hidrive-endpoint
+Endpoint for the service.
-### exportformats
+This is the URL that API-calls will be made to.
-Dump the export formats for debug purposes
+Properties:
- rclone backend exportformats remote: [options] [<arguments>+]
+- Config: endpoint
+- Env Var: RCLONE_HIDRIVE_ENDPOINT
+- Type: string
+- Default: "https://api.hidrive.strato.com/2.1"
-### importformats
+#### --hidrive-disable-fetching-member-count
-Dump the import formats for debug purposes
+Do not fetch number of objects in directories unless it is absolutely necessary.
- rclone backend importformats remote: [options] [<arguments>+]
+Requests may be faster if the number of objects in subdirectories is not fetched.
+Properties:
+- Config: disable_fetching_member_count
+- Env Var: RCLONE_HIDRIVE_DISABLE_FETCHING_MEMBER_COUNT
+- Type: bool
+- Default: false
-## Limitations
+#### --hidrive-chunk-size
-Drive has quite a lot of rate limiting. This causes rclone to be
-limited to transferring about 2 files per second only. Individual
-files may be transferred much faster at 100s of MiB/s but lots of
-small files can take a long time.
+Chunksize for chunked uploads.
-Server side copies are also subject to a separate rate limit. If you
-see User rate limit exceeded errors, wait at least 24 hours and retry.
-You can disable server-side copies with `--disable copy` to download
-and upload the files if you prefer.
+Any files larger than the configured cutoff (or files of unknown size) will be uploaded in chunks of this size.
-### Limitations of Google Docs
+The upper limit for this is 2147483647 bytes (about 2.000Gi).
+That is the maximum amount of bytes a single upload-operation will support.
+Setting this above the upper limit or to a negative value will cause uploads to fail.
-Google docs will appear as size -1 in `rclone ls`, `rclone ncdu` etc,
-and as size 0 in anything which uses the VFS layer, e.g. `rclone mount`
-and `rclone serve`. When calculating directory totals, e.g. in
-`rclone size` and `rclone ncdu`, they will be counted in as empty
-files.
+Setting this to larger values may increase the upload speed at the cost of using more memory.
+It can be set to smaller values smaller to save on memory.
-This is because rclone can't find out the size of the Google docs
-without downloading them.
+Properties:
-Google docs will transfer correctly with `rclone sync`, `rclone copy`
-etc as rclone knows to ignore the size when doing the transfer.
+- Config: chunk_size
+- Env Var: RCLONE_HIDRIVE_CHUNK_SIZE
+- Type: SizeSuffix
+- Default: 48Mi
-However an unfortunate consequence of this is that you may not be able
-to download Google docs using `rclone mount`. If it doesn't work you
-will get a 0 sized file. If you try again the doc may gain its
-correct size and be downloadable. Whether it will work on not depends
-on the application accessing the mount and the OS you are running -
-experiment to find out if it does work for you!
+#### --hidrive-upload-cutoff
-### Duplicated files
+Cutoff/Threshold for chunked uploads.
-Sometimes, for no reason I've been able to track down, drive will
-duplicate a file that rclone uploads. Drive unlike all the other
-remotes can have duplicated files.
+Any files larger than this will be uploaded in chunks of the configured chunksize.
-Duplicated files cause problems with the syncing and you will see
-messages in the log about duplicates.
+The upper limit for this is 2147483647 bytes (about 2.000Gi).
+That is the maximum amount of bytes a single upload-operation will support.
+Setting this above the upper limit will cause uploads to fail.
-Use `rclone dedupe` to fix duplicated files.
+Properties:
-Note that this isn't just a problem with rclone, even Google Photos on
-Android duplicates files on drive sometimes.
+- Config: upload_cutoff
+- Env Var: RCLONE_HIDRIVE_UPLOAD_CUTOFF
+- Type: SizeSuffix
+- Default: 96Mi
-### Rclone appears to be re-copying files it shouldn't
+#### --hidrive-upload-concurrency
-The most likely cause of this is the duplicated file issue above - run
-`rclone dedupe` and check your logs for duplicate object or directory
-messages.
+Concurrency for chunked uploads.
-This can also be caused by a delay/caching on google drive's end when
-comparing directory listings. Specifically with team drives used in
-combination with --fast-list. Files that were uploaded recently may
-not appear on the directory list sent to rclone when using --fast-list.
+This is the upper limit for how many transfers for the same file are running concurrently.
+Setting this above to a value smaller than 1 will cause uploads to deadlock.
-Waiting a moderate period of time between attempts (estimated to be
-approximately 1 hour) and/or not using --fast-list both seem to be
-effective in preventing the problem.
+If you are uploading small numbers of large files over high-speed links
+and these uploads do not fully utilize your bandwidth, then increasing
+this may help to speed up the transfers.
-## Making your own client_id
+Properties:
-When you use rclone with Google drive in its default configuration you
-are using rclone's client_id. This is shared between all the rclone
-users. There is a global rate limit on the number of queries per
-second that each client_id can do set by Google. rclone already has a
-high quota and I will continue to make sure it is high enough by
-contacting Google.
+- Config: upload_concurrency
+- Env Var: RCLONE_HIDRIVE_UPLOAD_CONCURRENCY
+- Type: int
+- Default: 4
-It is strongly recommended to use your own client ID as the default rclone ID is heavily used. If you have multiple services running, it is recommended to use an API key for each service. The default Google quota is 10 transactions per second so it is recommended to stay under that number as if you use more than that, it will cause rclone to rate limit and make things slower.
+#### --hidrive-encoding
-Here is how to create your own Google Drive client ID for rclone:
+The encoding for the backend.
-1. Log into the [Google API
-Console](https://console.developers.google.com/) with your Google
-account. It doesn't matter what Google account you use. (It need not
-be the same account as the Google Drive you want to access)
+See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info.
-2. Select a project or create a new project.
+Properties:
-3. Under "ENABLE APIS AND SERVICES" search for "Drive", and enable the
-"Google Drive API".
+- Config: encoding
+- Env Var: RCLONE_HIDRIVE_ENCODING
+- Type: Encoding
+- Default: Slash,Dot
-4. Click "Credentials" in the left-side panel (not "Create
-credentials", which opens the wizard).
-5. If you already configured an "Oauth Consent Screen", then skip
-to the next step; if not, click on "CONFIGURE CONSENT SCREEN" button
-(near the top right corner of the right panel), then select "External"
-and click on "CREATE"; on the next screen, enter an "Application name"
-("rclone" is OK); enter "User Support Email" (your own email is OK);
-enter "Developer Contact Email" (your own email is OK); then click on
-"Save" (all other data is optional). You will also have to add some scopes,
-including `.../auth/docs` and `.../auth/drive` in order to be able to edit,
-create and delete files with RClone. You may also want to include the
-`../auth/drive.metadata.readonly` scope. After adding scopes, click
-"Save and continue" to add test users. Be sure to add your own account to
-the test users. Once you've added yourself as a test user and saved the
-changes, click again on "Credentials" on the left panel to go back to
-the "Credentials" screen.
- (PS: if you are a GSuite user, you could also select "Internal" instead
-of "External" above, but this will restrict API use to Google Workspace
-users in your organisation).
+## Limitations
-6. Click on the "+ CREATE CREDENTIALS" button at the top of the screen,
-then select "OAuth client ID".
+### Symbolic links
-7. Choose an application type of "Desktop app" and click "Create". (the default name is fine)
+HiDrive is able to store symbolic links (*symlinks*) by design,
+for example, when unpacked from a zip archive.
-8. It will show you a client ID and client secret. Make a note of these.
-
- (If you selected "External" at Step 5 continue to Step 9.
- If you chose "Internal" you don't need to publish and can skip straight to
- Step 10 but your destination drive must be part of the same Google Workspace.)
+There exists no direct mechanism to manage native symlinks in remotes.
+As such this implementation has chosen to ignore any native symlinks present in the remote.
+rclone will not be able to access or show any symlinks stored in the hidrive-remote.
+This means symlinks cannot be individually removed, copied, or moved,
+except when removing, copying, or moving the parent folder.
-9. Go to "Oauth consent screen" and then click "PUBLISH APP" button and confirm.
- You will also want to add yourself as a test user.
+*This does not affect the `.rclonelink`-files
+that rclone uses to encode and store symbolic links.*
-10. Provide the noted client ID and client secret to rclone.
+### Sparse files
-Be aware that, due to the "enhanced security" recently introduced by
-Google, you are theoretically expected to "submit your app for verification"
-and then wait a few weeks(!) for their response; in practice, you can go right
-ahead and use the client ID and client secret with rclone, the only issue will
-be a very scary confirmation screen shown when you connect via your browser
-for rclone to be able to get its token-id (but as this only happens during
-the remote configuration, it's not such a big deal). Keeping the application in
-"Testing" will work as well, but the limitation is that any grants will expire
-after a week, which can be annoying to refresh constantly. If, for whatever
-reason, a short grant time is not a problem, then keeping the application in
-testing mode would also be sufficient.
+It is possible to store sparse files in HiDrive.
-(Thanks to @balazer on github for these instructions.)
+Note that copying a sparse file will expand the holes
+into null-byte (0x00) regions that will then consume disk space.
+Likewise, when downloading a sparse file,
+the resulting file will have null-byte regions in the place of file holes.
-Sometimes, creation of an OAuth consent in Google API Console fails due to an error message
-“The request failed because changes to one of the field of the resource is not supportedâ€.
-As a convenient workaround, the necessary Google Drive API key can be created on the
-[Python Quickstart](https://developers.google.com/drive/api/v3/quickstart/python) page.
-Just push the Enable the Drive API button to receive the Client ID and Secret.
-Note that it will automatically create a new project in the API Console.
+# HTTP
-# Google Photos
+The HTTP remote is a read only remote for reading files of a
+webserver. The webserver should provide file listings which rclone
+will read and turn into a remote. This has been tested with common
+webservers such as Apache/Nginx/Caddy and will likely work with file
+listings from most web servers. (If it doesn't then please file an
+issue, or send a pull request!)
-The rclone backend for [Google Photos](https://www.google.com/photos/about/) is
-a specialized backend for transferring photos and videos to and from
-Google Photos.
+Paths are specified as `remote:` or `remote:path`.
-**NB** The Google Photos API which rclone uses has quite a few
-limitations, so please read the [limitations section](#limitations)
-carefully to make sure it is suitable for your use.
+The `remote:` represents the configured [url](#http-url), and any path following
+it will be resolved relative to this url, according to the URL standard. This
+means with remote url `https://beta.rclone.org/branch` and path `fix`, the
+resolved URL will be `https://beta.rclone.org/branch/fix`, while with path
+`/fix` the resolved URL will be `https://beta.rclone.org/fix` as the absolute
+path is resolved from the root of the domain.
-## Configuration
+If the path following the `remote:` ends with `/` it will be assumed to point
+to a directory. If the path does not end with `/`, then a HEAD request is sent
+and the response used to decide if it it is treated as a file or a directory
+(run with `-vv` to see details). When [--http-no-head](#http-no-head) is
+specified, a path without ending `/` is always assumed to be a file. If rclone
+incorrectly assumes the path is a file, the solution is to specify the path with
+ending `/`. When you know the path is a directory, ending it with `/` is always
+better as it avoids the initial HEAD request.
-The initial setup for google cloud storage involves getting a token from Google Photos
-which you need to do in your browser. `rclone config` walks you
-through it.
+To just download a single file it is easier to use
+[copyurl](https://rclone.org/commands/rclone_copyurl/).
-Here is an example of how to make a remote called `remote`. First run:
+## Configuration
- rclone config
+Here is an example of how to make a remote called `remote`. First
+run:
-This will guide you through an interactive setup process:
-
-No remotes found, make a new one? n) New remote s) Set configuration password q) Quit config n/s/q> n name> remote Type of storage to configure. Enter a string value. Press Enter for the default (""). Choose a number from below, or type in your own value [snip] XX / Google Photos  "google photos" [snip] Storage> google photos ** See help for google photos backend at: https://rclone.org/googlephotos/ **
-Google Application Client Id Leave blank normally. Enter a string value. Press Enter for the default (""). client_id> Google Application Client Secret Leave blank normally. Enter a string value. Press Enter for the default (""). client_secret> Set to make the Google Photos backend read only.
-If you choose read only then rclone will only request read only access to your photos, otherwise rclone will request full access. Enter a boolean value (true or false). Press Enter for the default ("false"). read_only> Edit advanced config? (y/n) y) Yes n) No y/n> n Remote config Use web browser to automatically authenticate rclone with remote? * Say Y if the machine running rclone has a web browser you can use * Say N if running rclone on a (remote) machine without web browser access If not sure try Y. If Y failed, try N. y) Yes n) No y/n> y If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth Log in and authorize rclone for access Waiting for code... Got code
-*** IMPORTANT: All media items uploaded to Google Photos with rclone *** are stored in full resolution at original quality. These uploads *** will count towards storage in your Google Account.
-
-
-
-
-
-[remote] type = google photos token = {"access_token":"XXX","token_type":"Bearer","refresh_token":"XXX","expiry":"2019-06-28T17:38:04.644930156+01:00"}
-
-
-
-
-y) Yes this is OK e) Edit this remote d) Delete this remote y/e/d> y ```
-
-
-See the remote setup docs for how to set it up on a machine with no Internet browser available.
-
-
-Note that rclone runs a webserver on your local machine to collect the token as returned from Google if using web browser to automatically authenticate. This only runs from the moment it opens your browser to the moment you get back the verification code. This is on http://127.0.0.1:53682/ and this may require you to unblock it temporarily if you are running a host firewall, or use manual mode.
-
-
-This remote is called remote and can now be used like this
-
-
-See all the albums in your photos
-
-
-rclone lsd remote:album
-
-
-Make a new album
-
-
-rclone mkdir remote:album/newAlbum
-
-
-List the contents of an album
-
-
-rclone ls remote:album/newAlbum
-
-
-Sync /home/local/images to the Google Photos, removing any excess files in the album.
-
-
-rclone sync --interactive /home/local/image remote:album/newAlbum
-
-
-### Layout
-
-
-As Google Photos is not a general purpose cloud storage system, the backend is laid out to help you navigate it.
-
-
-The directories under media show different ways of categorizing the media. Each file will appear multiple times. So if you want to make a backup of your google photos you might choose to backup remote:media/by-month. (NB remote:media/by-day is rather slow at the moment so avoid for syncing.)
-
-
-Note that all your photos and videos will appear somewhere under media, but they may not appear under album unless you've put them into albums.
-
-
-/ - upload - file1.jpg - file2.jpg - ... - media - all - file1.jpg - file2.jpg - ... - by-year - 2000 - file1.jpg - ... - 2001 - file2.jpg - ... - ... - by-month - 2000 - 2000-01 - file1.jpg - ... - 2000-02 - file2.jpg - ... - ... - by-day - 2000 - 2000-01-01 - file1.jpg - ... - 2000-01-02 - file2.jpg - ... - ... - album - album name - album name/sub - shared-album - album name - album name/sub - feature - favorites - file1.jpg - file2.jpg
-
-There are two writable parts of the tree, the upload directory and sub directories of the album directory.
-
-
-The upload directory is for uploading files you don't want to put into albums. This will be empty to start with and will contain the files you've uploaded for one rclone session only, becoming empty again when you restart rclone. The use case for this would be if you have a load of files you just want to once off dump into Google Photos. For repeated syncing, uploading to album will work better.
-
-
-Directories within the album directory are also writeable and you may create new directories (albums) under album. If you copy files with a directory hierarchy in there then rclone will create albums with the / character in them. For example if you do
-
-
-rclone copy /path/to/images remote:album/images
-
-
-and the images directory contains
-
-
-images - file1.jpg dir file2.jpg dir2 dir3 file3.jpg
-
-Then rclone will create the following albums with the following files in
-
-
-- images - file1.jpg - images/dir - file2.jpg - images/dir2/dir3 - file3.jpg
-
-
-This means that you can use the album path pretty much like a normal filesystem and it is a good target for repeated syncing.
-
-
-The shared-album directory shows albums shared with you or by you. This is similar to the Sharing tab in the Google Photos web interface.
-
-
-### Standard options
-
-
-Here are the Standard options specific to google photos (Google Photos).
-
-
-#### --gphotos-client-id
-
-
-OAuth Client Id.
-
-
-Leave blank normally.
-
-
-Properties:
-
-
-- Config: client_id - Env Var: RCLONE_GPHOTOS_CLIENT_ID - Type: string - Required: false
-
-
-#### --gphotos-client-secret
-
-
-OAuth Client Secret.
-
-
-Leave blank normally.
-
-
-Properties:
-
-
-- Config: client_secret - Env Var: RCLONE_GPHOTOS_CLIENT_SECRET - Type: string - Required: false
-
-
-#### --gphotos-read-only
-
-
-Set to make the Google Photos backend read only.
-
-
-If you choose read only then rclone will only request read only access to your photos, otherwise rclone will request full access.
-
-
-Properties:
-
-
-- Config: read_only - Env Var: RCLONE_GPHOTOS_READ_ONLY - Type: bool - Default: false
-
-
-### Advanced options
-
-
-Here are the Advanced options specific to google photos (Google Photos).
-
-
-#### --gphotos-token
-
-
-OAuth Access Token as a JSON blob.
-
-
-Properties:
-
-
-- Config: token - Env Var: RCLONE_GPHOTOS_TOKEN - Type: string - Required: false
-
-
-#### --gphotos-auth-url
-
-
-Auth server URL.
-
-
-Leave blank to use the provider defaults.
-
-
-Properties:
-
-
-- Config: auth_url - Env Var: RCLONE_GPHOTOS_AUTH_URL - Type: string - Required: false
-
-
-#### --gphotos-token-url
-
-
-Token server url.
-
-
-Leave blank to use the provider defaults.
-
-
-Properties:
-
-
-- Config: token_url - Env Var: RCLONE_GPHOTOS_TOKEN_URL - Type: string - Required: false
-
-
-#### --gphotos-read-size
-
-
-Set to read the size of media items.
-
-
-Normally rclone does not read the size of media items since this takes another transaction. This isn't necessary for syncing. However rclone mount needs to know the size of files in advance of reading them, so setting this flag when using rclone mount is recommended if you want to read the media.
-
-
-Properties:
-
-
-- Config: read_size - Env Var: RCLONE_GPHOTOS_READ_SIZE - Type: bool - Default: false
-
-
-#### --gphotos-start-year
-
-
-Year limits the photos to be downloaded to those which are uploaded after the given year.
-
-
-Properties:
-
-
-- Config: start_year - Env Var: RCLONE_GPHOTOS_START_YEAR - Type: int - Default: 2000
-
-
-#### --gphotos-include-archived
-
-
-Also view and download archived media.
-
-
-By default, rclone does not request archived media. Thus, when syncing, archived media is not visible in directory listings or transferred.
-
-
-Note that media in albums is always visible and synced, no matter their archive status.
-
-
-With this flag, archived media are always visible in directory listings and transferred.
-
-
-Without this flag, archived media will not be visible in directory listings and won't be transferred.
-
-
-Properties:
-
-
-- Config: include_archived - Env Var: RCLONE_GPHOTOS_INCLUDE_ARCHIVED - Type: bool - Default: false
-
-
-#### --gphotos-encoding
-
-
-The encoding for the backend.
-
-
-See the encoding section in the overview for more info.
-
-
-Properties:
-
-
-- Config: encoding - Env Var: RCLONE_GPHOTOS_ENCODING - Type: MultiEncoder - Default: Slash,CrLf,InvalidUtf8,Dot
-
-
-## Limitations
-
-
-Only images and videos can be uploaded. If you attempt to upload non videos or images or formats that Google Photos doesn't understand, rclone will upload the file, then Google Photos will give an error when it is put turned into a media item.
-
-
-Note that all media items uploaded to Google Photos through the API are stored in full resolution at "original quality" and will count towards your storage quota in your Google Account. The API does not offer a way to upload in "high quality" mode..
-
-
-rclone about is not supported by the Google Photos backend. Backends without this capability cannot determine free space for an rclone mount or use policy mfs (most free space) as a member of an rclone union remote.
-
-See List of backends that do not support rclone about See rclone about
-
-
-### Downloading Images
-
-
-When Images are downloaded this strips EXIF location (according to the docs and my tests). This is a limitation of the Google Photos API and is covered by bug #112096115.
-
-
-The current google API does not allow photos to be downloaded at original resolution. This is very important if you are, for example, relying on "Google Photos" as a backup of your photos. You will not be able to use rclone to redownload original images. You could use 'google takeout' to recover the original photos as a last resort
-
-
-### Downloading Videos
-
-
-When videos are downloaded they are downloaded in a really compressed version of the video compared to downloading it via the Google Photos web interface. This is covered by bug #113672044.
-
-
-### Duplicates
-
-
-If a file name is duplicated in a directory then rclone will add the file ID into its name. So two files called file.jpg would then appear as file {123456}.jpg and file {ABCDEF}.jpg (the actual IDs are a lot longer alas!).
-
-
-If you upload the same image (with the same binary data) twice then Google Photos will deduplicate it. However it will retain the filename from the first upload which may confuse rclone. For example if you uploaded an image to upload then uploaded the same image to album/my_album the filename of the image in album/my_album will be what it was uploaded with initially, not what you uploaded it with to album. In practise this shouldn't cause too many problems.
-
-
-### Modified time
-
-
-The date shown of media in Google Photos is the creation date as determined by the EXIF information, or the upload date if that is not known.
-
-
-This is not changeable by rclone and is not the modification date of the media on local disk. This means that rclone cannot use the dates from Google Photos for syncing purposes.
-
-
-### Size
-
-
-The Google Photos API does not return the size of media. This means that when syncing to Google Photos, rclone can only do a file existence check.
-
-
-It is possible to read the size of the media, but this needs an extra HTTP HEAD request per media item so is very slow and uses up a lot of transactions. This can be enabled with the --gphotos-read-size option or the read_size = true config parameter.
-
-
-If you want to use the backend with rclone mount you may need to enable this flag (depending on your OS and application using the photos) otherwise you may not be able to read media off the mount. You'll need to experiment to see if it works for you without the flag.
-
-
-### Albums
-
-
-Rclone can only upload files to albums it created. This is a limitation of the Google Photos API.
-
-
-Rclone can remove files it uploaded from albums it created only.
-
-
-### Deleting files
-
-
-Rclone can remove files from albums it created, but note that the Google Photos API does not allow media to be deleted permanently so this media will still remain. See bug #109759781.
-
-
-Rclone cannot delete files anywhere except under album.
-
-
-### Deleting albums
-
-
-The Google Photos API does not support deleting albums - see bug #135714733.
-
-
-# Hasher
-
-
-Hasher is a special overlay backend to create remotes which handle checksums for other remotes. It's main functions include: - Emulate hash types unimplemented by backends - Cache checksums to help with slow hashing of large local or (S)FTP files - Warm up checksum cache from external SUM files
-
-
-## Getting started
-
-
-To use Hasher, first set up the underlying remote following the configuration instructions for that remote. You can also use a local pathname instead of a remote. Check that your base remote is working.
-
-
-Let's call the base remote myRemote:path here. Note that anything inside myRemote:path will be handled by hasher and anything outside won't. This means that if you are using a bucket based remote (S3, B2, Swift) then you should put the bucket in the remote s3:bucket.
-
-
-Now proceed to interactive or manual configuration.
-
-
-### Interactive configuration
-
-
-Run rclone config: ``` No remotes found, make a new one? n) New remote s) Set configuration password q) Quit config n/s/q> n name> Hasher1 Type of storage to configure. Choose a number from below, or type in your own value [snip] XX / Handle checksums for other remotes  "hasher" [snip] Storage> hasher Remote to cache checksums for, like myremote:mypath. Enter a string value. Press Enter for the default (""). remote> myRemote:path Comma separated list of supported checksum types. Enter a string value. Press Enter for the default ("md5,sha1"). hashsums> md5 Maximum time to keep checksums in cache. 0 = no cache, off = cache forever. max_age> off Edit advanced config? (y/n) y) Yes n) No y/n> n Remote config
-
-
-
-[Hasher1] type = hasher remote = myRemote:path hashsums = md5 max_age = off -------------------- y) Yes this is OK e) Edit this remote d) Delete this remote y/e/d> y
-
-### Manual configuration
+ rclone config
-Run `rclone config path` to see the path of current active config file,
-usually `YOURHOME/.config/rclone/rclone.conf`.
-Open it in your favorite text editor, find section for the base remote
-and create new section for hasher like in the following examples:
+This will guide you through an interactive setup process:
-[Hasher1] type = hasher remote = myRemote:path hashes = md5 max_age = off
-[Hasher2] type = hasher remote = /local/path hashes = dropbox,sha1 max_age = 24h
+No remotes found, make a new one? n) New remote s) Set configuration password q) Quit config n/s/q> n name> remote Type of storage to configure. Choose a number from below, or type in your own value [snip] XX / HTTP  "http" [snip] Storage> http URL of http host to connect to Choose a number from below, or type in your own value 1 / Connect to example.com  "https://example.com" url> https://beta.rclone.org Remote config -------------------- [remote] url = https://beta.rclone.org -------------------- y) Yes this is OK e) Edit this remote d) Delete this remote y/e/d> y Current remotes:
+Name Type ==== ==== remote http
+
+- Edit existing remote
+- New remote
+- Delete remote
+- Rename remote
+- Copy remote
+- Set configuration password
+- Quit config e/n/d/r/c/s/q> q
+
-Hasher takes basically the following parameters:
-- `remote` is required,
-- `hashes` is a comma separated list of supported checksums
- (by default `md5,sha1`),
-- `max_age` - maximum time to keep a checksum value in the cache,
- `0` will disable caching completely,
- `off` will cache "forever" (that is until the files get changed).
+This remote is called `remote` and can now be used like this
-Make sure the `remote` has `:` (colon) in. If you specify the remote without
-a colon then rclone will use a local directory of that name. So if you use
-a remote of `/local/path` then rclone will handle hashes for that directory.
-If you use `remote = name` literally then rclone will put files
-**in a directory called `name` located under current directory**.
+See all the top level directories
-## Usage
+ rclone lsd remote:
-### Basic operations
+List the contents of a directory
-Now you can use it as `Hasher2:subdir/file` instead of base remote.
-Hasher will transparently update cache with new checksums when a file
-is fully read or overwritten, like:
-rclone copy External:path/file Hasher:dest/path
-rclone cat Hasher:path/to/file > /dev/null
-
-The way to refresh **all** cached checksums (even unsupported by the base backend)
-for a subtree is to **re-download** all files in the subtree. For example,
-use `hashsum --download` using **any** supported hashsum on the command line
-(we just care to re-read):
-rclone hashsum MD5 --download Hasher:path/to/subtree > /dev/null
-rclone backend dump Hasher:path/to/subtree
-
-You can print or drop hashsum cache using custom backend commands:
-rclone backend dump Hasher:dir/subdir
-rclone backend drop Hasher:
-
-### Pre-Seed from a SUM File
+ rclone ls remote:directory
-Hasher supports two backend commands: generic SUM file `import` and faster
-but less consistent `stickyimport`.
-
-rclone backend import Hasher:dir/subdir SHA1 /path/to/SHA1SUM [--checkers 4]
-
-Instead of SHA1 it can be any hash supported by the remote. The last argument
-can point to either a local or an `other-remote:path` text file in SUM format.
-The command will parse the SUM file, then walk down the path given by the
-first argument, snapshot current fingerprints and fill in the cache entries
-correspondingly.
-- Paths in the SUM file are treated as relative to `hasher:dir/subdir`.
-- The command will **not** check that supplied values are correct.
- You **must know** what you are doing.
-- This is a one-time action. The SUM file will not get "attached" to the
- remote. Cache entries can still be overwritten later, should the object's
- fingerprint change.
-- The tree walk can take long depending on the tree size. You can increase
- `--checkers` to make it faster. Or use `stickyimport` if you don't care
- about fingerprints and consistency.
-
-rclone backend stickyimport hasher:path/to/data sha1 remote:/path/to/sum.sha1
-
-`stickyimport` is similar to `import` but works much faster because it
-does not need to stat existing files and skips initial tree walk.
-Instead of binding cache entries to file fingerprints it creates _sticky_
-entries bound to the file name alone ignoring size, modification time etc.
-Such hash entries can be replaced only by `purge`, `delete`, `backend drop`
-or by full re-read/re-write of the files.
+Sync the remote `directory` to `/home/local/directory`, deleting any excess files.
-## Configuration reference
+ rclone sync --interactive remote:directory /home/local/directory
+
+### Read only
+
+This remote is read only - you can't upload files to an HTTP server.
+
+### Modification times
+
+Most HTTP servers store time accurate to 1 second.
+
+### Checksum
+
+No checksums are stored.
+
+### Usage without a config file
+
+Since the http remote only has one config parameter it is easy to use
+without a config file:
+
+ rclone lsd --http-url https://beta.rclone.org :http:
+
+or:
+
+ rclone lsd :http,url='https://beta.rclone.org':
### Standard options
-Here are the Standard options specific to hasher (Better checksums for other remotes).
+Here are the Standard options specific to http (HTTP).
-#### --hasher-remote
+#### --http-url
-Remote to cache checksums for (e.g. myRemote:path).
+URL of HTTP host to connect to.
+
+E.g. "https://example.com", or "https://user:pass@example.com" to use a username and password.
Properties:
-- Config: remote
-- Env Var: RCLONE_HASHER_REMOTE
+- Config: url
+- Env Var: RCLONE_HTTP_URL
- Type: string
- Required: true
-#### --hasher-hashes
+### Advanced options
-Comma separated list of supported checksum types.
+Here are the Advanced options specific to http (HTTP).
-Properties:
+#### --http-headers
-- Config: hashes
-- Env Var: RCLONE_HASHER_HASHES
-- Type: CommaSepList
-- Default: md5,sha1
+Set HTTP headers for all transactions.
-#### --hasher-max-age
+Use this to set additional HTTP headers for all transactions.
-Maximum time to keep checksums in cache (0 = no cache, off = cache forever).
+The input format is comma separated list of key,value pairs. Standard
+[CSV encoding](https://godoc.org/encoding/csv) may be used.
+
+For example, to set a Cookie use 'Cookie,name=value', or '"Cookie","name=value"'.
+
+You can set multiple headers, e.g. '"Cookie","name=value","Authorization","xxx"'.
Properties:
-- Config: max_age
-- Env Var: RCLONE_HASHER_MAX_AGE
-- Type: Duration
-- Default: off
+- Config: headers
+- Env Var: RCLONE_HTTP_HEADERS
+- Type: CommaSepList
+- Default:
-### Advanced options
+#### --http-no-slash
-Here are the Advanced options specific to hasher (Better checksums for other remotes).
+Set this if the site doesn't end directories with /.
-#### --hasher-auto-size
+Use this if your target website does not use / on the end of
+directories.
-Auto-update checksum for files smaller than this size (disabled by default).
+A / on the end of a path is how rclone normally tells the difference
+between files and directories. If this flag is set, then rclone will
+treat all files with Content-Type: text/html as directories and read
+URLs from them rather than downloading them.
+
+Note that this may cause rclone to confuse genuine HTML files with
+directories.
Properties:
-- Config: auto_size
-- Env Var: RCLONE_HASHER_AUTO_SIZE
-- Type: SizeSuffix
-- Default: 0
+- Config: no_slash
+- Env Var: RCLONE_HTTP_NO_SLASH
+- Type: bool
+- Default: false
-### Metadata
+#### --http-no-head
-Any metadata supported by the underlying remote is read and written.
+Don't use HEAD requests.
-See the [metadata](https://rclone.org/docs/#metadata) docs for more info.
+HEAD requests are mainly used to find file sizes in dir listing.
+If your site is being very slow to load then you can try this option.
+Normally rclone does a HEAD request for each potential file in a
+directory listing to:
+
+- find its size
+- check it really exists
+- check to see if it is a directory
+
+If you set this option, rclone will not do the HEAD request. This will mean
+that directory listings are much quicker, but rclone won't have the times or
+sizes of any files, and some files that don't exist may be in the listing.
+
+Properties:
+
+- Config: no_head
+- Env Var: RCLONE_HTTP_NO_HEAD
+- Type: bool
+- Default: false
## Backend commands
-Here are the commands specific to the hasher backend.
+Here are the commands specific to the http backend.
Run them with
@@ -25403,263 +25379,383 @@ Synology C2 Object Storage
These can be run on a running backend using the rc command
[backend/command](https://rclone.org/rc/#backend-command).
-### drop
+### set
-Drop cache
+Set command for updating the config parameters.
- rclone backend drop remote: [options] [<arguments>+]
+ rclone backend set remote: [options] [<arguments>+]
-Completely drop checksum cache.
-Usage Example:
- rclone backend drop hasher:
+This set command can be used to update the config parameters
+for a running http backend.
+Usage Examples:
-### dump
+ rclone backend set remote: [-o opt_name=opt_value] [-o opt_name2=opt_value2]
+ rclone rc backend/command command=set fs=remote: [-o opt_name=opt_value] [-o opt_name2=opt_value2]
+ rclone rc backend/command command=set fs=remote: -o url=https://example.com
-Dump the database
+The option keys are named as they are in the config file.
- rclone backend dump remote: [options] [<arguments>+]
+This rebuilds the connection to the http backend when it is called with
+the new parameters. Only new parameters need be passed as the values
+will default to those currently in use.
-Dump cache records covered by the current remote
+It doesn't return anything.
-### fulldump
-Full dump of the database
- rclone backend fulldump remote: [options] [<arguments>+]
-Dump all cache records in the database
+## Limitations
-### import
+`rclone about` is not supported by the HTTP backend. Backends without
+this capability cannot determine free space for an rclone mount or
+use policy `mfs` (most free space) as a member of an rclone union
+remote.
-Import a SUM file
+See [List of backends that do not support rclone about](https://rclone.org/overview/#optional-features) and [rclone about](https://rclone.org/commands/rclone_about/)
- rclone backend import remote: [options] [<arguments>+]
+# ImageKit
+This is a backend for the [ImageKit.io](https://imagekit.io/) storage service.
-Amend hash cache from a SUM file and bind checksums to files by size/time.
-Usage Example:
- rclone backend import hasher:subdir md5 /path/to/sum.md5
+#### About ImageKit
+[ImageKit.io](https://imagekit.io/) provides real-time image and video optimizations, transformations, and CDN delivery. Over 1,000 businesses and 70,000 developers trust ImageKit with their images and videos on the web.
-### stickyimport
+#### Accounts & Pricing
-Perform fast import of a SUM file
+To use this backend, you need to [create an account](https://imagekit.io/registration/) on ImageKit. Start with a free plan with generous usage limits. Then, as your requirements grow, upgrade to a plan that best fits your needs. See [the pricing details](https://imagekit.io/plans).
- rclone backend stickyimport remote: [options] [<arguments>+]
+## Configuration
-Fill hash cache from a SUM file without verifying file fingerprints.
-Usage Example:
- rclone backend stickyimport hasher:subdir md5 remote:path/to/sum.md5
+Here is an example of making an imagekit configuration.
+Firstly create a [ImageKit.io](https://imagekit.io/) account and choose a plan.
+You will need to log in and get the `publicKey` and `privateKey` for your account from the developer section.
+Now run
+rclone config
+
+This will guide you through an interactive setup process:
+
+No remotes found, make a new one? n) New remote s) Set configuration password q) Quit config n/s/q> n
+Enter the name for the new remote. name> imagekit-media-library
+Option Storage. Type of storage to configure. Choose a number from below, or type in your own value. [snip] XX / ImageKit.io  (imagekit) [snip] Storage> imagekit
+Option endpoint. You can find your ImageKit.io URL endpoint in your dashboard Enter a value. endpoint> https://ik.imagekit.io/imagekit_id
+Option public_key. You can find your ImageKit.io public key in your dashboard Enter a value. public_key> public_****************************
+Option private_key. You can find your ImageKit.io private key in your dashboard Enter a value. private_key> private_****************************
+Edit advanced config? y) Yes n) No (default) y/n> n
+Configuration complete. Options: - type: imagekit - endpoint: https://ik.imagekit.io/imagekit_id - public_key: public_**************************** - private_key: private_****************************
+Keep this "imagekit-media-library" remote? y) Yes this is OK (default) e) Edit this remote d) Delete this remote y/e/d> y
+List directories in the top level of your Media Library
+rclone lsd imagekit-media-library:
+Make a new directory.
+rclone mkdir imagekit-media-library:directory
+List the contents of a directory.
+rclone ls imagekit-media-library:directory
+
+### Modified time and hashes
-## Implementation details (advanced)
+ImageKit does not support modification times or hashes yet.
-This section explains how various rclone operations work on a hasher remote.
+### Checksums
-**Disclaimer. This section describes current implementation which can
-change in future rclone versions!.**
+No checksums are supported.
-### Hashsum command
-The `rclone hashsum` (or `md5sum` or `sha1sum`) command will:
+### Standard options
-1. if requested hash is supported by lower level, just pass it.
-2. if object size is below `auto_size` then download object and calculate
- _requested_ hashes on the fly.
-3. if unsupported and the size is big enough, build object `fingerprint`
- (including size, modtime if supported, first-found _other_ hash if any).
-4. if the strict match is found in cache for the requested remote, return
- the stored hash.
-5. if remote found but fingerprint mismatched, then purge the entry and
- proceed to step 6.
-6. if remote not found or had no requested hash type or after step 5:
- download object, calculate all _supported_ hashes on the fly and store
- in cache; return requested hash.
+Here are the Standard options specific to imagekit (ImageKit.io).
-### Other operations
+#### --imagekit-endpoint
-- whenever a file is uploaded or downloaded **in full**, capture the stream
- to calculate all supported hashes on the fly and update database
-- server-side `move` will update keys of existing cache entries
-- `deletefile` will remove a single cache entry
-- `purge` will remove all cache entries under the purged path
+You can find your ImageKit.io URL endpoint in your [dashboard](https://imagekit.io/dashboard/developer/api-keys)
-Note that setting `max_age = 0` will disable checksum caching completely.
+Properties:
-If you set `max_age = off`, checksums in cache will never age, unless you
-fully rewrite or delete the file.
+- Config: endpoint
+- Env Var: RCLONE_IMAGEKIT_ENDPOINT
+- Type: string
+- Required: true
-### Cache storage
+#### --imagekit-public-key
-Cached checksums are stored as `bolt` database files under rclone cache
-directory, usually `~/.cache/rclone/kv/`. Databases are maintained
-one per _base_ backend, named like `BaseRemote~hasher.bolt`.
-Checksums for multiple `alias`-es into a single base backend
-will be stored in the single database. All local paths are treated as
-aliases into the `local` backend (unless encrypted or chunked) and stored
-in `~/.cache/rclone/kv/local~hasher.bolt`.
-Databases can be shared between multiple rclone processes.
+You can find your ImageKit.io public key in your [dashboard](https://imagekit.io/dashboard/developer/api-keys)
-# HDFS
+Properties:
-[HDFS](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html) is a
-distributed file-system, part of the [Apache Hadoop](https://hadoop.apache.org/) framework.
+- Config: public_key
+- Env Var: RCLONE_IMAGEKIT_PUBLIC_KEY
+- Type: string
+- Required: true
-Paths are specified as `remote:` or `remote:path/to/dir`.
+#### --imagekit-private-key
-## Configuration
+You can find your ImageKit.io private key in your [dashboard](https://imagekit.io/dashboard/developer/api-keys)
-Here is an example of how to make a remote called `remote`. First run:
+Properties:
- rclone config
+- Config: private_key
+- Env Var: RCLONE_IMAGEKIT_PRIVATE_KEY
+- Type: string
+- Required: true
-This will guide you through an interactive setup process:
-
-No remotes found, make a new one? n) New remote s) Set configuration password q) Quit config n/s/q> n name> remote Type of storage to configure. Enter a string value. Press Enter for the default (""). Choose a number from below, or type in your own value [skip] XX / Hadoop distributed file system  "hdfs" [skip] Storage> hdfs ** See help for hdfs backend at: https://rclone.org/hdfs/ **
-hadoop name node and port Enter a string value. Press Enter for the default (""). Choose a number from below, or type in your own value 1 / Connect to host namenode at port 8020  "namenode:8020" namenode> namenode.hadoop:8020 hadoop user name Enter a string value. Press Enter for the default (""). Choose a number from below, or type in your own value 1 / Connect to hdfs as root  "root" username> root Edit advanced config? (y/n) y) Yes n) No (default) y/n> n Remote config -------------------- [remote] type = hdfs namenode = namenode.hadoop:8020 username = root -------------------- y) Yes this is OK (default) e) Edit this remote d) Delete this remote y/e/d> y Current remotes:
-Name Type ==== ==== hadoop hdfs
-
-- Edit existing remote
-- New remote
-- Delete remote
-- Rename remote
-- Copy remote
-- Set configuration password
-- Quit config e/n/d/r/c/s/q> q
-
-
-This remote is called `remote` and can now be used like this
+### Advanced options
-See all the top level directories
+Here are the Advanced options specific to imagekit (ImageKit.io).
- rclone lsd remote:
+#### --imagekit-only-signed
-List the contents of a directory
+If you have configured `Restrict unsigned image URLs` in your dashboard settings, set this to true.
- rclone ls remote:directory
+Properties:
-Sync the remote `directory` to `/home/local/directory`, deleting any excess files.
+- Config: only_signed
+- Env Var: RCLONE_IMAGEKIT_ONLY_SIGNED
+- Type: bool
+- Default: false
- rclone sync --interactive remote:directory /home/local/directory
+#### --imagekit-versions
-### Setting up your own HDFS instance for testing
+Include old versions in directory listings.
-You may start with a [manual setup](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html)
-or use the docker image from the tests:
+Properties:
-If you want to build the docker image
-
-git clone https://github.com/rclone/rclone.git cd rclone/fstest/testserver/images/test-hdfs docker build --rm -t rclone/test-hdfs .
-
-Or you can just use the latest one pushed
-
-docker run --rm --name "rclone-hdfs" -p 127.0.0.1:9866:9866 -p 127.0.0.1:8020:8020 --hostname "rclone-hdfs" rclone/test-hdfs
-
-**NB** it need few seconds to startup.
+- Config: versions
+- Env Var: RCLONE_IMAGEKIT_VERSIONS
+- Type: bool
+- Default: false
-For this docker image the remote needs to be configured like this:
-
-[remote] type = hdfs namenode = 127.0.0.1:8020 username = root
-
-You can stop this image with `docker kill rclone-hdfs` (**NB** it does not use volumes, so all data
-uploaded will be lost.)
+#### --imagekit-upload-tags
-### Modified time
+Tags to add to the uploaded files, e.g. "tag1,tag2".
-Time accurate to 1 second is stored.
+Properties:
-### Checksum
+- Config: upload_tags
+- Env Var: RCLONE_IMAGEKIT_UPLOAD_TAGS
+- Type: string
+- Required: false
-No checksums are implemented.
+#### --imagekit-encoding
-### Usage information
+The encoding for the backend.
-You can use the `rclone about remote:` command which will display filesystem size and current usage.
+See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info.
-### Restricted filename characters
+Properties:
-In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters)
-the following characters are also replaced:
+- Config: encoding
+- Env Var: RCLONE_IMAGEKIT_ENCODING
+- Type: Encoding
+- Default: Slash,LtGt,DoubleQuote,Dollar,Question,Hash,Percent,BackSlash,Del,Ctl,InvalidUtf8,Dot,SquareBracket
-| Character | Value | Replacement |
-| --------- |:-----:|:-----------:|
-| : | 0x3A | ： |
+### Metadata
-Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8).
+Any metadata supported by the underlying remote is read and written.
+
+Here are the possible system metadata items for the imagekit backend.
+
+| Name | Help | Type | Example | Read Only |
+|------|------|------|---------|-----------|
+| aws-tags | AI generated tags by AWS Rekognition associated with the image | string | tag1,tag2 | **Y** |
+| btime | Time of file birth (creation) read from Last-Modified header | RFC 3339 | 2006-01-02T15:04:05.999999999Z07:00 | **Y** |
+| custom-coordinates | Custom coordinates of the file | string | 0,0,100,100 | **Y** |
+| file-type | Type of the file | string | image | **Y** |
+| google-tags | AI generated tags by Google Cloud Vision associated with the image | string | tag1,tag2 | **Y** |
+| has-alpha | Whether the image has alpha channel or not | bool | | **Y** |
+| height | Height of the image or video in pixels | int | | **Y** |
+| is-private-file | Whether the file is private or not | bool | | **Y** |
+| size | Size of the object in bytes | int64 | | **Y** |
+| tags | Tags associated with the file | string | tag1,tag2 | **Y** |
+| width | Width of the image or video in pixels | int | | **Y** |
+
+See the [metadata](https://rclone.org/docs/#metadata) docs for more info.
+
+
+
+# Internet Archive
+
+The Internet Archive backend utilizes Items on [archive.org](https://archive.org/)
+
+Refer to [IAS3 API documentation](https://archive.org/services/docs/api/ias3.html) for the API this backend uses.
+
+Paths are specified as `remote:bucket` (or `remote:` for the `lsd`
+command.) You may put subdirectories in too, e.g. `remote:item/path/to/dir`.
+
+Unlike S3, listing up all items uploaded by you isn't supported.
+Once you have made a remote, you can use it like this:
+
+Make a new item
+
+ rclone mkdir remote:item
+
+List the contents of a item
+
+ rclone ls remote:item
+
+Sync `/home/local/directory` to the remote item, deleting any excess
+files in the item.
+
+ rclone sync --interactive /home/local/directory remote:item
+
+## Notes
+Because of Internet Archive's architecture, it enqueues write operations (and extra post-processings) in a per-item queue. You can check item's queue at https://catalogd.archive.org/history/item-name-here . Because of that, all uploads/deletes will not show up immediately and takes some time to be available.
+The per-item queue is enqueued to an another queue, Item Deriver Queue. [You can check the status of Item Deriver Queue here.](https://catalogd.archive.org/catalog.php?whereami=1) This queue has a limit, and it may block you from uploading, or even deleting. You should avoid uploading a lot of small files for better behavior.
+
+You can optionally wait for the server's processing to finish, by setting non-zero value to `wait_archive` key.
+By making it wait, rclone can do normal file comparison.
+Make sure to set a large enough value (e.g. `30m0s` for smaller files) as it can take a long time depending on server's queue.
+
+## About metadata
+This backend supports setting, updating and reading metadata of each file.
+The metadata will appear as file metadata on Internet Archive.
+However, some fields are reserved by both Internet Archive and rclone.
+
+The following are reserved by Internet Archive:
+- `name`
+- `source`
+- `size`
+- `md5`
+- `crc32`
+- `sha1`
+- `format`
+- `old_version`
+- `viruscheck`
+- `summation`
+
+Trying to set values to these keys is ignored with a warning.
+Only setting `mtime` is an exception. Doing so make it the identical behavior as setting ModTime.
+
+rclone reserves all the keys starting with `rclone-`. Setting value for these keys will give you warnings, but values are set according to request.
+
+If there are multiple values for a key, only the first one is returned.
+This is a limitation of rclone, that supports one value per one key.
+It can be triggered when you did a server-side copy.
+
+Reading metadata will also provide custom (non-standard nor reserved) ones.
+
+## Filtering auto generated files
+
+The Internet Archive automatically creates metadata files after
+upload. These can cause problems when doing an `rclone sync` as rclone
+will try, and fail, to delete them. These metadata files are not
+changeable, as they are created by the Internet Archive automatically.
+
+These auto-created files can be excluded from the sync using [metadata
+filtering](https://rclone.org/filtering/#metadata).
+
+ rclone sync ... --metadata-exclude "source=metadata" --metadata-exclude "format=Metadata"
+
+Which excludes from the sync any files which have the
+`source=metadata` or `format=Metadata` flags which are added to
+Internet Archive auto-created files.
+
+## Configuration
+
+Here is an example of making an internetarchive configuration.
+Most applies to the other providers as well, any differences are described [below](#providers).
+
+First run
+
+ rclone config
+
+This will guide you through an interactive setup process.
+
+No remotes found, make a new one? n) New remote s) Set configuration password q) Quit config n/s/q> n name> remote Option Storage. Type of storage to configure. Choose a number from below, or type in your own value. XX / InternetArchive Items  (internetarchive) Storage> internetarchive Option access_key_id. IAS3 Access Key. Leave blank for anonymous access. You can find one here: https://archive.org/account/s3.php Enter a value. Press Enter to leave empty. access_key_id> XXXX Option secret_access_key. IAS3 Secret Key (password). Leave blank for anonymous access. Enter a value. Press Enter to leave empty. secret_access_key> XXXX Edit advanced config? y) Yes n) No (default) y/n> y Option endpoint. IAS3 Endpoint. Leave blank for default value. Enter a string value. Press Enter for the default (https://s3.us.archive.org). endpoint> Option front_endpoint. Host of InternetArchive Frontend. Leave blank for default value. Enter a string value. Press Enter for the default (https://archive.org). front_endpoint> Option disable_checksum. Don't store MD5 checksum with object metadata. Normally rclone will calculate the MD5 checksum of the input before uploading it so it can ask the server to check the object against checksum. This is great for data integrity checking but can cause long delays for large files to start uploading. Enter a boolean value (true or false). Press Enter for the default (true). disable_checksum> true Option encoding. The encoding for the backend. See the encoding section in the overview for more info. Enter a encoder.MultiEncoder value. Press Enter for the default (Slash,Question,Hash,Percent,Del,Ctl,InvalidUtf8,Dot). encoding> Edit advanced config? y) Yes n) No (default) y/n> n -------------------- [remote] type = internetarchive access_key_id = XXXX secret_access_key = XXXX -------------------- y) Yes this is OK (default) e) Edit this remote d) Delete this remote y/e/d> y
+
### Standard options
-Here are the Standard options specific to hdfs (Hadoop distributed file system).
+Here are the Standard options specific to internetarchive (Internet Archive).
-#### --hdfs-namenode
+#### --internetarchive-access-key-id
-Hadoop name node and port.
+IAS3 Access Key.
-E.g. "namenode:8020" to connect to host namenode at port 8020.
+Leave blank for anonymous access.
+You can find one here: https://archive.org/account/s3.php
Properties:
-- Config: namenode
-- Env Var: RCLONE_HDFS_NAMENODE
+- Config: access_key_id
+- Env Var: RCLONE_INTERNETARCHIVE_ACCESS_KEY_ID
- Type: string
-- Required: true
+- Required: false
-#### --hdfs-username
+#### --internetarchive-secret-access-key
-Hadoop user name.
+IAS3 Secret Key (password).
+
+Leave blank for anonymous access.
Properties:
-- Config: username
-- Env Var: RCLONE_HDFS_USERNAME
+- Config: secret_access_key
+- Env Var: RCLONE_INTERNETARCHIVE_SECRET_ACCESS_KEY
- Type: string
- Required: false
-- Examples:
- - "root"
- - Connect to hdfs as root.
### Advanced options
-Here are the Advanced options specific to hdfs (Hadoop distributed file system).
+Here are the Advanced options specific to internetarchive (Internet Archive).
-#### --hdfs-service-principal-name
+#### --internetarchive-endpoint
-Kerberos service principal name for the namenode.
+IAS3 Endpoint.
-Enables KERBEROS authentication. Specifies the Service Principal Name
-(SERVICE/FQDN) for the namenode. E.g. \"hdfs/namenode.hadoop.docker\"
-for namenode running as service 'hdfs' with FQDN 'namenode.hadoop.docker'.
+Leave blank for default value.
+
+Properties:
+
+- Config: endpoint
+- Env Var: RCLONE_INTERNETARCHIVE_ENDPOINT
+- Type: string
+- Default: "https://s3.us.archive.org"
+
+#### --internetarchive-front-endpoint
+
+Host of InternetArchive Frontend.
+
+Leave blank for default value.
Properties:
-- Config: service_principal_name
-- Env Var: RCLONE_HDFS_SERVICE_PRINCIPAL_NAME
+- Config: front_endpoint
+- Env Var: RCLONE_INTERNETARCHIVE_FRONT_ENDPOINT
- Type: string
-- Required: false
+- Default: "https://archive.org"
-#### --hdfs-data-transfer-protection
+#### --internetarchive-disable-checksum
-Kerberos data transfer protection: authentication|integrity|privacy.
+Don't ask the server to test against MD5 checksum calculated by rclone.
+Normally rclone will calculate the MD5 checksum of the input before
+uploading it so it can ask the server to check the object against checksum.
+This is great for data integrity checking but can cause long delays for
+large files to start uploading.
-Specifies whether or not authentication, data signature integrity
-checks, and wire encryption are required when communicating with
-the datanodes. Possible values are 'authentication', 'integrity'
-and 'privacy'. Used only with KERBEROS enabled.
+Properties:
+
+- Config: disable_checksum
+- Env Var: RCLONE_INTERNETARCHIVE_DISABLE_CHECKSUM
+- Type: bool
+- Default: true
+
+#### --internetarchive-wait-archive
+
+Timeout for waiting the server's processing tasks (specifically archive and book_op) to finish.
+Only enable if you need to be guaranteed to be reflected after write operations.
+0 to disable waiting. No errors to be thrown in case of timeout.
Properties:
-- Config: data_transfer_protection
-- Env Var: RCLONE_HDFS_DATA_TRANSFER_PROTECTION
-- Type: string
-- Required: false
-- Examples:
- - "privacy"
- - Ensure authentication, integrity and encryption enabled.
+- Config: wait_archive
+- Env Var: RCLONE_INTERNETARCHIVE_WAIT_ARCHIVE
+- Type: Duration
+- Default: 0s
-#### --hdfs-encoding
+#### --internetarchive-encoding
The encoding for the backend.
@@ -25668,172 +25764,263 @@ Synology C2 Object Storage
Properties:
- Config: encoding
-- Env Var: RCLONE_HDFS_ENCODING
-- Type: MultiEncoder
-- Default: Slash,Colon,Del,Ctl,InvalidUtf8,Dot
+- Env Var: RCLONE_INTERNETARCHIVE_ENCODING
+- Type: Encoding
+- Default: Slash,LtGt,CrLf,Del,Ctl,InvalidUtf8,Dot
+### Metadata
+Metadata fields provided by Internet Archive.
+If there are multiple values for a key, only the first one is returned.
+This is a limitation of Rclone, that supports one value per one key.
-## Limitations
+Owner is able to add custom keys. Metadata feature grabs all the keys including them.
-- No server-side `Move` or `DirMove`.
-- Checksums not implemented.
+Here are the possible system metadata items for the internetarchive backend.
-# HiDrive
+| Name | Help | Type | Example | Read Only |
+|------|------|------|---------|-----------|
+| crc32 | CRC32 calculated by Internet Archive | string | 01234567 | **Y** |
+| format | Name of format identified by Internet Archive | string | Comma-Separated Values | **Y** |
+| md5 | MD5 hash calculated by Internet Archive | string | 01234567012345670123456701234567 | **Y** |
+| mtime | Time of last modification, managed by Rclone | RFC 3339 | 2006-01-02T15:04:05.999999999Z | **Y** |
+| name | Full file path, without the bucket part | filename | backend/internetarchive/internetarchive.go | **Y** |
+| old_version | Whether the file was replaced and moved by keep-old-version flag | boolean | true | **Y** |
+| rclone-ia-mtime | Time of last modification, managed by Internet Archive | RFC 3339 | 2006-01-02T15:04:05.999999999Z | N |
+| rclone-mtime | Time of last modification, managed by Rclone | RFC 3339 | 2006-01-02T15:04:05.999999999Z | N |
+| rclone-update-track | Random value used by Rclone for tracking changes inside Internet Archive | string | aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa | N |
+| sha1 | SHA1 hash calculated by Internet Archive | string | 0123456701234567012345670123456701234567 | **Y** |
+| size | File size in bytes | decimal number | 123456 | **Y** |
+| source | The source of the file | string | original | **Y** |
+| summation | Check https://forum.rclone.org/t/31922 for how it is used | string | md5 | **Y** |
+| viruscheck | The last time viruscheck process was run for the file (?) | unixtime | 1654191352 | **Y** |
+
+See the [metadata](https://rclone.org/docs/#metadata) docs for more info.
+
+
+
+# Jottacloud
+
+Jottacloud is a cloud storage service provider from a Norwegian company, using its own datacenters
+in Norway. In addition to the official service at [jottacloud.com](https://www.jottacloud.com/),
+it also provides white-label solutions to different companies, such as:
+* Telia
+ * Telia Cloud (cloud.telia.se)
+ * Telia Sky (sky.telia.no)
+* Tele2
+ * Tele2 Cloud (mittcloud.tele2.se)
+* Onlime
+ * Onlime Cloud Storage (onlime.dk)
+* Elkjøp (with subsidiaries):
+ * Elkjøp Cloud (cloud.elkjop.no)
+ * Elgiganten Sweden (cloud.elgiganten.se)
+ * Elgiganten Denmark (cloud.elgiganten.dk)
+ * Giganti Cloud (cloud.gigantti.fi)
+ * ELKO Cloud (cloud.elko.is)
+
+Most of the white-label versions are supported by this backend, although may require different
+authentication setup - described below.
Paths are specified as `remote:path`
Paths may be as deep as required, e.g. `remote:directory/subdirectory`.
-The initial setup for hidrive involves getting a token from HiDrive
-which you need to do in your browser.
-`rclone config` walks you through it.
+## Authentication types
-## Configuration
+Some of the whitelabel versions uses a different authentication method than the official service,
+and you have to choose the correct one when setting up the remote.
-Here is an example of how to make a remote called `remote`. First run:
+### Standard authentication
- rclone config
+The standard authentication method used by the official service (jottacloud.com), as well as
+some of the whitelabel services, requires you to generate a single-use personal login token
+from the account security settings in the service's web interface. Log in to your account,
+go to "Settings" and then "Security", or use the direct link presented to you by rclone when
+configuring the remote: <https://www.jottacloud.com/web/secure>. Scroll down to the section
+"Personal login token", and click the "Generate" button. Note that if you are using a
+whitelabel service you probably can't use the direct link, you need to find the same page in
+their dedicated web interface, and also it may be in a different location than described above.
-This will guide you through an interactive setup process:
-
-No remotes found - make a new one n) New remote s) Set configuration password q) Quit config n/s/q> n name> remote Type of storage to configure. Choose a number from below, or type in your own value [snip] XX / HiDrive  "hidrive" [snip] Storage> hidrive OAuth Client Id - Leave blank normally. client_id> OAuth Client Secret - Leave blank normally. client_secret> Access permissions that rclone should use when requesting access from HiDrive. Leave blank normally. scope_access> Edit advanced config? y/n> n Use web browser to automatically authenticate rclone with remote? * Say Y if the machine running rclone has a web browser you can use * Say N if running rclone on a (remote) machine without web browser access If not sure try Y. If Y failed, try N. y/n> y If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth?state=xxxxxxxxxxxxxxxxxxxxxx Log in and authorize rclone for access Waiting for code... Got code -------------------- [remote] type = hidrive token = {"access_token":"xxxxxxxxxxxxxxxxxxxx","token_type":"Bearer","refresh_token":"xxxxxxxxxxxxxxxxxxxxxxx","expiry":"xxxxxxxxxxxxxxxxxxxxxxx"} -------------------- y) Yes this is OK (default) e) Edit this remote d) Delete this remote y/e/d> y
-
-**You should be aware that OAuth-tokens can be used to access your account
-and hence should not be shared with other persons.**
-See the [below section](#keeping-your-tokens-safe) for more information.
+To access your account from multiple instances of rclone, you need to configure each of them
+with a separate personal login token. E.g. you create a Jottacloud remote with rclone in one
+location, and copy the configuration file to a second location where you also want to run
+rclone and access the same remote. Then you need to replace the token for one of them, using
+the [config reconnect](https://rclone.org/commands/rclone_config_reconnect/) command, which
+requires you to generate a new personal login token and supply as input. If you do not
+do this, the token may easily end up being invalidated, resulting in both instances failing
+with an error message something along the lines of:
-See the [remote setup docs](https://rclone.org/remote_setup/) for how to set it up on a
-machine with no Internet browser available.
+ oauth2: cannot fetch token: 400 Bad Request
+ Response: {"error":"invalid_grant","error_description":"Stale token"}
-Note that rclone runs a webserver on your local machine to collect the
-token as returned from HiDrive. This only runs from the moment it opens
-your browser to the moment you get back the verification code.
-The webserver runs on `http://127.0.0.1:53682/`.
-If local port `53682` is protected by a firewall you may need to temporarily
-unblock the firewall to complete authorization.
+When this happens, you need to replace the token as described above to be able to use your
+remote again.
-Once configured you can then use `rclone` like this,
+All personal login tokens you have taken into use will be listed in the web interface under
+"My logged in devices", and from the right side of that list you can click the "X" button to
+revoke individual tokens.
-List directories in top level of your HiDrive root folder
+### Legacy authentication
- rclone lsd remote:
+If you are using one of the whitelabel versions (e.g. from Elkjøp) you may not have the option
+to generate a CLI token. In this case you'll have to use the legacy authentication. To do this select
+yes when the setup asks for legacy authentication and enter your username and password.
+The rest of the setup is identical to the default setup.
-List all the files in your HiDrive filesystem
+### Telia Cloud authentication
- rclone ls remote:
+Similar to other whitelabel versions Telia Cloud doesn't offer the option of creating a CLI token, and
+additionally uses a separate authentication flow where the username is generated internally. To setup
+rclone to use Telia Cloud, choose Telia Cloud authentication in the setup. The rest of the setup is
+identical to the default setup.
-To copy a local directory to a HiDrive directory called backup
+### Tele2 Cloud authentication
- rclone copy /home/source remote:backup
+As Tele2-Com Hem merger was completed this authentication can be used for former Com Hem Cloud and
+Tele2 Cloud customers as no support for creating a CLI token exists, and additionally uses a separate
+authentication flow where the username is generated internally. To setup rclone to use Tele2 Cloud,
+choose Tele2 Cloud authentication in the setup. The rest of the setup is identical to the default setup.
-### Keeping your tokens safe
+### Onlime Cloud Storage authentication
-Any OAuth-tokens will be stored by rclone in the remote's configuration file as unencrypted text.
-Anyone can use a valid refresh-token to access your HiDrive filesystem without knowing your password.
-Therefore you should make sure no one else can access your configuration.
+Onlime has sold access to Jottacloud proper, while providing localized support to Danish Customers, but
+have recently set up their own hosting, transferring their customers from Jottacloud servers to their
+own ones.
-It is possible to encrypt rclone's configuration file.
-You can find information on securing your configuration file by viewing the [configuration encryption docs](https://rclone.org/docs/#configuration-encryption).
+This, of course, necessitates using their servers for authentication, but otherwise functionality and
+architecture seems equivalent to Jottacloud.
-### Invalid refresh token
+To setup rclone to use Onlime Cloud Storage, choose Onlime Cloud authentication in the setup. The rest
+of the setup is identical to the default setup.
-As can be verified [here](https://developer.hidrive.com/basics-flows/),
-each `refresh_token` (for Native Applications) is valid for 60 days.
-If used to access HiDrivei, its validity will be automatically extended.
+## Configuration
-This means that if you
+Here is an example of how to make a remote called `remote` with the default setup. First run:
- * Don't use the HiDrive remote for 60 days
+ rclone config
-then rclone will return an error which includes a text
-that implies the refresh token is *invalid* or *expired*.
+This will guide you through an interactive setup process:
+
+No remotes found, make a new one? n) New remote s) Set configuration password q) Quit config n/s/q> n name> remote Option Storage. Type of storage to configure. Choose a number from below, or type in your own value. [snip] XX / Jottacloud  (jottacloud) [snip] Storage> jottacloud Edit advanced config? y) Yes n) No (default) y/n> n Option config_type. Select authentication type. Choose a number from below, or type in an existing string value. Press Enter for the default (standard). / Standard authentication. 1 | Use this if you're a normal Jottacloud user.  (standard) / Legacy authentication. 2 | This is only required for certain whitelabel versions of Jottacloud and not recommended for normal users.  (legacy) / Telia Cloud authentication. 3 | Use this if you are using Telia Cloud.  (telia) / Tele2 Cloud authentication. 4 | Use this if you are using Tele2 Cloud.  (tele2) / Onlime Cloud authentication. 5 | Use this if you are using Onlime Cloud.  (onlime) config_type> 1 Personal login token. Generate here: https://www.jottacloud.com/web/secure Login Token> Use a non-standard device/mountpoint? Choosing no, the default, will let you access the storage used for the archive section of the official Jottacloud client. If you instead want to access the sync or the backup section, for example, you must choose yes. y) Yes n) No (default) y/n> y Option config_device. The device to use. In standard setup the built-in Jotta device is used, which contains predefined mountpoints for archive, sync etc. All other devices are treated as backup devices by the official Jottacloud client. You may create a new by entering a unique name. Choose a number from below, or type in your own string value. Press Enter for the default (DESKTOP-3H31129). 1 > DESKTOP-3H31129 2 > Jotta config_device> 2 Option config_mountpoint. The mountpoint to use for the built-in device Jotta. The standard setup is to use the Archive mountpoint. Most other mountpoints have very limited support in rclone and should generally be avoided. Choose a number from below, or type in an existing string value. Press Enter for the default (Archive). 1 > Archive 2 > Shared 3 > Sync config_mountpoint> 1 -------------------- [remote] type = jottacloud configVersion = 1 client_id = jottacli client_secret = tokenURL = https://id.jottacloud.com/auth/realms/jottacloud/protocol/openid-connect/token token = {........} username = 2940e57271a93d987d6f8a21 device = Jotta mountpoint = Archive -------------------- y) Yes this is OK (default) e) Edit this remote d) Delete this remote y/e/d> y
+
+Once configured you can then use `rclone` like this,
-To fix this you will need to authorize rclone to access your HiDrive account again.
+List directories in top level of your Jottacloud
-Using
+ rclone lsd remote:
- rclone config reconnect remote:
+List all the files in your Jottacloud
-the process is very similar to the process of initial setup exemplified before.
+ rclone ls remote:
-### Modified time and hashes
+To copy a local directory to an Jottacloud directory called backup
-HiDrive allows modification times to be set on objects accurate to 1 second.
+ rclone copy /home/source remote:backup
-HiDrive supports [its own hash type](https://static.hidrive.com/dev/0001)
-which is used to verify the integrity of file contents after successful transfers.
+### Devices and Mountpoints
-### Restricted filename characters
+The official Jottacloud client registers a device for each computer you install
+it on, and shows them in the backup section of the user interface. For each
+folder you select for backup it will create a mountpoint within this device.
+A built-in device called Jotta is special, and contains mountpoints Archive,
+Sync and some others, used for corresponding features in official clients.
-HiDrive cannot store files or folders that include
-`/` (0x2F) or null-bytes (0x00) in their name.
-Any other characters can be used in the names of files or folders.
-Additionally, files or folders cannot be named either of the following: `.` or `..`
+With rclone you'll want to use the standard Jotta/Archive device/mountpoint in
+most cases. However, you may for example want to access files from the sync or
+backup functionality provided by the official clients, and rclone therefore
+provides the option to select other devices and mountpoints during config.
-Therefore rclone will automatically replace these characters,
-if files or folders are stored or accessed with such names.
+You are allowed to create new devices and mountpoints. All devices except the
+built-in Jotta device are treated as backup devices by official Jottacloud
+clients, and the mountpoints on them are individual backup sets.
-You can read about how this filename encoding works in general
-[here](overview/#restricted-filenames).
+With the built-in Jotta device, only existing, built-in, mountpoints can be
+selected. In addition to the mentioned Archive and Sync, it may contain
+several other mountpoints such as: Latest, Links, Shared and Trash. All of
+these are special mountpoints with a different internal representation than
+the "regular" mountpoints. Rclone will only to a very limited degree support
+them. Generally you should avoid these, unless you know what you are doing.
-Keep in mind that HiDrive only supports file or folder names
-with a length of 255 characters or less.
+### --fast-list
-### Transfers
+This backend supports `--fast-list` which allows you to use fewer
+transactions in exchange for more memory. See the [rclone
+docs](https://rclone.org/docs/#fast-list) for more details.
-HiDrive limits file sizes per single request to a maximum of 2 GiB.
-To allow storage of larger files and allow for better upload performance,
-the hidrive backend will use a chunked transfer for files larger than 96 MiB.
-Rclone will upload multiple parts/chunks of the file at the same time.
-Chunks in the process of being uploaded are buffered in memory,
-so you may want to restrict this behaviour on systems with limited resources.
+Note that the implementation in Jottacloud always uses only a single
+API request to get the entire list, so for large folders this could
+lead to long wait time before the first results are shown.
-You can customize this behaviour using the following options:
+Note also that with rclone version 1.58 and newer, information about
+[MIME types](https://rclone.org/overview/#mime-type) and metadata item [utime](#metadata)
+are not available when using `--fast-list`.
-* `chunk_size`: size of file parts
-* `upload_cutoff`: files larger or equal to this in size will use a chunked transfer
-* `upload_concurrency`: number of file-parts to upload at the same time
+### Modification times and hashes
-See the below section about configuration options for more details.
+Jottacloud allows modification times to be set on objects accurate to 1
+second. These will be used to detect whether objects need syncing or
+not.
-### Root folder
+Jottacloud supports MD5 type hashes, so you can use the `--checksum`
+flag.
-You can set the root folder for rclone.
-This is the directory that rclone considers to be the root of your HiDrive.
+Note that Jottacloud requires the MD5 hash before upload so if the
+source does not have an MD5 checksum then the file will be cached
+temporarily on disk (in location given by
+[--temp-dir](https://rclone.org/docs/#temp-dir-dir)) before it is uploaded.
+Small files will be cached in memory - see the
+[--jottacloud-md5-memory-limit](#jottacloud-md5-memory-limit) flag.
+When uploading from local disk the source checksum is always available,
+so this does not apply. Starting with rclone version 1.52 the same is
+true for encrypted remotes (in older versions the crypt backend would not
+calculate hashes for uploads from local disk, so the Jottacloud
+backend had to do it as described above).
-Usually, you will leave this blank, and rclone will use the root of the account.
+### Restricted filename characters
-However, you can set this to restrict rclone to a specific folder hierarchy.
+In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters)
+the following characters are also replaced:
-This works by prepending the contents of the `root_prefix` option
-to any paths accessed by rclone.
-For example, the following two ways to access the home directory are equivalent:
+| Character | Value | Replacement |
+| --------- |:-----:|:-----------:|
+| " | 0x22 | ＂ |
+| * | 0x2A | ＊ |
+| : | 0x3A | ： |
+| < | 0x3C | ＜ |
+| > | 0x3E | ＞ |
+| ? | 0x3F | ？ |
+| \| | 0x7C | ｜ |
- rclone lsd --hidrive-root-prefix="/users/test/" remote:path
+Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8),
+as they can't be used in XML strings.
- rclone lsd remote:/users/test/path
+### Deleting files
-See the below section about configuration options for more details.
+By default, rclone will send all files to the trash when deleting files. They will be permanently
+deleted automatically after 30 days. You may bypass the trash and permanently delete files immediately
+by using the [--jottacloud-hard-delete](#jottacloud-hard-delete) flag, or set the equivalent environment variable.
+Emptying the trash is supported by the [cleanup](https://rclone.org/commands/rclone_cleanup/) command.
-### Directory member count
+### Versions
-By default, rclone will know the number of directory members contained in a directory.
-For example, `rclone lsd` uses this information.
+Jottacloud supports file versioning. When rclone uploads a new version of a file it creates a new version of it.
+Currently rclone only supports retrieving the current version but older versions can be accessed via the Jottacloud Website.
-The acquisition of this information will result in additional time costs for HiDrive's API.
-When dealing with large directory structures, it may be desirable to circumvent this time cost,
-especially when this information is not explicitly needed.
-For this, the `disable_fetching_member_count` option can be used.
+Versioning can be disabled by `--jottacloud-no-versions` option. This is achieved by deleting the remote file prior to uploading
+a new version. If the upload the fails no version of the file will be available in the remote.
-See the below section about configuration options for more details.
+### Quota information
+
+To view your current quota you can use the `rclone about remote:`
+command which will display your usage limit (unless it is unlimited)
+and the current usage.
### Standard options
-Here are the Standard options specific to hidrive (HiDrive).
+Here are the Standard options specific to jottacloud (Jottacloud).
-#### --hidrive-client-id
+#### --jottacloud-client-id
OAuth Client Id.
@@ -25842,11 +26029,11 @@ Synology C2 Object Storage
Properties:
- Config: client_id
-- Env Var: RCLONE_HIDRIVE_CLIENT_ID
+- Env Var: RCLONE_JOTTACLOUD_CLIENT_ID
- Type: string
- Required: false
-#### --hidrive-client-secret
+#### --jottacloud-client-secret
OAuth Client Secret.
@@ -25855,42 +26042,26 @@ Synology C2 Object Storage
Properties:
- Config: client_secret
-- Env Var: RCLONE_HIDRIVE_CLIENT_SECRET
+- Env Var: RCLONE_JOTTACLOUD_CLIENT_SECRET
- Type: string
- Required: false
-#### --hidrive-scope-access
-
-Access permissions that rclone should use when requesting access from HiDrive.
-
-Properties:
-
-- Config: scope_access
-- Env Var: RCLONE_HIDRIVE_SCOPE_ACCESS
-- Type: string
-- Default: "rw"
-- Examples:
- - "rw"
- - Read and write access to resources.
- - "ro"
- - Read-only access to resources.
-
### Advanced options
-Here are the Advanced options specific to hidrive (HiDrive).
+Here are the Advanced options specific to jottacloud (Jottacloud).
-#### --hidrive-token
+#### --jottacloud-token
OAuth Access Token as a JSON blob.
Properties:
- Config: token
-- Env Var: RCLONE_HIDRIVE_TOKEN
+- Env Var: RCLONE_JOTTACLOUD_TOKEN
- Type: string
- Required: false
-#### --hidrive-auth-url
+#### --jottacloud-auth-url
Auth server URL.
@@ -25899,11 +26070,11 @@ Synology C2 Object Storage
Properties:
- Config: auth_url
-- Env Var: RCLONE_HIDRIVE_AUTH_URL
+- Env Var: RCLONE_JOTTACLOUD_AUTH_URL
- Type: string
- Required: false
-#### --hidrive-token-url
+#### --jottacloud-token-url
Token server url.
@@ -25912,134 +26083,255 @@ Synology C2 Object Storage
Properties:
- Config: token_url
-- Env Var: RCLONE_HIDRIVE_TOKEN_URL
+- Env Var: RCLONE_JOTTACLOUD_TOKEN_URL
- Type: string
- Required: false
-#### --hidrive-scope-role
+#### --jottacloud-md5-memory-limit
-User-level that rclone should use when requesting access from HiDrive.
+Files bigger than this will be cached on disk to calculate the MD5 if required.
Properties:
-- Config: scope_role
-- Env Var: RCLONE_HIDRIVE_SCOPE_ROLE
-- Type: string
-- Default: "user"
-- Examples:
- - "user"
- - User-level access to management permissions.
- - This will be sufficient in most cases.
- - "admin"
- - Extensive access to management permissions.
- - "owner"
- - Full access to management permissions.
+- Config: md5_memory_limit
+- Env Var: RCLONE_JOTTACLOUD_MD5_MEMORY_LIMIT
+- Type: SizeSuffix
+- Default: 10Mi
+
+#### --jottacloud-trashed-only
+
+Only show files that are in the trash.
+
+This will show trashed files in their original directory structure.
+
+Properties:
+
+- Config: trashed_only
+- Env Var: RCLONE_JOTTACLOUD_TRASHED_ONLY
+- Type: bool
+- Default: false
+
+#### --jottacloud-hard-delete
+
+Delete files permanently rather than putting them into the trash.
+
+Properties:
+
+- Config: hard_delete
+- Env Var: RCLONE_JOTTACLOUD_HARD_DELETE
+- Type: bool
+- Default: false
+
+#### --jottacloud-upload-resume-limit
+
+Files bigger than this can be resumed if the upload fail's.
+
+Properties:
+
+- Config: upload_resume_limit
+- Env Var: RCLONE_JOTTACLOUD_UPLOAD_RESUME_LIMIT
+- Type: SizeSuffix
+- Default: 10Mi
+
+#### --jottacloud-no-versions
+
+Avoid server side versioning by deleting files and recreating files instead of overwriting them.
+
+Properties:
+
+- Config: no_versions
+- Env Var: RCLONE_JOTTACLOUD_NO_VERSIONS
+- Type: bool
+- Default: false
+
+#### --jottacloud-encoding
+
+The encoding for the backend.
+
+See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info.
+
+Properties:
+
+- Config: encoding
+- Env Var: RCLONE_JOTTACLOUD_ENCODING
+- Type: Encoding
+- Default: Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,Del,Ctl,InvalidUtf8,Dot
+
+### Metadata
+
+Jottacloud has limited support for metadata, currently an extended set of timestamps.
+
+Here are the possible system metadata items for the jottacloud backend.
+
+| Name | Help | Type | Example | Read Only |
+|------|------|------|---------|-----------|
+| btime | Time of file birth (creation), read from rclone metadata | RFC 3339 | 2006-01-02T15:04:05.999999999Z07:00 | N |
+| content-type | MIME type, also known as media type | string | text/plain | **Y** |
+| mtime | Time of last modification, read from rclone metadata | RFC 3339 | 2006-01-02T15:04:05.999999999Z07:00 | N |
+| utime | Time of last upload, when current revision was created, generated by backend | RFC 3339 | 2006-01-02T15:04:05.999999999Z07:00 | **Y** |
+
+See the [metadata](https://rclone.org/docs/#metadata) docs for more info.
+
+
+
+## Limitations
+
+Note that Jottacloud is case insensitive so you can't have a file called
+"Hello.doc" and one called "hello.doc".
+
+There are quite a few characters that can't be in Jottacloud file names. Rclone will map these names to and from an identical
+looking unicode equivalent. For example if a file has a ? in it will be mapped to ？ instead.
+
+Jottacloud only supports filenames up to 255 characters in length.
+
+## Troubleshooting
+
+Jottacloud exhibits some inconsistent behaviours regarding deleted files and folders which may cause Copy, Move and DirMove
+operations to previously deleted paths to fail. Emptying the trash should help in such cases.
+
+# Koofr
+
+Paths are specified as `remote:path`
+
+Paths may be as deep as required, e.g. `remote:directory/subdirectory`.
+
+## Configuration
+
+The initial setup for Koofr involves creating an application password for
+rclone. You can do that by opening the Koofr
+[web application](https://app.koofr.net/app/admin/preferences/password),
+giving the password a nice name like `rclone` and clicking on generate.
+
+Here is an example of how to make a remote called `koofr`. First run:
+
+ rclone config
+
+This will guide you through an interactive setup process:
+
+No remotes found, make a new one? n) New remote s) Set configuration password q) Quit config n/s/q> n name> koofr Option Storage. Type of storage to configure. Choose a number from below, or type in your own value. [snip] 22 / Koofr, Digi Storage and other Koofr-compatible storage providers  (koofr) [snip] Storage> koofr Option provider. Choose your storage provider. Choose a number from below, or type in your own value. Press Enter to leave empty. 1 / Koofr, https://app.koofr.net/  (koofr) 2 / Digi Storage, https://storage.rcs-rds.ro/  (digistorage) 3 / Any other Koofr API compatible storage service  (other) provider> 1
+Option user. Your user name. Enter a value. user> USERNAME Option password. Your password for rclone (generate one at https://app.koofr.net/app/admin/preferences/password). Choose an alternative below. y) Yes, type in my own password g) Generate random password y/g> y Enter the password: password: Confirm the password: password: Edit advanced config? y) Yes n) No (default) y/n> n Remote config -------------------- [koofr] type = koofr provider = koofr user = USERNAME password = *** ENCRYPTED *** -------------------- y) Yes this is OK (default) e) Edit this remote d) Delete this remote y/e/d> y
+
+You can choose to edit advanced config in order to enter your own service URL
+if you use an on-premise or white label Koofr instance, or choose an alternative
+mount instead of your primary storage.
+
+Once configured you can then use `rclone` like this,
+
+List directories in top level of your Koofr
+
+ rclone lsd koofr:
+
+List all the files in your Koofr
+
+ rclone ls koofr:
+
+To copy a local directory to an Koofr directory called backup
+
+ rclone copy /home/source koofr:backup
+
+### Restricted filename characters
+
+In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters)
+the following characters are also replaced:
-#### --hidrive-root-prefix
+| Character | Value | Replacement |
+| --------- |:-----:|:-----------:|
+| \ | 0x5C | ï¼¼ |
-The root/parent folder for all paths.
+Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8),
+as they can't be used in XML strings.
-Fill in to use the specified folder as the parent for all paths given to the remote.
-This way rclone can use any folder as its starting point.
+
+### Standard options
+
+Here are the Standard options specific to koofr (Koofr, Digi Storage and other Koofr-compatible storage providers).
+
+#### --koofr-provider
+
+Choose your storage provider.
Properties:
-- Config: root_prefix
-- Env Var: RCLONE_HIDRIVE_ROOT_PREFIX
+- Config: provider
+- Env Var: RCLONE_KOOFR_PROVIDER
- Type: string
-- Default: "/"
+- Required: false
- Examples:
- - "/"
- - The topmost directory accessible by rclone.
- - This will be equivalent with "root" if rclone uses a regular HiDrive user account.
- - "root"
- - The topmost directory of the HiDrive user account
- - ""
- - This specifies that there is no root-prefix for your paths.
- - When using this you will always need to specify paths to this remote with a valid parent e.g. "remote:/path/to/dir" or "remote:root/path/to/dir".
-
-#### --hidrive-endpoint
+ - "koofr"
+ - Koofr, https://app.koofr.net/
+ - "digistorage"
+ - Digi Storage, https://storage.rcs-rds.ro/
+ - "other"
+ - Any other Koofr API compatible storage service
-Endpoint for the service.
+#### --koofr-endpoint
-This is the URL that API-calls will be made to.
+The Koofr API endpoint to use.
Properties:
- Config: endpoint
-- Env Var: RCLONE_HIDRIVE_ENDPOINT
+- Env Var: RCLONE_KOOFR_ENDPOINT
+- Provider: other
- Type: string
-- Default: "https://api.hidrive.strato.com/2.1"
-
-#### --hidrive-disable-fetching-member-count
+- Required: true
-Do not fetch number of objects in directories unless it is absolutely necessary.
+#### --koofr-user
-Requests may be faster if the number of objects in subdirectories is not fetched.
+Your user name.
Properties:
-- Config: disable_fetching_member_count
-- Env Var: RCLONE_HIDRIVE_DISABLE_FETCHING_MEMBER_COUNT
-- Type: bool
-- Default: false
-
-#### --hidrive-chunk-size
-
-Chunksize for chunked uploads.
+- Config: user
+- Env Var: RCLONE_KOOFR_USER
+- Type: string
+- Required: true
-Any files larger than the configured cutoff (or files of unknown size) will be uploaded in chunks of this size.
+#### --koofr-password
-The upper limit for this is 2147483647 bytes (about 2.000Gi).
-That is the maximum amount of bytes a single upload-operation will support.
-Setting this above the upper limit or to a negative value will cause uploads to fail.
+Your password for rclone (generate one at https://app.koofr.net/app/admin/preferences/password).
-Setting this to larger values may increase the upload speed at the cost of using more memory.
-It can be set to smaller values smaller to save on memory.
+**NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/).
Properties:
-- Config: chunk_size
-- Env Var: RCLONE_HIDRIVE_CHUNK_SIZE
-- Type: SizeSuffix
-- Default: 48Mi
+- Config: password
+- Env Var: RCLONE_KOOFR_PASSWORD
+- Provider: koofr
+- Type: string
+- Required: true
-#### --hidrive-upload-cutoff
+### Advanced options
-Cutoff/Threshold for chunked uploads.
+Here are the Advanced options specific to koofr (Koofr, Digi Storage and other Koofr-compatible storage providers).
-Any files larger than this will be uploaded in chunks of the configured chunksize.
+#### --koofr-mountid
-The upper limit for this is 2147483647 bytes (about 2.000Gi).
-That is the maximum amount of bytes a single upload-operation will support.
-Setting this above the upper limit will cause uploads to fail.
+Mount ID of the mount to use.
-Properties:
+If omitted, the primary mount is used.
-- Config: upload_cutoff
-- Env Var: RCLONE_HIDRIVE_UPLOAD_CUTOFF
-- Type: SizeSuffix
-- Default: 96Mi
+Properties:
-#### --hidrive-upload-concurrency
+- Config: mountid
+- Env Var: RCLONE_KOOFR_MOUNTID
+- Type: string
+- Required: false
-Concurrency for chunked uploads.
+#### --koofr-setmtime
-This is the upper limit for how many transfers for the same file are running concurrently.
-Setting this above to a value smaller than 1 will cause uploads to deadlock.
+Does the backend support setting modification time.
-If you are uploading small numbers of large files over high-speed links
-and these uploads do not fully utilize your bandwidth, then increasing
-this may help to speed up the transfers.
+Set this to false if you use a mount ID that points to a Dropbox or Amazon Drive backend.
Properties:
-- Config: upload_concurrency
-- Env Var: RCLONE_HIDRIVE_UPLOAD_CONCURRENCY
-- Type: int
-- Default: 4
+- Config: setmtime
+- Env Var: RCLONE_KOOFR_SETMTIME
+- Type: bool
+- Default: true
-#### --hidrive-encoding
+#### --koofr-encoding
The encoding for the backend.
@@ -26048,1996 +26340,1874 @@ Synology C2 Object Storage
Properties:
- Config: encoding
-- Env Var: RCLONE_HIDRIVE_ENCODING
-- Type: MultiEncoder
-- Default: Slash,Dot
+- Env Var: RCLONE_KOOFR_ENCODING
+- Type: Encoding
+- Default: Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot
## Limitations
-### Symbolic links
-
-HiDrive is able to store symbolic links (*symlinks*) by design,
-for example, when unpacked from a zip archive.
-
-There exists no direct mechanism to manage native symlinks in remotes.
-As such this implementation has chosen to ignore any native symlinks present in the remote.
-rclone will not be able to access or show any symlinks stored in the hidrive-remote.
-This means symlinks cannot be individually removed, copied, or moved,
-except when removing, copying, or moving the parent folder.
-
-*This does not affect the `.rclonelink`-files
-that rclone uses to encode and store symbolic links.*
-
-### Sparse files
-
-It is possible to store sparse files in HiDrive.
-
-Note that copying a sparse file will expand the holes
-into null-byte (0x00) regions that will then consume disk space.
-Likewise, when downloading a sparse file,
-the resulting file will have null-byte regions in the place of file holes.
-
-# HTTP
-
-The HTTP remote is a read only remote for reading files of a
-webserver. The webserver should provide file listings which rclone
-will read and turn into a remote. This has been tested with common
-webservers such as Apache/Nginx/Caddy and will likely work with file
-listings from most web servers. (If it doesn't then please file an
-issue, or send a pull request!)
+Note that Koofr is case insensitive so you can't have a file called
+"Hello.doc" and one called "hello.doc".
-Paths are specified as `remote:` or `remote:path`.
+## Providers
-The `remote:` represents the configured [url](#http-url), and any path following
-it will be resolved relative to this url, according to the URL standard. This
-means with remote url `https://beta.rclone.org/branch` and path `fix`, the
-resolved URL will be `https://beta.rclone.org/branch/fix`, while with path
-`/fix` the resolved URL will be `https://beta.rclone.org/fix` as the absolute
-path is resolved from the root of the domain.
+### Koofr
-If the path following the `remote:` ends with `/` it will be assumed to point
-to a directory. If the path does not end with `/`, then a HEAD request is sent
-and the response used to decide if it it is treated as a file or a directory
-(run with `-vv` to see details). When [--http-no-head](#http-no-head) is
-specified, a path without ending `/` is always assumed to be a file. If rclone
-incorrectly assumes the path is a file, the solution is to specify the path with
-ending `/`. When you know the path is a directory, ending it with `/` is always
-better as it avoids the initial HEAD request.
+This is the original [Koofr](https://koofr.eu) storage provider used as main example and described in the [configuration](#configuration) section above.
-To just download a single file it is easier to use
-[copyurl](https://rclone.org/commands/rclone_copyurl/).
+### Digi Storage
-## Configuration
+[Digi Storage](https://www.digi.ro/servicii/online/digi-storage) is a cloud storage service run by [Digi.ro](https://www.digi.ro/) that
+provides a Koofr API.
-Here is an example of how to make a remote called `remote`. First
-run:
+Here is an example of how to make a remote called `ds`. First run:
rclone config
This will guide you through an interactive setup process:
-No remotes found, make a new one? n) New remote s) Set configuration password q) Quit config n/s/q> n name> remote Type of storage to configure. Choose a number from below, or type in your own value [snip] XX / HTTP  "http" [snip] Storage> http URL of http host to connect to Choose a number from below, or type in your own value 1 / Connect to example.com  "https://example.com" url> https://beta.rclone.org Remote config -------------------- [remote] url = https://beta.rclone.org -------------------- y) Yes this is OK e) Edit this remote d) Delete this remote y/e/d> y Current remotes:
-Name Type ==== ==== remote http
-
-- Edit existing remote
-- New remote
-- Delete remote
-- Rename remote
-- Copy remote
-- Set configuration password
-- Quit config e/n/d/r/c/s/q> q
-
+No remotes found, make a new one? n) New remote s) Set configuration password q) Quit config n/s/q> n name> ds Option Storage. Type of storage to configure. Choose a number from below, or type in your own value. [snip] 22 / Koofr, Digi Storage and other Koofr-compatible storage providers  (koofr) [snip] Storage> koofr Option provider. Choose your storage provider. Choose a number from below, or type in your own value. Press Enter to leave empty. 1 / Koofr, https://app.koofr.net/  (koofr) 2 / Digi Storage, https://storage.rcs-rds.ro/  (digistorage) 3 / Any other Koofr API compatible storage service  (other) provider> 2 Option user. Your user name. Enter a value. user> USERNAME Option password. Your password for rclone (generate one at https://storage.rcs-rds.ro/app/admin/preferences/password). Choose an alternative below. y) Yes, type in my own password g) Generate random password y/g> y Enter the password: password: Confirm the password: password: Edit advanced config? y) Yes n) No (default) y/n> n -------------------- [ds] type = koofr provider = digistorage user = USERNAME password = *** ENCRYPTED *** -------------------- y) Yes this is OK (default) e) Edit this remote d) Delete this remote y/e/d> y
-This remote is called `remote` and can now be used like this
-
-See all the top level directories
-
- rclone lsd remote:
-
-List the contents of a directory
-
- rclone ls remote:directory
-
-Sync the remote `directory` to `/home/local/directory`, deleting any excess files.
-
- rclone sync --interactive remote:directory /home/local/directory
-
-### Read only
-
-This remote is read only - you can't upload files to an HTTP server.
+### Other
-### Modified time
+You may also want to use another, public or private storage provider that runs a Koofr API compatible service, by simply providing the base URL to connect to.
-Most HTTP servers store time accurate to 1 second.
+Here is an example of how to make a remote called `other`. First run:
-### Checksum
+ rclone config
-No checksums are stored.
+This will guide you through an interactive setup process:
+
+No remotes found, make a new one? n) New remote s) Set configuration password q) Quit config n/s/q> n name> other Option Storage. Type of storage to configure. Choose a number from below, or type in your own value. [snip] 22 / Koofr, Digi Storage and other Koofr-compatible storage providers  (koofr) [snip] Storage> koofr Option provider. Choose your storage provider. Choose a number from below, or type in your own value. Press Enter to leave empty. 1 / Koofr, https://app.koofr.net/  (koofr) 2 / Digi Storage, https://storage.rcs-rds.ro/  (digistorage) 3 / Any other Koofr API compatible storage service  (other) provider> 3 Option endpoint. The Koofr API endpoint to use. Enter a value. endpoint> https://koofr.other.org Option user. Your user name. Enter a value. user> USERNAME Option password. Your password for rclone (generate one at your service's settings page). Choose an alternative below. y) Yes, type in my own password g) Generate random password y/g> y Enter the password: password: Confirm the password: password: Edit advanced config? y) Yes n) No (default) y/n> n -------------------- [other] type = koofr provider = other endpoint = https://koofr.other.org user = USERNAME password = *** ENCRYPTED *** -------------------- y) Yes this is OK (default) e) Edit this remote d) Delete this remote y/e/d> y
+
+# Linkbox
-### Usage without a config file
+Linkbox is [a private cloud drive](https://linkbox.to/).
-Since the http remote only has one config parameter it is easy to use
-without a config file:
+## Configuration
- rclone lsd --http-url https://beta.rclone.org :http:
+Here is an example of making a remote for Linkbox.
-or:
+First run:
- rclone lsd :http,url='https://beta.rclone.org':
+ rclone config
+This will guide you through an interactive setup process:
+
+No remotes found, make a new one? n) New remote s) Set configuration password q) Quit config n/s/q> n
+Enter name for new remote. name> remote
+Option Storage. Type of storage to configure. Choose a number from below, or type in your own value. XX / Linkbox  (linkbox) Storage> XX
+Option token. Token from https://www.linkbox.to/admin/account Enter a value. token> testFromCLToken
+Configuration complete. Options: - type: linkbox - token: XXXXXXXXXXX Keep this "linkbox" remote? y) Yes this is OK (default) e) Edit this remote d) Delete this remote y/e/d> y
+
### Standard options
-Here are the Standard options specific to http (HTTP).
-
-#### --http-url
+Here are the Standard options specific to linkbox (Linkbox).
-URL of HTTP host to connect to.
+#### --linkbox-token
-E.g. "https://example.com", or "https://user:pass@example.com" to use a username and password.
+Token from https://www.linkbox.to/admin/account
Properties:
-- Config: url
-- Env Var: RCLONE_HTTP_URL
+- Config: token
+- Env Var: RCLONE_LINKBOX_TOKEN
- Type: string
- Required: true
-### Advanced options
-Here are the Advanced options specific to http (HTTP).
-#### --http-headers
+## Limitations
-Set HTTP headers for all transactions.
+Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8),
+as they can't be used in JSON strings.
-Use this to set additional HTTP headers for all transactions.
+# Mail.ru Cloud
-The input format is comma separated list of key,value pairs. Standard
-[CSV encoding](https://godoc.org/encoding/csv) may be used.
+[Mail.ru Cloud](https://cloud.mail.ru/) is a cloud storage provided by a Russian internet company [Mail.Ru Group](https://mail.ru). The official desktop client is [Disk-O:](https://disk-o.cloud/en), available on Windows and Mac OS.
-For example, to set a Cookie use 'Cookie,name=value', or '"Cookie","name=value"'.
+## Features highlights
-You can set multiple headers, e.g. '"Cookie","name=value","Authorization","xxx"'.
+- Paths may be as deep as required, e.g. `remote:directory/subdirectory`
+- Files have a `last modified time` property, directories don't
+- Deleted files are by default moved to the trash
+- Files and directories can be shared via public links
+- Partial uploads or streaming are not supported, file size must be known before upload
+- Maximum file size is limited to 2G for a free account, unlimited for paid accounts
+- Storage keeps hash for all files and performs transparent deduplication,
+ the hash algorithm is a modified SHA1
+- If a particular file is already present in storage, one can quickly submit file hash
+ instead of long file upload (this optimization is supported by rclone)
-Properties:
+## Configuration
-- Config: headers
-- Env Var: RCLONE_HTTP_HEADERS
-- Type: CommaSepList
-- Default:
+Here is an example of making a mailru configuration.
-#### --http-no-slash
+First create a Mail.ru Cloud account and choose a tariff.
-Set this if the site doesn't end directories with /.
+You will need to log in and create an app password for rclone. Rclone
+**will not work** with your normal username and password - it will
+give an error like `oauth2: server response missing access_token`.
-Use this if your target website does not use / on the end of
-directories.
+- Click on your user icon in the top right
+- Go to Security / "Пароль и безопаÑноÑÑ‚ÑŒ"
+- Click password for apps / "Пароли Ð´Ð»Ñ Ð²Ð½ÐµÑˆÐ½Ð¸Ñ… приложений"
+- Add the password - give it a name - eg "rclone"
+- Copy the password and use this password below - your normal login password won't work.
-A / on the end of a path is how rclone normally tells the difference
-between files and directories. If this flag is set, then rclone will
-treat all files with Content-Type: text/html as directories and read
-URLs from them rather than downloading them.
+Now run
-Note that this may cause rclone to confuse genuine HTML files with
-directories.
+ rclone config
-Properties:
+This will guide you through an interactive setup process:
+
+No remotes found, make a new one? n) New remote s) Set configuration password q) Quit config n/s/q> n name> remote Type of storage to configure. Type of storage to configure. Enter a string value. Press Enter for the default (""). Choose a number from below, or type in your own value [snip] XX / Mail.ru Cloud  "mailru" [snip] Storage> mailru User name (usually email) Enter a string value. Press Enter for the default (""). user> username@mail.ru Password
+This must be an app password - rclone will not work with your normal password. See the Configuration section in the docs for how to make an app password. y) Yes type in my own password g) Generate random password y/g> y Enter the password: password: Confirm the password: password: Skip full upload if there is another file with same data hash. This feature is called "speedup" or "put by hash". It is especially efficient in case of generally available files like popular books, video or audio clips [snip] Enter a boolean value (true or false). Press Enter for the default ("true"). Choose a number from below, or type in your own value 1 / Enable  "true" 2 / Disable  "false" speedup_enable> 1 Edit advanced config? (y/n) y) Yes n) No y/n> n Remote config -------------------- [remote] type = mailru user = username@mail.ru pass = *** ENCRYPTED *** speedup_enable = true -------------------- y) Yes this is OK e) Edit this remote d) Delete this remote y/e/d> y
+
+Configuration of this backend does not require a local web browser.
+You can use the configured backend as shown below:
-- Config: no_slash
-- Env Var: RCLONE_HTTP_NO_SLASH
-- Type: bool
-- Default: false
+See top level directories
-#### --http-no-head
+ rclone lsd remote:
-Don't use HEAD requests.
+Make a new directory
-HEAD requests are mainly used to find file sizes in dir listing.
-If your site is being very slow to load then you can try this option.
-Normally rclone does a HEAD request for each potential file in a
-directory listing to:
+ rclone mkdir remote:directory
-- find its size
-- check it really exists
-- check to see if it is a directory
+List the contents of a directory
-If you set this option, rclone will not do the HEAD request. This will mean
-that directory listings are much quicker, but rclone won't have the times or
-sizes of any files, and some files that don't exist may be in the listing.
+ rclone ls remote:directory
-Properties:
+Sync `/home/local/directory` to the remote path, deleting any
+excess files in the path.
-- Config: no_head
-- Env Var: RCLONE_HTTP_NO_HEAD
-- Type: bool
-- Default: false
+ rclone sync --interactive /home/local/directory remote:directory
+### Modification times and hashes
+Files support a modification time attribute with up to 1 second precision.
+Directories do not have a modification time, which is shown as "Jan 1 1970".
-## Limitations
+File hashes are supported, with a custom Mail.ru algorithm based on SHA1.
+If file size is less than or equal to the SHA1 block size (20 bytes),
+its hash is simply its data right-padded with zero bytes.
+Hashes of a larger file is computed as a SHA1 of the file data
+bytes concatenated with a decimal representation of the data length.
-`rclone about` is not supported by the HTTP backend. Backends without
-this capability cannot determine free space for an rclone mount or
-use policy `mfs` (most free space) as a member of an rclone union
-remote.
+### Emptying Trash
-See [List of backends that do not support rclone about](https://rclone.org/overview/#optional-features) and [rclone about](https://rclone.org/commands/rclone_about/)
+Removing a file or directory actually moves it to the trash, which is not
+visible to rclone but can be seen in a web browser. The trashed file
+still occupies part of total quota. If you wish to empty your trash
+and free some quota, you can use the `rclone cleanup remote:` command,
+which will permanently delete all your trashed files.
+This command does not take any path arguments.
-# Internet Archive
+### Quota information
-The Internet Archive backend utilizes Items on [archive.org](https://archive.org/)
+To view your current quota you can use the `rclone about remote:`
+command which will display your usage limit (quota) and the current usage.
-Refer to [IAS3 API documentation](https://archive.org/services/docs/api/ias3.html) for the API this backend uses.
+### Restricted filename characters
-Paths are specified as `remote:bucket` (or `remote:` for the `lsd`
-command.) You may put subdirectories in too, e.g. `remote:item/path/to/dir`.
+In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters)
+the following characters are also replaced:
-Unlike S3, listing up all items uploaded by you isn't supported.
+| Character | Value | Replacement |
+| --------- |:-----:|:-----------:|
+| " | 0x22 | ＂ |
+| * | 0x2A | ＊ |
+| : | 0x3A | ： |
+| < | 0x3C | ＜ |
+| > | 0x3E | ＞ |
+| ? | 0x3F | ？ |
+| \ | 0x5C | ï¼¼ |
+| \| | 0x7C | ｜ |
-Once you have made a remote, you can use it like this:
+Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8),
+as they can't be used in JSON strings.
-Make a new item
- rclone mkdir remote:item
+### Standard options
-List the contents of a item
+Here are the Standard options specific to mailru (Mail.ru Cloud).
- rclone ls remote:item
+#### --mailru-client-id
-Sync `/home/local/directory` to the remote item, deleting any excess
-files in the item.
+OAuth Client Id.
- rclone sync --interactive /home/local/directory remote:item
+Leave blank normally.
-## Notes
-Because of Internet Archive's architecture, it enqueues write operations (and extra post-processings) in a per-item queue. You can check item's queue at https://catalogd.archive.org/history/item-name-here . Because of that, all uploads/deletes will not show up immediately and takes some time to be available.
-The per-item queue is enqueued to an another queue, Item Deriver Queue. [You can check the status of Item Deriver Queue here.](https://catalogd.archive.org/catalog.php?whereami=1) This queue has a limit, and it may block you from uploading, or even deleting. You should avoid uploading a lot of small files for better behavior.
+Properties:
-You can optionally wait for the server's processing to finish, by setting non-zero value to `wait_archive` key.
-By making it wait, rclone can do normal file comparison.
-Make sure to set a large enough value (e.g. `30m0s` for smaller files) as it can take a long time depending on server's queue.
+- Config: client_id
+- Env Var: RCLONE_MAILRU_CLIENT_ID
+- Type: string
+- Required: false
-## About metadata
-This backend supports setting, updating and reading metadata of each file.
-The metadata will appear as file metadata on Internet Archive.
-However, some fields are reserved by both Internet Archive and rclone.
+#### --mailru-client-secret
-The following are reserved by Internet Archive:
-- `name`
-- `source`
-- `size`
-- `md5`
-- `crc32`
-- `sha1`
-- `format`
-- `old_version`
-- `viruscheck`
-- `summation`
+OAuth Client Secret.
-Trying to set values to these keys is ignored with a warning.
-Only setting `mtime` is an exception. Doing so make it the identical behavior as setting ModTime.
+Leave blank normally.
-rclone reserves all the keys starting with `rclone-`. Setting value for these keys will give you warnings, but values are set according to request.
+Properties:
-If there are multiple values for a key, only the first one is returned.
-This is a limitation of rclone, that supports one value per one key.
-It can be triggered when you did a server-side copy.
+- Config: client_secret
+- Env Var: RCLONE_MAILRU_CLIENT_SECRET
+- Type: string
+- Required: false
-Reading metadata will also provide custom (non-standard nor reserved) ones.
+#### --mailru-user
-## Filtering auto generated files
+User name (usually email).
-The Internet Archive automatically creates metadata files after
-upload. These can cause problems when doing an `rclone sync` as rclone
-will try, and fail, to delete them. These metadata files are not
-changeable, as they are created by the Internet Archive automatically.
+Properties:
-These auto-created files can be excluded from the sync using [metadata
-filtering](https://rclone.org/filtering/#metadata).
+- Config: user
+- Env Var: RCLONE_MAILRU_USER
+- Type: string
+- Required: true
- rclone sync ... --metadata-exclude "source=metadata" --metadata-exclude "format=Metadata"
+#### --mailru-pass
-Which excludes from the sync any files which have the
-`source=metadata` or `format=Metadata` flags which are added to
-Internet Archive auto-created files.
+Password.
-## Configuration
+This must be an app password - rclone will not work with your normal
+password. See the Configuration section in the docs for how to make an
+app password.
-Here is an example of making an internetarchive configuration.
-Most applies to the other providers as well, any differences are described [below](#providers).
-First run
+**NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/).
- rclone config
+Properties:
-This will guide you through an interactive setup process.
-
-No remotes found, make a new one? n) New remote s) Set configuration password q) Quit config n/s/q> n name> remote Option Storage. Type of storage to configure. Choose a number from below, or type in your own value. XX / InternetArchive Items  (internetarchive) Storage> internetarchive Option access_key_id. IAS3 Access Key. Leave blank for anonymous access. You can find one here: https://archive.org/account/s3.php Enter a value. Press Enter to leave empty. access_key_id> XXXX Option secret_access_key. IAS3 Secret Key (password). Leave blank for anonymous access. Enter a value. Press Enter to leave empty. secret_access_key> XXXX Edit advanced config? y) Yes n) No (default) y/n> y Option endpoint. IAS3 Endpoint. Leave blank for default value. Enter a string value. Press Enter for the default (https://s3.us.archive.org). endpoint> Option front_endpoint. Host of InternetArchive Frontend. Leave blank for default value. Enter a string value. Press Enter for the default (https://archive.org). front_endpoint> Option disable_checksum. Don't store MD5 checksum with object metadata. Normally rclone will calculate the MD5 checksum of the input before uploading it so it can ask the server to check the object against checksum. This is great for data integrity checking but can cause long delays for large files to start uploading. Enter a boolean value (true or false). Press Enter for the default (true). disable_checksum> true Option encoding. The encoding for the backend. See the encoding section in the overview for more info. Enter a encoder.MultiEncoder value. Press Enter for the default (Slash,Question,Hash,Percent,Del,Ctl,InvalidUtf8,Dot). encoding> Edit advanced config? y) Yes n) No (default) y/n> n -------------------- [remote] type = internetarchive access_key_id = XXXX secret_access_key = XXXX -------------------- y) Yes this is OK (default) e) Edit this remote d) Delete this remote y/e/d> y
-
+- Config: pass
+- Env Var: RCLONE_MAILRU_PASS
+- Type: string
+- Required: true
-### Standard options
+#### --mailru-speedup-enable
-Here are the Standard options specific to internetarchive (Internet Archive).
+Skip full upload if there is another file with same data hash.
-#### --internetarchive-access-key-id
+This feature is called "speedup" or "put by hash". It is especially efficient
+in case of generally available files like popular books, video or audio clips,
+because files are searched by hash in all accounts of all mailru users.
+It is meaningless and ineffective if source file is unique or encrypted.
+Please note that rclone may need local memory and disk space to calculate
+content hash in advance and decide whether full upload is required.
+Also, if rclone does not know file size in advance (e.g. in case of
+streaming or partial uploads), it will not even try this optimization.
-IAS3 Access Key.
+Properties:
-Leave blank for anonymous access.
-You can find one here: https://archive.org/account/s3.php
+- Config: speedup_enable
+- Env Var: RCLONE_MAILRU_SPEEDUP_ENABLE
+- Type: bool
+- Default: true
+- Examples:
+ - "true"
+ - Enable
+ - "false"
+ - Disable
+
+### Advanced options
+
+Here are the Advanced options specific to mailru (Mail.ru Cloud).
+
+#### --mailru-token
+
+OAuth Access Token as a JSON blob.
Properties:
-- Config: access_key_id
-- Env Var: RCLONE_INTERNETARCHIVE_ACCESS_KEY_ID
+- Config: token
+- Env Var: RCLONE_MAILRU_TOKEN
- Type: string
- Required: false
-#### --internetarchive-secret-access-key
+#### --mailru-auth-url
-IAS3 Secret Key (password).
+Auth server URL.
-Leave blank for anonymous access.
+Leave blank to use the provider defaults.
Properties:
-- Config: secret_access_key
-- Env Var: RCLONE_INTERNETARCHIVE_SECRET_ACCESS_KEY
+- Config: auth_url
+- Env Var: RCLONE_MAILRU_AUTH_URL
- Type: string
- Required: false
-### Advanced options
-
-Here are the Advanced options specific to internetarchive (Internet Archive).
-
-#### --internetarchive-endpoint
+#### --mailru-token-url
-IAS3 Endpoint.
+Token server url.
-Leave blank for default value.
+Leave blank to use the provider defaults.
Properties:
-- Config: endpoint
-- Env Var: RCLONE_INTERNETARCHIVE_ENDPOINT
+- Config: token_url
+- Env Var: RCLONE_MAILRU_TOKEN_URL
- Type: string
-- Default: "https://s3.us.archive.org"
+- Required: false
-#### --internetarchive-front-endpoint
+#### --mailru-speedup-file-patterns
-Host of InternetArchive Frontend.
+Comma separated list of file name patterns eligible for speedup (put by hash).
-Leave blank for default value.
+Patterns are case insensitive and can contain '*' or '?' meta characters.
Properties:
-- Config: front_endpoint
-- Env Var: RCLONE_INTERNETARCHIVE_FRONT_ENDPOINT
+- Config: speedup_file_patterns
+- Env Var: RCLONE_MAILRU_SPEEDUP_FILE_PATTERNS
- Type: string
-- Default: "https://archive.org"
-
-#### --internetarchive-disable-checksum
-
-Don't ask the server to test against MD5 checksum calculated by rclone.
-Normally rclone will calculate the MD5 checksum of the input before
-uploading it so it can ask the server to check the object against checksum.
-This is great for data integrity checking but can cause long delays for
-large files to start uploading.
-
-Properties:
+- Default: "*.mkv,*.avi,*.mp4,*.mp3,*.zip,*.gz,*.rar,*.pdf"
+- Examples:
+ - ""
+ - Empty list completely disables speedup (put by hash).
+ - "*"
+ - All files will be attempted for speedup.
+ - "*.mkv,*.avi,*.mp4,*.mp3"
+ - Only common audio/video files will be tried for put by hash.
+ - "*.zip,*.gz,*.rar,*.pdf"
+ - Only common archives or PDF books will be tried for speedup.
-- Config: disable_checksum
-- Env Var: RCLONE_INTERNETARCHIVE_DISABLE_CHECKSUM
-- Type: bool
-- Default: true
+#### --mailru-speedup-max-disk
-#### --internetarchive-wait-archive
+This option allows you to disable speedup (put by hash) for large files.
-Timeout for waiting the server's processing tasks (specifically archive and book_op) to finish.
-Only enable if you need to be guaranteed to be reflected after write operations.
-0 to disable waiting. No errors to be thrown in case of timeout.
+Reason is that preliminary hashing can exhaust your RAM or disk space.
Properties:
-- Config: wait_archive
-- Env Var: RCLONE_INTERNETARCHIVE_WAIT_ARCHIVE
-- Type: Duration
-- Default: 0s
-
-#### --internetarchive-encoding
+- Config: speedup_max_disk
+- Env Var: RCLONE_MAILRU_SPEEDUP_MAX_DISK
+- Type: SizeSuffix
+- Default: 3Gi
+- Examples:
+ - "0"
+ - Completely disable speedup (put by hash).
+ - "1G"
+ - Files larger than 1Gb will be uploaded directly.
+ - "3G"
+ - Choose this option if you have less than 3Gb free on local disk.
-The encoding for the backend.
+#### --mailru-speedup-max-memory
-See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info.
+Files larger than the size given below will always be hashed on disk.
Properties:
-- Config: encoding
-- Env Var: RCLONE_INTERNETARCHIVE_ENCODING
-- Type: MultiEncoder
-- Default: Slash,LtGt,CrLf,Del,Ctl,InvalidUtf8,Dot
-
-### Metadata
-
-Metadata fields provided by Internet Archive.
-If there are multiple values for a key, only the first one is returned.
-This is a limitation of Rclone, that supports one value per one key.
+- Config: speedup_max_memory
+- Env Var: RCLONE_MAILRU_SPEEDUP_MAX_MEMORY
+- Type: SizeSuffix
+- Default: 32Mi
+- Examples:
+ - "0"
+ - Preliminary hashing will always be done in a temporary disk location.
+ - "32M"
+ - Do not dedicate more than 32Mb RAM for preliminary hashing.
+ - "256M"
+ - You have at most 256Mb RAM free for hash calculations.
-Owner is able to add custom keys. Metadata feature grabs all the keys including them.
+#### --mailru-check-hash
-Here are the possible system metadata items for the internetarchive backend.
+What should copy do if file checksum is mismatched or invalid.
-| Name | Help | Type | Example | Read Only |
-|------|------|------|---------|-----------|
-| crc32 | CRC32 calculated by Internet Archive | string | 01234567 | **Y** |
-| format | Name of format identified by Internet Archive | string | Comma-Separated Values | **Y** |
-| md5 | MD5 hash calculated by Internet Archive | string | 01234567012345670123456701234567 | **Y** |
-| mtime | Time of last modification, managed by Rclone | RFC 3339 | 2006-01-02T15:04:05.999999999Z | **Y** |
-| name | Full file path, without the bucket part | filename | backend/internetarchive/internetarchive.go | **Y** |
-| old_version | Whether the file was replaced and moved by keep-old-version flag | boolean | true | **Y** |
-| rclone-ia-mtime | Time of last modification, managed by Internet Archive | RFC 3339 | 2006-01-02T15:04:05.999999999Z | N |
-| rclone-mtime | Time of last modification, managed by Rclone | RFC 3339 | 2006-01-02T15:04:05.999999999Z | N |
-| rclone-update-track | Random value used by Rclone for tracking changes inside Internet Archive | string | aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa | N |
-| sha1 | SHA1 hash calculated by Internet Archive | string | 0123456701234567012345670123456701234567 | **Y** |
-| size | File size in bytes | decimal number | 123456 | **Y** |
-| source | The source of the file | string | original | **Y** |
-| summation | Check https://forum.rclone.org/t/31922 for how it is used | string | md5 | **Y** |
-| viruscheck | The last time viruscheck process was run for the file (?) | unixtime | 1654191352 | **Y** |
+Properties:
-See the [metadata](https://rclone.org/docs/#metadata) docs for more info.
+- Config: check_hash
+- Env Var: RCLONE_MAILRU_CHECK_HASH
+- Type: bool
+- Default: true
+- Examples:
+ - "true"
+ - Fail with error.
+ - "false"
+ - Ignore and continue.
+#### --mailru-user-agent
+HTTP user agent used internally by client.
-# Jottacloud
+Defaults to "rclone/VERSION" or "--user-agent" provided on command line.
-Jottacloud is a cloud storage service provider from a Norwegian company, using its own datacenters
-in Norway. In addition to the official service at [jottacloud.com](https://www.jottacloud.com/),
-it also provides white-label solutions to different companies, such as:
-* Telia
- * Telia Cloud (cloud.telia.se)
- * Telia Sky (sky.telia.no)
-* Tele2
- * Tele2 Cloud (mittcloud.tele2.se)
-* Onlime
- * Onlime Cloud Storage (onlime.dk)
-* Elkjøp (with subsidiaries):
- * Elkjøp Cloud (cloud.elkjop.no)
- * Elgiganten Sweden (cloud.elgiganten.se)
- * Elgiganten Denmark (cloud.elgiganten.dk)
- * Giganti Cloud (cloud.gigantti.fi)
- * ELKO Cloud (cloud.elko.is)
+Properties:
-Most of the white-label versions are supported by this backend, although may require different
-authentication setup - described below.
+- Config: user_agent
+- Env Var: RCLONE_MAILRU_USER_AGENT
+- Type: string
+- Required: false
-Paths are specified as `remote:path`
+#### --mailru-quirks
-Paths may be as deep as required, e.g. `remote:directory/subdirectory`.
+Comma separated list of internal maintenance flags.
-## Authentication types
+This option must not be used by an ordinary user. It is intended only to
+facilitate remote troubleshooting of backend issues. Strict meaning of
+flags is not documented and not guaranteed to persist between releases.
+Quirks will be removed when the backend grows stable.
+Supported quirks: atomicmkdir binlist unknowndirs
-Some of the whitelabel versions uses a different authentication method than the official service,
-and you have to choose the correct one when setting up the remote.
+Properties:
-### Standard authentication
+- Config: quirks
+- Env Var: RCLONE_MAILRU_QUIRKS
+- Type: string
+- Required: false
-The standard authentication method used by the official service (jottacloud.com), as well as
-some of the whitelabel services, requires you to generate a single-use personal login token
-from the account security settings in the service's web interface. Log in to your account,
-go to "Settings" and then "Security", or use the direct link presented to you by rclone when
-configuring the remote: <https://www.jottacloud.com/web/secure>. Scroll down to the section
-"Personal login token", and click the "Generate" button. Note that if you are using a
-whitelabel service you probably can't use the direct link, you need to find the same page in
-their dedicated web interface, and also it may be in a different location than described above.
+#### --mailru-encoding
-To access your account from multiple instances of rclone, you need to configure each of them
-with a separate personal login token. E.g. you create a Jottacloud remote with rclone in one
-location, and copy the configuration file to a second location where you also want to run
-rclone and access the same remote. Then you need to replace the token for one of them, using
-the [config reconnect](https://rclone.org/commands/rclone_config_reconnect/) command, which
-requires you to generate a new personal login token and supply as input. If you do not
-do this, the token may easily end up being invalidated, resulting in both instances failing
-with an error message something along the lines of:
+The encoding for the backend.
- oauth2: cannot fetch token: 400 Bad Request
- Response: {"error":"invalid_grant","error_description":"Stale token"}
+See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info.
-When this happens, you need to replace the token as described above to be able to use your
-remote again.
+Properties:
-All personal login tokens you have taken into use will be listed in the web interface under
-"My logged in devices", and from the right side of that list you can click the "X" button to
-revoke individual tokens.
+- Config: encoding
+- Env Var: RCLONE_MAILRU_ENCODING
+- Type: Encoding
+- Default: Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Del,Ctl,InvalidUtf8,Dot
-### Legacy authentication
-If you are using one of the whitelabel versions (e.g. from Elkjøp) you may not have the option
-to generate a CLI token. In this case you'll have to use the legacy authentication. To do this select
-yes when the setup asks for legacy authentication and enter your username and password.
-The rest of the setup is identical to the default setup.
-### Telia Cloud authentication
+## Limitations
-Similar to other whitelabel versions Telia Cloud doesn't offer the option of creating a CLI token, and
-additionally uses a separate authentication flow where the username is generated internally. To setup
-rclone to use Telia Cloud, choose Telia Cloud authentication in the setup. The rest of the setup is
-identical to the default setup.
+File size limits depend on your account. A single file size is limited by 2G
+for a free account and unlimited for paid tariffs. Please refer to the Mail.ru
+site for the total uploaded size limits.
-### Tele2 Cloud authentication
+Note that Mailru is case insensitive so you can't have a file called
+"Hello.doc" and one called "hello.doc".
-As Tele2-Com Hem merger was completed this authentication can be used for former Com Hem Cloud and
-Tele2 Cloud customers as no support for creating a CLI token exists, and additionally uses a separate
-authentication flow where the username is generated internally. To setup rclone to use Tele2 Cloud,
-choose Tele2 Cloud authentication in the setup. The rest of the setup is identical to the default setup.
+# Mega
-### Onlime Cloud Storage authentication
+[Mega](https://mega.nz/) is a cloud storage and file hosting service
+known for its security feature where all files are encrypted locally
+before they are uploaded. This prevents anyone (including employees of
+Mega) from accessing the files without knowledge of the key used for
+encryption.
-Onlime has sold access to Jottacloud proper, while providing localized support to Danish Customers, but
-have recently set up their own hosting, transferring their customers from Jottacloud servers to their
-own ones.
+This is an rclone backend for Mega which supports the file transfer
+features of Mega using the same client side encryption.
-This, of course, necessitates using their servers for authentication, but otherwise functionality and
-architecture seems equivalent to Jottacloud.
+Paths are specified as `remote:path`
-To setup rclone to use Onlime Cloud Storage, choose Onlime Cloud authentication in the setup. The rest
-of the setup is identical to the default setup.
+Paths may be as deep as required, e.g. `remote:directory/subdirectory`.
## Configuration
-Here is an example of how to make a remote called `remote` with the default setup. First run:
+Here is an example of how to make a remote called `remote`. First run:
- rclone config
+ rclone config
This will guide you through an interactive setup process:
-No remotes found, make a new one? n) New remote s) Set configuration password q) Quit config n/s/q> n name> remote Option Storage. Type of storage to configure. Choose a number from below, or type in your own value. [snip] XX / Jottacloud  (jottacloud) [snip] Storage> jottacloud Edit advanced config? y) Yes n) No (default) y/n> n Option config_type. Select authentication type. Choose a number from below, or type in an existing string value. Press Enter for the default (standard). / Standard authentication. 1 | Use this if you're a normal Jottacloud user.  (standard) / Legacy authentication. 2 | This is only required for certain whitelabel versions of Jottacloud and not recommended for normal users.  (legacy) / Telia Cloud authentication. 3 | Use this if you are using Telia Cloud.  (telia) / Tele2 Cloud authentication. 4 | Use this if you are using Tele2 Cloud.  (tele2) / Onlime Cloud authentication. 5 | Use this if you are using Onlime Cloud.  (onlime) config_type> 1 Personal login token. Generate here: https://www.jottacloud.com/web/secure Login Token> Use a non-standard device/mountpoint? Choosing no, the default, will let you access the storage used for the archive section of the official Jottacloud client. If you instead want to access the sync or the backup section, for example, you must choose yes. y) Yes n) No (default) y/n> y Option config_device. The device to use. In standard setup the built-in Jotta device is used, which contains predefined mountpoints for archive, sync etc. All other devices are treated as backup devices by the official Jottacloud client. You may create a new by entering a unique name. Choose a number from below, or type in your own string value. Press Enter for the default (DESKTOP-3H31129). 1 > DESKTOP-3H31129 2 > Jotta config_device> 2 Option config_mountpoint. The mountpoint to use for the built-in device Jotta. The standard setup is to use the Archive mountpoint. Most other mountpoints have very limited support in rclone and should generally be avoided. Choose a number from below, or type in an existing string value. Press Enter for the default (Archive). 1 > Archive 2 > Shared 3 > Sync config_mountpoint> 1 -------------------- [remote] type = jottacloud configVersion = 1 client_id = jottacli client_secret = tokenURL = https://id.jottacloud.com/auth/realms/jottacloud/protocol/openid-connect/token token = {........} username = 2940e57271a93d987d6f8a21 device = Jotta mountpoint = Archive -------------------- y) Yes this is OK (default) e) Edit this remote d) Delete this remote y/e/d> y
+No remotes found, make a new one? n) New remote s) Set configuration password q) Quit config n/s/q> n name> remote Type of storage to configure. Choose a number from below, or type in your own value [snip] XX / Mega  "mega" [snip] Storage> mega User name user> you@example.com Password. y) Yes type in my own password g) Generate random password n) No leave this optional password blank y/g/n> y Enter the password: password: Confirm the password: password: Remote config -------------------- [remote] type = mega user = you@example.com pass = *** ENCRYPTED *** -------------------- y) Yes this is OK e) Edit this remote d) Delete this remote y/e/d> y
+**NOTE:** The encryption keys need to have been already generated after a regular login
+via the browser, otherwise attempting to use the credentials in `rclone` will fail.
+
Once configured you can then use `rclone` like this,
-List directories in top level of your Jottacloud
+List directories in top level of your Mega
rclone lsd remote:
-List all the files in your Jottacloud
+List all the files in your Mega
rclone ls remote:
-To copy a local directory to an Jottacloud directory called backup
+To copy a local directory to an Mega directory called backup
rclone copy /home/source remote:backup
-### Devices and Mountpoints
+### Modification times and hashes
-The official Jottacloud client registers a device for each computer you install
-it on, and shows them in the backup section of the user interface. For each
-folder you select for backup it will create a mountpoint within this device.
-A built-in device called Jotta is special, and contains mountpoints Archive,
-Sync and some others, used for corresponding features in official clients.
+Mega does not support modification times or hashes yet.
-With rclone you'll want to use the standard Jotta/Archive device/mountpoint in
-most cases. However, you may for example want to access files from the sync or
-backup functionality provided by the official clients, and rclone therefore
-provides the option to select other devices and mountpoints during config.
+### Restricted filename characters
-You are allowed to create new devices and mountpoints. All devices except the
-built-in Jotta device are treated as backup devices by official Jottacloud
-clients, and the mountpoints on them are individual backup sets.
+| Character | Value | Replacement |
+| --------- |:-----:|:-----------:|
+| NUL | 0x00 | †|
+| / | 0x2F | ï¼ |
-With the built-in Jotta device, only existing, built-in, mountpoints can be
-selected. In addition to the mentioned Archive and Sync, it may contain
-several other mountpoints such as: Latest, Links, Shared and Trash. All of
-these are special mountpoints with a different internal representation than
-the "regular" mountpoints. Rclone will only to a very limited degree support
-them. Generally you should avoid these, unless you know what you are doing.
+Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8),
+as they can't be used in JSON strings.
-### --fast-list
+### Duplicated files
-This remote supports `--fast-list` which allows you to use fewer
-transactions in exchange for more memory. See the [rclone
-docs](https://rclone.org/docs/#fast-list) for more details.
+Mega can have two files with exactly the same name and path (unlike a
+normal file system).
-Note that the implementation in Jottacloud always uses only a single
-API request to get the entire list, so for large folders this could
-lead to long wait time before the first results are shown.
+Duplicated files cause problems with the syncing and you will see
+messages in the log about duplicates.
-Note also that with rclone version 1.58 and newer information about
-[MIME types](https://rclone.org/overview/#mime-type) are not available when using `--fast-list`.
+Use `rclone dedupe` to fix duplicated files.
-### Modified time and hashes
+### Failure to log-in
-Jottacloud allows modification times to be set on objects accurate to 1
-second. These will be used to detect whether objects need syncing or
-not.
+#### Object not found
-Jottacloud supports MD5 type hashes, so you can use the `--checksum`
-flag.
+If you are connecting to your Mega remote for the first time,
+to test access and synchronization, you may receive an error such as
+
+Failed to create file system for "my-mega-remote:": couldn't login: Object (typically, node or user) not found
+
+The diagnostic steps often recommended in the [rclone forum](https://forum.rclone.org/search?q=mega)
+start with the **MEGAcmd** utility. Note that this refers to
+the official C++ command from https://github.com/meganz/MEGAcmd
+and not the go language built command from t3rm1n4l/megacmd
+that is no longer maintained.
-Note that Jottacloud requires the MD5 hash before upload so if the
-source does not have an MD5 checksum then the file will be cached
-temporarily on disk (in location given by
-[--temp-dir](https://rclone.org/docs/#temp-dir-dir)) before it is uploaded.
-Small files will be cached in memory - see the
-[--jottacloud-md5-memory-limit](#jottacloud-md5-memory-limit) flag.
-When uploading from local disk the source checksum is always available,
-so this does not apply. Starting with rclone version 1.52 the same is
-true for encrypted remotes (in older versions the crypt backend would not
-calculate hashes for uploads from local disk, so the Jottacloud
-backend had to do it as described above).
+Follow the instructions for installing MEGAcmd and try accessing
+your remote as they recommend. You can establish whether or not
+you can log in using MEGAcmd, and obtain diagnostic information
+to help you, and search or work with others in the forum.
+
+MEGA CMD> login me@example.com Password: Fetching nodes ... Loading transfers from local cache Login complete as me@example.com me@example.com:/$
+
+Note that some have found issues with passwords containing special
+characters. If you can not log on with rclone, but MEGAcmd logs on
+just fine, then consider changing your password temporarily to
+pure alphanumeric characters, in case that helps.
-### Restricted filename characters
-In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters)
-the following characters are also replaced:
+#### Repeated commands blocks access
-| Character | Value | Replacement |
-| --------- |:-----:|:-----------:|
-| " | 0x22 | ＂ |
-| * | 0x2A | ＊ |
-| : | 0x3A | ： |
-| < | 0x3C | ＜ |
-| > | 0x3E | ＞ |
-| ? | 0x3F | ？ |
-| \| | 0x7C | ｜ |
+Mega remotes seem to get blocked (reject logins) under "heavy use".
+We haven't worked out the exact blocking rules but it seems to be
+related to fast paced, successive rclone commands.
-Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8),
-as they can't be used in XML strings.
+For example, executing this command 90 times in a row `rclone link
+remote:file` will cause the remote to become "blocked". This is not an
+abnormal situation, for example if you wish to get the public links of
+a directory with hundred of files... After more or less a week, the
+remote will remote accept rclone logins normally again.
-### Deleting files
+You can mitigate this issue by mounting the remote it with `rclone
+mount`. This will log-in when mounting and a log-out when unmounting
+only. You can also run `rclone rcd` and then use `rclone rc` to run
+the commands over the API to avoid logging in each time.
-By default, rclone will send all files to the trash when deleting files. They will be permanently
-deleted automatically after 30 days. You may bypass the trash and permanently delete files immediately
-by using the [--jottacloud-hard-delete](#jottacloud-hard-delete) flag, or set the equivalent environment variable.
-Emptying the trash is supported by the [cleanup](https://rclone.org/commands/rclone_cleanup/) command.
+Rclone does not currently close mega sessions (you can see them in the
+web interface), however closing the sessions does not solve the issue.
-### Versions
+If you space rclone commands by 3 seconds it will avoid blocking the
+remote. We haven't identified the exact blocking rules, so perhaps one
+could execute the command 80 times without waiting and avoid blocking
+by waiting 3 seconds, then continuing...
-Jottacloud supports file versioning. When rclone uploads a new version of a file it creates a new version of it.
-Currently rclone only supports retrieving the current version but older versions can be accessed via the Jottacloud Website.
+Note that this has been observed by trial and error and might not be
+set in stone.
-Versioning can be disabled by `--jottacloud-no-versions` option. This is achieved by deleting the remote file prior to uploading
-a new version. If the upload the fails no version of the file will be available in the remote.
+Other tools seem not to produce this blocking effect, as they use a
+different working approach (state-based, using sessionIDs instead of
+log-in) which isn't compatible with the current stateless rclone
+approach.
-### Quota information
+Note that once blocked, the use of other tools (such as megacmd) is
+not a sure workaround: following megacmd login times have been
+observed in succession for blocked remote: 7 minutes, 20 min, 30min, 30
+min, 30min. Web access looks unaffected though.
-To view your current quota you can use the `rclone about remote:`
-command which will display your usage limit (unless it is unlimited)
-and the current usage.
+Investigation is continuing in relation to workarounds based on
+timeouts, pacers, retrials and tpslimits - if you discover something
+relevant, please post on the forum.
+
+So, if rclone was working nicely and suddenly you are unable to log-in
+and you are sure the user and the password are correct, likely you
+have got the remote blocked for a while.
### Standard options
-Here are the Standard options specific to jottacloud (Jottacloud).
-
-#### --jottacloud-client-id
+Here are the Standard options specific to mega (Mega).
-OAuth Client Id.
+#### --mega-user
-Leave blank normally.
+User name.
Properties:
-- Config: client_id
-- Env Var: RCLONE_JOTTACLOUD_CLIENT_ID
+- Config: user
+- Env Var: RCLONE_MEGA_USER
- Type: string
-- Required: false
+- Required: true
-#### --jottacloud-client-secret
+#### --mega-pass
-OAuth Client Secret.
+Password.
-Leave blank normally.
+**NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/).
Properties:
-- Config: client_secret
-- Env Var: RCLONE_JOTTACLOUD_CLIENT_SECRET
+- Config: pass
+- Env Var: RCLONE_MEGA_PASS
- Type: string
-- Required: false
+- Required: true
### Advanced options
-Here are the Advanced options specific to jottacloud (Jottacloud).
-
-#### --jottacloud-token
-
-OAuth Access Token as a JSON blob.
-
-Properties:
-
-- Config: token
-- Env Var: RCLONE_JOTTACLOUD_TOKEN
-- Type: string
-- Required: false
+Here are the Advanced options specific to mega (Mega).
-#### --jottacloud-auth-url
+#### --mega-debug
-Auth server URL.
+Output more debug from Mega.
-Leave blank to use the provider defaults.
+If this flag is set (along with -vv) it will print further debugging
+information from the mega backend.
Properties:
-- Config: auth_url
-- Env Var: RCLONE_JOTTACLOUD_AUTH_URL
-- Type: string
-- Required: false
-
-#### --jottacloud-token-url
-
-Token server url.
-
-Leave blank to use the provider defaults.
-
-Properties:
+- Config: debug
+- Env Var: RCLONE_MEGA_DEBUG
+- Type: bool
+- Default: false
-- Config: token_url
-- Env Var: RCLONE_JOTTACLOUD_TOKEN_URL
-- Type: string
-- Required: false
+#### --mega-hard-delete
-#### --jottacloud-md5-memory-limit
+Delete files permanently rather than putting them into the trash.
-Files bigger than this will be cached on disk to calculate the MD5 if required.
+Normally the mega backend will put all deletions into the trash rather
+than permanently deleting them. If you specify this then rclone will
+permanently delete objects instead.
Properties:
-- Config: md5_memory_limit
-- Env Var: RCLONE_JOTTACLOUD_MD5_MEMORY_LIMIT
-- Type: SizeSuffix
-- Default: 10Mi
+- Config: hard_delete
+- Env Var: RCLONE_MEGA_HARD_DELETE
+- Type: bool
+- Default: false
-#### --jottacloud-trashed-only
+#### --mega-use-https
-Only show files that are in the trash.
+Use HTTPS for transfers.
-This will show trashed files in their original directory structure.
+MEGA uses plain text HTTP connections by default.
+Some ISPs throttle HTTP connections, this causes transfers to become very slow.
+Enabling this will force MEGA to use HTTPS for all transfers.
+HTTPS is normally not necessary since all data is already encrypted anyway.
+Enabling it will increase CPU usage and add network overhead.
Properties:
-- Config: trashed_only
-- Env Var: RCLONE_JOTTACLOUD_TRASHED_ONLY
+- Config: use_https
+- Env Var: RCLONE_MEGA_USE_HTTPS
- Type: bool
- Default: false
-#### --jottacloud-hard-delete
+#### --mega-encoding
-Delete files permanently rather than putting them into the trash.
+The encoding for the backend.
+
+See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info.
Properties:
-- Config: hard_delete
-- Env Var: RCLONE_JOTTACLOUD_HARD_DELETE
-- Type: bool
-- Default: false
+- Config: encoding
+- Env Var: RCLONE_MEGA_ENCODING
+- Type: Encoding
+- Default: Slash,InvalidUtf8,Dot
-#### --jottacloud-upload-resume-limit
-Files bigger than this can be resumed if the upload fail's.
-Properties:
+### Process `killed`
-- Config: upload_resume_limit
-- Env Var: RCLONE_JOTTACLOUD_UPLOAD_RESUME_LIMIT
-- Type: SizeSuffix
-- Default: 10Mi
+On accounts with large files or something else, memory usage can significantly increase when executing list/sync instructions. When running on cloud providers (like AWS with EC2), check if the instance type has sufficient memory/CPU to execute the commands. Use the resource monitoring tools to inspect after sending the commands. Look [at this issue](https://forum.rclone.org/t/rclone-with-mega-appears-to-work-only-in-some-accounts/40233/4).
-#### --jottacloud-no-versions
+## Limitations
-Avoid server side versioning by deleting files and recreating files instead of overwriting them.
+This backend uses the [go-mega go library](https://github.com/t3rm1n4l/go-mega) which is an opensource
+go library implementing the Mega API. There doesn't appear to be any
+documentation for the mega protocol beyond the [mega C++ SDK](https://github.com/meganz/sdk) source code
+so there are likely quite a few errors still remaining in this library.
-Properties:
+Mega allows duplicate files which may confuse rclone.
-- Config: no_versions
-- Env Var: RCLONE_JOTTACLOUD_NO_VERSIONS
-- Type: bool
-- Default: false
+# Memory
-#### --jottacloud-encoding
+The memory backend is an in RAM backend. It does not persist its
+data - use the local backend for that.
-The encoding for the backend.
+The memory backend behaves like a bucket-based remote (e.g. like
+s3). Because it has no parameters you can just use it with the
+`:memory:` remote name.
-See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info.
+## Configuration
-Properties:
+You can configure it as a remote like this with `rclone config` too if
+you want to:
+
+No remotes found, make a new one? n) New remote s) Set configuration password q) Quit config n/s/q> n name> remote Type of storage to configure. Enter a string value. Press Enter for the default (""). Choose a number from below, or type in your own value [snip] XX / Memory  "memory" [snip] Storage> memory ** See help for memory backend at: https://rclone.org/memory/ **
+Remote config
+
+
+
+[remote]
+
+
+type = memory
+
+
+
+
+- Yes this is OK (default)
+- Edit this remote
+- Delete this remote y/e/d> y
+
+
+Because the memory backend isn't persistent it is most useful for
+testing or with an rclone server or rclone mount, e.g.
-- Config: encoding
-- Env Var: RCLONE_JOTTACLOUD_ENCODING
-- Type: MultiEncoder
-- Default: Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,Del,Ctl,InvalidUtf8,Dot
+ rclone mount :memory: /mnt/tmp
+ rclone serve webdav :memory:
+ rclone serve sftp :memory:
+### Modification times and hashes
+The memory backend supports MD5 hashes and modification times accurate to 1 nS.
-## Limitations
+### Restricted filename characters
-Note that Jottacloud is case insensitive so you can't have a file called
-"Hello.doc" and one called "hello.doc".
+The memory backend replaces the [default restricted characters
+set](https://rclone.org/overview/#restricted-characters).
-There are quite a few characters that can't be in Jottacloud file names. Rclone will map these names to and from an identical
-looking unicode equivalent. For example if a file has a ? in it will be mapped to ？ instead.
-Jottacloud only supports filenames up to 255 characters in length.
-## Troubleshooting
-Jottacloud exhibits some inconsistent behaviours regarding deleted files and folders which may cause Copy, Move and DirMove
-operations to previously deleted paths to fail. Emptying the trash should help in such cases.
+# Akamai NetStorage
-# Koofr
+Paths are specified as `remote:`
+You may put subdirectories in too, e.g. `remote:/path/to/dir`.
+If you have a CP code you can use that as the folder after the domain such as \<domain>\/\<cpcode>\/\<internal directories within cpcode>.
-Paths are specified as `remote:path`
+For example, this is commonly configured with or without a CP code:
+* **With a CP code**. `[your-domain-prefix]-nsu.akamaihd.net/123456/subdirectory/`
+* **Without a CP code**. `[your-domain-prefix]-nsu.akamaihd.net`
-Paths may be as deep as required, e.g. `remote:directory/subdirectory`.
-## Configuration
+See all buckets
+ rclone lsd remote:
+The initial setup for Netstorage involves getting an account and secret. Use `rclone config` to walk you through the setup process.
-The initial setup for Koofr involves creating an application password for
-rclone. You can do that by opening the Koofr
-[web application](https://app.koofr.net/app/admin/preferences/password),
-giving the password a nice name like `rclone` and clicking on generate.
+## Configuration
-Here is an example of how to make a remote called `koofr`. First run:
+Here's an example of how to make a remote called `ns1`.
- rclone config
+1. To begin the interactive configuration process, enter this command:
+
+rclone config
+
+2. Type `n` to create a new remote.
+
+
+- New remote
+- Delete remote
+- Quit config e/n/d/q> n
+
+
+3. For this example, enter `ns1` when you reach the name> prompt.
+
+name> ns1
+
+4. Enter `netstorage` as the type of storage to configure.
+
+Type of storage to configure. Enter a string value. Press Enter for the default (""). Choose a number from below, or type in your own value XX / NetStorage  "netstorage" Storage> netstorage
+
+5. Select between the HTTP or HTTPS protocol. Most users should choose HTTPS, which is the default. HTTP is provided primarily for debugging purposes.
-This will guide you through an interactive setup process:
-No remotes found, make a new one? n) New remote s) Set configuration password q) Quit config n/s/q> n name> koofr Option Storage. Type of storage to configure. Choose a number from below, or type in your own value. [snip] 22 / Koofr, Digi Storage and other Koofr-compatible storage providers  (koofr) [snip] Storage> koofr Option provider. Choose your storage provider. Choose a number from below, or type in your own value. Press Enter to leave empty. 1 / Koofr, https://app.koofr.net/  (koofr) 2 / Digi Storage, https://storage.rcs-rds.ro/  (digistorage) 3 / Any other Koofr API compatible storage service  (other) provider> 1
-Option user. Your user name. Enter a value. user> USERNAME Option password. Your password for rclone (generate one at https://app.koofr.net/app/admin/preferences/password). Choose an alternative below. y) Yes, type in my own password g) Generate random password y/g> y Enter the password: password: Confirm the password: password: Edit advanced config? y) Yes n) No (default) y/n> n Remote config -------------------- [koofr] type = koofr provider = koofr user = USERNAME password = *** ENCRYPTED *** -------------------- y) Yes this is OK (default) e) Edit this remote d) Delete this remote y/e/d> y
+Enter a string value. Press Enter for the default (""). Choose a number from below, or type in your own value 1 / HTTP protocol  "http" 2 / HTTPS protocol  "https" protocol> 1
-You can choose to edit advanced config in order to enter your own service URL
-if you use an on-premise or white label Koofr instance, or choose an alternative
-mount instead of your primary storage.
+6. Specify your NetStorage host, CP code, and any necessary content paths using this format: `<domain>/<cpcode>/<content>/`
+
+Enter a string value. Press Enter for the default (""). host> baseball-nsu.akamaihd.net/123456/content/
+
+7. Set the netstorage account name
+Enter a string value. Press Enter for the default (""). account> username
+
+8. Set the Netstorage account secret/G2O key which will be used for authentication purposes. Select the `y` option to set your own password then enter your secret.
+Note: The secret is stored in the `rclone.conf` file with hex-encoded encryption.
+
+
+- Yes type in my own password
+- Generate random password y/g> y Enter the password: password: Confirm the password: password:
+
+
+9. View the summary and confirm your remote configuration.
+
+[ns1] type = netstorage protocol = http host = baseball-nsu.akamaihd.net/123456/content/ account = username secret = *** ENCRYPTED *** -------------------- y) Yes this is OK (default) e) Edit this remote d) Delete this remote y/e/d> y
+
+This remote is called `ns1` and can now be used.
-Once configured you can then use `rclone` like this,
+## Example operations
-List directories in top level of your Koofr
+Get started with rclone and NetStorage with these examples. For additional rclone commands, visit https://rclone.org/commands/.
- rclone lsd koofr:
+### See contents of a directory in your project
-List all the files in your Koofr
+ rclone lsd ns1:/974012/testing/
- rclone ls koofr:
+### Sync the contents local with remote
-To copy a local directory to an Koofr directory called backup
+ rclone sync . ns1:/974012/testing/
- rclone copy /home/source koofr:backup
+### Upload local content to remote
+ rclone copy notes.txt ns1:/974012/testing/
-### Restricted filename characters
+### Delete content on remote
+ rclone delete ns1:/974012/testing/notes.txt
-In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters)
-the following characters are also replaced:
+### Move or copy content between CP codes.
-| Character | Value | Replacement |
-| --------- |:-----:|:-----------:|
-| \ | 0x5C | ï¼¼ |
+Your credentials must have access to two CP codes on the same remote. You can't perform operations between different remotes.
-Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8),
-as they can't be used in XML strings.
+ rclone move ns1:/974012/testing/notes.txt ns1:/974450/testing2/
+## Features
-### Standard options
+### Symlink Support
-Here are the Standard options specific to koofr (Koofr, Digi Storage and other Koofr-compatible storage providers).
+The Netstorage backend changes the rclone `--links, -l` behavior. When uploading, instead of creating the .rclonelink file, use the "symlink" API in order to create the corresponding symlink on the remote. The .rclonelink file will not be created, the upload will be intercepted and only the symlink file that matches the source file name with no suffix will be created on the remote.
-#### --koofr-provider
+This will effectively allow commands like copy/copyto, move/moveto and sync to upload from local to remote and download from remote to local directories with symlinks. Due to internal rclone limitations, it is not possible to upload an individual symlink file to any remote backend. You can always use the "backend symlink" command to create a symlink on the NetStorage server, refer to "symlink" section below.
-Choose your storage provider.
+Individual symlink files on the remote can be used with the commands like "cat" to print the destination name, or "delete" to delete symlink, or copy, copy/to and move/moveto to download from the remote to local. Note: individual symlink files on the remote should be specified including the suffix .rclonelink.
-Properties:
+**Note**: No file with the suffix .rclonelink should ever exist on the server since it is not possible to actually upload/create a file with .rclonelink suffix with rclone, it can only exist if it is manually created through a non-rclone method on the remote.
-- Config: provider
-- Env Var: RCLONE_KOOFR_PROVIDER
-- Type: string
-- Required: false
-- Examples:
- - "koofr"
- - Koofr, https://app.koofr.net/
- - "digistorage"
- - Digi Storage, https://storage.rcs-rds.ro/
- - "other"
- - Any other Koofr API compatible storage service
+### Implicit vs. Explicit Directories
-#### --koofr-endpoint
+With NetStorage, directories can exist in one of two forms:
-The Koofr API endpoint to use.
+1. **Explicit Directory**. This is an actual, physical directory that you have created in a storage group.
+2. **Implicit Directory**. This refers to a directory within a path that has not been physically created. For example, during upload of a file, nonexistent subdirectories can be specified in the target path. NetStorage creates these as "implicit." While the directories aren't physically created, they exist implicitly and the noted path is connected with the uploaded file.
-Properties:
+Rclone will intercept all file uploads and mkdir commands for the NetStorage remote and will explicitly issue the mkdir command for each directory in the uploading path. This will help with the interoperability with the other Akamai services such as SFTP and the Content Management Shell (CMShell). Rclone will not guarantee correctness of operations with implicit directories which might have been created as a result of using an upload API directly.
-- Config: endpoint
-- Env Var: RCLONE_KOOFR_ENDPOINT
-- Provider: other
-- Type: string
-- Required: true
+### `--fast-list` / ListR support
-#### --koofr-user
+NetStorage remote supports the ListR feature by using the "list" NetStorage API action to return a lexicographical list of all objects within the specified CP code, recursing into subdirectories as they're encountered.
-Your user name.
+* **Rclone will use the ListR method for some commands by default**. Commands such as `lsf -R` will use ListR by default. To disable this, include the `--disable listR` option to use the non-recursive method of listing objects.
-Properties:
+* **Rclone will not use the ListR method for some commands**. Commands such as `sync` don't use ListR by default. To force using the ListR method, include the `--fast-list` option.
-- Config: user
-- Env Var: RCLONE_KOOFR_USER
-- Type: string
-- Required: true
+There are pros and cons of using the ListR method, refer to [rclone documentation](https://rclone.org/docs/#fast-list). In general, the sync command over an existing deep tree on the remote will run faster with the "--fast-list" flag but with extra memory usage as a side effect. It might also result in higher CPU utilization but the whole task can be completed faster.
-#### --koofr-password
+**Note**: There is a known limitation that "lsf -R" will display number of files in the directory and directory size as -1 when ListR method is used. The workaround is to pass "--disable listR" flag if these numbers are important in the output.
-Your password for rclone (generate one at https://app.koofr.net/app/admin/preferences/password).
+### Purge
-**NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/).
+NetStorage remote supports the purge feature by using the "quick-delete" NetStorage API action. The quick-delete action is disabled by default for security reasons and can be enabled for the account through the Akamai portal. Rclone will first try to use quick-delete action for the purge command and if this functionality is disabled then will fall back to a standard delete method.
+
+**Note**: Read the [NetStorage Usage API](https://learn.akamai.com/en-us/webhelp/netstorage/netstorage-http-api-developer-guide/GUID-15836617-9F50-405A-833C-EA2556756A30.html) for considerations when using "quick-delete". In general, using quick-delete method will not delete the tree immediately and objects targeted for quick-delete may still be accessible.
+
+
+### Standard options
+
+Here are the Standard options specific to netstorage (Akamai NetStorage).
+
+#### --netstorage-host
+
+Domain+path of NetStorage host to connect to.
+
+Format should be `<domain>/<internal folders>`
Properties:
-- Config: password
-- Env Var: RCLONE_KOOFR_PASSWORD
-- Provider: koofr
+- Config: host
+- Env Var: RCLONE_NETSTORAGE_HOST
- Type: string
- Required: true
-#### --koofr-password
-
-Your password for rclone (generate one at https://storage.rcs-rds.ro/app/admin/preferences/password).
+#### --netstorage-account
-**NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/).
+Set the NetStorage account name
Properties:
-- Config: password
-- Env Var: RCLONE_KOOFR_PASSWORD
-- Provider: digistorage
+- Config: account
+- Env Var: RCLONE_NETSTORAGE_ACCOUNT
- Type: string
- Required: true
-#### --koofr-password
+#### --netstorage-secret
-Your password for rclone (generate one at your service's settings page).
+Set the NetStorage account secret/G2O key for authentication.
+
+Please choose the 'y' option to set your own password then enter your secret.
**NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/).
Properties:
-- Config: password
-- Env Var: RCLONE_KOOFR_PASSWORD
-- Provider: other
+- Config: secret
+- Env Var: RCLONE_NETSTORAGE_SECRET
- Type: string
- Required: true
### Advanced options
-Here are the Advanced options specific to koofr (Koofr, Digi Storage and other Koofr-compatible storage providers).
+Here are the Advanced options specific to netstorage (Akamai NetStorage).
-#### --koofr-mountid
+#### --netstorage-protocol
-Mount ID of the mount to use.
+Select between HTTP or HTTPS protocol.
-If omitted, the primary mount is used.
+Most users should choose HTTPS, which is the default.
+HTTP is provided primarily for debugging purposes.
Properties:
-- Config: mountid
-- Env Var: RCLONE_KOOFR_MOUNTID
+- Config: protocol
+- Env Var: RCLONE_NETSTORAGE_PROTOCOL
- Type: string
-- Required: false
-
-#### --koofr-setmtime
-
-Does the backend support setting modification time.
-
-Set this to false if you use a mount ID that points to a Dropbox or Amazon Drive backend.
+- Default: "https"
+- Examples:
+ - "http"
+ - HTTP protocol
+ - "https"
+ - HTTPS protocol
-Properties:
+## Backend commands
-- Config: setmtime
-- Env Var: RCLONE_KOOFR_SETMTIME
-- Type: bool
-- Default: true
+Here are the commands specific to the netstorage backend.
-#### --koofr-encoding
+Run them with
-The encoding for the backend.
+ rclone backend COMMAND remote:
-See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info.
+The help below will explain what arguments each command takes.
-Properties:
+See the [backend](https://rclone.org/commands/rclone_backend/) command for more
+info on how to pass options and arguments.
-- Config: encoding
-- Env Var: RCLONE_KOOFR_ENCODING
-- Type: MultiEncoder
-- Default: Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot
+These can be run on a running backend using the rc command
+[backend/command](https://rclone.org/rc/#backend-command).
+### du
+Return disk usage information for a specified directory
-## Limitations
+ rclone backend du remote: [options] [<arguments>+]
-Note that Koofr is case insensitive so you can't have a file called
-"Hello.doc" and one called "hello.doc".
+The usage information returned, includes the targeted directory as well as all
+files stored in any sub-directories that may exist.
-## Providers
+### symlink
-### Koofr
+You can create a symbolic link in ObjectStore with the symlink action.
-This is the original [Koofr](https://koofr.eu) storage provider used as main example and described in the [configuration](#configuration) section above.
+ rclone backend symlink remote: [options] [<arguments>+]
-### Digi Storage
+The desired path location (including applicable sub-directories) ending in
+the object that will be the target of the symlink (for example, /links/mylink).
+Include the file extension for the object, if applicable.
+`rclone backend symlink <src> <path>`
-[Digi Storage](https://www.digi.ro/servicii/online/digi-storage) is a cloud storage service run by [Digi.ro](https://www.digi.ro/) that
-provides a Koofr API.
-Here is an example of how to make a remote called `ds`. First run:
- rclone config
+# Microsoft Azure Blob Storage
-This will guide you through an interactive setup process:
-
-No remotes found, make a new one? n) New remote s) Set configuration password q) Quit config n/s/q> n name> ds Option Storage. Type of storage to configure. Choose a number from below, or type in your own value. [snip] 22 / Koofr, Digi Storage and other Koofr-compatible storage providers  (koofr) [snip] Storage> koofr Option provider. Choose your storage provider. Choose a number from below, or type in your own value. Press Enter to leave empty. 1 / Koofr, https://app.koofr.net/  (koofr) 2 / Digi Storage, https://storage.rcs-rds.ro/  (digistorage) 3 / Any other Koofr API compatible storage service  (other) provider> 2 Option user. Your user name. Enter a value. user> USERNAME Option password. Your password for rclone (generate one at https://storage.rcs-rds.ro/app/admin/preferences/password). Choose an alternative below. y) Yes, type in my own password g) Generate random password y/g> y Enter the password: password: Confirm the password: password: Edit advanced config? y) Yes n) No (default) y/n> n -------------------- [ds] type = koofr provider = digistorage user = USERNAME password = *** ENCRYPTED *** -------------------- y) Yes this is OK (default) e) Edit this remote d) Delete this remote y/e/d> y
-
-### Other
+Paths are specified as `remote:container` (or `remote:` for the `lsd`
+command.) You may put subdirectories in too, e.g.
+`remote:container/path/to/dir`.
-You may also want to use another, public or private storage provider that runs a Koofr API compatible service, by simply providing the base URL to connect to.
+## Configuration
-Here is an example of how to make a remote called `other`. First run:
+Here is an example of making a Microsoft Azure Blob Storage
+configuration. For a remote called `remote`. First run:
rclone config
This will guide you through an interactive setup process:
-No remotes found, make a new one? n) New remote s) Set configuration password q) Quit config n/s/q> n name> other Option Storage. Type of storage to configure. Choose a number from below, or type in your own value. [snip] 22 / Koofr, Digi Storage and other Koofr-compatible storage providers  (koofr) [snip] Storage> koofr Option provider. Choose your storage provider. Choose a number from below, or type in your own value. Press Enter to leave empty. 1 / Koofr, https://app.koofr.net/  (koofr) 2 / Digi Storage, https://storage.rcs-rds.ro/  (digistorage) 3 / Any other Koofr API compatible storage service  (other) provider> 3 Option endpoint. The Koofr API endpoint to use. Enter a value. endpoint> https://koofr.other.org Option user. Your user name. Enter a value. user> USERNAME Option password. Your password for rclone (generate one at your service's settings page). Choose an alternative below. y) Yes, type in my own password g) Generate random password y/g> y Enter the password: password: Confirm the password: password: Edit advanced config? y) Yes n) No (default) y/n> n -------------------- [other] type = koofr provider = other endpoint = https://koofr.other.org user = USERNAME password = *** ENCRYPTED *** -------------------- y) Yes this is OK (default) e) Edit this remote d) Delete this remote y/e/d> y
+No remotes found, make a new one? n) New remote s) Set configuration password q) Quit config n/s/q> n name> remote Type of storage to configure. Choose a number from below, or type in your own value [snip] XX / Microsoft Azure Blob Storage  "azureblob" [snip] Storage> azureblob Storage Account Name account> account_name Storage Account Key key> base64encodedkey== Endpoint for the service - leave blank normally. endpoint> Remote config -------------------- [remote] account = account_name key = base64encodedkey== endpoint = -------------------- y) Yes this is OK e) Edit this remote d) Delete this remote y/e/d> y
-# Mail.ru Cloud
+See all containers
-[Mail.ru Cloud](https://cloud.mail.ru/) is a cloud storage provided by a Russian internet company [Mail.Ru Group](https://mail.ru). The official desktop client is [Disk-O:](https://disk-o.cloud/en), available on Windows and Mac OS.
+ rclone lsd remote:
-## Features highlights
+Make a new container
-- Paths may be as deep as required, e.g. `remote:directory/subdirectory`
-- Files have a `last modified time` property, directories don't
-- Deleted files are by default moved to the trash
-- Files and directories can be shared via public links
-- Partial uploads or streaming are not supported, file size must be known before upload
-- Maximum file size is limited to 2G for a free account, unlimited for paid accounts
-- Storage keeps hash for all files and performs transparent deduplication,
- the hash algorithm is a modified SHA1
-- If a particular file is already present in storage, one can quickly submit file hash
- instead of long file upload (this optimization is supported by rclone)
+ rclone mkdir remote:container
-## Configuration
+List the contents of a container
-Here is an example of making a mailru configuration.
+ rclone ls remote:container
-First create a Mail.ru Cloud account and choose a tariff.
+Sync `/home/local/directory` to the remote container, deleting any excess
+files in the container.
-You will need to log in and create an app password for rclone. Rclone
-**will not work** with your normal username and password - it will
-give an error like `oauth2: server response missing access_token`.
+ rclone sync --interactive /home/local/directory remote:container
-- Click on your user icon in the top right
-- Go to Security / "Пароль и безопаÑноÑÑ‚ÑŒ"
-- Click password for apps / "Пароли Ð´Ð»Ñ Ð²Ð½ÐµÑˆÐ½Ð¸Ñ… приложений"
-- Add the password - give it a name - eg "rclone"
-- Copy the password and use this password below - your normal login password won't work.
+### --fast-list
-Now run
+This remote supports `--fast-list` which allows you to use fewer
+transactions in exchange for more memory. See the [rclone
+docs](https://rclone.org/docs/#fast-list) for more details.
- rclone config
+### Modification times and hashes
-This will guide you through an interactive setup process:
-
-No remotes found, make a new one? n) New remote s) Set configuration password q) Quit config n/s/q> n name> remote Type of storage to configure. Type of storage to configure. Enter a string value. Press Enter for the default (""). Choose a number from below, or type in your own value [snip] XX / Mail.ru Cloud  "mailru" [snip] Storage> mailru User name (usually email) Enter a string value. Press Enter for the default (""). user> username@mail.ru Password
-This must be an app password - rclone will not work with your normal password. See the Configuration section in the docs for how to make an app password. y) Yes type in my own password g) Generate random password y/g> y Enter the password: password: Confirm the password: password: Skip full upload if there is another file with same data hash. This feature is called "speedup" or "put by hash". It is especially efficient in case of generally available files like popular books, video or audio clips [snip] Enter a boolean value (true or false). Press Enter for the default ("true"). Choose a number from below, or type in your own value 1 / Enable  "true" 2 / Disable  "false" speedup_enable> 1 Edit advanced config? (y/n) y) Yes n) No y/n> n Remote config -------------------- [remote] type = mailru user = username@mail.ru pass = *** ENCRYPTED *** speedup_enable = true -------------------- y) Yes this is OK e) Edit this remote d) Delete this remote y/e/d> y
-
-Configuration of this backend does not require a local web browser.
-You can use the configured backend as shown below:
+The modification time is stored as metadata on the object with the
+`mtime` key. It is stored using RFC3339 Format time with nanosecond
+precision. The metadata is supplied during directory listings so
+there is no performance overhead to using it.
-See top level directories
+If you wish to use the Azure standard `LastModified` time stored on
+the object as the modified time, then use the `--use-server-modtime`
+flag. Note that rclone can't set `LastModified`, so using the
+`--update` flag when syncing is recommended if using
+`--use-server-modtime`.
- rclone lsd remote:
+MD5 hashes are stored with blobs. However blobs that were uploaded in
+chunks only have an MD5 if the source remote was capable of MD5
+hashes, e.g. the local disk.
-Make a new directory
+### Performance
- rclone mkdir remote:directory
+When uploading large files, increasing the value of
+`--azureblob-upload-concurrency` will increase performance at the cost
+of using more memory. The default of 16 is set quite conservatively to
+use less memory. It maybe be necessary raise it to 64 or higher to
+fully utilize a 1 GBit/s link with a single file transfer.
-List the contents of a directory
+### Restricted filename characters
- rclone ls remote:directory
+In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters)
+the following characters are also replaced:
-Sync `/home/local/directory` to the remote path, deleting any
-excess files in the path.
+| Character | Value | Replacement |
+| --------- |:-----:|:-----------:|
+| / | 0x2F | ï¼ |
+| \ | 0x5C | ï¼¼ |
- rclone sync --interactive /home/local/directory remote:directory
+File names can also not end with the following characters.
+These only get replaced if they are the last character in the name:
-### Modified time
+| Character | Value | Replacement |
+| --------- |:-----:|:-----------:|
+| . | 0x2E | ． |
-Files support a modification time attribute with up to 1 second precision.
-Directories do not have a modification time, which is shown as "Jan 1 1970".
+Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8),
+as they can't be used in JSON strings.
-### Hash checksums
+### Authentication {#authentication}
-Hash sums use a custom Mail.ru algorithm based on SHA1.
-If file size is less than or equal to the SHA1 block size (20 bytes),
-its hash is simply its data right-padded with zero bytes.
-Hash sum of a larger file is computed as a SHA1 sum of the file data
-bytes concatenated with a decimal representation of the data length.
+There are a number of ways of supplying credentials for Azure Blob
+Storage. Rclone tries them in the order of the sections below.
-### Emptying Trash
+#### Env Auth
-Removing a file or directory actually moves it to the trash, which is not
-visible to rclone but can be seen in a web browser. The trashed file
-still occupies part of total quota. If you wish to empty your trash
-and free some quota, you can use the `rclone cleanup remote:` command,
-which will permanently delete all your trashed files.
-This command does not take any path arguments.
+If the `env_auth` config parameter is `true` then rclone will pull
+credentials from the environment or runtime.
-### Quota information
+It tries these authentication methods in this order:
-To view your current quota you can use the `rclone about remote:`
-command which will display your usage limit (quota) and the current usage.
+1. Environment Variables
+2. Managed Service Identity Credentials
+3. Azure CLI credentials (as used by the az tool)
-### Restricted filename characters
+These are described in the following sections
-In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters)
-the following characters are also replaced:
+##### Env Auth: 1. Environment Variables
-| Character | Value | Replacement |
-| --------- |:-----:|:-----------:|
-| " | 0x22 | ＂ |
-| * | 0x2A | ＊ |
-| : | 0x3A | ： |
-| < | 0x3C | ＜ |
-| > | 0x3E | ＞ |
-| ? | 0x3F | ？ |
-| \ | 0x5C | ï¼¼ |
-| \| | 0x7C | ｜ |
+If `env_auth` is set and environment variables are present rclone
+authenticates a service principal with a secret or certificate, or a
+user with a password, depending on which environment variable are set.
+It reads configuration from these variables, in the following order:
-Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8),
-as they can't be used in JSON strings.
+1. Service principal with client secret
+ - `AZURE_TENANT_ID`: ID of the service principal's tenant. Also called its "directory" ID.
+ - `AZURE_CLIENT_ID`: the service principal's client ID
+ - `AZURE_CLIENT_SECRET`: one of the service principal's client secrets
+2. Service principal with certificate
+ - `AZURE_TENANT_ID`: ID of the service principal's tenant. Also called its "directory" ID.
+ - `AZURE_CLIENT_ID`: the service principal's client ID
+ - `AZURE_CLIENT_CERTIFICATE_PATH`: path to a PEM or PKCS12 certificate file including the private key.
+ - `AZURE_CLIENT_CERTIFICATE_PASSWORD`: (optional) password for the certificate file.
+ - `AZURE_CLIENT_SEND_CERTIFICATE_CHAIN`: (optional) Specifies whether an authentication request will include an x5c header to support subject name / issuer based authentication. When set to "true" or "1", authentication requests include the x5c header.
+3. User with username and password
+ - `AZURE_TENANT_ID`: (optional) tenant to authenticate in. Defaults to "organizations".
+ - `AZURE_CLIENT_ID`: client ID of the application the user will authenticate to
+ - `AZURE_USERNAME`: a username (usually an email address)
+ - `AZURE_PASSWORD`: the user's password
+4. Workload Identity
+ - `AZURE_TENANT_ID`: Tenant to authenticate in.
+ - `AZURE_CLIENT_ID`: Client ID of the application the user will authenticate to.
+ - `AZURE_FEDERATED_TOKEN_FILE`: Path to projected service account token file.
+ - `AZURE_AUTHORITY_HOST`: Authority of an Azure Active Directory endpoint (default: login.microsoftonline.com).
-### Standard options
+##### Env Auth: 2. Managed Service Identity Credentials
-Here are the Standard options specific to mailru (Mail.ru Cloud).
+When using Managed Service Identity if the VM(SS) on which this
+program is running has a system-assigned identity, it will be used by
+default. If the resource has no system-assigned but exactly one
+user-assigned identity, the user-assigned identity will be used by
+default.
-#### --mailru-client-id
+If the resource has multiple user-assigned identities you will need to
+unset `env_auth` and set `use_msi` instead. See the [`use_msi`
+section](#use_msi).
-OAuth Client Id.
+##### Env Auth: 3. Azure CLI credentials (as used by the az tool)
-Leave blank normally.
+Credentials created with the `az` tool can be picked up using `env_auth`.
-Properties:
+For example if you were to login with a service principal like this:
-- Config: client_id
-- Env Var: RCLONE_MAILRU_CLIENT_ID
-- Type: string
-- Required: false
+ az login --service-principal -u XXX -p XXX --tenant XXX
-#### --mailru-client-secret
+Then you could access rclone resources like this:
-OAuth Client Secret.
+ rclone lsf :azureblob,env_auth,account=ACCOUNT:CONTAINER
-Leave blank normally.
+Or
-Properties:
+ rclone lsf --azureblob-env-auth --azureblob-account=ACCOUNT :azureblob:CONTAINER
-- Config: client_secret
-- Env Var: RCLONE_MAILRU_CLIENT_SECRET
-- Type: string
-- Required: false
+Which is analogous to using the `az` tool:
-#### --mailru-user
+ az storage blob list --container-name CONTAINER --account-name ACCOUNT --auth-mode login
-User name (usually email).
+#### Account and Shared Key
-Properties:
+This is the most straight forward and least flexible way. Just fill
+in the `account` and `key` lines and leave the rest blank.
-- Config: user
-- Env Var: RCLONE_MAILRU_USER
-- Type: string
-- Required: true
+#### SAS URL
-#### --mailru-pass
+This can be an account level SAS URL or container level SAS URL.
-Password.
+To use it leave `account` and `key` blank and fill in `sas_url`.
-This must be an app password - rclone will not work with your normal
-password. See the Configuration section in the docs for how to make an
-app password.
+An account level SAS URL or container level SAS URL can be obtained
+from the Azure portal or the Azure Storage Explorer. To get a
+container level SAS URL right click on a container in the Azure Blob
+explorer in the Azure portal.
+If you use a container level SAS URL, rclone operations are permitted
+only on a particular container, e.g.
-**NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/).
+ rclone ls azureblob:container
-Properties:
+You can also list the single container from the root. This will only
+show the container specified by the SAS URL.
-- Config: pass
-- Env Var: RCLONE_MAILRU_PASS
-- Type: string
-- Required: true
+ $ rclone lsd azureblob:
+ container/
-#### --mailru-speedup-enable
+Note that you can't see or access any other containers - this will
+fail
-Skip full upload if there is another file with same data hash.
+ rclone ls azureblob:othercontainer
-This feature is called "speedup" or "put by hash". It is especially efficient
-in case of generally available files like popular books, video or audio clips,
-because files are searched by hash in all accounts of all mailru users.
-It is meaningless and ineffective if source file is unique or encrypted.
-Please note that rclone may need local memory and disk space to calculate
-content hash in advance and decide whether full upload is required.
-Also, if rclone does not know file size in advance (e.g. in case of
-streaming or partial uploads), it will not even try this optimization.
+Container level SAS URLs are useful for temporarily allowing third
+parties access to a single container or putting credentials into an
+untrusted environment such as a CI build server.
-Properties:
+#### Service principal with client secret
-- Config: speedup_enable
-- Env Var: RCLONE_MAILRU_SPEEDUP_ENABLE
-- Type: bool
-- Default: true
-- Examples:
- - "true"
- - Enable
- - "false"
- - Disable
+If these variables are set, rclone will authenticate with a service principal with a client secret.
-### Advanced options
+- `tenant`: ID of the service principal's tenant. Also called its "directory" ID.
+- `client_id`: the service principal's client ID
+- `client_secret`: one of the service principal's client secrets
-Here are the Advanced options specific to mailru (Mail.ru Cloud).
+The credentials can also be placed in a file using the
+`service_principal_file` configuration option.
-#### --mailru-token
+#### Service principal with certificate
-OAuth Access Token as a JSON blob.
+If these variables are set, rclone will authenticate with a service principal with certificate.
-Properties:
+- `tenant`: ID of the service principal's tenant. Also called its "directory" ID.
+- `client_id`: the service principal's client ID
+- `client_certificate_path`: path to a PEM or PKCS12 certificate file including the private key.
+- `client_certificate_password`: (optional) password for the certificate file.
+- `client_send_certificate_chain`: (optional) Specifies whether an authentication request will include an x5c header to support subject name / issuer based authentication. When set to "true" or "1", authentication requests include the x5c header.
-- Config: token
-- Env Var: RCLONE_MAILRU_TOKEN
-- Type: string
-- Required: false
+**NB** `client_certificate_password` must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/).
-#### --mailru-auth-url
+#### User with username and password
-Auth server URL.
+If these variables are set, rclone will authenticate with username and password.
-Leave blank to use the provider defaults.
+- `tenant`: (optional) tenant to authenticate in. Defaults to "organizations".
+- `client_id`: client ID of the application the user will authenticate to
+- `username`: a username (usually an email address)
+- `password`: the user's password
-Properties:
+Microsoft doesn't recommend this kind of authentication, because it's
+less secure than other authentication flows. This method is not
+interactive, so it isn't compatible with any form of multi-factor
+authentication, and the application must already have user or admin
+consent. This credential can only authenticate work and school
+accounts; it can't authenticate Microsoft accounts.
-- Config: auth_url
-- Env Var: RCLONE_MAILRU_AUTH_URL
-- Type: string
-- Required: false
+**NB** `password` must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/).
-#### --mailru-token-url
+#### Managed Service Identity Credentials {#use_msi}
-Token server url.
+If `use_msi` is set then managed service identity credentials are
+used. This authentication only works when running in an Azure service.
+`env_auth` needs to be unset to use this.
-Leave blank to use the provider defaults.
+However if you have multiple user identities to choose from these must
+be explicitly specified using exactly one of the `msi_object_id`,
+`msi_client_id`, or `msi_mi_res_id` parameters.
-Properties:
+If none of `msi_object_id`, `msi_client_id`, or `msi_mi_res_id` is
+set, this is is equivalent to using `env_auth`.
-- Config: token_url
-- Env Var: RCLONE_MAILRU_TOKEN_URL
-- Type: string
-- Required: false
-#### --mailru-speedup-file-patterns
+### Standard options
-Comma separated list of file name patterns eligible for speedup (put by hash).
+Here are the Standard options specific to azureblob (Microsoft Azure Blob Storage).
-Patterns are case insensitive and can contain '*' or '?' meta characters.
+#### --azureblob-account
-Properties:
+Azure Storage Account Name.
-- Config: speedup_file_patterns
-- Env Var: RCLONE_MAILRU_SPEEDUP_FILE_PATTERNS
-- Type: string
-- Default: "*.mkv,*.avi,*.mp4,*.mp3,*.zip,*.gz,*.rar,*.pdf"
-- Examples:
- - ""
- - Empty list completely disables speedup (put by hash).
- - "*"
- - All files will be attempted for speedup.
- - "*.mkv,*.avi,*.mp4,*.mp3"
- - Only common audio/video files will be tried for put by hash.
- - "*.zip,*.gz,*.rar,*.pdf"
- - Only common archives or PDF books will be tried for speedup.
+Set this to the Azure Storage Account Name in use.
-#### --mailru-speedup-max-disk
+Leave blank to use SAS URL or Emulator, otherwise it needs to be set.
-This option allows you to disable speedup (put by hash) for large files.
+If this is blank and if env_auth is set it will be read from the
+environment variable `AZURE_STORAGE_ACCOUNT_NAME` if possible.
-Reason is that preliminary hashing can exhaust your RAM or disk space.
Properties:
-- Config: speedup_max_disk
-- Env Var: RCLONE_MAILRU_SPEEDUP_MAX_DISK
-- Type: SizeSuffix
-- Default: 3Gi
-- Examples:
- - "0"
- - Completely disable speedup (put by hash).
- - "1G"
- - Files larger than 1Gb will be uploaded directly.
- - "3G"
- - Choose this option if you have less than 3Gb free on local disk.
-
-#### --mailru-speedup-max-memory
-
-Files larger than the size given below will always be hashed on disk.
-
-Properties:
+- Config: account
+- Env Var: RCLONE_AZUREBLOB_ACCOUNT
+- Type: string
+- Required: false
-- Config: speedup_max_memory
-- Env Var: RCLONE_MAILRU_SPEEDUP_MAX_MEMORY
-- Type: SizeSuffix
-- Default: 32Mi
-- Examples:
- - "0"
- - Preliminary hashing will always be done in a temporary disk location.
- - "32M"
- - Do not dedicate more than 32Mb RAM for preliminary hashing.
- - "256M"
- - You have at most 256Mb RAM free for hash calculations.
+#### --azureblob-env-auth
-#### --mailru-check-hash
+Read credentials from runtime (environment variables, CLI or MSI).
-What should copy do if file checksum is mismatched or invalid.
+See the [authentication docs](/azureblob#authentication) for full info.
Properties:
-- Config: check_hash
-- Env Var: RCLONE_MAILRU_CHECK_HASH
+- Config: env_auth
+- Env Var: RCLONE_AZUREBLOB_ENV_AUTH
- Type: bool
-- Default: true
-- Examples:
- - "true"
- - Fail with error.
- - "false"
- - Ignore and continue.
+- Default: false
-#### --mailru-user-agent
+#### --azureblob-key
-HTTP user agent used internally by client.
+Storage Account Shared Key.
-Defaults to "rclone/VERSION" or "--user-agent" provided on command line.
+Leave blank to use SAS URL or Emulator.
Properties:
-- Config: user_agent
-- Env Var: RCLONE_MAILRU_USER_AGENT
+- Config: key
+- Env Var: RCLONE_AZUREBLOB_KEY
- Type: string
- Required: false
-#### --mailru-quirks
+#### --azureblob-sas-url
-Comma separated list of internal maintenance flags.
+SAS URL for container level access only.
-This option must not be used by an ordinary user. It is intended only to
-facilitate remote troubleshooting of backend issues. Strict meaning of
-flags is not documented and not guaranteed to persist between releases.
-Quirks will be removed when the backend grows stable.
-Supported quirks: atomicmkdir binlist unknowndirs
+Leave blank if using account/key or Emulator.
Properties:
-- Config: quirks
-- Env Var: RCLONE_MAILRU_QUIRKS
+- Config: sas_url
+- Env Var: RCLONE_AZUREBLOB_SAS_URL
- Type: string
- Required: false
-#### --mailru-encoding
-
-The encoding for the backend.
-
-See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info.
-
-Properties:
-
-- Config: encoding
-- Env Var: RCLONE_MAILRU_ENCODING
-- Type: MultiEncoder
-- Default: Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Del,Ctl,InvalidUtf8,Dot
+#### --azureblob-tenant
+ID of the service principal's tenant. Also called its directory ID.
+Set this if using
+- Service principal with client secret
+- Service principal with certificate
+- User with username and password
-## Limitations
-File size limits depend on your account. A single file size is limited by 2G
-for a free account and unlimited for paid tariffs. Please refer to the Mail.ru
-site for the total uploaded size limits.
+Properties:
-Note that Mailru is case insensitive so you can't have a file called
-"Hello.doc" and one called "hello.doc".
+- Config: tenant
+- Env Var: RCLONE_AZUREBLOB_TENANT
+- Type: string
+- Required: false
-# Mega
+#### --azureblob-client-id
-[Mega](https://mega.nz/) is a cloud storage and file hosting service
-known for its security feature where all files are encrypted locally
-before they are uploaded. This prevents anyone (including employees of
-Mega) from accessing the files without knowledge of the key used for
-encryption.
+The ID of the client in use.
-This is an rclone backend for Mega which supports the file transfer
-features of Mega using the same client side encryption.
+Set this if using
+- Service principal with client secret
+- Service principal with certificate
+- User with username and password
-Paths are specified as `remote:path`
-Paths may be as deep as required, e.g. `remote:directory/subdirectory`.
+Properties:
-## Configuration
+- Config: client_id
+- Env Var: RCLONE_AZUREBLOB_CLIENT_ID
+- Type: string
+- Required: false
-Here is an example of how to make a remote called `remote`. First run:
+#### --azureblob-client-secret
- rclone config
+One of the service principal's client secrets
-This will guide you through an interactive setup process:
-
-No remotes found, make a new one? n) New remote s) Set configuration password q) Quit config n/s/q> n name> remote Type of storage to configure. Choose a number from below, or type in your own value [snip] XX / Mega  "mega" [snip] Storage> mega User name user> you@example.com Password. y) Yes type in my own password g) Generate random password n) No leave this optional password blank y/g/n> y Enter the password: password: Confirm the password: password: Remote config -------------------- [remote] type = mega user = you@example.com pass = *** ENCRYPTED *** -------------------- y) Yes this is OK e) Edit this remote d) Delete this remote y/e/d> y
-
-**NOTE:** The encryption keys need to have been already generated after a regular login
-via the browser, otherwise attempting to use the credentials in `rclone` will fail.
+Set this if using
+- Service principal with client secret
-Once configured you can then use `rclone` like this,
-List directories in top level of your Mega
+Properties:
- rclone lsd remote:
+- Config: client_secret
+- Env Var: RCLONE_AZUREBLOB_CLIENT_SECRET
+- Type: string
+- Required: false
-List all the files in your Mega
+#### --azureblob-client-certificate-path
- rclone ls remote:
+Path to a PEM or PKCS12 certificate file including the private key.
-To copy a local directory to an Mega directory called backup
+Set this if using
+- Service principal with certificate
- rclone copy /home/source remote:backup
-### Modified time and hashes
+Properties:
-Mega does not support modification times or hashes yet.
+- Config: client_certificate_path
+- Env Var: RCLONE_AZUREBLOB_CLIENT_CERTIFICATE_PATH
+- Type: string
+- Required: false
-### Restricted filename characters
+#### --azureblob-client-certificate-password
-| Character | Value | Replacement |
-| --------- |:-----:|:-----------:|
-| NUL | 0x00 | †|
-| / | 0x2F | ï¼ |
+Password for the certificate file (optional).
-Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8),
-as they can't be used in JSON strings.
+Optionally set this if using
+- Service principal with certificate
-### Duplicated files
+And the certificate has a password.
-Mega can have two files with exactly the same name and path (unlike a
-normal file system).
-Duplicated files cause problems with the syncing and you will see
-messages in the log about duplicates.
+**NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/).
-Use `rclone dedupe` to fix duplicated files.
+Properties:
-### Failure to log-in
+- Config: client_certificate_password
+- Env Var: RCLONE_AZUREBLOB_CLIENT_CERTIFICATE_PASSWORD
+- Type: string
+- Required: false
-#### Object not found
+### Advanced options
-If you are connecting to your Mega remote for the first time,
-to test access and synchronization, you may receive an error such as
-
-Failed to create file system for "my-mega-remote:": couldn't login: Object (typically, node or user) not found
-
-The diagnostic steps often recommended in the [rclone forum](https://forum.rclone.org/search?q=mega)
-start with the **MEGAcmd** utility. Note that this refers to
-the official C++ command from https://github.com/meganz/MEGAcmd
-and not the go language built command from t3rm1n4l/megacmd
-that is no longer maintained.
+Here are the Advanced options specific to azureblob (Microsoft Azure Blob Storage).
-Follow the instructions for installing MEGAcmd and try accessing
-your remote as they recommend. You can establish whether or not
-you can log in using MEGAcmd, and obtain diagnostic information
-to help you, and search or work with others in the forum.
-
-MEGA CMD> login me@example.com Password: Fetching nodes ... Loading transfers from local cache Login complete as me@example.com me@example.com:/$
-
-Note that some have found issues with passwords containing special
-characters. If you can not log on with rclone, but MEGAcmd logs on
-just fine, then consider changing your password temporarily to
-pure alphanumeric characters, in case that helps.
+#### --azureblob-client-send-certificate-chain
+Send the certificate chain when using certificate auth.
-#### Repeated commands blocks access
+Specifies whether an authentication request will include an x5c header
+to support subject name / issuer based authentication. When set to
+true, authentication requests include the x5c header.
-Mega remotes seem to get blocked (reject logins) under "heavy use".
-We haven't worked out the exact blocking rules but it seems to be
-related to fast paced, successive rclone commands.
+Optionally set this if using
+- Service principal with certificate
-For example, executing this command 90 times in a row `rclone link
-remote:file` will cause the remote to become "blocked". This is not an
-abnormal situation, for example if you wish to get the public links of
-a directory with hundred of files... After more or less a week, the
-remote will remote accept rclone logins normally again.
-You can mitigate this issue by mounting the remote it with `rclone
-mount`. This will log-in when mounting and a log-out when unmounting
-only. You can also run `rclone rcd` and then use `rclone rc` to run
-the commands over the API to avoid logging in each time.
+Properties:
-Rclone does not currently close mega sessions (you can see them in the
-web interface), however closing the sessions does not solve the issue.
+- Config: client_send_certificate_chain
+- Env Var: RCLONE_AZUREBLOB_CLIENT_SEND_CERTIFICATE_CHAIN
+- Type: bool
+- Default: false
-If you space rclone commands by 3 seconds it will avoid blocking the
-remote. We haven't identified the exact blocking rules, so perhaps one
-could execute the command 80 times without waiting and avoid blocking
-by waiting 3 seconds, then continuing...
+#### --azureblob-username
-Note that this has been observed by trial and error and might not be
-set in stone.
+User name (usually an email address)
-Other tools seem not to produce this blocking effect, as they use a
-different working approach (state-based, using sessionIDs instead of
-log-in) which isn't compatible with the current stateless rclone
-approach.
+Set this if using
+- User with username and password
-Note that once blocked, the use of other tools (such as megacmd) is
-not a sure workaround: following megacmd login times have been
-observed in succession for blocked remote: 7 minutes, 20 min, 30min, 30
-min, 30min. Web access looks unaffected though.
-Investigation is continuing in relation to workarounds based on
-timeouts, pacers, retrials and tpslimits - if you discover something
-relevant, please post on the forum.
+Properties:
-So, if rclone was working nicely and suddenly you are unable to log-in
-and you are sure the user and the password are correct, likely you
-have got the remote blocked for a while.
+- Config: username
+- Env Var: RCLONE_AZUREBLOB_USERNAME
+- Type: string
+- Required: false
+#### --azureblob-password
-### Standard options
+The user's password
-Here are the Standard options specific to mega (Mega).
+Set this if using
+- User with username and password
-#### --mega-user
-User name.
+**NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/).
Properties:
-- Config: user
-- Env Var: RCLONE_MEGA_USER
+- Config: password
+- Env Var: RCLONE_AZUREBLOB_PASSWORD
- Type: string
-- Required: true
-
-#### --mega-pass
-
-Password.
-
-**NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/).
+- Required: false
-Properties:
+#### --azureblob-service-principal-file
-- Config: pass
-- Env Var: RCLONE_MEGA_PASS
-- Type: string
-- Required: true
+Path to file containing credentials for use with a service principal.
-### Advanced options
+Leave blank normally. Needed only if you want to use a service principal instead of interactive login.
-Here are the Advanced options specific to mega (Mega).
+ $ az ad sp create-for-rbac --name "<name>" \
+ --role "Storage Blob Data Owner" \
+ --scopes "/subscriptions/<subscription>/resourceGroups/<resource-group>/providers/Microsoft.Storage/storageAccounts/<storage-account>/blobServices/default/containers/<container>" \
+ > azure-principal.json
-#### --mega-debug
+See ["Create an Azure service principal"](https://docs.microsoft.com/en-us/cli/azure/create-an-azure-service-principal-azure-cli) and ["Assign an Azure role for access to blob data"](https://docs.microsoft.com/en-us/azure/storage/common/storage-auth-aad-rbac-cli) pages for more details.
-Output more debug from Mega.
+It may be more convenient to put the credentials directly into the
+rclone config file under the `client_id`, `tenant` and `client_secret`
+keys instead of setting `service_principal_file`.
-If this flag is set (along with -vv) it will print further debugging
-information from the mega backend.
Properties:
-- Config: debug
-- Env Var: RCLONE_MEGA_DEBUG
-- Type: bool
-- Default: false
+- Config: service_principal_file
+- Env Var: RCLONE_AZUREBLOB_SERVICE_PRINCIPAL_FILE
+- Type: string
+- Required: false
-#### --mega-hard-delete
+#### --azureblob-use-msi
-Delete files permanently rather than putting them into the trash.
+Use a managed service identity to authenticate (only works in Azure).
-Normally the mega backend will put all deletions into the trash rather
-than permanently deleting them. If you specify this then rclone will
-permanently delete objects instead.
+When true, use a [managed service identity](https://docs.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/)
+to authenticate to Azure Storage instead of a SAS token or account key.
+
+If the VM(SS) on which this program is running has a system-assigned identity, it will
+be used by default. If the resource has no system-assigned but exactly one user-assigned identity,
+the user-assigned identity will be used by default. If the resource has multiple user-assigned
+identities, the identity to use must be explicitly specified using exactly one of the msi_object_id,
+msi_client_id, or msi_mi_res_id parameters.
Properties:
-- Config: hard_delete
-- Env Var: RCLONE_MEGA_HARD_DELETE
+- Config: use_msi
+- Env Var: RCLONE_AZUREBLOB_USE_MSI
- Type: bool
- Default: false
-#### --mega-use-https
+#### --azureblob-msi-object-id
-Use HTTPS for transfers.
+Object ID of the user-assigned MSI to use, if any.
-MEGA uses plain text HTTP connections by default.
-Some ISPs throttle HTTP connections, this causes transfers to become very slow.
-Enabling this will force MEGA to use HTTPS for all transfers.
-HTTPS is normally not necessary since all data is already encrypted anyway.
-Enabling it will increase CPU usage and add network overhead.
+Leave blank if msi_client_id or msi_mi_res_id specified.
Properties:
-- Config: use_https
-- Env Var: RCLONE_MEGA_USE_HTTPS
-- Type: bool
-- Default: false
+- Config: msi_object_id
+- Env Var: RCLONE_AZUREBLOB_MSI_OBJECT_ID
+- Type: string
+- Required: false
-#### --mega-encoding
+#### --azureblob-msi-client-id
-The encoding for the backend.
+Object ID of the user-assigned MSI to use, if any.
-See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info.
+Leave blank if msi_object_id or msi_mi_res_id specified.
Properties:
-- Config: encoding
-- Env Var: RCLONE_MEGA_ENCODING
-- Type: MultiEncoder
-- Default: Slash,InvalidUtf8,Dot
+- Config: msi_client_id
+- Env Var: RCLONE_AZUREBLOB_MSI_CLIENT_ID
+- Type: string
+- Required: false
+#### --azureblob-msi-mi-res-id
+Azure resource ID of the user-assigned MSI to use, if any.
-### Process `killed`
+Leave blank if msi_client_id or msi_object_id specified.
-On accounts with large files or something else, memory usage can significantly increase when executing list/sync instructions. When running on cloud providers (like AWS with EC2), check if the instance type has sufficient memory/CPU to execute the commands. Use the resource monitoring tools to inspect after sending the commands. Look [at this issue](https://forum.rclone.org/t/rclone-with-mega-appears-to-work-only-in-some-accounts/40233/4).
+Properties:
-## Limitations
+- Config: msi_mi_res_id
+- Env Var: RCLONE_AZUREBLOB_MSI_MI_RES_ID
+- Type: string
+- Required: false
-This backend uses the [go-mega go library](https://github.com/t3rm1n4l/go-mega) which is an opensource
-go library implementing the Mega API. There doesn't appear to be any
-documentation for the mega protocol beyond the [mega C++ SDK](https://github.com/meganz/sdk) source code
-so there are likely quite a few errors still remaining in this library.
+#### --azureblob-use-emulator
-Mega allows duplicate files which may confuse rclone.
+Uses local storage emulator if provided as 'true'.
-# Memory
+Leave blank if using real azure storage endpoint.
-The memory backend is an in RAM backend. It does not persist its
-data - use the local backend for that.
+Properties:
-The memory backend behaves like a bucket-based remote (e.g. like
-s3). Because it has no parameters you can just use it with the
-`:memory:` remote name.
+- Config: use_emulator
+- Env Var: RCLONE_AZUREBLOB_USE_EMULATOR
+- Type: bool
+- Default: false
-## Configuration
+#### --azureblob-endpoint
-You can configure it as a remote like this with `rclone config` too if
-you want to:
-
-No remotes found, make a new one? n) New remote s) Set configuration password q) Quit config n/s/q> n name> remote Type of storage to configure. Enter a string value. Press Enter for the default (""). Choose a number from below, or type in your own value [snip] XX / Memory  "memory" [snip] Storage> memory ** See help for memory backend at: https://rclone.org/memory/ **
-Remote config
-
-
-
-[remote]
-
-
-type = memory
-
-
-
-
-- Yes this is OK (default)
-- Edit this remote
-- Delete this remote y/e/d> y
-
-
-Because the memory backend isn't persistent it is most useful for
-testing or with an rclone server or rclone mount, e.g.
+Endpoint for the service.
- rclone mount :memory: /mnt/tmp
- rclone serve webdav :memory:
- rclone serve sftp :memory:
+Leave blank normally.
-### Modified time and hashes
+Properties:
-The memory backend supports MD5 hashes and modification times accurate to 1 nS.
+- Config: endpoint
+- Env Var: RCLONE_AZUREBLOB_ENDPOINT
+- Type: string
+- Required: false
-### Restricted filename characters
+#### --azureblob-upload-cutoff
-The memory backend replaces the [default restricted characters
-set](https://rclone.org/overview/#restricted-characters).
+Cutoff for switching to chunked upload (<= 256 MiB) (deprecated).
+Properties:
+- Config: upload_cutoff
+- Env Var: RCLONE_AZUREBLOB_UPLOAD_CUTOFF
+- Type: string
+- Required: false
+#### --azureblob-chunk-size
-# Akamai NetStorage
+Upload chunk size.
-Paths are specified as `remote:`
-You may put subdirectories in too, e.g. `remote:/path/to/dir`.
-If you have a CP code you can use that as the folder after the domain such as \<domain>\/\<cpcode>\/\<internal directories within cpcode>.
+Note that this is stored in memory and there may be up to
+"--transfers" * "--azureblob-upload-concurrency" chunks stored at once
+in memory.
-For example, this is commonly configured with or without a CP code:
-* **With a CP code**. `[your-domain-prefix]-nsu.akamaihd.net/123456/subdirectory/`
-* **Without a CP code**. `[your-domain-prefix]-nsu.akamaihd.net`
+Properties:
+- Config: chunk_size
+- Env Var: RCLONE_AZUREBLOB_CHUNK_SIZE
+- Type: SizeSuffix
+- Default: 4Mi
-See all buckets
- rclone lsd remote:
-The initial setup for Netstorage involves getting an account and secret. Use `rclone config` to walk you through the setup process.
+#### --azureblob-upload-concurrency
-## Configuration
+Concurrency for multipart uploads.
-Here's an example of how to make a remote called `ns1`.
+This is the number of chunks of the same file that are uploaded
+concurrently.
-1. To begin the interactive configuration process, enter this command:
-
-rclone config
-
-2. Type `n` to create a new remote.
-
-
-- New remote
-- Delete remote
-- Quit config e/n/d/q> n
-
-
-3. For this example, enter `ns1` when you reach the name> prompt.
-
-name> ns1
-
-4. Enter `netstorage` as the type of storage to configure.
-
-Type of storage to configure. Enter a string value. Press Enter for the default (""). Choose a number from below, or type in your own value XX / NetStorage  "netstorage" Storage> netstorage
-
-5. Select between the HTTP or HTTPS protocol. Most users should choose HTTPS, which is the default. HTTP is provided primarily for debugging purposes.
+If you are uploading small numbers of large files over high-speed
+links and these uploads do not fully utilize your bandwidth, then
+increasing this may help to speed up the transfers.
-
-Enter a string value. Press Enter for the default (""). Choose a number from below, or type in your own value 1 / HTTP protocol  "http" 2 / HTTPS protocol  "https" protocol> 1
-
-6. Specify your NetStorage host, CP code, and any necessary content paths using this format: `<domain>/<cpcode>/<content>/`
-
-Enter a string value. Press Enter for the default (""). host> baseball-nsu.akamaihd.net/123456/content/
-
-7. Set the netstorage account name
-Enter a string value. Press Enter for the default (""). account> username
-
-8. Set the Netstorage account secret/G2O key which will be used for authentication purposes. Select the `y` option to set your own password then enter your secret.
-Note: The secret is stored in the `rclone.conf` file with hex-encoded encryption.
-
-
-- Yes type in my own password
-- Generate random password y/g> y Enter the password: password: Confirm the password: password:
-
-
-9. View the summary and confirm your remote configuration.
-
-[ns1] type = netstorage protocol = http host = baseball-nsu.akamaihd.net/123456/content/ account = username secret = *** ENCRYPTED *** -------------------- y) Yes this is OK (default) e) Edit this remote d) Delete this remote y/e/d> y
-
-This remote is called `ns1` and can now be used.
+In tests, upload speed increases almost linearly with upload
+concurrency. For example to fill a gigabit pipe it may be necessary to
+raise this to 64. Note that this will use more memory.
-## Example operations
+Note that chunks are stored in memory and there may be up to
+"--transfers" * "--azureblob-upload-concurrency" chunks stored at once
+in memory.
-Get started with rclone and NetStorage with these examples. For additional rclone commands, visit https://rclone.org/commands/.
+Properties:
-### See contents of a directory in your project
+- Config: upload_concurrency
+- Env Var: RCLONE_AZUREBLOB_UPLOAD_CONCURRENCY
+- Type: int
+- Default: 16
- rclone lsd ns1:/974012/testing/
+#### --azureblob-list-chunk
-### Sync the contents local with remote
+Size of blob list.
- rclone sync . ns1:/974012/testing/
+This sets the number of blobs requested in each listing chunk. Default
+is the maximum, 5000. "List blobs" requests are permitted 2 minutes
+per megabyte to complete. If an operation is taking longer than 2
+minutes per megabyte on average, it will time out (
+[source](https://docs.microsoft.com/en-us/rest/api/storageservices/setting-timeouts-for-blob-service-operations#exceptions-to-default-timeout-interval)
+). This can be used to limit the number of blobs items to return, to
+avoid the time out.
-### Upload local content to remote
- rclone copy notes.txt ns1:/974012/testing/
+Properties:
-### Delete content on remote
- rclone delete ns1:/974012/testing/notes.txt
+- Config: list_chunk
+- Env Var: RCLONE_AZUREBLOB_LIST_CHUNK
+- Type: int
+- Default: 5000
-### Move or copy content between CP codes.
+#### --azureblob-access-tier
-Your credentials must have access to two CP codes on the same remote. You can't perform operations between different remotes.
+Access tier of blob: hot, cool, cold or archive.
- rclone move ns1:/974012/testing/notes.txt ns1:/974450/testing2/
+Archived blobs can be restored by setting access tier to hot, cool or
+cold. Leave blank if you intend to use default access tier, which is
+set at account level
-## Features
+If there is no "access tier" specified, rclone doesn't apply any tier.
+rclone performs "Set Tier" operation on blobs while uploading, if objects
+are not modified, specifying "access tier" to new one will have no effect.
+If blobs are in "archive tier" at remote, trying to perform data transfer
+operations from remote will not be allowed. User should first restore by
+tiering blob to "Hot", "Cool" or "Cold".
-### Symlink Support
+Properties:
-The Netstorage backend changes the rclone `--links, -l` behavior. When uploading, instead of creating the .rclonelink file, use the "symlink" API in order to create the corresponding symlink on the remote. The .rclonelink file will not be created, the upload will be intercepted and only the symlink file that matches the source file name with no suffix will be created on the remote.
+- Config: access_tier
+- Env Var: RCLONE_AZUREBLOB_ACCESS_TIER
+- Type: string
+- Required: false
-This will effectively allow commands like copy/copyto, move/moveto and sync to upload from local to remote and download from remote to local directories with symlinks. Due to internal rclone limitations, it is not possible to upload an individual symlink file to any remote backend. You can always use the "backend symlink" command to create a symlink on the NetStorage server, refer to "symlink" section below.
+#### --azureblob-archive-tier-delete
-Individual symlink files on the remote can be used with the commands like "cat" to print the destination name, or "delete" to delete symlink, or copy, copy/to and move/moveto to download from the remote to local. Note: individual symlink files on the remote should be specified including the suffix .rclonelink.
+Delete archive tier blobs before overwriting.
-**Note**: No file with the suffix .rclonelink should ever exist on the server since it is not possible to actually upload/create a file with .rclonelink suffix with rclone, it can only exist if it is manually created through a non-rclone method on the remote.
+Archive tier blobs cannot be updated. So without this flag, if you
+attempt to update an archive tier blob, then rclone will produce the
+error:
-### Implicit vs. Explicit Directories
+ can't update archive tier blob without --azureblob-archive-tier-delete
-With NetStorage, directories can exist in one of two forms:
+With this flag set then before rclone attempts to overwrite an archive
+tier blob, it will delete the existing blob before uploading its
+replacement. This has the potential for data loss if the upload fails
+(unlike updating a normal blob) and also may cost more since deleting
+archive tier blobs early may be chargable.
-1. **Explicit Directory**. This is an actual, physical directory that you have created in a storage group.
-2. **Implicit Directory**. This refers to a directory within a path that has not been physically created. For example, during upload of a file, nonexistent subdirectories can be specified in the target path. NetStorage creates these as "implicit." While the directories aren't physically created, they exist implicitly and the noted path is connected with the uploaded file.
-Rclone will intercept all file uploads and mkdir commands for the NetStorage remote and will explicitly issue the mkdir command for each directory in the uploading path. This will help with the interoperability with the other Akamai services such as SFTP and the Content Management Shell (CMShell). Rclone will not guarantee correctness of operations with implicit directories which might have been created as a result of using an upload API directly.
+Properties:
-### `--fast-list` / ListR support
+- Config: archive_tier_delete
+- Env Var: RCLONE_AZUREBLOB_ARCHIVE_TIER_DELETE
+- Type: bool
+- Default: false
-NetStorage remote supports the ListR feature by using the "list" NetStorage API action to return a lexicographical list of all objects within the specified CP code, recursing into subdirectories as they're encountered.
+#### --azureblob-disable-checksum
-* **Rclone will use the ListR method for some commands by default**. Commands such as `lsf -R` will use ListR by default. To disable this, include the `--disable listR` option to use the non-recursive method of listing objects.
+Don't store MD5 checksum with object metadata.
-* **Rclone will not use the ListR method for some commands**. Commands such as `sync` don't use ListR by default. To force using the ListR method, include the `--fast-list` option.
+Normally rclone will calculate the MD5 checksum of the input before
+uploading it so it can add it to metadata on the object. This is great
+for data integrity checking but can cause long delays for large files
+to start uploading.
-There are pros and cons of using the ListR method, refer to [rclone documentation](https://rclone.org/docs/#fast-list). In general, the sync command over an existing deep tree on the remote will run faster with the "--fast-list" flag but with extra memory usage as a side effect. It might also result in higher CPU utilization but the whole task can be completed faster.
+Properties:
-**Note**: There is a known limitation that "lsf -R" will display number of files in the directory and directory size as -1 when ListR method is used. The workaround is to pass "--disable listR" flag if these numbers are important in the output.
+- Config: disable_checksum
+- Env Var: RCLONE_AZUREBLOB_DISABLE_CHECKSUM
+- Type: bool
+- Default: false
-### Purge
+#### --azureblob-memory-pool-flush-time
-NetStorage remote supports the purge feature by using the "quick-delete" NetStorage API action. The quick-delete action is disabled by default for security reasons and can be enabled for the account through the Akamai portal. Rclone will first try to use quick-delete action for the purge command and if this functionality is disabled then will fall back to a standard delete method.
+How often internal memory buffer pools will be flushed. (no longer used)
-**Note**: Read the [NetStorage Usage API](https://learn.akamai.com/en-us/webhelp/netstorage/netstorage-http-api-developer-guide/GUID-15836617-9F50-405A-833C-EA2556756A30.html) for considerations when using "quick-delete". In general, using quick-delete method will not delete the tree immediately and objects targeted for quick-delete may still be accessible.
+Properties:
+- Config: memory_pool_flush_time
+- Env Var: RCLONE_AZUREBLOB_MEMORY_POOL_FLUSH_TIME
+- Type: Duration
+- Default: 1m0s
-### Standard options
+#### --azureblob-memory-pool-use-mmap
-Here are the Standard options specific to netstorage (Akamai NetStorage).
+Whether to use mmap buffers in internal memory pool. (no longer used)
-#### --netstorage-host
+Properties:
-Domain+path of NetStorage host to connect to.
+- Config: memory_pool_use_mmap
+- Env Var: RCLONE_AZUREBLOB_MEMORY_POOL_USE_MMAP
+- Type: bool
+- Default: false
-Format should be `<domain>/<internal folders>`
+#### --azureblob-encoding
+
+The encoding for the backend.
+
+See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info.
Properties:
-- Config: host
-- Env Var: RCLONE_NETSTORAGE_HOST
-- Type: string
-- Required: true
+- Config: encoding
+- Env Var: RCLONE_AZUREBLOB_ENCODING
+- Type: Encoding
+- Default: Slash,BackSlash,Del,Ctl,RightPeriod,InvalidUtf8
-#### --netstorage-account
+#### --azureblob-public-access
-Set the NetStorage account name
+Public access level of a container: blob or container.
Properties:
-- Config: account
-- Env Var: RCLONE_NETSTORAGE_ACCOUNT
+- Config: public_access
+- Env Var: RCLONE_AZUREBLOB_PUBLIC_ACCESS
- Type: string
-- Required: true
+- Required: false
+- Examples:
+ - ""
+ - The container and its blobs can be accessed only with an authorized request.
+ - It's a default value.
+ - "blob"
+ - Blob data within this container can be read via anonymous request.
+ - "container"
+ - Allow full public read access for container and blob data.
-#### --netstorage-secret
+#### --azureblob-directory-markers
-Set the NetStorage account secret/G2O key for authentication.
+Upload an empty object with a trailing slash when a new directory is created
-Please choose the 'y' option to set your own password then enter your secret.
+Empty folders are unsupported for bucket based remotes, this option
+creates an empty object ending with "/", to persist the folder.
-**NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/).
+This object also has the metadata "hdi_isfolder = true" to conform to
+the Microsoft standard.
+
Properties:
-- Config: secret
-- Env Var: RCLONE_NETSTORAGE_SECRET
-- Type: string
-- Required: true
-
-### Advanced options
+- Config: directory_markers
+- Env Var: RCLONE_AZUREBLOB_DIRECTORY_MARKERS
+- Type: bool
+- Default: false
-Here are the Advanced options specific to netstorage (Akamai NetStorage).
+#### --azureblob-no-check-container
-#### --netstorage-protocol
+If set, don't attempt to check the container exists or create it.
-Select between HTTP or HTTPS protocol.
+This can be useful when trying to minimise the number of transactions
+rclone does if you know the container exists already.
-Most users should choose HTTPS, which is the default.
-HTTP is provided primarily for debugging purposes.
Properties:
-- Config: protocol
-- Env Var: RCLONE_NETSTORAGE_PROTOCOL
-- Type: string
-- Default: "https"
-- Examples:
- - "http"
- - HTTP protocol
- - "https"
- - HTTPS protocol
+- Config: no_check_container
+- Env Var: RCLONE_AZUREBLOB_NO_CHECK_CONTAINER
+- Type: bool
+- Default: false
-## Backend commands
+#### --azureblob-no-head-object
-Here are the commands specific to the netstorage backend.
+If set, do not do HEAD before GET when getting objects.
-Run them with
+Properties:
- rclone backend COMMAND remote:
+- Config: no_head_object
+- Env Var: RCLONE_AZUREBLOB_NO_HEAD_OBJECT
+- Type: bool
+- Default: false
-The help below will explain what arguments each command takes.
-See the [backend](https://rclone.org/commands/rclone_backend/) command for more
-info on how to pass options and arguments.
-These can be run on a running backend using the rc command
-[backend/command](https://rclone.org/rc/#backend-command).
+### Custom upload headers
-### du
+You can set custom upload headers with the `--header-upload` flag.
-Return disk usage information for a specified directory
+- Cache-Control
+- Content-Disposition
+- Content-Encoding
+- Content-Language
+- Content-Type
- rclone backend du remote: [options] [<arguments>+]
+Eg `--header-upload "Content-Type: text/potato"`
-The usage information returned, includes the targeted directory as well as all
-files stored in any sub-directories that may exist.
+## Limitations
-### symlink
+MD5 sums are only uploaded with chunked files if the source has an MD5
+sum. This will always be the case for a local to azure copy.
-You can create a symbolic link in ObjectStore with the symlink action.
+`rclone about` is not supported by the Microsoft Azure Blob storage backend. Backends without
+this capability cannot determine free space for an rclone mount or
+use policy `mfs` (most free space) as a member of an rclone union
+remote.
- rclone backend symlink remote: [options] [<arguments>+]
+See [List of backends that do not support rclone about](https://rclone.org/overview/#optional-features) and [rclone about](https://rclone.org/commands/rclone_about/)
-The desired path location (including applicable sub-directories) ending in
-the object that will be the target of the symlink (for example, /links/mylink).
-Include the file extension for the object, if applicable.
-`rclone backend symlink <src> <path>`
+## Azure Storage Emulator Support
+You can run rclone with the storage emulator (usually _azurite_).
+To do this, just set up a new remote with `rclone config` following
+the instructions in the introduction and set `use_emulator` in the
+advanced settings as `true`. You do not need to provide a default
+account name nor an account key. But you can override them in the
+`account` and `key` options. (Prior to v1.61 they were hard coded to
+_azurite_'s `devstoreaccount1`.)
-# Microsoft Azure Blob Storage
+Also, if you want to access a storage emulator instance running on a
+different machine, you can override the `endpoint` parameter in the
+advanced settings, setting it to
+`http(s)://<host>:<port>/devstoreaccount1`
+(e.g. `http://10.254.2.5:10000/devstoreaccount1`).
-Paths are specified as `remote:container` (or `remote:` for the `lsd`
-command.) You may put subdirectories in too, e.g.
-`remote:container/path/to/dir`.
+# Microsoft Azure Files Storage
+
+Paths are specified as `remote:` You may put subdirectories in too,
+e.g. `remote:path/to/dir`.
## Configuration
-Here is an example of making a Microsoft Azure Blob Storage
+Here is an example of making a Microsoft Azure Files Storage
configuration. For a remote called `remote`. First run:
rclone config
This will guide you through an interactive setup process:
-No remotes found, make a new one? n) New remote s) Set configuration password q) Quit config n/s/q> n name> remote Type of storage to configure. Choose a number from below, or type in your own value [snip] XX / Microsoft Azure Blob Storage  "azureblob" [snip] Storage> azureblob Storage Account Name account> account_name Storage Account Key key> base64encodedkey== Endpoint for the service - leave blank normally. endpoint> Remote config -------------------- [remote] account = account_name key = base64encodedkey== endpoint = -------------------- y) Yes this is OK e) Edit this remote d) Delete this remote y/e/d> y
+No remotes found, make a new one? n) New remote s) Set configuration password q) Quit config n/s/q> n name> remote Type of storage to configure. Choose a number from below, or type in your own value [snip] XX / Microsoft Azure Files Storage  "azurefiles" [snip]
+Option account. Azure Storage Account Name. Set this to the Azure Storage Account Name in use. Leave blank to use SAS URL or connection string, otherwise it needs to be set. If this is blank and if env_auth is set it will be read from the environment variable AZURE_STORAGE_ACCOUNT_NAME if possible. Enter a value. Press Enter to leave empty. account> account_name
+Option share_name. Azure Files Share Name. This is required and is the name of the share to access. Enter a value. Press Enter to leave empty. share_name> share_name
+Option env_auth. Read credentials from runtime (environment variables, CLI or MSI). See the authentication docs for full info. Enter a boolean value (true or false). Press Enter for the default (false). env_auth>
+Option key. Storage Account Shared Key. Leave blank to use SAS URL or connection string. Enter a value. Press Enter to leave empty. key> base64encodedkey==
+Option sas_url. SAS URL. Leave blank if using account/key or connection string. Enter a value. Press Enter to leave empty. sas_url>
+Option connection_string. Azure Files Connection String. Enter a value. Press Enter to leave empty. connection_string> [snip]
+Configuration complete. Options: - type: azurefiles - account: account_name - share_name: share_name - key: base64encodedkey== Keep this "remote" remote? y) Yes this is OK (default) e) Edit this remote d) Delete this remote y/e/d>
-See all containers
-
- rclone lsd remote:
+Once configured you can use rclone.
-Make a new container
+See all files in the top level:
- rclone mkdir remote:container
+ rclone lsf remote:
-List the contents of a container
+Make a new directory in the root:
- rclone ls remote:container
+ rclone mkdir remote:dir
-Sync `/home/local/directory` to the remote container, deleting any excess
-files in the container.
+Recursively List the contents:
- rclone sync --interactive /home/local/directory remote:container
+ rclone ls remote:
-### --fast-list
+Sync `/home/local/directory` to the remote directory, deleting any
+excess files in the directory.
-This remote supports `--fast-list` which allows you to use fewer
-transactions in exchange for more memory. See the [rclone
-docs](https://rclone.org/docs/#fast-list) for more details.
+ rclone sync --interactive /home/local/directory remote:dir
### Modified time
-The modified time is stored as metadata on the object with the `mtime`
-key. It is stored using RFC3339 Format time with nanosecond
-precision. The metadata is supplied during directory listings so
-there is no performance overhead to using it.
-
-If you wish to use the Azure standard `LastModified` time stored on
-the object as the modified time, then use the `--use-server-modtime`
-flag. Note that rclone can't set `LastModified`, so using the
-`--update` flag when syncing is recommended if using
-`--use-server-modtime`.
+The modified time is stored as Azure standard `LastModified` time on
+files
### Performance
When uploading large files, increasing the value of
-`--azureblob-upload-concurrency` will increase performance at the cost
+`--azurefiles-upload-concurrency` will increase performance at the cost
of using more memory. The default of 16 is set quite conservatively to
use less memory. It maybe be necessary raise it to 64 or higher to
fully utilize a 1 GBit/s link with a single file transfer.
@@ -28049,8 +28219,14 @@ Synology C2 Object Storage
| Character | Value | Replacement |
| --------- |:-----:|:-----------:|
-| / | 0x2F | ï¼ |
-| \ | 0x5C | ï¼¼ |
+| " | 0x22 | ＂ |
+| * | 0x2A | ＊ |
+| : | 0x3A | ： |
+| < | 0x3C | ＜ |
+| > | 0x3E | ＞ |
+| ? | 0x3F | ？ |
+| \ | 0x5C | ï¼¼ |
+| \| | 0x7C | ｜ |
File names can also not end with the following characters.
These only get replaced if they are the last character in the name:
@@ -28064,13 +28240,12 @@ Synology C2 Object Storage
### Hashes
-MD5 hashes are stored with blobs. However blobs that were uploaded in
-chunks only have an MD5 if the source remote was capable of MD5
-hashes, e.g. the local disk.
+MD5 hashes are stored with files. Not all files will have MD5 hashes
+as these have to be uploaded with the file.
### Authentication {#authentication}
-There are a number of ways of supplying credentials for Azure Blob
+There are a number of ways of supplying credentials for Azure Files
Storage. Rclone tries them in the order of the sections below.
#### Env Auth
@@ -28137,15 +28312,11 @@ Synology C2 Object Storage
Then you could access rclone resources like this:
- rclone lsf :azureblob,env_auth,account=ACCOUNT:CONTAINER
+ rclone lsf :azurefiles,env_auth,account=ACCOUNT:
Or
- rclone lsf --azureblob-env-auth --azureblob-account=ACCOUNT :azureblob:CONTAINER
-
-Which is analogous to using the `az` tool:
-
- az storage blob list --container-name CONTAINER --account-name ACCOUNT --auth-mode login
+ rclone lsf --azurefiles-env-auth --azurefiles-account=ACCOUNT :azurefiles:
#### Account and Shared Key
@@ -28154,34 +28325,11 @@ Synology C2 Object Storage
#### SAS URL
-This can be an account level SAS URL or container level SAS URL.
-
-To use it leave `account` and `key` blank and fill in `sas_url`.
-
-An account level SAS URL or container level SAS URL can be obtained
-from the Azure portal or the Azure Storage Explorer. To get a
-container level SAS URL right click on a container in the Azure Blob
-explorer in the Azure portal.
-
-If you use a container level SAS URL, rclone operations are permitted
-only on a particular container, e.g.
-
- rclone ls azureblob:container
-
-You can also list the single container from the root. This will only
-show the container specified by the SAS URL.
-
- $ rclone lsd azureblob:
- container/
-
-Note that you can't see or access any other containers - this will
-fail
+To use it leave `account`, `key` and `connection_string` blank and fill in `sas_url`.
- rclone ls azureblob:othercontainer
+#### Connection String
-Container level SAS URLs are useful for temporarily allowing third
-parties access to a single container or putting credentials into an
-untrusted environment such as a CI build server.
+To use it leave `account`, `key` and "sas_url" blank and fill in `connection_string`.
#### Service principal with client secret
@@ -28240,15 +28388,15 @@ Synology C2 Object Storage
### Standard options
-Here are the Standard options specific to azureblob (Microsoft Azure Blob Storage).
+Here are the Standard options specific to azurefiles (Microsoft Azure Files).
-#### --azureblob-account
+#### --azurefiles-account
Azure Storage Account Name.
Set this to the Azure Storage Account Name in use.
-Leave blank to use SAS URL or Emulator, otherwise it needs to be set.
+Leave blank to use SAS URL or connection string, otherwise it needs to be set.
If this is blank and if env_auth is set it will be read from the
environment variable `AZURE_STORAGE_ACCOUNT_NAME` if possible.
@@ -28257,50 +28405,75 @@ Synology C2 Object Storage
Properties:
- Config: account
-- Env Var: RCLONE_AZUREBLOB_ACCOUNT
+- Env Var: RCLONE_AZUREFILES_ACCOUNT
- Type: string
- Required: false
-#### --azureblob-env-auth
+#### --azurefiles-share-name
+
+Azure Files Share Name.
+
+This is required and is the name of the share to access.
+
+
+Properties:
+
+- Config: share_name
+- Env Var: RCLONE_AZUREFILES_SHARE_NAME
+- Type: string
+- Required: false
+
+#### --azurefiles-env-auth
Read credentials from runtime (environment variables, CLI or MSI).
-See the [authentication docs](/azureblob#authentication) for full info.
+See the [authentication docs](/azurefiles#authentication) for full info.
Properties:
- Config: env_auth
-- Env Var: RCLONE_AZUREBLOB_ENV_AUTH
+- Env Var: RCLONE_AZUREFILES_ENV_AUTH
- Type: bool
- Default: false
-#### --azureblob-key
+#### --azurefiles-key
Storage Account Shared Key.
-Leave blank to use SAS URL or Emulator.
+Leave blank to use SAS URL or connection string.
Properties:
- Config: key
-- Env Var: RCLONE_AZUREBLOB_KEY
+- Env Var: RCLONE_AZUREFILES_KEY
- Type: string
- Required: false
-#### --azureblob-sas-url
+#### --azurefiles-sas-url
-SAS URL for container level access only.
+SAS URL.
-Leave blank if using account/key or Emulator.
+Leave blank if using account/key or connection string.
Properties:
- Config: sas_url
-- Env Var: RCLONE_AZUREBLOB_SAS_URL
+- Env Var: RCLONE_AZUREFILES_SAS_URL
- Type: string
- Required: false
-#### --azureblob-tenant
+#### --azurefiles-connection-string
+
+Azure Files Connection String.
+
+Properties:
+
+- Config: connection_string
+- Env Var: RCLONE_AZUREFILES_CONNECTION_STRING
+- Type: string
+- Required: false
+
+#### --azurefiles-tenant
ID of the service principal's tenant. Also called its directory ID.
@@ -28313,11 +28486,11 @@ Synology C2 Object Storage
Properties:
- Config: tenant
-- Env Var: RCLONE_AZUREBLOB_TENANT
+- Env Var: RCLONE_AZUREFILES_TENANT
- Type: string
- Required: false
-#### --azureblob-client-id
+#### --azurefiles-client-id
The ID of the client in use.
@@ -28330,11 +28503,11 @@ Synology C2 Object Storage
Properties:
- Config: client_id
-- Env Var: RCLONE_AZUREBLOB_CLIENT_ID
+- Env Var: RCLONE_AZUREFILES_CLIENT_ID
- Type: string
- Required: false
-#### --azureblob-client-secret
+#### --azurefiles-client-secret
One of the service principal's client secrets
@@ -28345,11 +28518,11 @@ Synology C2 Object Storage
Properties:
- Config: client_secret
-- Env Var: RCLONE_AZUREBLOB_CLIENT_SECRET
+- Env Var: RCLONE_AZUREFILES_CLIENT_SECRET
- Type: string
- Required: false
-#### --azureblob-client-certificate-path
+#### --azurefiles-client-certificate-path
Path to a PEM or PKCS12 certificate file including the private key.
@@ -28360,11 +28533,11 @@ Synology C2 Object Storage
Properties:
- Config: client_certificate_path
-- Env Var: RCLONE_AZUREBLOB_CLIENT_CERTIFICATE_PATH
+- Env Var: RCLONE_AZUREFILES_CLIENT_CERTIFICATE_PATH
- Type: string
- Required: false
-#### --azureblob-client-certificate-password
+#### --azurefiles-client-certificate-password
Password for the certificate file (optional).
@@ -28379,15 +28552,15 @@ Synology C2 Object Storage
Properties:
- Config: client_certificate_password
-- Env Var: RCLONE_AZUREBLOB_CLIENT_CERTIFICATE_PASSWORD
+- Env Var: RCLONE_AZUREFILES_CLIENT_CERTIFICATE_PASSWORD
- Type: string
- Required: false
### Advanced options
-Here are the Advanced options specific to azureblob (Microsoft Azure Blob Storage).
+Here are the Advanced options specific to azurefiles (Microsoft Azure Files).
-#### --azureblob-client-send-certificate-chain
+#### --azurefiles-client-send-certificate-chain
Send the certificate chain when using certificate auth.
@@ -28402,11 +28575,11 @@ Synology C2 Object Storage
Properties:
- Config: client_send_certificate_chain
-- Env Var: RCLONE_AZUREBLOB_CLIENT_SEND_CERTIFICATE_CHAIN
+- Env Var: RCLONE_AZUREFILES_CLIENT_SEND_CERTIFICATE_CHAIN
- Type: bool
- Default: false
-#### --azureblob-username
+#### --azurefiles-username
User name (usually an email address)
@@ -28417,11 +28590,11 @@ Synology C2 Object Storage
Properties:
- Config: username
-- Env Var: RCLONE_AZUREBLOB_USERNAME
+- Env Var: RCLONE_AZUREFILES_USERNAME
- Type: string
- Required: false
-#### --azureblob-password
+#### --azurefiles-password
The user's password
@@ -28434,22 +28607,24 @@ Synology C2 Object Storage
Properties:
- Config: password
-- Env Var: RCLONE_AZUREBLOB_PASSWORD
+- Env Var: RCLONE_AZUREFILES_PASSWORD
- Type: string
- Required: false
-#### --azureblob-service-principal-file
+#### --azurefiles-service-principal-file
Path to file containing credentials for use with a service principal.
Leave blank normally. Needed only if you want to use a service principal instead of interactive login.
$ az ad sp create-for-rbac --name "<name>" \
- --role "Storage Blob Data Owner" \
+ --role "Storage Files Data Owner" \
--scopes "/subscriptions/<subscription>/resourceGroups/<resource-group>/providers/Microsoft.Storage/storageAccounts/<storage-account>/blobServices/default/containers/<container>" \
> azure-principal.json
-See ["Create an Azure service principal"](https://docs.microsoft.com/en-us/cli/azure/create-an-azure-service-principal-azure-cli) and ["Assign an Azure role for access to blob data"](https://docs.microsoft.com/en-us/azure/storage/common/storage-auth-aad-rbac-cli) pages for more details.
+See ["Create an Azure service principal"](https://docs.microsoft.com/en-us/cli/azure/create-an-azure-service-principal-azure-cli) and ["Assign an Azure role for access to files data"](https://docs.microsoft.com/en-us/azure/storage/common/storage-auth-aad-rbac-cli) pages for more details.
+
+**NB** this section needs updating for Azure Files - pull requests appreciated!
It may be more convenient to put the credentials directly into the
rclone config file under the `client_id`, `tenant` and `client_secret`
@@ -28459,11 +28634,11 @@ Synology C2 Object Storage
Properties:
- Config: service_principal_file
-- Env Var: RCLONE_AZUREBLOB_SERVICE_PRINCIPAL_FILE
+- Env Var: RCLONE_AZUREFILES_SERVICE_PRINCIPAL_FILE
- Type: string
- Required: false
-#### --azureblob-use-msi
+#### --azurefiles-use-msi
Use a managed service identity to authenticate (only works in Azure).
@@ -28479,11 +28654,11 @@ Synology C2 Object Storage
Properties:
- Config: use_msi
-- Env Var: RCLONE_AZUREBLOB_USE_MSI
+- Env Var: RCLONE_AZUREFILES_USE_MSI
- Type: bool
- Default: false
-#### --azureblob-msi-object-id
+#### --azurefiles-msi-object-id
Object ID of the user-assigned MSI to use, if any.
@@ -28492,11 +28667,11 @@ Synology C2 Object Storage
Properties:
- Config: msi_object_id
-- Env Var: RCLONE_AZUREBLOB_MSI_OBJECT_ID
+- Env Var: RCLONE_AZUREFILES_MSI_OBJECT_ID
- Type: string
- Required: false
-#### --azureblob-msi-client-id
+#### --azurefiles-msi-client-id
Object ID of the user-assigned MSI to use, if any.
@@ -28505,11 +28680,11 @@ Synology C2 Object Storage
Properties:
- Config: msi_client_id
-- Env Var: RCLONE_AZUREBLOB_MSI_CLIENT_ID
+- Env Var: RCLONE_AZUREFILES_MSI_CLIENT_ID
- Type: string
- Required: false
-#### --azureblob-msi-mi-res-id
+#### --azurefiles-msi-mi-res-id
Azure resource ID of the user-assigned MSI to use, if any.
@@ -28518,24 +28693,11 @@ Synology C2 Object Storage
Properties:
- Config: msi_mi_res_id
-- Env Var: RCLONE_AZUREBLOB_MSI_MI_RES_ID
+- Env Var: RCLONE_AZUREFILES_MSI_MI_RES_ID
- Type: string
- Required: false
-#### --azureblob-use-emulator
-
-Uses local storage emulator if provided as 'true'.
-
-Leave blank if using real azure storage endpoint.
-
-Properties:
-
-- Config: use_emulator
-- Env Var: RCLONE_AZUREBLOB_USE_EMULATOR
-- Type: bool
-- Default: false
-
-#### --azureblob-endpoint
+#### --azurefiles-endpoint
Endpoint for the service.
@@ -28544,37 +28706,26 @@ Synology C2 Object Storage
Properties:
- Config: endpoint
-- Env Var: RCLONE_AZUREBLOB_ENDPOINT
-- Type: string
-- Required: false
-
-#### --azureblob-upload-cutoff
-
-Cutoff for switching to chunked upload (<= 256 MiB) (deprecated).
-
-Properties:
-
-- Config: upload_cutoff
-- Env Var: RCLONE_AZUREBLOB_UPLOAD_CUTOFF
+- Env Var: RCLONE_AZUREFILES_ENDPOINT
- Type: string
- Required: false
-#### --azureblob-chunk-size
+#### --azurefiles-chunk-size
Upload chunk size.
Note that this is stored in memory and there may be up to
-"--transfers" * "--azureblob-upload-concurrency" chunks stored at once
+"--transfers" * "--azurefile-upload-concurrency" chunks stored at once
in memory.
Properties:
- Config: chunk_size
-- Env Var: RCLONE_AZUREBLOB_CHUNK_SIZE
+- Env Var: RCLONE_AZUREFILES_CHUNK_SIZE
- Type: SizeSuffix
- Default: 4Mi
-#### --azureblob-upload-concurrency
+#### --azurefiles-upload-concurrency
Concurrency for multipart uploads.
@@ -28585,125 +28736,41 @@ Synology C2 Object Storage
links and these uploads do not fully utilize your bandwidth, then
increasing this may help to speed up the transfers.
-In tests, upload speed increases almost linearly with upload
-concurrency. For example to fill a gigabit pipe it may be necessary to
-raise this to 64. Note that this will use more memory.
-
Note that chunks are stored in memory and there may be up to
-"--transfers" * "--azureblob-upload-concurrency" chunks stored at once
+"--transfers" * "--azurefile-upload-concurrency" chunks stored at once
in memory.
Properties:
- Config: upload_concurrency
-- Env Var: RCLONE_AZUREBLOB_UPLOAD_CONCURRENCY
+- Env Var: RCLONE_AZUREFILES_UPLOAD_CONCURRENCY
- Type: int
- Default: 16
-#### --azureblob-list-chunk
+#### --azurefiles-max-stream-size
-Size of blob list.
+Max size for streamed files.
-This sets the number of blobs requested in each listing chunk. Default
-is the maximum, 5000. "List blobs" requests are permitted 2 minutes
-per megabyte to complete. If an operation is taking longer than 2
-minutes per megabyte on average, it will time out (
-[source](https://docs.microsoft.com/en-us/rest/api/storageservices/setting-timeouts-for-blob-service-operations#exceptions-to-default-timeout-interval)
-). This can be used to limit the number of blobs items to return, to
-avoid the time out.
+Azure files needs to know in advance how big the file will be. When
+rclone doesn't know it uses this value instead.
-Properties:
-
-- Config: list_chunk
-- Env Var: RCLONE_AZUREBLOB_LIST_CHUNK
-- Type: int
-- Default: 5000
+This will be used when rclone is streaming data, the most common uses are:
-#### --azureblob-access-tier
+- Uploading files with `--vfs-cache-mode off` with `rclone mount`
+- Using `rclone rcat`
+- Copying files with unknown length
-Access tier of blob: hot, cool or archive.
+You will need this much free space in the share as the file will be this size temporarily.
-Archived blobs can be restored by setting access tier to hot or
-cool. Leave blank if you intend to use default access tier, which is
-set at account level
-
-If there is no "access tier" specified, rclone doesn't apply any tier.
-rclone performs "Set Tier" operation on blobs while uploading, if objects
-are not modified, specifying "access tier" to new one will have no effect.
-If blobs are in "archive tier" at remote, trying to perform data transfer
-operations from remote will not be allowed. User should first restore by
-tiering blob to "Hot" or "Cool".
-
-Properties:
-
-- Config: access_tier
-- Env Var: RCLONE_AZUREBLOB_ACCESS_TIER
-- Type: string
-- Required: false
-
-#### --azureblob-archive-tier-delete
-
-Delete archive tier blobs before overwriting.
-
-Archive tier blobs cannot be updated. So without this flag, if you
-attempt to update an archive tier blob, then rclone will produce the
-error:
-
- can't update archive tier blob without --azureblob-archive-tier-delete
-
-With this flag set then before rclone attempts to overwrite an archive
-tier blob, it will delete the existing blob before uploading its
-replacement. This has the potential for data loss if the upload fails
-(unlike updating a normal blob) and also may cost more since deleting
-archive tier blobs early may be chargable.
-
-
-Properties:
-
-- Config: archive_tier_delete
-- Env Var: RCLONE_AZUREBLOB_ARCHIVE_TIER_DELETE
-- Type: bool
-- Default: false
-
-#### --azureblob-disable-checksum
-
-Don't store MD5 checksum with object metadata.
-
-Normally rclone will calculate the MD5 checksum of the input before
-uploading it so it can add it to metadata on the object. This is great
-for data integrity checking but can cause long delays for large files
-to start uploading.
Properties:
-- Config: disable_checksum
-- Env Var: RCLONE_AZUREBLOB_DISABLE_CHECKSUM
-- Type: bool
-- Default: false
-
-#### --azureblob-memory-pool-flush-time
-
-How often internal memory buffer pools will be flushed. (no longer used)
-
-Properties:
-
-- Config: memory_pool_flush_time
-- Env Var: RCLONE_AZUREBLOB_MEMORY_POOL_FLUSH_TIME
-- Type: Duration
-- Default: 1m0s
-
-#### --azureblob-memory-pool-use-mmap
-
-Whether to use mmap buffers in internal memory pool. (no longer used)
-
-Properties:
-
-- Config: memory_pool_use_mmap
-- Env Var: RCLONE_AZUREBLOB_MEMORY_POOL_USE_MMAP
-- Type: bool
-- Default: false
+- Config: max_stream_size
+- Env Var: RCLONE_AZUREFILES_MAX_STREAM_SIZE
+- Type: SizeSuffix
+- Default: 10Gi
-#### --azureblob-encoding
+#### --azurefiles-encoding
The encoding for the backend.
@@ -28712,72 +28779,9 @@ Synology C2 Object Storage
Properties:
- Config: encoding
-- Env Var: RCLONE_AZUREBLOB_ENCODING
-- Type: MultiEncoder
-- Default: Slash,BackSlash,Del,Ctl,RightPeriod,InvalidUtf8
-
-#### --azureblob-public-access
-
-Public access level of a container: blob or container.
-
-Properties:
-
-- Config: public_access
-- Env Var: RCLONE_AZUREBLOB_PUBLIC_ACCESS
-- Type: string
-- Required: false
-- Examples:
- - ""
- - The container and its blobs can be accessed only with an authorized request.
- - It's a default value.
- - "blob"
- - Blob data within this container can be read via anonymous request.
- - "container"
- - Allow full public read access for container and blob data.
-
-#### --azureblob-directory-markers
-
-Upload an empty object with a trailing slash when a new directory is created
-
-Empty folders are unsupported for bucket based remotes, this option
-creates an empty object ending with "/", to persist the folder.
-
-This object also has the metadata "hdi_isfolder = true" to conform to
-the Microsoft standard.
-
-
-Properties:
-
-- Config: directory_markers
-- Env Var: RCLONE_AZUREBLOB_DIRECTORY_MARKERS
-- Type: bool
-- Default: false
-
-#### --azureblob-no-check-container
-
-If set, don't attempt to check the container exists or create it.
-
-This can be useful when trying to minimise the number of transactions
-rclone does if you know the container exists already.
-
-
-Properties:
-
-- Config: no_check_container
-- Env Var: RCLONE_AZUREBLOB_NO_CHECK_CONTAINER
-- Type: bool
-- Default: false
-
-#### --azureblob-no-head-object
-
-If set, do not do HEAD before GET when getting objects.
-
-Properties:
-
-- Config: no_head_object
-- Env Var: RCLONE_AZUREBLOB_NO_HEAD_OBJECT
-- Type: bool
-- Default: false
+- Env Var: RCLONE_AZUREFILES_ENCODING
+- Type: Encoding
+- Default: Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Del,Ctl,RightPeriod,InvalidUtf8,Dot
@@ -28798,30 +28802,6 @@ Synology C2 Object Storage
MD5 sums are only uploaded with chunked files if the source has an MD5
sum. This will always be the case for a local to azure copy.
-`rclone about` is not supported by the Microsoft Azure Blob storage backend. Backends without
-this capability cannot determine free space for an rclone mount or
-use policy `mfs` (most free space) as a member of an rclone union
-remote.
-
-See [List of backends that do not support rclone about](https://rclone.org/overview/#optional-features) and [rclone about](https://rclone.org/commands/rclone_about/)
-
-## Azure Storage Emulator Support
-
-You can run rclone with the storage emulator (usually _azurite_).
-
-To do this, just set up a new remote with `rclone config` following
-the instructions in the introduction and set `use_emulator` in the
-advanced settings as `true`. You do not need to provide a default
-account name nor an account key. But you can override them in the
-`account` and `key` options. (Prior to v1.61 they were hard coded to
-_azurite_'s `devstoreaccount1`.)
-
-Also, if you want to access a storage emulator instance running on a
-different machine, you can override the `endpoint` parameter in the
-advanced settings, setting it to
-`http(s)://<host>:<port>/devstoreaccount1`
-(e.g. `http://10.254.2.5:10000/devstoreaccount1`).
-
# Microsoft OneDrive
Paths are specified as `remote:path`
@@ -28930,7 +28910,7 @@ Synology C2 Object Storage
Note: If you have a special region, you may need a different host in step 4 and 5. Here are [some hints](https://github.com/rclone/rclone/blob/bc23bf11db1c78c6ebbf8ea538fbebf7058b4176/backend/onedrive/onedrive.go#L86).
-### Modification time and hashes
+### Modification times and hashes
OneDrive allows modification times to be set on objects accurate to 1
second. These will be used to detect whether objects need syncing or
@@ -28951,6 +28931,32 @@ Synology C2 Object Storage
For all types of OneDrive you can use the `--checksum` flag.
+### --fast-list
+
+This remote supports `--fast-list` which allows you to use fewer
+transactions in exchange for more memory. See the [rclone
+docs](https://rclone.org/docs/#fast-list) for more details.
+
+This must be enabled with the `--onedrive-delta` flag (or `delta =
+true` in the config file) as it can cause performance degradation.
+
+It does this by using the delta listing facilities of OneDrive which
+returns all the files in the remote very efficiently. This is much
+more efficient than listing directories recursively and is Microsoft's
+recommended way of reading all the file information from a drive.
+
+This can be useful with `rclone mount` and [rclone rc vfs/refresh
+recursive=true](https://rclone.org/rc/#vfs-refresh)) to very quickly fill the mount with
+information about all the files.
+
+The API used for the recursive listing (`ListR`) only supports listing
+from the root of the drive. This will become increasingly inefficient
+the further away you get from the root as rclone will have to discard
+files outside of the directory you are using.
+
+Some commands (like `rclone lsf -R`) will use `ListR` by default - you
+can turn this off with `--disable ListR` if you need to.
+
### Restricted filename characters
In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters)
@@ -29362,6 +29368,43 @@ Synology C2 Object Storage
- Type: bool
- Default: false
+#### --onedrive-delta
+
+If set rclone will use delta listing to implement recursive listings.
+
+If this flag is set the the onedrive backend will advertise `ListR`
+support for recursive listings.
+
+Setting this flag speeds up these things greatly:
+
+ rclone lsf -R onedrive:
+ rclone size onedrive:
+ rclone rc vfs/refresh recursive=true
+
+**However** the delta listing API **only** works at the root of the
+drive. If you use it not at the root then it recurses from the root
+and discards all the data that is not under the directory you asked
+for. So it will be correct but may not be very efficient.
+
+This is why this flag is not set as the default.
+
+As a rule of thumb if nearly all of your data is under rclone's root
+directory (the `root/directory` in `onedrive:root/directory`) then
+using this flag will be be a big performance win. If your data is
+mostly not under the root then using this flag will be a big
+performance loss.
+
+It is recommended if you are mounting your onedrive at the root
+(or near the root when using crypt) and using rclone `rc vfs/refresh`.
+
+
+Properties:
+
+- Config: delta
+- Env Var: RCLONE_ONEDRIVE_DELTA
+- Type: bool
+- Default: false
+
#### --onedrive-encoding
The encoding for the backend.
@@ -29372,7 +29415,7 @@ Synology C2 Object Storage
- Config: encoding
- Env Var: RCLONE_ONEDRIVE_ENCODING
-- Type: MultiEncoder
+- Type: Encoding
- Default: Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Del,Ctl,LeftSpace,LeftTilde,RightSpace,RightPeriod,InvalidUtf8,Dot
@@ -29631,12 +29674,14 @@ Synology C2 Object Storage
rclone copy /home/source remote:backup
-### Modified time and MD5SUMs
+### Modification times and hashes
OpenDrive allows modification times to be set on objects accurate to 1
second. These will be used to detect whether objects need syncing or
not.
+The MD5 hash algorithm is supported.
+
### Restricted filename characters
| Character | Value | Replacement |
@@ -29710,7 +29755,7 @@ Synology C2 Object Storage
- Config: encoding
- Env Var: RCLONE_OPENDRIVE_ENCODING
-- Type: MultiEncoder
+- Type: Encoding
- Default: Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,LeftSpace,LeftCrLfHtVt,RightSpace,RightCrLfHtVt,InvalidUtf8,Dot
#### --opendrive-chunk-size
@@ -29818,6 +29863,7 @@ Synology C2 Object Storage
No authentication
### User Principal
+
Sample rclone config file for Authentication Provider User Principal:
[oos]
@@ -29838,6 +29884,7 @@ Synology C2 Object Storage
- If the user is deleted, the config file will no longer work and may cause automation regressions that use the user's credentials.
### Instance Principal
+
An OCI compute instance can be authorized to use rclone by using it's identity and certificates as an instance principal.
With this approach no credentials have to be stored and managed.
@@ -29867,6 +29914,7 @@ Synology C2 Object Storage
- It is applicable for oci compute instances only. It cannot be used on external instance or resources.
### Resource Principal
+
Resource principal auth is very similar to instance principal auth but used for resources that are not
compute instances such as [serverless functions](https://docs.oracle.com/en-us/iaas/Content/Functions/Concepts/functionsoverview.htm).
To use resource principal ensure Rclone process is started with these environment variables set in its process.
@@ -29886,6 +29934,7 @@ Synology C2 Object Storage
provider = resource_principal_auth
### No authentication
+
Public buckets do not require any authentication mechanism to read objects.
Sample rclone configuration file for No authentication:
@@ -29896,10 +29945,9 @@ Synology C2 Object Storage
region = us-ashburn-1
provider = no_auth
-## Options
-### Modified time
+### Modification times and hashes
-The modified time is stored as metadata on the object as
+The modification time is stored as metadata on the object as
`opc-meta-mtime` as floating point since the epoch, accurate to 1 ns.
If the modification time needs to be updated rclone will attempt to perform a server
@@ -29909,6 +29957,8 @@ Synology C2 Object Storage
Note that reading this from the object takes an additional `HEAD` request as the metadata
isn't returned in object listings.
+The MD5 hash algorithm is supported.
+
### Multipart uploads
rclone supports multipart uploads with OOS which means that it can
@@ -30211,7 +30261,7 @@ Synology C2 Object Storage
- Config: encoding
- Env Var: RCLONE_OOS_ENCODING
-- Type: MultiEncoder
+- Type: Encoding
- Default: Slash,InvalidUtf8,Dot
#### --oos-leave-parts-on-error
@@ -30682,7 +30732,7 @@ Synology C2 Object Storage
- Config: encoding
- Env Var: RCLONE_QINGSTOR_ENCODING
-- Type: MultiEncoder
+- Type: Encoding
- Default: Slash,Ctl,InvalidUtf8
@@ -30784,7 +30834,7 @@ Synology C2 Object Storage
y) Yes this is OK e) Edit this remote d) Delete this remote y/e/d> y ```
-### Modified time and hashes
+### Modification times and hashes
Quatrix allows modification times to be set on objects accurate to 1 microsecond. These will be used to detect whether objects need syncing or not.
@@ -30859,7 +30909,7 @@ Synology C2 Object Storage
Properties:
-- Config: encoding - Env Var: RCLONE_QUATRIX_ENCODING - Type: MultiEncoder - Default: Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot
+- Config: encoding - Env Var: RCLONE_QUATRIX_ENCODING - Type: Encoding - Default: Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot
#### --quatrix-effective-upload-time
@@ -31036,7 +31086,7 @@ Synology C2 Object Storage
- Config: encoding
- Env Var: RCLONE_SIA_ENCODING
-- Type: MultiEncoder
+- Type: Encoding
- Default: Slash,Question,Hash,Percent,Del,Ctl,InvalidUtf8,Dot
@@ -31162,7 +31212,7 @@ Synology C2 Object Storage
`--use-server-modtime`, you can avoid the extra API call and simply upload
files whose local modtime is newer than the time it was last uploaded.
-### Modified time
+### Modification times and hashes
The modified time is stored as metadata on the object as
`X-Object-Meta-Mtime` as floating point since the epoch accurate to 1
@@ -31171,6 +31221,8 @@ Synology C2 Object Storage
This is a de facto standard (used in the official python-swiftclient
amongst others) for storing the modification time for an object.
+The MD5 hash algorithm is supported.
+
### Restricted filename characters
| Character | Value | Replacement |
@@ -31517,7 +31569,7 @@ Synology C2 Object Storage
- Config: encoding
- Env Var: RCLONE_SWIFT_ENCODING
-- Type: MultiEncoder
+- Type: Encoding
- Default: Slash,InvalidUtf8
@@ -31605,7 +31657,7 @@ Synology C2 Object Storage
rclone copy /home/source remote:backup
-### Modified time and hashes ###
+### Modification times and hashes
pCloud allows modification times to be set on objects accurate to 1
second. These will be used to detect whether objects need syncing or
@@ -31744,7 +31796,7 @@ Synology C2 Object Storage
- Config: encoding
- Env Var: RCLONE_PCLOUD_ENCODING
-- Type: MultiEncoder
+- Type: Encoding
- Default: Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot
#### --pcloud-root-folder-id
@@ -31833,6 +31885,13 @@ Synology C2 Object Storage
Edit advanced config? y) Yes n) No (default) y/n>
Configuration complete. Options: - type: pikpak - user: USERNAME - pass: *** ENCRYPTED *** - token: {"access_token":"eyJ...","token_type":"Bearer","refresh_token":"os...","expiry":"2023-01-26T18:54:32.170582647+09:00"} Keep this "remote" remote? y) Yes this is OK (default) e) Edit this remote d) Delete this remote y/e/d> y
+### Modification times and hashes
+
+PikPak keeps modification times on objects, and updates them when uploading objects,
+but it does not support changing only the modification time
+
+The MD5 hash algorithm is supported.
+
### Standard options
@@ -31992,7 +32051,7 @@ Synology C2 Object Storage
- Config: encoding
- Env Var: RCLONE_PIKPAK_ENCODING
-- Type: MultiEncoder
+- Type: Encoding
- Default: Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,LeftSpace,RightSpace,RightPeriod,InvalidUtf8,Dot
## Backend commands
@@ -32056,15 +32115,16 @@ Synology C2 Object Storage
-## Limitations ##
+## Limitations
-### Hashes ###
+### Hashes may be empty
PikPak supports MD5 hash, but sometimes given empty especially for user-uploaded files.
-### Deleted files ###
+### Deleted files still visible with trashed-only
-Deleted files will still be visible with `--pikpak-trashed-only` even after the trash emptied. This goes away after few days.
+Deleted files will still be visible with `--pikpak-trashed-only` even after the
+trash emptied. This goes away after few days.
# premiumize.me
@@ -32109,7 +32169,7 @@ Synology C2 Object Storage
rclone copy /home/source remote:backup
-### Modified time and hashes
+### Modification times and hashes
premiumize.me does not support modification times or hashes, therefore
syncing will default to `--size-only` checking. Note that using
@@ -32224,7 +32284,7 @@ Synology C2 Object Storage
- Config: encoding
- Env Var: RCLONE_PREMIUMIZEME_ENCODING
-- Type: MultiEncoder
+- Type: Encoding
- Default: Slash,DoubleQuote,BackSlash,Del,Ctl,InvalidUtf8,Dot
@@ -32290,10 +32350,12 @@ Synology C2 Object Storage
rclone copy /home/source remote:backup
-### Modified time
+### Modification times and hashes
Proton Drive Bridge does not support updating modification times yet.
+The SHA1 hash algorithm is supported.
+
### Restricted filename characters
Invalid UTF-8 bytes will be [replaced](https://rclone.org/overview/#invalid-utf8), also left and
@@ -32439,7 +32501,7 @@ Synology C2 Object Storage
- Config: encoding
- Env Var: RCLONE_PROTONDRIVE_ENCODING
-- Type: MultiEncoder
+- Type: Encoding
- Default: Slash,LeftSpace,RightSpace,InvalidUtf8,Dot
#### --protondrive-original-file-size
@@ -32702,7 +32764,7 @@ Synology C2 Object Storage
- Config: encoding
- Env Var: RCLONE_PUTIO_ENCODING
-- Type: MultiEncoder
+- Type: Encoding
- Default: Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot
@@ -32766,10 +32828,12 @@ Synology C2 Object Storage
rclone copy /home/source remote:backup
-### Modified time
+### Modification times and hashes
Proton Drive Bridge does not support updating modification times yet.
+The SHA1 hash algorithm is supported.
+
### Restricted filename characters
Invalid UTF-8 bytes will be [replaced](https://rclone.org/overview/#invalid-utf8), also left and
@@ -32915,7 +32979,7 @@ Synology C2 Object Storage
- Config: encoding
- Env Var: RCLONE_PROTONDRIVE_ENCODING
-- Type: MultiEncoder
+- Type: Encoding
- Default: Slash,LeftSpace,RightSpace,InvalidUtf8,Dot
#### --protondrive-original-file-size
@@ -33272,7 +33336,7 @@ Synology C2 Object Storage
- Config: encoding
- Env Var: RCLONE_SEAFILE_ENCODING
-- Type: MultiEncoder
+- Type: Encoding
- Default: Slash,DoubleQuote,BackSlash,Ctl,InvalidUtf8
@@ -33457,7 +33521,7 @@ Checksum
The options md5sum_command and sha1_command can be used to customize the command to be executed for calculation of checksums. You can for example set a specific path to where md5sum and sha1sum executables are located, or use them to specify some other tools that print checksums in compatible format. The value can include command-line arguments, or even shell script blocks as with PowerShell. Rclone has subcommands md5sum and sha1sum that use compatible format, which means if you have an rclone executable on the server it can be used. As mentioned above, they will be automatically picked up if found in PATH, but if not you can set something like /path/to/rclone md5sum as the value of option md5sum_command to make sure a specific executable is used.
Remote checksumming is recommended and enabled by default. First time rclone is using a SFTP remote, if options md5sum_command or sha1_command are not set, it will check if any of the default commands for each of them, as described above, can be used. The result will be saved in the remote configuration, so next time it will use the same. Value none will be set if none of the default commands could be used for a specific algorithm, and this algorithm will not be supported by the remote.
Disabling the checksumming may be required if you are connecting to SFTP servers which are not under your control, and to which the execution of remote shell commands is prohibited. Set the configuration option disable_hashcheck to true to disable checksumming entirely, or set shell_type to none to disable all functionality based on remote shell command execution.
-Modified time
+Modification times and hashes
Modified times are stored on the server to 1 second precision.
Modified times are used in syncing and are fully supported.
Some SFTP servers disable setting/modifying the file modification time after upload (for example, certain configurations of ProFTPd with mod_sftp). If you are using one of these servers, you can set the option set_modtime = false in your RClone backend configuration to disable this behaviour.
@@ -33903,7 +33967,21 @@ --sftp-socks-proxy
- Type: string
- Required: false
-Limitations
+--sftp-copy-is-hardlink
+Set to enable server side copies using hardlinks.
+The SFTP protocol does not define a copy command so normally server side copies are not allowed with the sftp backend.
+However the SFTP protocol does support hardlinking, and if you enable this flag then the sftp backend will support server side copies. These will be implemented by doing a hardlink from the source to the destination.
+Not all sftp servers support this.
+Note that hardlinking two files together will use no additional space as the source and the destination will be the same file.
+This feature may be useful backups made with --copy-dest.
+Properties:
+
+- Config: copy_is_hardlink
+- Env Var: RCLONE_SFTP_COPY_IS_HARDLINK
+- Type: bool
+- Default: false
+
+Limitations
On some SFTP servers (e.g. Synology) the paths are different for SSH and SFTP so the hashes can't be calculated properly. For them using disable_hashcheck is a good idea.
The only ssh agent supported under Windows is Putty's pageant.
The Go SSH library disables the use of the aes128-cbc cipher by default, due to security concerns. This can be re-enabled on a per-connection basis by setting the use_insecure_cipher setting in the configuration file to true. Further details on the insecurity of this cipher can be found in this paper.
@@ -33920,7 +33998,7 @@ SMB
SMB is a communication protocol to share files over network.
This relies on go-smb2 library for communication with SMB protocol.
Paths are specified as remote:sharename (or remote: for the lsd command.) You may put subdirectories in too, e.g. remote:item/path/to/dir.
-Notes
+Notes
The first path segment must be the name of the share, which you entered when you started to share on Windows. On smbd, it's the section title in smb.conf (usually in /etc/samba/) file. You can find shares by querying the root if you're unsure (e.g. rclone lsd remote:).
You can't access to the shared printers from rclone, obviously.
You can't use Anonymous access for logging in. You have to use the guest user with an empty password instead. The rclone client tries to avoid 8.3 names when uploading files by encoding trailing spaces and periods. Alternatively, the local backend on Windows can access SMB servers using UNC paths, by \\server\share. This doesn't apply to non-Windows OSes, such as Linux and macOS.
@@ -34099,7 +34177,7 @@ --smb-encoding
- Config: encoding
- Env Var: RCLONE_SMB_ENCODING
-- Type: MultiEncoder
+- Type: Encoding
- Default: Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot
Storj
@@ -34391,7 +34469,7 @@ Sync two Locations
rclone sync --interactive --progress remote-us:bucket/path/to/dir/ remote-europe:bucket/path/to/dir/
Or even between another cloud storage and Storj.
rclone sync --interactive --progress s3:bucket/path/to/dir/ storj:bucket/path/to/dir/
-Limitations
+Limitations
rclone about is not supported by the rclone Storj backend. Backends without this capability cannot determine free space for an rclone mount or use policy mfs (most free space) as a member of an rclone union remote.
See List of backends that do not support rclone about and rclone about
Known issues
@@ -34464,7 +34542,7 @@ Configuration
Paths are specified as remote:path
Paths may be as deep as required, e.g. remote:directory/subdirectory.
NB you can't create files in the top level folder you have to create a folder, which rclone will create as a "Sync Folder" with SugarSync.
-Modified time and hashes
+Modification times and hashes
SugarSync does not support modification times or hashes, therefore syncing will default to --size-only checking. Note that using --update will work as rclone can read the time files were uploaded.
Restricted filename characters
SugarSync replaces the default restricted characters set except for DEL.
@@ -34582,10 +34660,10 @@ --sugarsync-encoding
- Config: encoding
- Env Var: RCLONE_SUGARSYNC_ENCODING
-- Type: MultiEncoder
+- Type: Encoding
- Default: Slash,Ctl,InvalidUtf8,Dot
-Limitations
+Limitations
rclone about is not supported by the SugarSync backend. Backends without this capability cannot determine free space for an rclone mount or use policy mfs (most free space) as a member of an rclone union remote.
See List of backends that do not support rclone about and rclone about
Tardigrade
@@ -34648,7 +34726,7 @@ Configuration
rclone ls remote:
To copy a local directory to an Uptobox directory called backup
rclone copy /home/source remote:backup
-Modified time and hashes
+Modification times and hashes
Uptobox supports neither modified times nor checksums. All timestamps will read as that set by --default-time.
Restricted filename characters
In addition to the default restricted characters set the following characters are also replaced:
@@ -34704,16 +34782,16 @@ --uptobox-encoding
- Config: encoding
- Env Var: RCLONE_UPTOBOX_ENCODING
-- Type: MultiEncoder
+- Type: Encoding
- Default: Slash,LtGt,DoubleQuote,BackQuote,Del,Ctl,LeftSpace,InvalidUtf8,Dot
-Limitations
+Limitations
Uptobox will delete inactive files that have not been accessed in 60 days.
rclone about is not supported by this backend an overview of used space can however been seen in the uptobox web interface.
Union
The union backend joins several remotes together to make a single unified view of them.
During the initial setup with rclone config you will specify the upstream remotes as a space separated list. The upstream remotes can either be a local paths or other remotes.
-The attributes :ro, :nc and :nc can be attached to the end of the remote to tag the remote as read only, no create or writeback, e.g. remote:directory/subdirectory:ro or remote:directory/subdirectory:nc.
+The attributes :ro, :nc and :writeback can be attached to the end of the remote to tag the remote as read only, no create or writeback, e.g. remote:directory/subdirectory:ro or remote:directory/subdirectory:nc.
:ro means files will only be read from here and never written:nc means new files or directories won't be created hereConfiguration
\ (sharepoint)
5 / Sharepoint with NTLM authentication, usually self-hosted or on-premises
\ (sharepoint-ntlm)
- 6 / Other site/service or software
+ 6 / rclone WebDAV server to serve a remote over HTTP via the WebDAV protocol
+ \ (rclone)
+ 7 / Other site/service or software
\ (other)
vendor> 2
User name
@@ -35093,7 +35173,7 @@ Configuration
rclone ls remote:
To copy a local directory to an WebDAV directory called backup
rclone copy /home/source remote:backup
-Modified time and hashes
+Modification times and hashes
Plain WebDAV does not support modified times. However when used with Fastmail Files, Owncloud or Nextcloud rclone will support modified times.
Likewise plain WebDAV does not support hashes, however when used with Fastmail Files, Owncloud or Nextcloud rclone will support SHA1 and MD5 hashes. Depending on the exact version of Owncloud or Nextcloud hashes may appear on all objects, or only on objects which had a hash uploaded with them.
Standard options
@@ -35138,6 +35218,10 @@ --webdav-vendor
- Sharepoint with NTLM authentication, usually self-hosted or on-premises
+- "rclone"
+
+- rclone WebDAV server to serve a remote over HTTP via the WebDAV protocol
+
- "other"
- Other site/service or software
@@ -35273,6 +35357,9 @@ Required Flags for SharePoint
As SharePoint does some special things with uploaded documents, you won't be able to use the documents size or the documents hash to compare if a file has been changed since the upload / which file is newer.
For Rclone calls copying files (especially Office files such as .docx, .xlsx, etc.) from/to SharePoint (like copy, sync, etc.), you should append these flags to ensure Rclone uses the "Last Modified" datetime property to compare your documents:
--ignore-size --ignore-checksum --update
+Rclone
+Use this option if you are hosting remotes over WebDAV provided by rclone. Read rclone serve webdav for more details.
+rclone serve supports modified times using the X-OC-Mtime header.
dCache
dCache is a storage system that supports many protocols and authentication/authorisation schemes. For WebDAV clients, it allows users to authenticate with username and password (BASIC), X.509, Kerberos, and various bearer tokens, including Macaroons and OpenID-Connect access tokens.
Configure as normal using the other type. Don't enter a username or password, instead enter your Macaroon as the bearer_token.
@@ -35357,10 +35444,9 @@ Configuration
Sync /home/local/directory to the remote path, deleting any excess files in the path.
rclone sync --interactive /home/local/directory remote:directory
Yandex paths may be as deep as required, e.g. remote:directory/subdirectory.
-Modified time
+Modification times and hashes
Modified times are supported and are stored accurate to 1 ns in custom metadata called rclone_modified in RFC3339 with nanoseconds format.
-MD5 checksums
-MD5 checksums are natively supported by Yandex Disk.
+The MD5 hash algorithm is natively supported by Yandex Disk.
Emptying Trash
If you wish to empty your trash you can use the rclone cleanup remote: command which will permanently delete all your trashed files. This command does not take any path arguments.
Quota information
@@ -35437,10 +35523,10 @@ --yandex-encoding
- Config: encoding
- Env Var: RCLONE_YANDEX_ENCODING
-- Type: MultiEncoder
+- Type: Encoding
- Default: Slash,Del,Ctl,InvalidUtf8,Dot
-Limitations
+Limitations
When uploading very large files (bigger than about 5 GiB) you will need to increase the --timeout parameter. This is because Yandex pauses (perhaps to calculate the MD5SUM for the entire file) before returning confirmation that the file has been uploaded. The default handling of timeouts in rclone is to assume a 5 minute pause is an error and close the connection - you'll see net/http: timeout awaiting response headers errors in the logs if this is happening. Setting the timeout to twice the max size of file in GiB should be enough, so if you want to upload a 30 GiB file set a timeout of 2 * 30 = 60m, that is --timeout 60m.
Having a Yandex Mail account is mandatory to use the Yandex.Disk subscription. Token generation will work without a mail account, but Rclone won't be able to complete any actions.
[403 - DiskUnsupportedUserAccountTypeError] User account type is not supported.
@@ -35519,10 +35605,9 @@ Configuration
Sync /home/local/directory to the remote path, deleting any excess files in the path.
rclone sync --interactive /home/local/directory remote:directory
Zoho paths may be as deep as required, eg remote:directory/subdirectory.
-Modified time
+Modification times and hashes
Modified times are currently not supported for Zoho Workdrive
-Checksums
-No checksums are supported.
+No hash algorithms are supported.
Usage information
To view your current quota you can use the rclone about remote: command which will display your current usage.
Restricted filename characters
@@ -35624,7 +35709,7 @@ --zoho-encoding
- Config: encoding
- Env Var: RCLONE_ZOHO_ENCODING
-- Type: MultiEncoder
+- Type: Encoding
- Default: Del,Ctl,InvalidUtf8
Setting up your own client_id
@@ -35641,8 +35726,8 @@ Local Filesystem
Will sync /home/source to /tmp/destination.
Configuration
For consistencies sake one can also configure a remote of type local in the config file, and access the local filesystem using rclone remote paths, e.g. remote:path/to/wherever, but it is probably easier not to.
-Modified time
-Rclone reads and writes the modified time using an accuracy determined by the OS. Typically this is 1ns on Linux, 10 ns on Windows and 1 Second on OS X.
+Modification times
+Rclone reads and writes the modification times using an accuracy determined by the OS. Typically this is 1ns on Linux, 10 ns on Windows and 1 Second on OS X.
Filenames
Filenames should be encoded in UTF-8 on disk. This is the normal case for Windows and OS X.
There is a bit more uncertainty in the Linux world, but new distributions will have UTF-8 encoded files names. If you are using an old Linux filesystem with non UTF-8 file names (e.g. latin1) then you can use the convmv tool to convert the filesystem to UTF-8. This tool is available in most distributions' package managers.
@@ -36097,6 +36182,7 @@ --local-no-check-updated
- Only checksum the size that stat gave
- Don't update the stat info for the file
+NB do not use this flag on a Windows Volume Shadow (VSS). For some unknown reason, files in a VSS sometimes show different sizes from the directory listing (where the initial stat value comes from on Windows) and when stat is called on them directly. Other copy tools always use the direct stat value and setting this flag will disable that.
Properties:
- Config: no_check_updated
@@ -36170,7 +36256,7 @@ --local-encoding
- Config: encoding
- Env Var: RCLONE_LOCAL_ENCODING
-- Type: MultiEncoder
+- Type: Encoding
- Default: Slash,Dot
Metadata
@@ -36264,6 +36350,228 @@ noop
- "error": return an error based on option value
Changelog
+v1.65.0 - 2023-11-26
+
+
+- New backends
+
+- Azure Files (karan, moongdal, Nick Craig-Wood)
+- ImageKit (Abhinav Dhiman)
+- Linkbox (viktor, Nick Craig-Wood)
+
+- New commands
+
+serve s3: Let rclone act as an S3 compatible server (Mikubill, Artur Neumann, Saw-jan, Nick Craig-Wood)nfsmount: mount command to provide mount mechanism on macOS without FUSE (Saleh Dindar)serve nfs: to serve a remote for use by nfsmount (Saleh Dindar)
+- New Features
+
+- install.sh: Clean up temp files in install script (Jacob Hands)
+- build
+
+- Update all dependencies (Nick Craig-Wood)
+- Refactor version info and icon resource handling on windows (albertony)
+
+- doc updates (albertony, alfish2000, asdffdsazqqq, Dimitri Papadopoulos, Herby Gillot, Joda Stößer, Manoj Ghosh, Nick Craig-Wood)
+- Implement
--metadata-mapper to transform metatadata with a user supplied program (Nick Craig-Wood)
+- Add
ChunkWriterDoesntSeek feature flag and set it for b2 (Nick Craig-Wood)
+- lib/http: Export basic go string functions for use in
--template (Gabriel Espinoza)
+- makefile: Use POSIX compatible install arguments (Mina Galić)
+- operations
+
+- Use less memory when doing multithread uploads (Nick Craig-Wood)
+- Implement
--partial-suffix to control extension of temporary file names (Volodymyr)
+
+- rc
+
+- Add
operations/check to the rc API (Nick Craig-Wood)
+- Always report an error as JSON (Nick Craig-Wood)
+- Set
Last-Modified header for files served by --rc-serve (Nikita Shoshin)
+
+- size: Dont show duplicate object count when less than 1k (albertony)
+
+- Bug Fixes
+
+- fshttp: Fix
--contimeout being ignored (ä½ çŸ¥é“未æ¥å—)
+- march: Fix excessive parallelism when using
--no-traverse (Nick Craig-Wood)
+- ncdu: Fix crash when re-entering changed directory after rescan (Nick Craig-Wood)
+- operations
+
+- Fix overwrite of destination when multi-thread transfer fails (Nick Craig-Wood)
+- Fix invalid UTF-8 when truncating file names when not using
--inplace (Nick Craig-Wood)
+
+- serve dnla: Fix crash on graceful exit (wuxingzhong)
+
+- Mount
+
+- Disable mount for freebsd and alias cmount as mount on that platform (Nick Craig-Wood)
+
+- VFS
+
+- Add
--vfs-refresh flag to read all the directories on start (Beyond Meat)
+- Implement Name() method in WriteFileHandle and ReadFileHandle (Saleh Dindar)
+- Add go-billy dependency and make sure vfs.Handle implements billy.File (Saleh Dindar)
+- Error out early if can't upload 0 length file (Nick Craig-Wood)
+
+- Local
+
+- Fix copying from Windows Volume Shadows (Nick Craig-Wood)
+
+- Azure Blob
+
+- Add support for cold tier (Ivan Yanitra)
+
+- B2
+
+- Implement "rclone backend lifecycle" to read and set bucket lifecycles (Nick Craig-Wood)
+- Implement
--b2-lifecycle to control lifecycle when creating buckets (Nick Craig-Wood)
+- Fix listing all buckets when not needed (Nick Craig-Wood)
+- Fix multi-thread upload with copyto going to wrong name (Nick Craig-Wood)
+- Fix server side chunked copy when file size was exactly
--b2-copy-cutoff (Nick Craig-Wood)
+- Fix streaming chunked files an exact multiple of chunk size (Nick Craig-Wood)
+
+- Box
+
+- Filter more EventIDs when polling (David Sze)
+- Add more logging for polling (David Sze)
+- Fix performance problem reading metadata for single files (Nick Craig-Wood)
+
+- Drive
+
+- Add read/write metadata support (Nick Craig-Wood)
+- Add support for SHA-1 and SHA-256 checksums (rinsuki)
+- Add
--drive-show-all-gdocs to allow unexportable gdocs to be server side copied (Nick Craig-Wood)
+- Add a note that
--drive-scope accepts comma-separated list of scopes (Keigo Imai)
+- Fix error updating created time metadata on existing object (Nick Craig-Wood)
+- Fix integration tests by enabling metadata support from the context (Nick Craig-Wood)
+
+- Dropbox
+
+- Factor batcher into lib/batcher (Nick Craig-Wood)
+- Fix missing encoding for rclone purge (Nick Craig-Wood)
+
+- Google Cloud Storage
+
+- Fix 400 Bad request errors when using multi-thread copy (Nick Craig-Wood)
+
+- Googlephotos
+
+- Implement batcher for uploads (Nick Craig-Wood)
+
+- Hdfs
+
+- Added support for list of namenodes in hdfs remote config (Tayo-pasedaRJ)
+
+- HTTP
+
+- Implement set backend command to update running backend (Nick Craig-Wood)
+- Enable methods used with WebDAV (Alen Å iljak)
+
+- Jottacloud
+
+- Add support for reading and writing metadata (albertony)
+
+- Onedrive
+
+- Implement ListR method which gives
--fast-list support (Nick Craig-Wood)
+
+- This must be enabled with the
--onedrive-delta flag
+
+
+- Quatrix
+
+- Add partial upload support (Oksana Zhykina)
+- Overwrite files on conflict during server-side move (Oksana Zhykina)
+
+- S3
+
+- Add Linode provider (Nick Craig-Wood)
+- Add docs on how to add a new provider (Nick Craig-Wood)
+- Fix no error being returned when creating a bucket we don't own (Nick Craig-Wood)
+- Emit a debug message if anonymous credentials are in use (Nick Craig-Wood)
+- Add
--s3-disable-multipart-uploads flag (Nick Craig-Wood)
+- Detect looping when using gcs and versions (Nick Craig-Wood)
+
+- SFTP
+
+- Implement
--sftp-copy-is-hardlink to server side copy as hardlink (Nick Craig-Wood)
+
+- Smb
+
+- Fix incorrect
about size by switching to github.com/cloudsoda/go-smb2 fork (Nick Craig-Wood)
+- Fix modtime of multithread uploads by setting PartialUploads (Nick Craig-Wood)
+
+- WebDAV
+
+- Added an rclone vendor to work with
rclone serve webdav (Adithya Kumar)
+
+
+v1.64.2 - 2023-10-19
+
+
+- Bug Fixes
+
+- selfupdate: Fix "invalid hashsum signature" error (Nick Craig-Wood)
+- build: Fix docker build running out of space (Nick Craig-Wood)
+
+
+v1.64.1 - 2023-10-17
+
+
+- Bug Fixes
+
+- cmd: Make
--progress output logs in the same format as without (Nick Craig-Wood)
+- docs fixes (Dimitri Papadopoulos Orfanos, Herby Gillot, Manoj Ghosh, Nick Craig-Wood)
+- lsjson: Make sure we set the global metadata flag too (Nick Craig-Wood)
+- operations
+
+- Ensure concurrency is no greater than the number of chunks (Pat Patterson)
+- Fix OpenOptions ignored in copy if operation was a multiThreadCopy (Vitor Gomes)
+- Fix error message on delete to have file name (Nick Craig-Wood)
+
+- serve sftp: Return not supported error for not supported commands (Nick Craig-Wood)
+- build: Upgrade golang.org/x/net to v0.17.0 to fix HTTP/2 rapid reset (Nick Craig-Wood)
+- pacer: Fix b2 deadlock by defaulting max connections to unlimited (Nick Craig-Wood)
+
+- Mount
+
+- Fix automount not detecting drive is ready (Nick Craig-Wood)
+
+- VFS
+
+- Fix update dir modification time (Saleh Dindar)
+
+- Azure Blob
+
+- Fix "fatal error: concurrent map writes" (Nick Craig-Wood)
+
+- B2
+
+- Fix multipart upload: corrupted on transfer: sizes differ XXX vs 0 (Nick Craig-Wood)
+- Fix locking window when getting mutipart upload URL (Nick Craig-Wood)
+- Fix server side copies greater than 4GB (Nick Craig-Wood)
+- Fix chunked streaming uploads (Nick Craig-Wood)
+- Reduce default
--b2-upload-concurrency to 4 to reduce memory usage (Nick Craig-Wood)
+
+- Onedrive
+
+- Fix the configurator to allow
/teams/ID in the config (Nick Craig-Wood)
+
+- Oracleobjectstorage
+
+- Fix OpenOptions being ignored in uploadMultipart with chunkWriter (Nick Craig-Wood)
+
+- S3
+
+- Fix slice bounds out of range error when listing (Nick Craig-Wood)
+- Fix OpenOptions being ignored in uploadMultipart with chunkWriter (Vitor Gomes)
+
+- Storj
+
+- Update storj.io/uplink to v1.12.0 (Kaloyan Raev)
+
+
v1.64.0 - 2023-09-11
@@ -36414,7 +36722,7 @@ v1.64.0 - 2023-09-11
- Hdfs
- Retry "replication in progress" errors when uploading (Nick Craig-Wood)
-- Fix uploading to the wrong object on Update with overriden remote name (Nick Craig-Wood)
+- Fix uploading to the wrong object on Update with overridden remote name (Nick Craig-Wood)
- HTTP
@@ -36427,7 +36735,7 @@ v1.64.0 - 2023-09-11
- Oracleobjectstorage
-- Use rclone's rate limiter in mutipart transfers (Manoj Ghosh)
+- Use rclone's rate limiter in multipart transfers (Manoj Ghosh)
- Implement
OpenChunkWriter and multi-thread uploads (Manoj Ghosh)
- S3
@@ -36690,7 +36998,7 @@
v1.63.0 - 2023-06-30
- Putio
-- Fix uploading to the wrong object on Update with overriden remote name (Nick Craig-Wood)
+- Fix uploading to the wrong object on Update with overridden remote name (Nick Craig-Wood)
- Fix modification times not being preserved for server side copy and move (Nick Craig-Wood)
- Fix server side copy failures (400 errors) (Nick Craig-Wood)
@@ -36699,7 +37007,7 @@ v1.63.0 - 2023-06-30
- Empty directory markers (JÄnis BebrÄ«tis, Nick Craig-Wood)
- Update Scaleway storage classes (Brian Starkey)
- Fix
--s3-versions on individual objects (Nick Craig-Wood)
-- Fix hang on aborting multpart upload with iDrive e2 (Nick Craig-Wood)
+- Fix hang on aborting multipart upload with iDrive e2 (Nick Craig-Wood)
- Fix missing "tier" metadata (Nick Craig-Wood)
- Fix V3sign: add missing subresource delete (cc)
- Fix Arvancloud Domain and region changes and alphabetise the provider (Ehsan Tadayon)
@@ -36724,7 +37032,7 @@ v1.63.0 - 2023-06-30
- Storj
- Fix "uplink: too many requests" errors when uploading to the same file (Nick Craig-Wood)
-- Fix uploading to the wrong object on Update with overriden remote name (Nick Craig-Wood)
+- Fix uploading to the wrong object on Update with overridden remote name (Nick Craig-Wood)
- Swift
@@ -42141,7 +42449,7 @@ v1.38 - 2017-09-30
- Mount
-- Re-use
rcat internals to support uploads from all remotes
+- Reuse
rcat internals to support uploads from all remotes
- Dropbox
@@ -43316,7 +43624,7 @@ v0.00 - 2012-11-18
- Project started
Bugs and Limitations
-Limitations
+Limitations
Directory timestamps aren't preserved
Rclone doesn't currently preserve the timestamps of directories. This is because rclone only really considers objects when syncing.
Rclone struggles with millions of files in a directory/bucket
@@ -43325,7 +43633,7 @@ Rclone str
Bucket-based remotes and folders
Bucket-based remotes (e.g. S3/GCS/Swift/B2) do not have a concept of directories. Rclone therefore cannot create directories in them which means that empty directories on a bucket-based remote will tend to disappear.
Some software creates empty keys ending in / as directory markers. Rclone doesn't do this as it potentially creates more objects and costs more. This ability may be added in the future (probably via a flag/option).
-Bugs
+Bugs
Bugs are stored in rclone's GitHub project:
- Reported bugs
@@ -43959,7 +44267,7 @@ Contributors
- HNGamingUK connor@earnshawhome.co.uk
- Jonta 359397+Jonta@users.noreply.github.com
- YenForYang YenForYang@users.noreply.github.com
-- Joda Stößer stoesser@yay-digital.de services+github@simjo.st
+- SimJoSt / Joda Stößer git@simjo.st
- Logeshwaran waranlogesh@gmail.com
- Rajat Goel rajat@dropbox.com
- r0kk3rz r0kk3rz@gmail.com
@@ -44206,6 +44514,38 @@ Contributors
- Volodymyr Kit v.kit@maytech.net
- David Pedersen limero@me.com
- Drew Stinnett drew@drewlink.com
+- Pat Patterson pat@backblaze.com
+- Herby Gillot herby.gillot@gmail.com
+- Nikita Shoshin shoshin_nikita@fastmail.com
+- rinsuki 428rinsuki+git@gmail.com
+- Beyond Meat 51850644+beyondmeat@users.noreply.github.com
+- Saleh Dindar salh@fb.com
+- Volodymyr 142890760+vkit-maytech@users.noreply.github.com
+- Gabriel Espinoza 31670639+gspinoza@users.noreply.github.com
+- Keigo Imai keigo.imai@gmail.com
+- Ivan Yanitra iyanitra@tesla-consulting.com
+- alfish2000 alfish2000@gmail.com
+- wuxingzhong qq330332812@gmail.com
+- Adithya Kumar akumar42@protonmail.com
+- Tayo-pasedaRJ 138471223+Tayo-pasedaRJ@users.noreply.github.com
+- Peter Kreuser logo@kreuser.name
+- Piyush
+- fotile96 fotile96@users.noreply.github.com
+- Luc Ritchie luc.ritchie@gmail.com
+- cynful cynful@users.noreply.github.com
+- wjielai wjielai@tencent.com
+- Jack Deng jackdeng@gmail.com
+- Mikubill 31246794+Mikubill@users.noreply.github.com
+- Artur Neumann artur@jankaritech.com
+- Saw-jan saw.jan.grg3e@gmail.com
+- Oksana Zhykina o.zhykina@maytech.net
+- karan karan.gupta92@gmail.com
+- viktor viktor@yakovchuk.net
+- moongdal moongdal@tutanota.com
+- Mina Galić freebsd@igalic.co
+- Alen Å iljak dev@alensiljak.eu.org
+- ä½ çŸ¥é“未æ¥å— rkonfj@gmail.com
+- Abhinav Dhiman 8640877+ahnv@users.noreply.github.com
Contact the rclone project
Forum
diff --git a/MANUAL.md b/MANUAL.md
index 9b4c2e21df34b..13bee7b753b3e 100644
--- a/MANUAL.md
+++ b/MANUAL.md
@@ -1,6 +1,6 @@
% rclone(1) User Manual
% Nick Craig-Wood
-% Sep 11, 2023
+% Nov 26, 2023
# Rclone syncs your files to cloud storage
@@ -135,11 +135,14 @@ WebDAV or S3, that work out of the box.)
- Koofr
- Leviia Object Storage
- Liara Object Storage
+- Linkbox
+- Linode Object Storage
- Mail.ru Cloud
- Memset Memstore
- Mega
- Memory
- Microsoft Azure Blob Storage
+- Microsoft Azure Files Storage
- Microsoft OneDrive
- Minio
- Nextcloud
@@ -279,6 +282,19 @@ developers so it may be out of date. Its current version is as below.
[](https://repology.org/project/rclone/versions)
+### Installation with MacPorts (#macos-macports)
+
+On macOS, rclone can also be installed via [MacPorts](https://www.macports.org):
+
+ sudo port install rclone
+
+Note that this is a third party installer not controlled by the rclone
+developers so it may be out of date. Its current version is as below.
+
+[](https://repology.org/project/rclone/versions)
+
+More information [here](https://ports.macports.org/port/rclone/).
+
### Precompiled binary, using curl {#macos-precompiled}
To avoid problems with macOS gatekeeper enforcing the binary to be signed and
@@ -501,7 +517,7 @@ Make sure you have [Snapd installed](https://snapcraft.io/docs/installing-snapd)
```bash
$ sudo snap install rclone
```
-Due to the strict confinement of Snap, rclone snap cannot acess real /home/$USER/.config/rclone directory, default config path is as below.
+Due to the strict confinement of Snap, rclone snap cannot access real /home/$USER/.config/rclone directory, default config path is as below.
- Default config directory:
- /home/$USER/snap/rclone/current/.config/rclone
@@ -518,7 +534,7 @@ Note that this is controlled by [community maintainer](https://github.com/bouken
## Source installation {#source}
Make sure you have git and [Go](https://golang.org/) installed.
-Go version 1.17 or newer is required, latest release is recommended.
+Go version 1.18 or newer is required, the latest release is recommended.
You can get it from your package manager, or download it from
[golang.org/dl](https://golang.org/dl/). Then you can run the following:
@@ -552,26 +568,59 @@ port of GCC, e.g. by installing it in a [MSYS2](https://www.msys2.org)
distribution (make sure you install it in the classic mingw64 subsystem, the
ucrt64 version is not compatible).
-Additionally, on Windows, you must install the third party utility
-[WinFsp](https://winfsp.dev/), with the "Developer" feature selected.
+Additionally, to build with mount on Windows, you must install the third party
+utility [WinFsp](https://winfsp.dev/), with the "Developer" feature selected.
If building with cgo, you must also set environment variable CPATH pointing to
the fuse include directory within the WinFsp installation
(normally `C:\Program Files (x86)\WinFsp\inc\fuse`).
-You may also add arguments `-ldflags -s` (with or without `-tags cmount`),
-to omit symbol table and debug information, making the executable file smaller,
-and `-trimpath` to remove references to local file system paths. This is how
-the official rclone releases are built.
+You may add arguments `-ldflags -s` to omit symbol table and debug information,
+making the executable file smaller, and `-trimpath` to remove references to
+local file system paths. The official rclone releases are built with both of these.
```
go build -trimpath -ldflags -s -tags cmount
```
+If you want to customize the version string, as reported by
+the `rclone version` command, you can set one of the variables `fs.Version`,
+`fs.VersionTag` (to keep default suffix but customize the number),
+or `fs.VersionSuffix` (to keep default number but customize the suffix).
+This can be done from the build command, by adding to the `-ldflags`
+argument value as shown below.
+
+```
+go build -trimpath -ldflags "-s -X github.com/rclone/rclone/fs.Version=v9.9.9-test" -tags cmount
+```
+
+On Windows, the official executables also have the version information,
+as well as a file icon, embedded as binary resources. To get that with your
+own build you need to run the following command **before** the build command.
+It generates a Windows resource system object file, with extension .syso, e.g.
+`resource_windows_amd64.syso`, that will be automatically picked up by
+future build commands.
+
+```
+go run bin/resource_windows.go
+```
+
+The above command will generate a resource file containing version information
+based on the fs.Version variable in source at the time you run the command,
+which means if the value of this variable changes you need to re-run the
+command for it to be reflected in the version information. Also, if you
+override this version variable in the build command as described above, you
+need to do that also when generating the resource file, or else it will still
+use the value from the source.
+
+```
+go run bin/resource_windows.go -version v9.9.9-test
+```
+
Instead of executing the `go build` command directly, you can run it via the
-Makefile. It changes the version number suffix from "-DEV" to "-beta" and
-appends commit details. It also copies the resulting rclone executable into
-your GOPATH bin folder (`$(go env GOPATH)/bin`, which corresponds to
-`~/go/bin/rclone` by default).
+Makefile. The default target changes the version suffix from "-DEV" to "-beta"
+followed by additional commit details, embeds version information binary resources
+on Windows, and copies the resulting rclone executable into your GOPATH bin folder
+(`$(go env GOPATH)/bin`, which corresponds to `~/go/bin/rclone` by default).
```
make
@@ -584,32 +633,22 @@ make GOTAGS=cmount
```
There are other make targets that can be used for more advanced builds,
-such as cross-compiling for all supported os/architectures, embedding
-icon and version info resources into windows executable, and packaging
+such as cross-compiling for all supported os/architectures, and packaging
results into release artifacts.
See [Makefile](https://github.com/rclone/rclone/blob/master/Makefile)
and [cross-compile.go](https://github.com/rclone/rclone/blob/master/bin/cross-compile.go)
for details.
-Another alternative is to download the source, build and install rclone in one
-operation, as a regular Go package. The source will be stored it in the Go
-module cache, and the resulting executable will be in your GOPATH bin folder
-(`$(go env GOPATH)/bin`, which corresponds to `~/go/bin/rclone` by default).
-
-With Go version 1.17 or newer:
+Another alternative method for source installation is to download the source,
+build and install rclone - all in one operation, as a regular Go package.
+The source will be stored it in the Go module cache, and the resulting
+executable will be in your GOPATH bin folder (`$(go env GOPATH)/bin`,
+which corresponds to `~/go/bin/rclone` by default).
```
go install github.com/rclone/rclone@latest
```
-With Go versions older than 1.17 (do **not** use the `-u` flag, it causes Go to
-try to update the dependencies that rclone uses and sometimes these don't work
-with the current version):
-
-```
-go get github.com/rclone/rclone
-```
-
## Ansible installation {#ansible}
This can be done with [Stefan Weichinger's ansible
@@ -771,7 +810,7 @@ It requires .NET Framework, but it is preinstalled on newer versions of Windows,
also provides alternative standalone distributions which includes necessary runtime (.NET 5).
WinSW is a command-line only utility, where you have to manually create an XML file with
service configuration. This may be a drawback for some, but it can also be an advantage
-as it is easy to back up and re-use the configuration
+as it is easy to back up and reuse the configuration
settings, without having go through manual steps in a GUI. One thing to note is that
by default it does not restart the service on error, one have to explicit enable this
in the configuration file (via the "onfailure" parameter).
@@ -841,10 +880,12 @@ See the following for detailed instructions for
* [Internet Archive](https://rclone.org/internetarchive/)
* [Jottacloud](https://rclone.org/jottacloud/)
* [Koofr](https://rclone.org/koofr/)
+ * [Linkbox](https://rclone.org/linkbox/)
* [Mail.ru Cloud](https://rclone.org/mailru/)
* [Mega](https://rclone.org/mega/)
* [Memory](https://rclone.org/memory/)
* [Microsoft Azure Blob Storage](https://rclone.org/azureblob/)
+ * [Microsoft Azure Files Storage](https://rclone.org/azurefiles/)
* [Microsoft OneDrive](https://rclone.org/onedrive/)
* [OpenStack Swift / Rackspace Cloudfiles / Blomp Cloud Storage / Memset Memstore](https://rclone.org/swift/)
* [OpenDrive](https://rclone.org/opendrive/)
@@ -1024,11 +1065,11 @@ Flags for anything which can Copy a file.
-c, --checksum Check for changes with size & checksum (if available, or fallback to size only).
--compare-dest stringArray Include additional comma separated server-side paths during comparison
--copy-dest stringArray Implies --compare-dest but also copies files from paths into destination
- --cutoff-mode string Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default "HARD")
+ --cutoff-mode HARD|SOFT|CAUTIOUS Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default HARD)
--ignore-case-sync Ignore case when synchronizing
--ignore-checksum Skip post copy check of checksums
--ignore-existing Skip all files that exist on destination
- --ignore-size Ignore size when skipping use mod-time or checksum
+ --ignore-size Ignore size when skipping use modtime or checksum
-I, --ignore-times Don't skip files that match size and time - transfer all files
--immutable Do not modify files, fail if existing files have been modified
--inplace Download directly to destination file instead of atomic download to temp/rename
@@ -1043,11 +1084,12 @@ Flags for anything which can Copy a file.
--multi-thread-write-buffer-size SizeSuffix In memory buffer size for writing when in multi-thread mode (default 128Ki)
--no-check-dest Don't check the destination, copy regardless
--no-traverse Don't traverse destination file system on copy
- --no-update-modtime Don't update destination mod-time if files identical
+ --no-update-modtime Don't update destination modtime if files identical
--order-by string Instructions on how to order the transfers, e.g. 'size,descending'
+ --partial-suffix string Add partial-suffix to temporary file name when --inplace is not used (default ".partial")
--refresh-times Refresh the modtime of remote files
--server-side-across-configs Allow server-side operations (e.g. copy) to work across different configs
- --size-only Skip based on size only, not mod-time or checksum
+ --size-only Skip based on size only, not modtime or checksum
--streaming-upload-cutoff SizeSuffix Cutoff for switching to chunked upload if file size is unknown, upload starts after reaching cutoff or when file ends (default 100Ki)
-u, --update Skip files that are newer on the destination
```
@@ -1170,11 +1212,11 @@ Flags for anything which can Copy a file.
-c, --checksum Check for changes with size & checksum (if available, or fallback to size only).
--compare-dest stringArray Include additional comma separated server-side paths during comparison
--copy-dest stringArray Implies --compare-dest but also copies files from paths into destination
- --cutoff-mode string Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default "HARD")
+ --cutoff-mode HARD|SOFT|CAUTIOUS Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default HARD)
--ignore-case-sync Ignore case when synchronizing
--ignore-checksum Skip post copy check of checksums
--ignore-existing Skip all files that exist on destination
- --ignore-size Ignore size when skipping use mod-time or checksum
+ --ignore-size Ignore size when skipping use modtime or checksum
-I, --ignore-times Don't skip files that match size and time - transfer all files
--immutable Do not modify files, fail if existing files have been modified
--inplace Download directly to destination file instead of atomic download to temp/rename
@@ -1189,11 +1231,12 @@ Flags for anything which can Copy a file.
--multi-thread-write-buffer-size SizeSuffix In memory buffer size for writing when in multi-thread mode (default 128Ki)
--no-check-dest Don't check the destination, copy regardless
--no-traverse Don't traverse destination file system on copy
- --no-update-modtime Don't update destination mod-time if files identical
+ --no-update-modtime Don't update destination modtime if files identical
--order-by string Instructions on how to order the transfers, e.g. 'size,descending'
+ --partial-suffix string Add partial-suffix to temporary file name when --inplace is not used (default ".partial")
--refresh-times Refresh the modtime of remote files
--server-side-across-configs Allow server-side operations (e.g. copy) to work across different configs
- --size-only Skip based on size only, not mod-time or checksum
+ --size-only Skip based on size only, not modtime or checksum
--streaming-upload-cutoff SizeSuffix Cutoff for switching to chunked upload if file size is unknown, upload starts after reaching cutoff or when file ends (default 100Ki)
-u, --update Skip files that are newer on the destination
```
@@ -1330,11 +1373,11 @@ Flags for anything which can Copy a file.
-c, --checksum Check for changes with size & checksum (if available, or fallback to size only).
--compare-dest stringArray Include additional comma separated server-side paths during comparison
--copy-dest stringArray Implies --compare-dest but also copies files from paths into destination
- --cutoff-mode string Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default "HARD")
+ --cutoff-mode HARD|SOFT|CAUTIOUS Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default HARD)
--ignore-case-sync Ignore case when synchronizing
--ignore-checksum Skip post copy check of checksums
--ignore-existing Skip all files that exist on destination
- --ignore-size Ignore size when skipping use mod-time or checksum
+ --ignore-size Ignore size when skipping use modtime or checksum
-I, --ignore-times Don't skip files that match size and time - transfer all files
--immutable Do not modify files, fail if existing files have been modified
--inplace Download directly to destination file instead of atomic download to temp/rename
@@ -1349,11 +1392,12 @@ Flags for anything which can Copy a file.
--multi-thread-write-buffer-size SizeSuffix In memory buffer size for writing when in multi-thread mode (default 128Ki)
--no-check-dest Don't check the destination, copy regardless
--no-traverse Don't traverse destination file system on copy
- --no-update-modtime Don't update destination mod-time if files identical
+ --no-update-modtime Don't update destination modtime if files identical
--order-by string Instructions on how to order the transfers, e.g. 'size,descending'
+ --partial-suffix string Add partial-suffix to temporary file name when --inplace is not used (default ".partial")
--refresh-times Refresh the modtime of remote files
--server-side-across-configs Allow server-side operations (e.g. copy) to work across different configs
- --size-only Skip based on size only, not mod-time or checksum
+ --size-only Skip based on size only, not modtime or checksum
--streaming-upload-cutoff SizeSuffix Cutoff for switching to chunked upload if file size is unknown, upload starts after reaching cutoff or when file ends (default 100Ki)
-u, --update Skip files that are newer on the destination
```
@@ -2775,11 +2819,11 @@ Flags for anything which can Copy a file.
-c, --checksum Check for changes with size & checksum (if available, or fallback to size only).
--compare-dest stringArray Include additional comma separated server-side paths during comparison
--copy-dest stringArray Implies --compare-dest but also copies files from paths into destination
- --cutoff-mode string Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default "HARD")
+ --cutoff-mode HARD|SOFT|CAUTIOUS Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default HARD)
--ignore-case-sync Ignore case when synchronizing
--ignore-checksum Skip post copy check of checksums
--ignore-existing Skip all files that exist on destination
- --ignore-size Ignore size when skipping use mod-time or checksum
+ --ignore-size Ignore size when skipping use modtime or checksum
-I, --ignore-times Don't skip files that match size and time - transfer all files
--immutable Do not modify files, fail if existing files have been modified
--inplace Download directly to destination file instead of atomic download to temp/rename
@@ -2794,11 +2838,12 @@ Flags for anything which can Copy a file.
--multi-thread-write-buffer-size SizeSuffix In memory buffer size for writing when in multi-thread mode (default 128Ki)
--no-check-dest Don't check the destination, copy regardless
--no-traverse Don't traverse destination file system on copy
- --no-update-modtime Don't update destination mod-time if files identical
+ --no-update-modtime Don't update destination modtime if files identical
--order-by string Instructions on how to order the transfers, e.g. 'size,descending'
+ --partial-suffix string Add partial-suffix to temporary file name when --inplace is not used (default ".partial")
--refresh-times Refresh the modtime of remote files
--server-side-across-configs Allow server-side operations (e.g. copy) to work across different configs
- --size-only Skip based on size only, not mod-time or checksum
+ --size-only Skip based on size only, not modtime or checksum
--streaming-upload-cutoff SizeSuffix Cutoff for switching to chunked upload if file size is unknown, upload starts after reaching cutoff or when file ends (default 100Ki)
-u, --update Skip files that are newer on the destination
```
@@ -2950,17 +2995,20 @@ See the [global flags page](https://rclone.org/flags/) for global options not li
# rclone checksum
-Checks the files in the source against a SUM file.
+Checks the files in the destination against a SUM file.
## Synopsis
-Checks that hashsums of source files match the SUM file.
+Checks that hashsums of destination files match the SUM file.
It compares hashes (MD5, SHA1, etc) and logs a report of files which
don't match. It doesn't alter the file system.
-If you supply the `--download` flag, it will download the data from remote
-and calculate the contents hash on the fly. This can be useful for remotes
+The sumfile is treated as the source and the dst:path is treated as
+the destination for the purposes of the output.
+
+If you supply the `--download` flag, it will download the data from the remote
+and calculate the content hash on the fly. This can be useful for remotes
that don't support hashes or if you really want to check all the data.
Note that hash values in the SUM file are treated as case insensitive.
@@ -2991,7 +3039,7 @@ option for more information.
```
-rclone checksum sumfile src:path [flags]
+rclone checksum sumfile dst:path [flags]
```
## Options
@@ -3903,11 +3951,11 @@ Flags for anything which can Copy a file.
-c, --checksum Check for changes with size & checksum (if available, or fallback to size only).
--compare-dest stringArray Include additional comma separated server-side paths during comparison
--copy-dest stringArray Implies --compare-dest but also copies files from paths into destination
- --cutoff-mode string Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default "HARD")
+ --cutoff-mode HARD|SOFT|CAUTIOUS Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default HARD)
--ignore-case-sync Ignore case when synchronizing
--ignore-checksum Skip post copy check of checksums
--ignore-existing Skip all files that exist on destination
- --ignore-size Ignore size when skipping use mod-time or checksum
+ --ignore-size Ignore size when skipping use modtime or checksum
-I, --ignore-times Don't skip files that match size and time - transfer all files
--immutable Do not modify files, fail if existing files have been modified
--inplace Download directly to destination file instead of atomic download to temp/rename
@@ -3922,11 +3970,12 @@ Flags for anything which can Copy a file.
--multi-thread-write-buffer-size SizeSuffix In memory buffer size for writing when in multi-thread mode (default 128Ki)
--no-check-dest Don't check the destination, copy regardless
--no-traverse Don't traverse destination file system on copy
- --no-update-modtime Don't update destination mod-time if files identical
+ --no-update-modtime Don't update destination modtime if files identical
--order-by string Instructions on how to order the transfers, e.g. 'size,descending'
+ --partial-suffix string Add partial-suffix to temporary file name when --inplace is not used (default ".partial")
--refresh-times Refresh the modtime of remote files
--server-side-across-configs Allow server-side operations (e.g. copy) to work across different configs
- --size-only Skip based on size only, not mod-time or checksum
+ --size-only Skip based on size only, not modtime or checksum
--streaming-upload-cutoff SizeSuffix Cutoff for switching to chunked upload if file size is unknown, upload starts after reaching cutoff or when file ends (default 100Ki)
-u, --update Skip files that are newer on the destination
```
@@ -4454,10 +4503,6 @@ Run without a hash to see the list of all supported hashes, e.g.
* whirlpool
* crc32
* sha256
- * dropbox
- * hidrive
- * mailru
- * quickxor
Then
@@ -4467,7 +4512,7 @@ Note that hash names are case insensitive and values are output in lower case.
```
-rclone hashsum remote:path [flags]
+rclone hashsum [ remote:path] [flags]
```
## Options
@@ -4967,7 +5012,6 @@ Mount the remote as file system on a mountpoint.
## Synopsis
-
rclone mount allows Linux, FreeBSD, macOS and Windows to
mount any of Rclone's cloud storage systems as a file system with
FUSE.
@@ -5222,11 +5266,17 @@ does not suffer from the same limitations.
## Mounting on macOS
-Mounting on macOS can be done either via [macFUSE](https://osxfuse.github.io/)
+Mounting on macOS can be done either via [built-in NFS server](https://rclone.org/commands/rclone_serve_nfs/), [macFUSE](https://osxfuse.github.io/)
(also known as osxfuse) or [FUSE-T](https://www.fuse-t.org/). macFUSE is a traditional
FUSE driver utilizing a macOS kernel extension (kext). FUSE-T is an alternative FUSE system
which "mounts" via an NFSv4 local server.
+# NFS mount
+
+This method spins up an NFS server using [serve nfs](https://rclone.org/commands/rclone_serve_nfs/) command and mounts
+it to the specified mountpoint. If you run this in background mode using |--daemon|, you will need to
+send SIGTERM signal to the rclone process using |kill| command to stop the mount.
+
### macFUSE Notes
If installing macFUSE using [dmg packages](https://github.com/osxfuse/osxfuse/releases) from
@@ -5276,6 +5326,8 @@ sequentially, it can only seek when reading. This means that many
applications won't work with their files on an rclone mount without
`--vfs-cache-mode writes` or `--vfs-cache-mode full`.
See the [VFS File Caching](#vfs-file-caching) section for more info.
+When using NFS mount on macOS, if you don't specify |--vfs-cache-mode|
+the mount point will be read-only.
The bucket-based remotes (e.g. Swift, S3, Google Compute Storage, B2)
do not support the concept of empty directories, so empty
@@ -5422,7 +5474,6 @@ Mount option syntax includes a few extra options treated specially:
- `vv...` will be transformed into appropriate `--verbose=N`
- standard mount options like `x-systemd.automount`, `_netdev`, `nosuid` and alike
are intended only for Automountd and ignored by rclone.
-
## VFS - Virtual File System
This command uses the VFS layer. This adapts the cloud storage objects
@@ -5804,6 +5855,7 @@ rclone mount remote:path /path/to/mountpoint [flags]
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi)
--vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off)
--vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms)
+ --vfs-refresh Refreshes the directory cache recursively on start
--vfs-used-is-size rclone size Use the rclone size algorithm for Used size
--vfs-write-back Duration Time to writeback files after last use when using cache (default 5s)
--vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s)
@@ -5906,11 +5958,11 @@ Flags for anything which can Copy a file.
-c, --checksum Check for changes with size & checksum (if available, or fallback to size only).
--compare-dest stringArray Include additional comma separated server-side paths during comparison
--copy-dest stringArray Implies --compare-dest but also copies files from paths into destination
- --cutoff-mode string Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default "HARD")
+ --cutoff-mode HARD|SOFT|CAUTIOUS Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default HARD)
--ignore-case-sync Ignore case when synchronizing
--ignore-checksum Skip post copy check of checksums
--ignore-existing Skip all files that exist on destination
- --ignore-size Ignore size when skipping use mod-time or checksum
+ --ignore-size Ignore size when skipping use modtime or checksum
-I, --ignore-times Don't skip files that match size and time - transfer all files
--immutable Do not modify files, fail if existing files have been modified
--inplace Download directly to destination file instead of atomic download to temp/rename
@@ -5925,11 +5977,12 @@ Flags for anything which can Copy a file.
--multi-thread-write-buffer-size SizeSuffix In memory buffer size for writing when in multi-thread mode (default 128Ki)
--no-check-dest Don't check the destination, copy regardless
--no-traverse Don't traverse destination file system on copy
- --no-update-modtime Don't update destination mod-time if files identical
+ --no-update-modtime Don't update destination modtime if files identical
--order-by string Instructions on how to order the transfers, e.g. 'size,descending'
+ --partial-suffix string Add partial-suffix to temporary file name when --inplace is not used (default ".partial")
--refresh-times Refresh the modtime of remote files
--server-side-across-configs Allow server-side operations (e.g. copy) to work across different configs
- --size-only Skip based on size only, not mod-time or checksum
+ --size-only Skip based on size only, not modtime or checksum
--streaming-upload-cutoff SizeSuffix Cutoff for switching to chunked upload if file size is unknown, upload starts after reaching cutoff or when file ends (default 100Ki)
-u, --update Skip files that are newer on the destination
```
@@ -6395,6 +6448,17 @@ to be used within the template to server pages:
|-- .Size | Size in Bytes of the entry. |
|-- .ModTime | The UTC timestamp of an entry. |
+The server also makes the following functions available so that they can be used within the
+template. These functions help extend the options for dynamic rendering of HTML. They can
+be used to render HTML based on specific conditions.
+
+| Function | Description |
+| :---------- | :---------- |
+| afterEpoch | Returns the time since the epoch for the given time. |
+| contains | Checks whether a given substring is present or not in a given string. |
+| hasPrefix | Checks whether the given string begins with the specified prefix. |
+| hasSuffix | Checks whether the given string end with the specified suffix. |
+
### Authentication
By default this will serve files without needing a login.
@@ -6536,7 +6600,6 @@ Update the rclone binary.
## Synopsis
-
This command downloads the latest release of rclone and replaces the
currently running binary. The download is verified with a hashsum and
cryptographically signed signature; see [the release signing
@@ -6643,7 +6706,9 @@ See the [global flags page](https://rclone.org/flags/) for global options not li
* [rclone serve docker](https://rclone.org/commands/rclone_serve_docker/) - Serve any remote on docker's volume plugin API.
* [rclone serve ftp](https://rclone.org/commands/rclone_serve_ftp/) - Serve remote:path over FTP.
* [rclone serve http](https://rclone.org/commands/rclone_serve_http/) - Serve the remote over HTTP.
+* [rclone serve nfs](https://rclone.org/commands/rclone_serve_nfs/) - Serve the remote as an NFS mount
* [rclone serve restic](https://rclone.org/commands/rclone_serve_restic/) - Serve the remote for restic's REST API.
+* [rclone serve s3](https://rclone.org/commands/rclone_serve_s3/) - Serve remote:path over s3.
* [rclone serve sftp](https://rclone.org/commands/rclone_serve_sftp/) - Serve the remote over SFTP.
* [rclone serve webdav](https://rclone.org/commands/rclone_serve_webdav/) - Serve remote:path over WebDAV.
@@ -6676,7 +6741,6 @@ default "rclone (hostname)".
Use `--log-trace` in conjunction with `-vv` to enable additional debug
logging of all UPNP traffic.
-
## VFS - Virtual File System
This command uses the VFS layer. This adapts the cloud storage objects
@@ -7045,6 +7109,7 @@ rclone serve dlna remote:path [flags]
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi)
--vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off)
--vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms)
+ --vfs-refresh Refreshes the directory cache recursively on start
--vfs-used-is-size rclone size Use the rclone size algorithm for Used size
--vfs-write-back Duration Time to writeback files after last use when using cache (default 5s)
--vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s)
@@ -7092,7 +7157,6 @@ Serve any remote on docker's volume plugin API.
## Synopsis
-
This command implements the Docker volume plugin API allowing docker to use
rclone as a data storage mechanism for various cloud providers.
rclone provides [docker volume plugin](/docker) based on it.
@@ -7131,7 +7195,6 @@ directory with book-keeping records of created and mounted volumes.
All mount and VFS options are submitted by the docker daemon via API, but
you can also provide defaults on the command line as well as set path to the
config file and cache directory or adjust logging verbosity.
-
## VFS - Virtual File System
This command uses the VFS layer. This adapts the cloud storage objects
@@ -7518,6 +7581,7 @@ rclone serve docker [flags]
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi)
--vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off)
--vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms)
+ --vfs-refresh Refreshes the directory cache recursively on start
--vfs-used-is-size rclone size Use the rclone size algorithm for Used size
--vfs-write-back Duration Time to writeback files after last use when using cache (default 5s)
--vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s)
@@ -7587,7 +7651,6 @@ then using Authentication is advised - see the next section for info.
By default this will serve files without needing a login.
You can set a single username and password with the --user and --pass flags.
-
## VFS - Virtual File System
This command uses the VFS layer. This adapts the cloud storage objects
@@ -8040,6 +8103,7 @@ rclone serve ftp remote:path [flags]
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi)
--vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off)
--vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms)
+ --vfs-refresh Refreshes the directory cache recursively on start
--vfs-used-is-size rclone size Use the rclone size algorithm for Used size
--vfs-write-back Duration Time to writeback files after last use when using cache (default 5s)
--vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s)
@@ -8171,6 +8235,17 @@ to be used within the template to server pages:
|-- .Size | Size in Bytes of the entry. |
|-- .ModTime | The UTC timestamp of an entry. |
+The server also makes the following functions available so that they can be used within the
+template. These functions help extend the options for dynamic rendering of HTML. They can
+be used to render HTML based on specific conditions.
+
+| Function | Description |
+| :---------- | :---------- |
+| afterEpoch | Returns the time since the epoch for the given time. |
+| contains | Checks whether a given substring is present or not in a given string. |
+| hasPrefix | Checks whether the given string begins with the specified prefix. |
+| hasSuffix | Checks whether the given string end with the specified suffix. |
+
### Authentication
By default this will serve files without needing a login.
@@ -8197,7 +8272,6 @@ The password file can be updated while rclone is running.
Use `--realm` to set the authentication realm.
Use `--salt` to change the password hashing salt from the default.
-
## VFS - Virtual File System
This command uses the VFS layer. This adapts the cloud storage objects
@@ -8659,6 +8733,7 @@ rclone serve http remote:path [flags]
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi)
--vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off)
--vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms)
+ --vfs-refresh Refreshes the directory cache recursively on start
--vfs-used-is-size rclone size Use the rclone size algorithm for Used size
--vfs-write-back Duration Time to writeback files after last use when using cache (default 5s)
--vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s)
@@ -8700,263 +8775,40 @@ See the [global flags page](https://rclone.org/flags/) for global options not li
* [rclone serve](https://rclone.org/commands/rclone_serve/) - Serve a remote over a protocol.
-# rclone serve restic
+# rclone serve nfs
-Serve the remote for restic's REST API.
+Serve the remote as an NFS mount
## Synopsis
-Run a basic web server to serve a remote over restic's REST backend
-API over HTTP. This allows restic to use rclone as a data storage
-mechanism for cloud providers that restic does not support directly.
-
-[Restic](https://restic.net/) is a command-line program for doing
-backups.
-
-The server will log errors. Use -v to see access logs.
-
-`--bwlimit` will be respected for file transfers.
-Use `--stats` to control the stats printing.
-
-## Setting up rclone for use by restic ###
-
-First [set up a remote for your chosen cloud provider](https://rclone.org/docs/#configure).
-
-Once you have set up the remote, check it is working with, for example
-"rclone lsd remote:". You may have called the remote something other
-than "remote:" - just substitute whatever you called it in the
-following instructions.
-
-Now start the rclone restic server
-
- rclone serve restic -v remote:backup
-
-Where you can replace "backup" in the above by whatever path in the
-remote you wish to use.
-
-By default this will serve on "localhost:8080" you can change this
-with use of the `--addr` flag.
-
-You might wish to start this server on boot.
-
-Adding `--cache-objects=false` will cause rclone to stop caching objects
-returned from the List call. Caching is normally desirable as it speeds
-up downloading objects, saves transactions and uses very little memory.
-
-## Setting up restic to use rclone ###
-
-Now you can [follow the restic
-instructions](http://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#rest-server)
-on setting up restic.
-
-Note that you will need restic 0.8.2 or later to interoperate with
-rclone.
-
-For the example above you will want to use "http://localhost:8080/" as
-the URL for the REST server.
-
-For example:
-
- $ export RESTIC_REPOSITORY=rest:http://localhost:8080/
- $ export RESTIC_PASSWORD=yourpassword
- $ restic init
- created restic backend 8b1a4b56ae at rest:http://localhost:8080/
-
- Please note that knowledge of your password is required to access
- the repository. Losing your password means that your data is
- irrecoverably lost.
- $ restic backup /path/to/files/to/backup
- scan [/path/to/files/to/backup]
- scanned 189 directories, 312 files in 0:00
- [0:00] 100.00% 38.128 MiB / 38.128 MiB 501 / 501 items 0 errors ETA 0:00
- duration: 0:00
- snapshot 45c8fdd8 saved
-
-### Multiple repositories ####
-
-Note that you can use the endpoint to host multiple repositories. Do
-this by adding a directory name or path after the URL. Note that
-these **must** end with /. Eg
-
- $ export RESTIC_REPOSITORY=rest:http://localhost:8080/user1repo/
- # backup user1 stuff
- $ export RESTIC_REPOSITORY=rest:http://localhost:8080/user2repo/
- # backup user2 stuff
-
-### Private repositories ####
-
-The`--private-repos` flag can be used to limit users to repositories starting
-with a path of `//`.
-
-## Server options
-
-Use `--addr` to specify which IP address and port the server should
-listen on, eg `--addr 1.2.3.4:8000` or `--addr :8080` to listen to all
-IPs. By default it only listens on localhost. You can use port
-:0 to let the OS choose an available port.
-
-If you set `--addr` to listen on a public or LAN accessible IP address
-then using Authentication is advised - see the next section for info.
-
-You can use a unix socket by setting the url to `unix:///path/to/socket`
-or just by using an absolute path name. Note that unix sockets bypass the
-authentication - this is expected to be done with file system permissions.
-
-`--addr` may be repeated to listen on multiple IPs/ports/sockets.
-
-`--server-read-timeout` and `--server-write-timeout` can be used to
-control the timeouts on the server. Note that this is the total time
-for a transfer.
-
-`--max-header-bytes` controls the maximum number of bytes the server will
-accept in the HTTP header.
-
-`--baseurl` controls the URL prefix that rclone serves from. By default
-rclone will serve from the root. If you used `--baseurl "/rclone"` then
-rclone would serve from a URL starting with "/rclone/". This is
-useful if you wish to proxy rclone serve. Rclone automatically
-inserts leading and trailing "/" on `--baseurl`, so `--baseurl "rclone"`,
-`--baseurl "/rclone"` and `--baseurl "/rclone/"` are all treated
-identically.
-
-### TLS (SSL)
-
-By default this will serve over http. If you want you can serve over
-https. You will need to supply the `--cert` and `--key` flags.
-If you wish to do client side certificate validation then you will need to
-supply `--client-ca` also.
-
-`--cert` should be a either a PEM encoded certificate or a concatenation
-of that with the CA certificate. `--key` should be the PEM encoded
-private key and `--client-ca` should be the PEM encoded client
-certificate authority certificate.
-
---min-tls-version is minimum TLS version that is acceptable. Valid
- values are "tls1.0", "tls1.1", "tls1.2" and "tls1.3" (default
- "tls1.0").
-
-### Authentication
-
-By default this will serve files without needing a login.
-
-You can either use an htpasswd file which can take lots of users, or
-set a single username and password with the `--user` and `--pass` flags.
-
-If no static users are configured by either of the above methods, and client
-certificates are required by the `--client-ca` flag passed to the server, the
-client certificate common name will be considered as the username.
-
-Use `--htpasswd /path/to/htpasswd` to provide an htpasswd file. This is
-in standard apache format and supports MD5, SHA1 and BCrypt for basic
-authentication. Bcrypt is recommended.
-
-To create an htpasswd file:
-
- touch htpasswd
- htpasswd -B htpasswd user
- htpasswd -B htpasswd anotherUser
-
-The password file can be updated while rclone is running.
-
-Use `--realm` to set the authentication realm.
-
-Use `--salt` to change the password hashing salt from the default.
-
-
-```
-rclone serve restic remote:path [flags]
-```
-
-## Options
-
-```
- --addr stringArray IPaddress:Port or :Port to bind server to (default [127.0.0.1:8080])
- --allow-origin string Origin which cross-domain request (CORS) can be executed from
- --append-only Disallow deletion of repository data
- --baseurl string Prefix for URLs - leave blank for root
- --cache-objects Cache listed objects (default true)
- --cert string TLS PEM key (concatenation of certificate and CA certificate)
- --client-ca string Client certificate authority to verify clients with
- -h, --help help for restic
- --htpasswd string A htpasswd file - if not provided no authentication is done
- --key string TLS PEM Private key
- --max-header-bytes int Maximum size of request header (default 4096)
- --min-tls-version string Minimum TLS version that is acceptable (default "tls1.0")
- --pass string Password for authentication
- --private-repos Users can only access their private repo
- --realm string Realm for authentication
- --salt string Password hashing salt (default "dlPL2MqE")
- --server-read-timeout Duration Timeout for server reading data (default 1h0m0s)
- --server-write-timeout Duration Timeout for server writing data (default 1h0m0s)
- --stdio Run an HTTP2 server on stdin/stdout
- --user string User name for authentication
-```
-
-
-See the [global flags page](https://rclone.org/flags/) for global options not listed here.
-
-# SEE ALSO
-
-* [rclone serve](https://rclone.org/commands/rclone_serve/) - Serve a remote over a protocol.
-
-# rclone serve sftp
-
-Serve the remote over SFTP.
-
-## Synopsis
-
-Run an SFTP server to serve a remote over SFTP. This can be used
-with an SFTP client or you can make a remote of type [sftp](/sftp) to use with it.
-
-You can use the [filter](/filtering) flags (e.g. `--include`, `--exclude`)
-to control what is served.
-
-The server will respond to a small number of shell commands, mainly
-md5sum, sha1sum and df, which enable it to provide support for checksums
-and the about feature when accessed from an sftp remote.
-
-Note that this server uses standard 32 KiB packet payload size, which
-means you must not configure the client to expect anything else, e.g.
-with the [chunk_size](https://rclone.org/sftp/#sftp-chunk-size) option on an sftp remote.
-
-The server will log errors. Use `-v` to see access logs.
-
-`--bwlimit` will be respected for file transfers.
-Use `--stats` to control the stats printing.
-
-You must provide some means of authentication, either with
-`--user`/`--pass`, an authorized keys file (specify location with
-`--authorized-keys` - the default is the same as ssh), an
-`--auth-proxy`, or set the `--no-auth` flag for no
-authentication when logging in.
+Create an NFS server that serves the given remote over the network.
+
+The primary purpose for this command is to enable [mount command](https://rclone.org/commands/rclone_mount/) on recent macOS versions where
+installing FUSE is very cumbersome.
-If you don't supply a host `--key` then rclone will generate rsa, ecdsa
-and ed25519 variants, and cache them for later use in rclone's cache
-directory (see `rclone help flags cache-dir`) in the "serve-sftp"
-directory.
+Since this is running on NFSv3, no authentication method is available. Any client
+will be able to access the data. To limit access, you can use serve NFS on loopback address
+and rely on secure tunnels (such as SSH). For this reason, by default, a random TCP port is chosen and loopback interface is used for the listening address;
+meaning that it is only available to the local machine. If you want other machines to access the
+NFS mount over local network, you need to specify the listening address and port using `--addr` flag.
-By default the server binds to localhost:2022 - if you want it to be
-reachable externally then supply `--addr :2022` for example.
+Modifying files through NFS protocol requires VFS caching. Usually you will need to specify `--vfs-cache-mode`
+in order to be able to write to the mountpoint (full is recommended). If you don't specify VFS cache mode,
+the mount will be read-only.
-Note that the default of `--vfs-cache-mode off` is fine for the rclone
-sftp backend, but it may not be with other SFTP clients.
+To serve NFS over the network use following command:
-If `--stdio` is specified, rclone will serve SFTP over stdio, which can
-be used with sshd via ~/.ssh/authorized_keys, for example:
+ rclone serve nfs remote: --addr 0.0.0.0:$PORT --vfs-cache-mode=full
- restrict,command="rclone serve sftp --stdio ./photos" ssh-rsa ...
+We specify a specific port that we can use in the mount command:
-On the client you need to set `--transfers 1` when using `--stdio`.
-Otherwise multiple instances of the rclone server are started by OpenSSH
-which can lead to "corrupted on transfer" errors. This is the case because
-the client chooses indiscriminately which server to send commands to while
-the servers all have different views of the state of the filing system.
+To mount the server under Linux/macOS, use the following command:
+
+ mount -oport=$PORT,mountport=$PORT $HOSTNAME: path/to/mountpoint
-The "restrict" in authorized_keys prevents SHA1SUMs and MD5SUMs from being
-used. Omitting "restrict" and using `--sftp-path-override` to enable
-checksumming is possible but less secure and you could use the SFTP server
-provided by OpenSSH in this case.
+Where `$PORT` is the same port number we used in the serve nfs command.
+This feature is only available on Unix platforms.
## VFS - Virtual File System
@@ -9289,115 +9141,27 @@ _WARNING._ Contrary to `rclone size`, this flag ignores filters so that the
result is accurate. However, this is very inefficient and may cost lots of API
calls resulting in extra charges. Use it as a last resort and only with caching.
-## Auth Proxy
-
-If you supply the parameter `--auth-proxy /path/to/program` then
-rclone will use that program to generate backends on the fly which
-then are used to authenticate incoming requests. This uses a simple
-JSON based protocol with input on STDIN and output on STDOUT.
-
-**PLEASE NOTE:** `--auth-proxy` and `--authorized-keys` cannot be used
-together, if `--auth-proxy` is set the authorized keys option will be
-ignored.
-
-There is an example program
-[bin/test_proxy.py](https://github.com/rclone/rclone/blob/master/test_proxy.py)
-in the rclone source code.
-
-The program's job is to take a `user` and `pass` on the input and turn
-those into the config for a backend on STDOUT in JSON format. This
-config will have any default parameters for the backend added, but it
-won't use configuration from environment variables or command line
-options - it is the job of the proxy program to make a complete
-config.
-
-This config generated must have this extra parameter
-- `_root` - root to use for the backend
-
-And it may have this parameter
-- `_obscure` - comma separated strings for parameters to obscure
-
-If password authentication was used by the client, input to the proxy
-process (on STDIN) would look similar to this:
-
-```
-{
- "user": "me",
- "pass": "mypassword"
-}
-```
-
-If public-key authentication was used by the client, input to the
-proxy process (on STDIN) would look similar to this:
-
-```
-{
- "user": "me",
- "public_key": "AAAAB3NzaC1yc2EAAAADAQABAAABAQDuwESFdAe14hVS6omeyX7edc...JQdf"
-}
-```
-
-And as an example return this on STDOUT
-
-```
-{
- "type": "sftp",
- "_root": "",
- "_obscure": "pass",
- "user": "me",
- "pass": "mypassword",
- "host": "sftp.example.com"
-}
-```
-
-This would mean that an SFTP backend would be created on the fly for
-the `user` and `pass`/`public_key` returned in the output to the host given. Note
-that since `_obscure` is set to `pass`, rclone will obscure the `pass`
-parameter before creating the backend (which is required for sftp
-backends).
-
-The program can manipulate the supplied `user` in any way, for example
-to make proxy to many different sftp backends, you could make the
-`user` be `user@example.com` and then set the `host` to `example.com`
-in the output and the user to `user`. For security you'd probably want
-to restrict the `host` to a limited list.
-
-Note that an internal cache is keyed on `user` so only use that for
-configuration, don't use `pass` or `public_key`. This also means that if a user's
-password or public-key is changed the cache will need to expire (which takes 5 mins)
-before it takes effect.
-
-This can be used to build general purpose proxies to any kind of
-backend that rclone supports.
-
```
-rclone serve sftp remote:path [flags]
+rclone serve nfs remote:path [flags]
```
## Options
```
- --addr string IPaddress:Port or :Port to bind server to (default "localhost:2022")
- --auth-proxy string A program to use to create the backend from the auth
- --authorized-keys string Authorized keys file (default "~/.ssh/authorized_keys")
+ --addr string IPaddress:Port or :Port to bind server to
--dir-cache-time Duration Time to cache directory entries for (default 5m0s)
--dir-perms FileMode Directory permissions (default 0777)
--file-perms FileMode File permissions (default 0666)
--gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000)
- -h, --help help for sftp
- --key stringArray SSH private host key file (Can be multi-valued, leave blank to auto generate)
- --no-auth Allow connections with no authentication if set
+ -h, --help help for nfs
--no-checksum Don't compare checksums on up/download
--no-modtime Don't read/write the modification time (can speed things up)
--no-seek Don't allow seeking in files
- --pass string Password for authentication
--poll-interval Duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s)
--read-only Only allow read-only access
- --stdio Run an sftp server on stdin/stdout
--uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000)
--umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2)
- --user string User name for authentication
--vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s)
--vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off)
--vfs-cache-min-free-space SizeSuffix Target minimum free space on the disk containing the cache (default off)
@@ -9410,6 +9174,7 @@ rclone serve sftp remote:path [flags]
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi)
--vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off)
--vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms)
+ --vfs-refresh Refreshes the directory cache recursively on start
--vfs-used-is-size rclone size Use the rclone size algorithm for Used size
--vfs-write-back Duration Time to writeback files after last use when using cache (default 5s)
--vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s)
@@ -9451,52 +9216,93 @@ See the [global flags page](https://rclone.org/flags/) for global options not li
* [rclone serve](https://rclone.org/commands/rclone_serve/) - Serve a remote over a protocol.
-# rclone serve webdav
+# rclone serve restic
-Serve remote:path over WebDAV.
+Serve the remote for restic's REST API.
## Synopsis
-Run a basic WebDAV server to serve a remote over HTTP via the
-WebDAV protocol. This can be viewed with a WebDAV client, through a web
-browser, or you can make a remote of type WebDAV to read and write it.
+Run a basic web server to serve a remote over restic's REST backend
+API over HTTP. This allows restic to use rclone as a data storage
+mechanism for cloud providers that restic does not support directly.
-## WebDAV options
+[Restic](https://restic.net/) is a command-line program for doing
+backups.
-### --etag-hash
+The server will log errors. Use -v to see access logs.
-This controls the ETag header. Without this flag the ETag will be
-based on the ModTime and Size of the object.
+`--bwlimit` will be respected for file transfers.
+Use `--stats` to control the stats printing.
-If this flag is set to "auto" then rclone will choose the first
-supported hash on the backend or you can use a named hash such as
-"MD5" or "SHA-1". Use the [hashsum](https://rclone.org/commands/rclone_hashsum/) command
-to see the full list.
+## Setting up rclone for use by restic ###
-## Access WebDAV on Windows
-WebDAV shared folder can be mapped as a drive on Windows, however the default settings prevent it.
-Windows will fail to connect to the server using insecure Basic authentication.
-It will not even display any login dialog. Windows requires SSL / HTTPS connection to be used with Basic.
-If you try to connect via Add Network Location Wizard you will get the following error:
-"The folder you entered does not appear to be valid. Please choose another".
-However, you still can connect if you set the following registry key on a client machine:
-HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\WebClient\Parameters\BasicAuthLevel to 2.
-The BasicAuthLevel can be set to the following values:
- 0 - Basic authentication disabled
- 1 - Basic authentication enabled for SSL connections only
- 2 - Basic authentication enabled for SSL connections and for non-SSL connections
-If required, increase the FileSizeLimitInBytes to a higher value.
-Navigate to the Services interface, then restart the WebClient service.
+First [set up a remote for your chosen cloud provider](https://rclone.org/docs/#configure).
-## Access Office applications on WebDAV
-Navigate to following registry HKEY_CURRENT_USER\Software\Microsoft\Office\[14.0/15.0/16.0]\Common\Internet
-Create a new DWORD BasicAuthLevel with value 2.
- 0 - Basic authentication disabled
- 1 - Basic authentication enabled for SSL connections only
- 2 - Basic authentication enabled for SSL and for non-SSL connections
+Once you have set up the remote, check it is working with, for example
+"rclone lsd remote:". You may have called the remote something other
+than "remote:" - just substitute whatever you called it in the
+following instructions.
-https://learn.microsoft.com/en-us/office/troubleshoot/powerpoint/office-opens-blank-from-sharepoint
+Now start the rclone restic server
+
+ rclone serve restic -v remote:backup
+
+Where you can replace "backup" in the above by whatever path in the
+remote you wish to use.
+
+By default this will serve on "localhost:8080" you can change this
+with use of the `--addr` flag.
+
+You might wish to start this server on boot.
+
+Adding `--cache-objects=false` will cause rclone to stop caching objects
+returned from the List call. Caching is normally desirable as it speeds
+up downloading objects, saves transactions and uses very little memory.
+
+## Setting up restic to use rclone ###
+
+Now you can [follow the restic
+instructions](http://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#rest-server)
+on setting up restic.
+
+Note that you will need restic 0.8.2 or later to interoperate with
+rclone.
+
+For the example above you will want to use "http://localhost:8080/" as
+the URL for the REST server.
+
+For example:
+
+ $ export RESTIC_REPOSITORY=rest:http://localhost:8080/
+ $ export RESTIC_PASSWORD=yourpassword
+ $ restic init
+ created restic backend 8b1a4b56ae at rest:http://localhost:8080/
+
+ Please note that knowledge of your password is required to access
+ the repository. Losing your password means that your data is
+ irrecoverably lost.
+ $ restic backup /path/to/files/to/backup
+ scan [/path/to/files/to/backup]
+ scanned 189 directories, 312 files in 0:00
+ [0:00] 100.00% 38.128 MiB / 38.128 MiB 501 / 501 items 0 errors ETA 0:00
+ duration: 0:00
+ snapshot 45c8fdd8 saved
+
+### Multiple repositories ####
+
+Note that you can use the endpoint to host multiple repositories. Do
+this by adding a directory name or path after the URL. Note that
+these **must** end with /. Eg
+
+ $ export RESTIC_REPOSITORY=rest:http://localhost:8080/user1repo/
+ # backup user1 stuff
+ $ export RESTIC_REPOSITORY=rest:http://localhost:8080/user2repo/
+ # backup user2 stuff
+### Private repositories ####
+
+The`--private-repos` flag can be used to limit users to repositories starting
+with a path of `//`.
## Server options
@@ -9545,31 +9351,6 @@ certificate authority certificate.
values are "tls1.0", "tls1.1", "tls1.2" and "tls1.3" (default
"tls1.0").
-### Template
-
-`--template` allows a user to specify a custom markup template for HTTP
-and WebDAV serve functions. The server exports the following markup
-to be used within the template to server pages:
-
-| Parameter | Description |
-| :---------- | :---------- |
-| .Name | The full path of a file/directory. |
-| .Title | Directory listing of .Name |
-| .Sort | The current sort used. This is changeable via ?sort= parameter |
-| | Sort Options: namedirfirst,name,size,time (default namedirfirst) |
-| .Order | The current ordering used. This is changeable via ?order= parameter |
-| | Order Options: asc,desc (default asc) |
-| .Query | Currently unused. |
-| .Breadcrumb | Allows for creating a relative navigation |
-|-- .Link | The relative to the root link of the Text. |
-|-- .Text | The Name of the directory. |
-| .Entries | Information about a specific file/directory. |
-|-- .URL | The 'url' of an entry. |
-|-- .Leaf | Currently same as 'URL' but intended to be 'just' the name. |
-|-- .IsDir | Boolean for if an entry is a directory or not. |
-|-- .Size | Size in Bytes of the entry. |
-|-- .ModTime | The UTC timestamp of an entry. |
-
### Authentication
By default this will serve files without needing a login.
@@ -9597,6 +9378,211 @@ Use `--realm` to set the authentication realm.
Use `--salt` to change the password hashing salt from the default.
+
+```
+rclone serve restic remote:path [flags]
+```
+
+## Options
+
+```
+ --addr stringArray IPaddress:Port or :Port to bind server to (default [127.0.0.1:8080])
+ --allow-origin string Origin which cross-domain request (CORS) can be executed from
+ --append-only Disallow deletion of repository data
+ --baseurl string Prefix for URLs - leave blank for root
+ --cache-objects Cache listed objects (default true)
+ --cert string TLS PEM key (concatenation of certificate and CA certificate)
+ --client-ca string Client certificate authority to verify clients with
+ -h, --help help for restic
+ --htpasswd string A htpasswd file - if not provided no authentication is done
+ --key string TLS PEM Private key
+ --max-header-bytes int Maximum size of request header (default 4096)
+ --min-tls-version string Minimum TLS version that is acceptable (default "tls1.0")
+ --pass string Password for authentication
+ --private-repos Users can only access their private repo
+ --realm string Realm for authentication
+ --salt string Password hashing salt (default "dlPL2MqE")
+ --server-read-timeout Duration Timeout for server reading data (default 1h0m0s)
+ --server-write-timeout Duration Timeout for server writing data (default 1h0m0s)
+ --stdio Run an HTTP2 server on stdin/stdout
+ --user string User name for authentication
+```
+
+
+See the [global flags page](https://rclone.org/flags/) for global options not listed here.
+
+# SEE ALSO
+
+* [rclone serve](https://rclone.org/commands/rclone_serve/) - Serve a remote over a protocol.
+
+# rclone serve s3
+
+Serve remote:path over s3.
+
+## Synopsis
+
+`serve s3` implements a basic s3 server that serves a remote via s3.
+This can be viewed with an s3 client, or you can make an [s3 type
+remote](https://rclone.org/s3/) to read and write to it with rclone.
+
+`serve s3` is considered **Experimental** so use with care.
+
+S3 server supports Signature Version 4 authentication. Just use
+`--auth-key accessKey,secretKey` and set the `Authorization`
+header correctly in the request. (See the [AWS
+docs](https://docs.aws.amazon.com/general/latest/gr/signature-version-4.html)).
+
+`--auth-key` can be repeated for multiple auth pairs. If
+`--auth-key` is not provided then `serve s3` will allow anonymous
+access.
+
+Please note that some clients may require HTTPS endpoints. See [the
+SSL docs](#ssl-tls) for more information.
+
+This command uses the [VFS directory cache](#vfs-virtual-file-system).
+All the functionality will work with `--vfs-cache-mode off`. Using
+`--vfs-cache-mode full` (or `writes`) can be used to cache objects
+locally to improve performance.
+
+Use `--force-path-style=false` if you want to use the bucket name as a
+part of the hostname (such as mybucket.local)
+
+Use `--etag-hash` if you want to change the hash uses for the `ETag`.
+Note that using anything other than `MD5` (the default) is likely to
+cause problems for S3 clients which rely on the Etag being the MD5.
+
+## Quickstart
+
+For a simple set up, to serve `remote:path` over s3, run the server
+like this:
+
+```
+rclone serve s3 --auth-key ACCESS_KEY_ID,SECRET_ACCESS_KEY remote:path
+```
+
+This will be compatible with an rclone remote which is defined like this:
+
+```
+[serves3]
+type = s3
+provider = Rclone
+endpoint = http://127.0.0.1:8080/
+access_key_id = ACCESS_KEY_ID
+secret_access_key = SECRET_ACCESS_KEY
+use_multipart_uploads = false
+```
+
+Note that setting `disable_multipart_uploads = true` is to work around
+[a bug](#bugs) which will be fixed in due course.
+
+## Bugs
+
+When uploading multipart files `serve s3` holds all the parts in
+memory (see [#7453](https://github.com/rclone/rclone/issues/7453)).
+This is a limitaton of the library rclone uses for serving S3 and will
+hopefully be fixed at some point.
+
+Multipart server side copies do not work (see
+[#7454](https://github.com/rclone/rclone/issues/7454)). These take a
+very long time and eventually fail. The default threshold for
+multipart server side copies is 5G which is the maximum it can be, so
+files above this side will fail to be server side copied.
+
+For a current list of `serve s3` bugs see the [serve
+s3](https://github.com/rclone/rclone/labels/serve%20s3) bug category
+on GitHub.
+
+## Limitations
+
+`serve s3` will treat all directories in the root as buckets and
+ignore all files in the root. You can use `CreateBucket` to create
+folders under the root, but you can't create empty folders under other
+folders not in the root.
+
+When using `PutObject` or `DeleteObject`, rclone will automatically
+create or clean up empty folders. If you don't want to clean up empty
+folders automatically, use `--no-cleanup`.
+
+When using `ListObjects`, rclone will use `/` when the delimiter is
+empty. This reduces backend requests with no effect on most
+operations, but if the delimiter is something other than `/` and
+empty, rclone will do a full recursive search of the backend, which
+can take some time.
+
+Versioning is not currently supported.
+
+Metadata will only be saved in memory other than the rclone `mtime`
+metadata which will be set as the modification time of the file.
+
+## Supported operations
+
+`serve s3` currently supports the following operations.
+
+- Bucket
+ - `ListBuckets`
+ - `CreateBucket`
+ - `DeleteBucket`
+- Object
+ - `HeadObject`
+ - `ListObjects`
+ - `GetObject`
+ - `PutObject`
+ - `DeleteObject`
+ - `DeleteObjects`
+ - `CreateMultipartUpload`
+ - `CompleteMultipartUpload`
+ - `AbortMultipartUpload`
+ - `CopyObject`
+ - `UploadPart`
+
+Other operations will return error `Unimplemented`.
+
+## Server options
+
+Use `--addr` to specify which IP address and port the server should
+listen on, eg `--addr 1.2.3.4:8000` or `--addr :8080` to listen to all
+IPs. By default it only listens on localhost. You can use port
+:0 to let the OS choose an available port.
+
+If you set `--addr` to listen on a public or LAN accessible IP address
+then using Authentication is advised - see the next section for info.
+
+You can use a unix socket by setting the url to `unix:///path/to/socket`
+or just by using an absolute path name. Note that unix sockets bypass the
+authentication - this is expected to be done with file system permissions.
+
+`--addr` may be repeated to listen on multiple IPs/ports/sockets.
+
+`--server-read-timeout` and `--server-write-timeout` can be used to
+control the timeouts on the server. Note that this is the total time
+for a transfer.
+
+`--max-header-bytes` controls the maximum number of bytes the server will
+accept in the HTTP header.
+
+`--baseurl` controls the URL prefix that rclone serves from. By default
+rclone will serve from the root. If you used `--baseurl "/rclone"` then
+rclone would serve from a URL starting with "/rclone/". This is
+useful if you wish to proxy rclone serve. Rclone automatically
+inserts leading and trailing "/" on `--baseurl`, so `--baseurl "rclone"`,
+`--baseurl "/rclone"` and `--baseurl "/rclone/"` are all treated
+identically.
+
+### TLS (SSL)
+
+By default this will serve over http. If you want you can serve over
+https. You will need to supply the `--cert` and `--key` flags.
+If you wish to do client side certificate validation then you will need to
+supply `--client-ca` also.
+
+`--cert` should be a either a PEM encoded certificate or a concatenation
+of that with the CA certificate. `--key` should be the PEM encoded
+private key and `--client-ca` should be the PEM encoded client
+certificate authority certificate.
+
+--min-tls-version is minimum TLS version that is acceptable. Valid
+ values are "tls1.0", "tls1.1", "tls1.2" and "tls1.3" (default
+ "tls1.0").
## VFS - Virtual File System
This command uses the VFS layer. This adapts the cloud storage objects
@@ -9928,90 +9914,9 @@ _WARNING._ Contrary to `rclone size`, this flag ignores filters so that the
result is accurate. However, this is very inefficient and may cost lots of API
calls resulting in extra charges. Use it as a last resort and only with caching.
-## Auth Proxy
-
-If you supply the parameter `--auth-proxy /path/to/program` then
-rclone will use that program to generate backends on the fly which
-then are used to authenticate incoming requests. This uses a simple
-JSON based protocol with input on STDIN and output on STDOUT.
-
-**PLEASE NOTE:** `--auth-proxy` and `--authorized-keys` cannot be used
-together, if `--auth-proxy` is set the authorized keys option will be
-ignored.
-
-There is an example program
-[bin/test_proxy.py](https://github.com/rclone/rclone/blob/master/test_proxy.py)
-in the rclone source code.
-
-The program's job is to take a `user` and `pass` on the input and turn
-those into the config for a backend on STDOUT in JSON format. This
-config will have any default parameters for the backend added, but it
-won't use configuration from environment variables or command line
-options - it is the job of the proxy program to make a complete
-config.
-
-This config generated must have this extra parameter
-- `_root` - root to use for the backend
-
-And it may have this parameter
-- `_obscure` - comma separated strings for parameters to obscure
-
-If password authentication was used by the client, input to the proxy
-process (on STDIN) would look similar to this:
-
-```
-{
- "user": "me",
- "pass": "mypassword"
-}
-```
-
-If public-key authentication was used by the client, input to the
-proxy process (on STDIN) would look similar to this:
-
-```
-{
- "user": "me",
- "public_key": "AAAAB3NzaC1yc2EAAAADAQABAAABAQDuwESFdAe14hVS6omeyX7edc...JQdf"
-}
-```
-
-And as an example return this on STDOUT
-
-```
-{
- "type": "sftp",
- "_root": "",
- "_obscure": "pass",
- "user": "me",
- "pass": "mypassword",
- "host": "sftp.example.com"
-}
-```
-
-This would mean that an SFTP backend would be created on the fly for
-the `user` and `pass`/`public_key` returned in the output to the host given. Note
-that since `_obscure` is set to `pass`, rclone will obscure the `pass`
-parameter before creating the backend (which is required for sftp
-backends).
-
-The program can manipulate the supplied `user` in any way, for example
-to make proxy to many different sftp backends, you could make the
-`user` be `user@example.com` and then set the `host` to `example.com`
-in the output and the user to `user`. For security you'd probably want
-to restrict the `host` to a limited list.
-
-Note that an internal cache is keyed on `user` so only use that for
-configuration, don't use `pass` or `public_key`. This also means that if a user's
-password or public-key is changed the cache will need to expire (which takes 5 mins)
-before it takes effect.
-
-This can be used to build general purpose proxies to any kind of
-backend that rclone supports.
-
```
-rclone serve webdav remote:path [flags]
+rclone serve s3 remote:path [flags]
```
## Options
@@ -10019,35 +9924,30 @@ rclone serve webdav remote:path [flags]
```
--addr stringArray IPaddress:Port or :Port to bind server to (default [127.0.0.1:8080])
--allow-origin string Origin which cross-domain request (CORS) can be executed from
- --auth-proxy string A program to use to create the backend from the auth
+ --auth-key stringArray Set key pair for v4 authorization: access_key_id,secret_access_key
--baseurl string Prefix for URLs - leave blank for root
--cert string TLS PEM key (concatenation of certificate and CA certificate)
--client-ca string Client certificate authority to verify clients with
--dir-cache-time Duration Time to cache directory entries for (default 5m0s)
--dir-perms FileMode Directory permissions (default 0777)
- --disable-dir-list Disable HTML directory list on GET request for a directory
- --etag-hash string Which hash to use for the ETag, or auto or blank for off
+ --etag-hash string Which hash to use for the ETag, or auto or blank for off (default "MD5")
--file-perms FileMode File permissions (default 0666)
+ --force-path-style If true use path style access if false use virtual hosted style (default true) (default true)
--gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000)
- -h, --help help for webdav
- --htpasswd string A htpasswd file - if not provided no authentication is done
+ -h, --help help for s3
--key string TLS PEM Private key
--max-header-bytes int Maximum size of request header (default 4096)
--min-tls-version string Minimum TLS version that is acceptable (default "tls1.0")
--no-checksum Don't compare checksums on up/download
+ --no-cleanup Not to cleanup empty folder after object is deleted
--no-modtime Don't read/write the modification time (can speed things up)
--no-seek Don't allow seeking in files
- --pass string Password for authentication
--poll-interval Duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s)
--read-only Only allow read-only access
- --realm string Realm for authentication
- --salt string Password hashing salt (default "dlPL2MqE")
--server-read-timeout Duration Timeout for server reading data (default 1h0m0s)
--server-write-timeout Duration Timeout for server writing data (default 1h0m0s)
- --template string User-specified template
--uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000)
--umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2)
- --user string User name for authentication
--vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s)
--vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off)
--vfs-cache-min-free-space SizeSuffix Target minimum free space on the disk containing the cache (default off)
@@ -10060,6 +9960,7 @@ rclone serve webdav remote:path [flags]
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi)
--vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off)
--vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms)
+ --vfs-refresh Refreshes the directory cache recursively on start
--vfs-used-is-size rclone size Use the rclone size algorithm for Used size
--vfs-write-back Duration Time to writeback files after last use when using cache (default 5s)
--vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s)
@@ -10101,316 +10002,523 @@ See the [global flags page](https://rclone.org/flags/) for global options not li
* [rclone serve](https://rclone.org/commands/rclone_serve/) - Serve a remote over a protocol.
-# rclone settier
+# rclone serve sftp
-Changes storage class/tier of objects in remote.
+Serve the remote over SFTP.
## Synopsis
+Run an SFTP server to serve a remote over SFTP. This can be used
+with an SFTP client or you can make a remote of type [sftp](/sftp) to use with it.
-rclone settier changes storage tier or class at remote if supported.
-Few cloud storage services provides different storage classes on objects,
-for example AWS S3 and Glacier, Azure Blob storage - Hot, Cool and Archive,
-Google Cloud Storage, Regional Storage, Nearline, Coldline etc.
+You can use the [filter](/filtering) flags (e.g. `--include`, `--exclude`)
+to control what is served.
-Note that, certain tier changes make objects not available to access immediately.
-For example tiering to archive in azure blob storage makes objects in frozen state,
-user can restore by setting tier to Hot/Cool, similarly S3 to Glacier makes object
-inaccessible.true
+The server will respond to a small number of shell commands, mainly
+md5sum, sha1sum and df, which enable it to provide support for checksums
+and the about feature when accessed from an sftp remote.
-You can use it to tier single object
+Note that this server uses standard 32 KiB packet payload size, which
+means you must not configure the client to expect anything else, e.g.
+with the [chunk_size](https://rclone.org/sftp/#sftp-chunk-size) option on an sftp remote.
- rclone settier Cool remote:path/file
+The server will log errors. Use `-v` to see access logs.
-Or use rclone filters to set tier on only specific files
+`--bwlimit` will be respected for file transfers.
+Use `--stats` to control the stats printing.
- rclone --include "*.txt" settier Hot remote:path/dir
+You must provide some means of authentication, either with
+`--user`/`--pass`, an authorized keys file (specify location with
+`--authorized-keys` - the default is the same as ssh), an
+`--auth-proxy`, or set the `--no-auth` flag for no
+authentication when logging in.
-Or just provide remote directory and all files in directory will be tiered
+If you don't supply a host `--key` then rclone will generate rsa, ecdsa
+and ed25519 variants, and cache them for later use in rclone's cache
+directory (see `rclone help flags cache-dir`) in the "serve-sftp"
+directory.
- rclone settier tier remote:path/dir
+By default the server binds to localhost:2022 - if you want it to be
+reachable externally then supply `--addr :2022` for example.
+Note that the default of `--vfs-cache-mode off` is fine for the rclone
+sftp backend, but it may not be with other SFTP clients.
-```
-rclone settier tier remote:path [flags]
-```
+If `--stdio` is specified, rclone will serve SFTP over stdio, which can
+be used with sshd via ~/.ssh/authorized_keys, for example:
-## Options
+ restrict,command="rclone serve sftp --stdio ./photos" ssh-rsa ...
-```
- -h, --help help for settier
-```
+On the client you need to set `--transfers 1` when using `--stdio`.
+Otherwise multiple instances of the rclone server are started by OpenSSH
+which can lead to "corrupted on transfer" errors. This is the case because
+the client chooses indiscriminately which server to send commands to while
+the servers all have different views of the state of the filing system.
+The "restrict" in authorized_keys prevents SHA1SUMs and MD5SUMs from being
+used. Omitting "restrict" and using `--sftp-path-override` to enable
+checksumming is possible but less secure and you could use the SFTP server
+provided by OpenSSH in this case.
-See the [global flags page](https://rclone.org/flags/) for global options not listed here.
+## VFS - Virtual File System
-# SEE ALSO
+This command uses the VFS layer. This adapts the cloud storage objects
+that rclone uses into something which looks much more like a disk
+filing system.
-* [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends.
+Cloud storage objects have lots of properties which aren't like disk
+files - you can't extend them or write to the middle of them, so the
+VFS layer has to deal with that. Because there is no one right way of
+doing this there are various options explained below.
-# rclone test
+The VFS layer also implements a directory cache - this caches info
+about files and directories (but not the data) in memory.
-Run a test command
+## VFS Directory Cache
-## Synopsis
+Using the `--dir-cache-time` flag, you can control how long a
+directory should be considered up to date and not refreshed from the
+backend. Changes made through the VFS will appear immediately or
+invalidate the cache.
-Rclone test is used to run test commands.
+ --dir-cache-time duration Time to cache directory entries for (default 5m0s)
+ --poll-interval duration Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable (default 1m0s)
-Select which test command you want with the subcommand, eg
+However, changes made directly on the cloud storage by the web
+interface or a different copy of rclone will only be picked up once
+the directory cache expires if the backend configured does not support
+polling for changes. If the backend supports polling, changes will be
+picked up within the polling interval.
- rclone test memory remote:
+You can send a `SIGHUP` signal to rclone for it to flush all
+directory caches, regardless of how old they are. Assuming only one
+rclone instance is running, you can reset the cache like this:
-Each subcommand has its own options which you can see in their help.
+ kill -SIGHUP $(pidof rclone)
-**NB** Be careful running these commands, they may do strange things
-so reading their documentation first is recommended.
+If you configure rclone with a [remote control](/rc) then you can use
+rclone rc to flush the whole directory cache:
+ rclone rc vfs/forget
-## Options
+Or individual files or directories:
-```
- -h, --help help for test
-```
+ rclone rc vfs/forget file=path/to/file dir=path/to/dir
+## VFS File Buffering
-See the [global flags page](https://rclone.org/flags/) for global options not listed here.
+The `--buffer-size` flag determines the amount of memory,
+that will be used to buffer data in advance.
-# SEE ALSO
+Each open file will try to keep the specified amount of data in memory
+at all times. The buffered data is bound to one open file and won't be
+shared.
-* [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends.
-* [rclone test changenotify](https://rclone.org/commands/rclone_test_changenotify/) - Log any change notify requests for the remote passed in.
-* [rclone test histogram](https://rclone.org/commands/rclone_test_histogram/) - Makes a histogram of file name characters.
-* [rclone test info](https://rclone.org/commands/rclone_test_info/) - Discovers file name or other limitations for paths.
-* [rclone test makefile](https://rclone.org/commands/rclone_test_makefile/) - Make files with random contents of the size given
-* [rclone test makefiles](https://rclone.org/commands/rclone_test_makefiles/) - Make a random file hierarchy in a directory
-* [rclone test memory](https://rclone.org/commands/rclone_test_memory/) - Load all the objects at remote:path into memory and report memory stats.
+This flag is a upper limit for the used memory per open file. The
+buffer will only use memory for data that is downloaded but not not
+yet read. If the buffer is empty, only a small amount of memory will
+be used.
-# rclone test changenotify
+The maximum memory used by rclone for buffering can be up to
+`--buffer-size * open files`.
-Log any change notify requests for the remote passed in.
+## VFS File Caching
-```
-rclone test changenotify remote: [flags]
-```
+These flags control the VFS file caching options. File caching is
+necessary to make the VFS layer appear compatible with a normal file
+system. It can be disabled at the cost of some compatibility.
-## Options
+For example you'll need to enable VFS caching if you want to read and
+write simultaneously to a file. See below for more details.
-```
- -h, --help help for changenotify
- --poll-interval Duration Time to wait between polling for changes (default 10s)
-```
+Note that the VFS cache is separate from the cache backend and you may
+find that you need one or the other or both.
+ --cache-dir string Directory rclone will use for caching.
+ --vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off)
+ --vfs-cache-max-age duration Max time since last access of objects in the cache (default 1h0m0s)
+ --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off)
+ --vfs-cache-min-free-space SizeSuffix Target minimum free space on the disk containing the cache (default off)
+ --vfs-cache-poll-interval duration Interval to poll the cache for stale objects (default 1m0s)
+ --vfs-write-back duration Time to writeback files after last use when using cache (default 5s)
-See the [global flags page](https://rclone.org/flags/) for global options not listed here.
+If run with `-vv` rclone will print the location of the file cache. The
+files are stored in the user cache file area which is OS dependent but
+can be controlled with `--cache-dir` or setting the appropriate
+environment variable.
-# SEE ALSO
+The cache has 4 different modes selected by `--vfs-cache-mode`.
+The higher the cache mode the more compatible rclone becomes at the
+cost of using disk space.
-* [rclone test](https://rclone.org/commands/rclone_test/) - Run a test command
+Note that files are written back to the remote only when they are
+closed and if they haven't been accessed for `--vfs-write-back`
+seconds. If rclone is quit or dies with files that haven't been
+uploaded, these will be uploaded next time rclone is run with the same
+flags.
-# rclone test histogram
+If using `--vfs-cache-max-size` or `--vfs-cache-min-free-size` note
+that the cache may exceed these quotas for two reasons. Firstly
+because it is only checked every `--vfs-cache-poll-interval`. Secondly
+because open files cannot be evicted from the cache. When
+`--vfs-cache-max-size` or `--vfs-cache-min-free-size` is exceeded,
+rclone will attempt to evict the least accessed files from the cache
+first. rclone will start with files that haven't been accessed for the
+longest. This cache flushing strategy is efficient and more relevant
+files are likely to remain cached.
-Makes a histogram of file name characters.
+The `--vfs-cache-max-age` will evict files from the cache
+after the set time since last access has passed. The default value of
+1 hour will start evicting files from cache that haven't been accessed
+for 1 hour. When a cached file is accessed the 1 hour timer is reset to 0
+and will wait for 1 more hour before evicting. Specify the time with
+standard notation, s, m, h, d, w .
-## Synopsis
+You **should not** run two copies of rclone using the same VFS cache
+with the same or overlapping remotes if using `--vfs-cache-mode > off`.
+This can potentially cause data corruption if you do. You can work
+around this by giving each rclone its own cache hierarchy with
+`--cache-dir`. You don't need to worry about this if the remotes in
+use don't overlap.
-This command outputs JSON which shows the histogram of characters used
-in filenames in the remote:path specified.
+### --vfs-cache-mode off
-The data doesn't contain any identifying information but is useful for
-the rclone developers when developing filename compression.
+In this mode (the default) the cache will read directly from the remote and write
+directly to the remote without caching anything on disk.
+This will mean some operations are not possible
-```
-rclone test histogram [remote:path] [flags]
-```
+ * Files can't be opened for both read AND write
+ * Files opened for write can't be seeked
+ * Existing files opened for write must have O_TRUNC set
+ * Files open for read with O_TRUNC will be opened write only
+ * Files open for write only will behave as if O_TRUNC was supplied
+ * Open modes O_APPEND, O_TRUNC are ignored
+ * If an upload fails it can't be retried
-## Options
+### --vfs-cache-mode minimal
-```
- -h, --help help for histogram
-```
+This is very similar to "off" except that files opened for read AND
+write will be buffered to disk. This means that files opened for
+write will be a lot more compatible, but uses the minimal disk space.
+These operations are not possible
-See the [global flags page](https://rclone.org/flags/) for global options not listed here.
+ * Files opened for write only can't be seeked
+ * Existing files opened for write must have O_TRUNC set
+ * Files opened for write only will ignore O_APPEND, O_TRUNC
+ * If an upload fails it can't be retried
-# SEE ALSO
+### --vfs-cache-mode writes
-* [rclone test](https://rclone.org/commands/rclone_test/) - Run a test command
+In this mode files opened for read only are still read directly from
+the remote, write only and read/write files are buffered to disk
+first.
-# rclone test info
+This mode should support all normal file system operations.
-Discovers file name or other limitations for paths.
+If an upload fails it will be retried at exponentially increasing
+intervals up to 1 minute.
-## Synopsis
+### --vfs-cache-mode full
-rclone info discovers what filenames and upload methods are possible
-to write to the paths passed in and how long they can be. It can take some
-time. It will write test files into the remote:path passed in. It outputs
-a bit of go code for each one.
+In this mode all reads and writes are buffered to and from disk. When
+data is read from the remote this is buffered to disk as well.
-**NB** this can create undeletable files and other hazards - use with care
+In this mode the files in the cache will be sparse files and rclone
+will keep track of which bits of the files it has downloaded.
+So if an application only reads the starts of each file, then rclone
+will only buffer the start of the file. These files will appear to be
+their full size in the cache, but they will be sparse files with only
+the data that has been downloaded present in them.
-```
-rclone test info [remote:path]+ [flags]
-```
+This mode should support all normal file system operations and is
+otherwise identical to `--vfs-cache-mode` writes.
-## Options
+When reading a file rclone will read `--buffer-size` plus
+`--vfs-read-ahead` bytes ahead. The `--buffer-size` is buffered in memory
+whereas the `--vfs-read-ahead` is buffered on disk.
-```
- --all Run all tests
- --check-base32768 Check can store all possible base32768 characters
- --check-control Check control characters
- --check-length Check max filename length
- --check-normalization Check UTF-8 Normalization
- --check-streaming Check uploads with indeterminate file size
- -h, --help help for info
- --upload-wait Duration Wait after writing a file (default 0s)
- --write-json string Write results to file
-```
+When using this mode it is recommended that `--buffer-size` is not set
+too large and `--vfs-read-ahead` is set large if required.
+**IMPORTANT** not all file systems support sparse files. In particular
+FAT/exFAT do not. Rclone will perform very badly if the cache
+directory is on a filesystem which doesn't support sparse files and it
+will log an ERROR message if one is detected.
-See the [global flags page](https://rclone.org/flags/) for global options not listed here.
+### Fingerprinting
-# SEE ALSO
+Various parts of the VFS use fingerprinting to see if a local file
+copy has changed relative to a remote file. Fingerprints are made
+from:
-* [rclone test](https://rclone.org/commands/rclone_test/) - Run a test command
+- size
+- modification time
+- hash
-# rclone test makefile
+where available on an object.
-Make files with random contents of the size given
+On some backends some of these attributes are slow to read (they take
+an extra API call per object, or extra work per object).
-```
-rclone test makefile []+ [flags]
-```
+For example `hash` is slow with the `local` and `sftp` backends as
+they have to read the entire file and hash it, and `modtime` is slow
+with the `s3`, `swift`, `ftp` and `qinqstor` backends because they
+need to do an extra API call to fetch it.
-## Options
+If you use the `--vfs-fast-fingerprint` flag then rclone will not
+include the slow operations in the fingerprint. This makes the
+fingerprinting less accurate but much faster and will improve the
+opening time of cached files.
-```
- --ascii Fill files with random ASCII printable bytes only
- --chargen Fill files with a ASCII chargen pattern
- -h, --help help for makefile
- --pattern Fill files with a periodic pattern
- --seed int Seed for the random number generator (0 for random) (default 1)
- --sparse Make the files sparse (appear to be filled with ASCII 0x00)
- --zero Fill files with ASCII 0x00
-```
+If you are running a vfs cache over `local`, `s3` or `swift` backends
+then using this flag is recommended.
+Note that if you change the value of this flag, the fingerprints of
+the files in the cache may be invalidated and the files will need to
+be downloaded again.
-See the [global flags page](https://rclone.org/flags/) for global options not listed here.
+## VFS Chunked Reading
-# SEE ALSO
+When rclone reads files from a remote it reads them in chunks. This
+means that rather than requesting the whole file rclone reads the
+chunk specified. This can reduce the used download quota for some
+remotes by requesting only chunks from the remote that are actually
+read, at the cost of an increased number of requests.
-* [rclone test](https://rclone.org/commands/rclone_test/) - Run a test command
+These flags control the chunking:
-# rclone test makefiles
+ --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M)
+ --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off)
-Make a random file hierarchy in a directory
+Rclone will start reading a chunk of size `--vfs-read-chunk-size`,
+and then double the size for each read. When `--vfs-read-chunk-size-limit` is
+specified, and greater than `--vfs-read-chunk-size`, the chunk size for each
+open file will get doubled only until the specified value is reached. If the
+value is "off", which is the default, the limit is disabled and the chunk size
+will grow indefinitely.
-```
-rclone test makefiles [flags]
-```
+With `--vfs-read-chunk-size 100M` and `--vfs-read-chunk-size-limit 0`
+the following parts will be downloaded: 0-100M, 100M-200M, 200M-300M, 300M-400M and so on.
+When `--vfs-read-chunk-size-limit 500M` is specified, the result would be
+0-100M, 100M-300M, 300M-700M, 700M-1200M, 1200M-1700M and so on.
-## Options
+Setting `--vfs-read-chunk-size` to `0` or "off" disables chunked reading.
-```
- --ascii Fill files with random ASCII printable bytes only
- --chargen Fill files with a ASCII chargen pattern
- --files int Number of files to create (default 1000)
- --files-per-directory int Average number of files per directory (default 10)
- -h, --help help for makefiles
- --max-depth int Maximum depth of directory hierarchy (default 10)
- --max-file-size SizeSuffix Maximum size of files to create (default 100)
- --max-name-length int Maximum size of file names (default 12)
- --min-file-size SizeSuffix Minimum size of file to create
- --min-name-length int Minimum size of file names (default 4)
- --pattern Fill files with a periodic pattern
- --seed int Seed for the random number generator (0 for random) (default 1)
- --sparse Make the files sparse (appear to be filled with ASCII 0x00)
- --zero Fill files with ASCII 0x00
-```
+## VFS Performance
+These flags may be used to enable/disable features of the VFS for
+performance or other reasons. See also the [chunked reading](#vfs-chunked-reading)
+feature.
-See the [global flags page](https://rclone.org/flags/) for global options not listed here.
+In particular S3 and Swift benefit hugely from the `--no-modtime` flag
+(or use `--use-server-modtime` for a slightly different effect) as each
+read of the modification time takes a transaction.
-# SEE ALSO
+ --no-checksum Don't compare checksums on up/download.
+ --no-modtime Don't read/write the modification time (can speed things up).
+ --no-seek Don't allow seeking in files.
+ --read-only Only allow read-only access.
-* [rclone test](https://rclone.org/commands/rclone_test/) - Run a test command
+Sometimes rclone is delivered reads or writes out of order. Rather
+than seeking rclone will wait a short time for the in sequence read or
+write to come in. These flags only come into effect when not using an
+on disk cache file.
-# rclone test memory
+ --vfs-read-wait duration Time to wait for in-sequence read before seeking (default 20ms)
+ --vfs-write-wait duration Time to wait for in-sequence write before giving error (default 1s)
-Load all the objects at remote:path into memory and report memory stats.
+When using VFS write caching (`--vfs-cache-mode` with value writes or full),
+the global flag `--transfers` can be set to adjust the number of parallel uploads of
+modified files from the cache (the related global flag `--checkers` has no effect on the VFS).
-```
-rclone test memory remote:path [flags]
-```
+ --transfers int Number of file transfers to run in parallel (default 4)
-## Options
+## VFS Case Sensitivity
-```
- -h, --help help for memory
-```
+Linux file systems are case-sensitive: two files can differ only
+by case, and the exact case must be used when opening a file.
+File systems in modern Windows are case-insensitive but case-preserving:
+although existing files can be opened using any case, the exact case used
+to create the file is preserved and available for programs to query.
+It is not allowed for two files in the same directory to differ only by case.
-See the [global flags page](https://rclone.org/flags/) for global options not listed here.
+Usually file systems on macOS are case-insensitive. It is possible to make macOS
+file systems case-sensitive but that is not the default.
-# SEE ALSO
+The `--vfs-case-insensitive` VFS flag controls how rclone handles these
+two cases. If its value is "false", rclone passes file names to the remote
+as-is. If the flag is "true" (or appears without a value on the
+command line), rclone may perform a "fixup" as explained below.
-* [rclone test](https://rclone.org/commands/rclone_test/) - Run a test command
+The user may specify a file name to open/delete/rename/etc with a case
+different than what is stored on the remote. If an argument refers
+to an existing file with exactly the same name, then the case of the existing
+file on the disk will be used. However, if a file name with exactly the same
+name is not found but a name differing only by case exists, rclone will
+transparently fixup the name. This fixup happens only when an existing file
+is requested. Case sensitivity of file names created anew by rclone is
+controlled by the underlying remote.
-# rclone touch
+Note that case sensitivity of the operating system running rclone (the target)
+may differ from case sensitivity of a file system presented by rclone (the source).
+The flag controls whether "fixup" is performed to satisfy the target.
-Create new file or change file modification time.
+If the flag is not provided on the command line, then its default value depends
+on the operating system where rclone runs: "true" on Windows and macOS, "false"
+otherwise. If the flag is provided without a value, then it is "true".
-## Synopsis
+## VFS Disk Options
+This flag allows you to manually set the statistics about the filing system.
+It can be useful when those statistics cannot be read correctly automatically.
-Set the modification time on file(s) as specified by remote:path to
-have the current time.
+ --vfs-disk-space-total-size Manually set the total disk space size (example: 256G, default: -1)
-If remote:path does not exist then a zero sized file will be created,
-unless `--no-create` or `--recursive` is provided.
+## Alternate report of used bytes
-If `--recursive` is used then recursively sets the modification
-time on all existing files that is found under the path. Filters are supported,
-and you can test with the `--dry-run` or the `--interactive`/`-i` flag.
+Some backends, most notably S3, do not report the amount of bytes used.
+If you need this information to be available when running `df` on the
+filesystem, then pass the flag `--vfs-used-is-size` to rclone.
+With this flag set, instead of relying on the backend to report this
+information, rclone will scan the whole remote similar to `rclone size`
+and compute the total used space itself.
-If `--timestamp` is used then sets the modification time to that
-time instead of the current time. Times may be specified as one of:
+_WARNING._ Contrary to `rclone size`, this flag ignores filters so that the
+result is accurate. However, this is very inefficient and may cost lots of API
+calls resulting in extra charges. Use it as a last resort and only with caching.
-- 'YYMMDD' - e.g. 17.10.30
-- 'YYYY-MM-DDTHH:MM:SS' - e.g. 2006-01-02T15:04:05
-- 'YYYY-MM-DDTHH:MM:SS.SSS' - e.g. 2006-01-02T15:04:05.123456789
+## Auth Proxy
-Note that value of `--timestamp` is in UTC. If you want local time
-then add the `--localtime` flag.
+If you supply the parameter `--auth-proxy /path/to/program` then
+rclone will use that program to generate backends on the fly which
+then are used to authenticate incoming requests. This uses a simple
+JSON based protocol with input on STDIN and output on STDOUT.
+
+**PLEASE NOTE:** `--auth-proxy` and `--authorized-keys` cannot be used
+together, if `--auth-proxy` is set the authorized keys option will be
+ignored.
+
+There is an example program
+[bin/test_proxy.py](https://github.com/rclone/rclone/blob/master/test_proxy.py)
+in the rclone source code.
+
+The program's job is to take a `user` and `pass` on the input and turn
+those into the config for a backend on STDOUT in JSON format. This
+config will have any default parameters for the backend added, but it
+won't use configuration from environment variables or command line
+options - it is the job of the proxy program to make a complete
+config.
+
+This config generated must have this extra parameter
+- `_root` - root to use for the backend
+And it may have this parameter
+- `_obscure` - comma separated strings for parameters to obscure
+
+If password authentication was used by the client, input to the proxy
+process (on STDIN) would look similar to this:
```
-rclone touch remote:path [flags]
+{
+ "user": "me",
+ "pass": "mypassword"
+}
```
-## Options
+If public-key authentication was used by the client, input to the
+proxy process (on STDIN) would look similar to this:
```
- -h, --help help for touch
- --localtime Use localtime for timestamp, not UTC
- -C, --no-create Do not create the file if it does not exist (implied with --recursive)
- -R, --recursive Recursively touch all files
- -t, --timestamp string Use specified time instead of the current time of day
+{
+ "user": "me",
+ "public_key": "AAAAB3NzaC1yc2EAAAADAQABAAABAQDuwESFdAe14hVS6omeyX7edc...JQdf"
+}
```
+And as an example return this on STDOUT
-## Important Options
+```
+{
+ "type": "sftp",
+ "_root": "",
+ "_obscure": "pass",
+ "user": "me",
+ "pass": "mypassword",
+ "host": "sftp.example.com"
+}
+```
+
+This would mean that an SFTP backend would be created on the fly for
+the `user` and `pass`/`public_key` returned in the output to the host given. Note
+that since `_obscure` is set to `pass`, rclone will obscure the `pass`
+parameter before creating the backend (which is required for sftp
+backends).
+
+The program can manipulate the supplied `user` in any way, for example
+to make proxy to many different sftp backends, you could make the
+`user` be `user@example.com` and then set the `host` to `example.com`
+in the output and the user to `user`. For security you'd probably want
+to restrict the `host` to a limited list.
+
+Note that an internal cache is keyed on `user` so only use that for
+configuration, don't use `pass` or `public_key`. This also means that if a user's
+password or public-key is changed the cache will need to expire (which takes 5 mins)
+before it takes effect.
+
+This can be used to build general purpose proxies to any kind of
+backend that rclone supports.
-Important flags useful for most commands.
```
- -n, --dry-run Do a trial run with no permanent changes
- -i, --interactive Enable interactive mode
- -v, --verbose count Print lots more stuff (repeat for more)
+rclone serve sftp remote:path [flags]
+```
+
+## Options
+
+```
+ --addr string IPaddress:Port or :Port to bind server to (default "localhost:2022")
+ --auth-proxy string A program to use to create the backend from the auth
+ --authorized-keys string Authorized keys file (default "~/.ssh/authorized_keys")
+ --dir-cache-time Duration Time to cache directory entries for (default 5m0s)
+ --dir-perms FileMode Directory permissions (default 0777)
+ --file-perms FileMode File permissions (default 0666)
+ --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000)
+ -h, --help help for sftp
+ --key stringArray SSH private host key file (Can be multi-valued, leave blank to auto generate)
+ --no-auth Allow connections with no authentication if set
+ --no-checksum Don't compare checksums on up/download
+ --no-modtime Don't read/write the modification time (can speed things up)
+ --no-seek Don't allow seeking in files
+ --pass string Password for authentication
+ --poll-interval Duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s)
+ --read-only Only allow read-only access
+ --stdio Run an sftp server on stdin/stdout
+ --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000)
+ --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2)
+ --user string User name for authentication
+ --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s)
+ --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off)
+ --vfs-cache-min-free-space SizeSuffix Target minimum free space on the disk containing the cache (default off)
+ --vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off)
+ --vfs-cache-poll-interval Duration Interval to poll the cache for stale objects (default 1m0s)
+ --vfs-case-insensitive If a file name not found, find a case insensitive match
+ --vfs-disk-space-total-size SizeSuffix Specify the total space of disk (default off)
+ --vfs-fast-fingerprint Use fast (less accurate) fingerprints for change detection
+ --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full
+ --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi)
+ --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off)
+ --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms)
+ --vfs-refresh Refreshes the directory cache recursively on start
+ --vfs-used-is-size rclone size Use the rclone size algorithm for Used size
+ --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s)
+ --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s)
```
+
## Filter Options
Flags for filtering directory listings.
@@ -10440,6928 +10548,8172 @@ Flags for filtering directory listings.
--min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off)
```
-## Listing Options
+See the [global flags page](https://rclone.org/flags/) for global options not listed here.
-Flags for listing directories.
+# SEE ALSO
-```
- --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z)
- --fast-list Use recursive list if available; uses more memory but fewer transactions
-```
-
-See the [global flags page](https://rclone.org/flags/) for global options not listed here.
-
-# SEE ALSO
-
-* [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends.
+* [rclone serve](https://rclone.org/commands/rclone_serve/) - Serve a remote over a protocol.
-# rclone tree
+# rclone serve webdav
-List the contents of the remote in a tree like fashion.
+Serve remote:path over WebDAV.
## Synopsis
+Run a basic WebDAV server to serve a remote over HTTP via the
+WebDAV protocol. This can be viewed with a WebDAV client, through a web
+browser, or you can make a remote of type WebDAV to read and write it.
-rclone tree lists the contents of a remote in a similar way to the
-unix tree command.
+## WebDAV options
-For example
+### --etag-hash
- $ rclone tree remote:path
- /
- ├── file1
- ├── file2
- ├── file3
- └── subdir
- ├── file4
- └── file5
+This controls the ETag header. Without this flag the ETag will be
+based on the ModTime and Size of the object.
- 1 directories, 5 files
+If this flag is set to "auto" then rclone will choose the first
+supported hash on the backend or you can use a named hash such as
+"MD5" or "SHA-1". Use the [hashsum](https://rclone.org/commands/rclone_hashsum/) command
+to see the full list.
-You can use any of the filtering options with the tree command (e.g.
-`--include` and `--exclude`. You can also use `--fast-list`.
+## Access WebDAV on Windows
+WebDAV shared folder can be mapped as a drive on Windows, however the default settings prevent it.
+Windows will fail to connect to the server using insecure Basic authentication.
+It will not even display any login dialog. Windows requires SSL / HTTPS connection to be used with Basic.
+If you try to connect via Add Network Location Wizard you will get the following error:
+"The folder you entered does not appear to be valid. Please choose another".
+However, you still can connect if you set the following registry key on a client machine:
+HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\WebClient\Parameters\BasicAuthLevel to 2.
+The BasicAuthLevel can be set to the following values:
+ 0 - Basic authentication disabled
+ 1 - Basic authentication enabled for SSL connections only
+ 2 - Basic authentication enabled for SSL connections and for non-SSL connections
+If required, increase the FileSizeLimitInBytes to a higher value.
+Navigate to the Services interface, then restart the WebClient service.
-The tree command has many options for controlling the listing which
-are compatible with the tree command, for example you can include file
-sizes with `--size`. Note that not all of them have
-short options as they conflict with rclone's short options.
+## Access Office applications on WebDAV
+Navigate to following registry HKEY_CURRENT_USER\Software\Microsoft\Office\[14.0/15.0/16.0]\Common\Internet
+Create a new DWORD BasicAuthLevel with value 2.
+ 0 - Basic authentication disabled
+ 1 - Basic authentication enabled for SSL connections only
+ 2 - Basic authentication enabled for SSL and for non-SSL connections
-For a more interactive navigation of the remote see the
-[ncdu](https://rclone.org/commands/rclone_ncdu/) command.
+https://learn.microsoft.com/en-us/office/troubleshoot/powerpoint/office-opens-blank-from-sharepoint
-```
-rclone tree remote:path [flags]
-```
+## Server options
-## Options
+Use `--addr` to specify which IP address and port the server should
+listen on, eg `--addr 1.2.3.4:8000` or `--addr :8080` to listen to all
+IPs. By default it only listens on localhost. You can use port
+:0 to let the OS choose an available port.
-```
- -a, --all All files are listed (list . files too)
- -d, --dirs-only List directories only
- --dirsfirst List directories before files (-U disables)
- --full-path Print the full path prefix for each file
- -h, --help help for tree
- --level int Descend only level directories deep
- -D, --modtime Print the date of last modification.
- --noindent Don't print indentation lines
- --noreport Turn off file/directory count at end of tree listing
- -o, --output string Output to file instead of stdout
- -p, --protections Print the protections for each file.
- -Q, --quote Quote filenames with double quotes.
- -s, --size Print the size in bytes of each file.
- --sort string Select sort: name,version,size,mtime,ctime
- --sort-ctime Sort files by last status change time
- -t, --sort-modtime Sort files by last modification time
- -r, --sort-reverse Reverse the order of the sort
- -U, --unsorted Leave files unsorted
- --version Sort files alphanumerically by version
-```
+If you set `--addr` to listen on a public or LAN accessible IP address
+then using Authentication is advised - see the next section for info.
+You can use a unix socket by setting the url to `unix:///path/to/socket`
+or just by using an absolute path name. Note that unix sockets bypass the
+authentication - this is expected to be done with file system permissions.
-## Filter Options
+`--addr` may be repeated to listen on multiple IPs/ports/sockets.
-Flags for filtering directory listings.
+`--server-read-timeout` and `--server-write-timeout` can be used to
+control the timeouts on the server. Note that this is the total time
+for a transfer.
-```
- --delete-excluded Delete files on dest excluded from sync
- --exclude stringArray Exclude files matching pattern
- --exclude-from stringArray Read file exclude patterns from file (use - to read from stdin)
- --exclude-if-present stringArray Exclude directories if filename is present
- --files-from stringArray Read list of source-file names from file (use - to read from stdin)
- --files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin)
- -f, --filter stringArray Add a file filtering rule
- --filter-from stringArray Read file filtering patterns from a file (use - to read from stdin)
- --ignore-case Ignore case in filters (case insensitive)
- --include stringArray Include files matching pattern
- --include-from stringArray Read file include patterns from file (use - to read from stdin)
- --max-age Duration Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off)
- --max-depth int If set limits the recursion depth to this (default -1)
- --max-size SizeSuffix Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off)
- --metadata-exclude stringArray Exclude metadatas matching pattern
- --metadata-exclude-from stringArray Read metadata exclude patterns from file (use - to read from stdin)
- --metadata-filter stringArray Add a metadata filtering rule
- --metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin)
- --metadata-include stringArray Include metadatas matching pattern
- --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin)
- --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off)
- --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off)
-```
+`--max-header-bytes` controls the maximum number of bytes the server will
+accept in the HTTP header.
-## Listing Options
+`--baseurl` controls the URL prefix that rclone serves from. By default
+rclone will serve from the root. If you used `--baseurl "/rclone"` then
+rclone would serve from a URL starting with "/rclone/". This is
+useful if you wish to proxy rclone serve. Rclone automatically
+inserts leading and trailing "/" on `--baseurl`, so `--baseurl "rclone"`,
+`--baseurl "/rclone"` and `--baseurl "/rclone/"` are all treated
+identically.
-Flags for listing directories.
+### TLS (SSL)
-```
- --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z)
- --fast-list Use recursive list if available; uses more memory but fewer transactions
-```
+By default this will serve over http. If you want you can serve over
+https. You will need to supply the `--cert` and `--key` flags.
+If you wish to do client side certificate validation then you will need to
+supply `--client-ca` also.
-See the [global flags page](https://rclone.org/flags/) for global options not listed here.
+`--cert` should be a either a PEM encoded certificate or a concatenation
+of that with the CA certificate. `--key` should be the PEM encoded
+private key and `--client-ca` should be the PEM encoded client
+certificate authority certificate.
-# SEE ALSO
+--min-tls-version is minimum TLS version that is acceptable. Valid
+ values are "tls1.0", "tls1.1", "tls1.2" and "tls1.3" (default
+ "tls1.0").
-* [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends.
+### Template
+`--template` allows a user to specify a custom markup template for HTTP
+and WebDAV serve functions. The server exports the following markup
+to be used within the template to server pages:
-Copying single files
---------------------
+| Parameter | Description |
+| :---------- | :---------- |
+| .Name | The full path of a file/directory. |
+| .Title | Directory listing of .Name |
+| .Sort | The current sort used. This is changeable via ?sort= parameter |
+| | Sort Options: namedirfirst,name,size,time (default namedirfirst) |
+| .Order | The current ordering used. This is changeable via ?order= parameter |
+| | Order Options: asc,desc (default asc) |
+| .Query | Currently unused. |
+| .Breadcrumb | Allows for creating a relative navigation |
+|-- .Link | The relative to the root link of the Text. |
+|-- .Text | The Name of the directory. |
+| .Entries | Information about a specific file/directory. |
+|-- .URL | The 'url' of an entry. |
+|-- .Leaf | Currently same as 'URL' but intended to be 'just' the name. |
+|-- .IsDir | Boolean for if an entry is a directory or not. |
+|-- .Size | Size in Bytes of the entry. |
+|-- .ModTime | The UTC timestamp of an entry. |
-rclone normally syncs or copies directories. However, if the source
-remote points to a file, rclone will just copy that file. The
-destination remote must point to a directory - rclone will give the
-error `Failed to create file system for "remote:file": is a file not a
-directory` if it isn't.
+The server also makes the following functions available so that they can be used within the
+template. These functions help extend the options for dynamic rendering of HTML. They can
+be used to render HTML based on specific conditions.
-For example, suppose you have a remote with a file in called
-`test.jpg`, then you could copy just that file like this
+| Function | Description |
+| :---------- | :---------- |
+| afterEpoch | Returns the time since the epoch for the given time. |
+| contains | Checks whether a given substring is present or not in a given string. |
+| hasPrefix | Checks whether the given string begins with the specified prefix. |
+| hasSuffix | Checks whether the given string end with the specified suffix. |
- rclone copy remote:test.jpg /tmp/download
+### Authentication
-The file `test.jpg` will be placed inside `/tmp/download`.
+By default this will serve files without needing a login.
-This is equivalent to specifying
+You can either use an htpasswd file which can take lots of users, or
+set a single username and password with the `--user` and `--pass` flags.
- rclone copy --files-from /tmp/files remote: /tmp/download
+If no static users are configured by either of the above methods, and client
+certificates are required by the `--client-ca` flag passed to the server, the
+client certificate common name will be considered as the username.
-Where `/tmp/files` contains the single line
+Use `--htpasswd /path/to/htpasswd` to provide an htpasswd file. This is
+in standard apache format and supports MD5, SHA1 and BCrypt for basic
+authentication. Bcrypt is recommended.
- test.jpg
+To create an htpasswd file:
-It is recommended to use `copy` when copying individual files, not `sync`.
-They have pretty much the same effect but `copy` will use a lot less
-memory.
+ touch htpasswd
+ htpasswd -B htpasswd user
+ htpasswd -B htpasswd anotherUser
-Syntax of remote paths
-----------------------
+The password file can be updated while rclone is running.
-The syntax of the paths passed to the rclone command are as follows.
+Use `--realm` to set the authentication realm.
-### /path/to/dir
+Use `--salt` to change the password hashing salt from the default.
+## VFS - Virtual File System
-This refers to the local file system.
+This command uses the VFS layer. This adapts the cloud storage objects
+that rclone uses into something which looks much more like a disk
+filing system.
-On Windows `\` may be used instead of `/` in local paths **only**,
-non local paths must use `/`. See [local filesystem](https://rclone.org/local/#paths-on-windows)
-documentation for more about Windows-specific paths.
+Cloud storage objects have lots of properties which aren't like disk
+files - you can't extend them or write to the middle of them, so the
+VFS layer has to deal with that. Because there is no one right way of
+doing this there are various options explained below.
-These paths needn't start with a leading `/` - if they don't then they
-will be relative to the current directory.
+The VFS layer also implements a directory cache - this caches info
+about files and directories (but not the data) in memory.
-### remote:path/to/dir
+## VFS Directory Cache
-This refers to a directory `path/to/dir` on `remote:` as defined in
-the config file (configured with `rclone config`).
+Using the `--dir-cache-time` flag, you can control how long a
+directory should be considered up to date and not refreshed from the
+backend. Changes made through the VFS will appear immediately or
+invalidate the cache.
-### remote:/path/to/dir
+ --dir-cache-time duration Time to cache directory entries for (default 5m0s)
+ --poll-interval duration Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable (default 1m0s)
-On most backends this is refers to the same directory as
-`remote:path/to/dir` and that format should be preferred. On a very
-small number of remotes (FTP, SFTP, Dropbox for business) this will
-refer to a different directory. On these, paths without a leading `/`
-will refer to your "home" directory and paths with a leading `/` will
-refer to the root.
+However, changes made directly on the cloud storage by the web
+interface or a different copy of rclone will only be picked up once
+the directory cache expires if the backend configured does not support
+polling for changes. If the backend supports polling, changes will be
+picked up within the polling interval.
-### :backend:path/to/dir
+You can send a `SIGHUP` signal to rclone for it to flush all
+directory caches, regardless of how old they are. Assuming only one
+rclone instance is running, you can reset the cache like this:
-This is an advanced form for creating remotes on the fly. `backend`
-should be the name or prefix of a backend (the `type` in the config
-file) and all the configuration for the backend should be provided on
-the command line (or in environment variables).
+ kill -SIGHUP $(pidof rclone)
-Here are some examples:
+If you configure rclone with a [remote control](/rc) then you can use
+rclone rc to flush the whole directory cache:
- rclone lsd --http-url https://pub.rclone.org :http:
+ rclone rc vfs/forget
-To list all the directories in the root of `https://pub.rclone.org/`.
+Or individual files or directories:
- rclone lsf --http-url https://example.com :http:path/to/dir
+ rclone rc vfs/forget file=path/to/file dir=path/to/dir
-To list files and directories in `https://example.com/path/to/dir/`
+## VFS File Buffering
- rclone copy --http-url https://example.com :http:path/to/dir /tmp/dir
+The `--buffer-size` flag determines the amount of memory,
+that will be used to buffer data in advance.
-To copy files and directories in `https://example.com/path/to/dir` to `/tmp/dir`.
+Each open file will try to keep the specified amount of data in memory
+at all times. The buffered data is bound to one open file and won't be
+shared.
- rclone copy --sftp-host example.com :sftp:path/to/dir /tmp/dir
+This flag is a upper limit for the used memory per open file. The
+buffer will only use memory for data that is downloaded but not not
+yet read. If the buffer is empty, only a small amount of memory will
+be used.
-To copy files and directories from `example.com` in the relative
-directory `path/to/dir` to `/tmp/dir` using sftp.
+The maximum memory used by rclone for buffering can be up to
+`--buffer-size * open files`.
-### Connection strings {#connection-strings}
+## VFS File Caching
-The above examples can also be written using a connection string
-syntax, so instead of providing the arguments as command line
-parameters `--http-url https://pub.rclone.org` they are provided as
-part of the remote specification as a kind of connection string.
+These flags control the VFS file caching options. File caching is
+necessary to make the VFS layer appear compatible with a normal file
+system. It can be disabled at the cost of some compatibility.
- rclone lsd ":http,url='https://pub.rclone.org':"
- rclone lsf ":http,url='https://example.com':path/to/dir"
- rclone copy ":http,url='https://example.com':path/to/dir" /tmp/dir
- rclone copy :sftp,host=example.com:path/to/dir /tmp/dir
+For example you'll need to enable VFS caching if you want to read and
+write simultaneously to a file. See below for more details.
-These can apply to modify existing remotes as well as create new
-remotes with the on the fly syntax. This example is equivalent to
-adding the `--drive-shared-with-me` parameter to the remote `gdrive:`.
+Note that the VFS cache is separate from the cache backend and you may
+find that you need one or the other or both.
- rclone lsf "gdrive,shared_with_me:path/to/dir"
+ --cache-dir string Directory rclone will use for caching.
+ --vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off)
+ --vfs-cache-max-age duration Max time since last access of objects in the cache (default 1h0m0s)
+ --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off)
+ --vfs-cache-min-free-space SizeSuffix Target minimum free space on the disk containing the cache (default off)
+ --vfs-cache-poll-interval duration Interval to poll the cache for stale objects (default 1m0s)
+ --vfs-write-back duration Time to writeback files after last use when using cache (default 5s)
-The major advantage to using the connection string style syntax is
-that it only applies to the remote, not to all the remotes of that
-type of the command line. A common confusion is this attempt to copy a
-file shared on google drive to the normal drive which **does not
-work** because the `--drive-shared-with-me` flag applies to both the
-source and the destination.
+If run with `-vv` rclone will print the location of the file cache. The
+files are stored in the user cache file area which is OS dependent but
+can be controlled with `--cache-dir` or setting the appropriate
+environment variable.
- rclone copy --drive-shared-with-me gdrive:shared-file.txt gdrive:
+The cache has 4 different modes selected by `--vfs-cache-mode`.
+The higher the cache mode the more compatible rclone becomes at the
+cost of using disk space.
-However using the connection string syntax, this does work.
+Note that files are written back to the remote only when they are
+closed and if they haven't been accessed for `--vfs-write-back`
+seconds. If rclone is quit or dies with files that haven't been
+uploaded, these will be uploaded next time rclone is run with the same
+flags.
- rclone copy "gdrive,shared_with_me:shared-file.txt" gdrive:
+If using `--vfs-cache-max-size` or `--vfs-cache-min-free-size` note
+that the cache may exceed these quotas for two reasons. Firstly
+because it is only checked every `--vfs-cache-poll-interval`. Secondly
+because open files cannot be evicted from the cache. When
+`--vfs-cache-max-size` or `--vfs-cache-min-free-size` is exceeded,
+rclone will attempt to evict the least accessed files from the cache
+first. rclone will start with files that haven't been accessed for the
+longest. This cache flushing strategy is efficient and more relevant
+files are likely to remain cached.
-Note that the connection string only affects the options of the immediate
-backend. If for example gdriveCrypt is a crypt based on gdrive, then the
-following command **will not work** as intended, because
-`shared_with_me` is ignored by the crypt backend:
+The `--vfs-cache-max-age` will evict files from the cache
+after the set time since last access has passed. The default value of
+1 hour will start evicting files from cache that haven't been accessed
+for 1 hour. When a cached file is accessed the 1 hour timer is reset to 0
+and will wait for 1 more hour before evicting. Specify the time with
+standard notation, s, m, h, d, w .
- rclone copy "gdriveCrypt,shared_with_me:shared-file.txt" gdriveCrypt:
+You **should not** run two copies of rclone using the same VFS cache
+with the same or overlapping remotes if using `--vfs-cache-mode > off`.
+This can potentially cause data corruption if you do. You can work
+around this by giving each rclone its own cache hierarchy with
+`--cache-dir`. You don't need to worry about this if the remotes in
+use don't overlap.
-The connection strings have the following syntax
+### --vfs-cache-mode off
- remote,parameter=value,parameter2=value2:path/to/dir
- :backend,parameter=value,parameter2=value2:path/to/dir
+In this mode (the default) the cache will read directly from the remote and write
+directly to the remote without caching anything on disk.
-If the `parameter` has a `:` or `,` then it must be placed in quotes `"` or
-`'`, so
+This will mean some operations are not possible
- remote,parameter="colon:value",parameter2="comma,value":path/to/dir
- :backend,parameter='colon:value',parameter2='comma,value':path/to/dir
+ * Files can't be opened for both read AND write
+ * Files opened for write can't be seeked
+ * Existing files opened for write must have O_TRUNC set
+ * Files open for read with O_TRUNC will be opened write only
+ * Files open for write only will behave as if O_TRUNC was supplied
+ * Open modes O_APPEND, O_TRUNC are ignored
+ * If an upload fails it can't be retried
-If a quoted value needs to include that quote, then it should be
-doubled, so
+### --vfs-cache-mode minimal
- remote,parameter="with""quote",parameter2='with''quote':path/to/dir
+This is very similar to "off" except that files opened for read AND
+write will be buffered to disk. This means that files opened for
+write will be a lot more compatible, but uses the minimal disk space.
-This will make `parameter` be `with"quote` and `parameter2` be
-`with'quote`.
+These operations are not possible
-If you leave off the `=parameter` then rclone will substitute `=true`
-which works very well with flags. For example, to use s3 configured in
-the environment you could use:
+ * Files opened for write only can't be seeked
+ * Existing files opened for write must have O_TRUNC set
+ * Files opened for write only will ignore O_APPEND, O_TRUNC
+ * If an upload fails it can't be retried
- rclone lsd :s3,env_auth:
+### --vfs-cache-mode writes
-Which is equivalent to
+In this mode files opened for read only are still read directly from
+the remote, write only and read/write files are buffered to disk
+first.
- rclone lsd :s3,env_auth=true:
+This mode should support all normal file system operations.
-Note that on the command line you might need to surround these
-connection strings with `"` or `'` to stop the shell interpreting any
-special characters within them.
+If an upload fails it will be retried at exponentially increasing
+intervals up to 1 minute.
-If you are a shell master then you'll know which strings are OK and
-which aren't, but if you aren't sure then enclose them in `"` and use
-`'` as the inside quote. This syntax works on all OSes.
+### --vfs-cache-mode full
- rclone copy ":http,url='https://example.com':path/to/dir" /tmp/dir
+In this mode all reads and writes are buffered to and from disk. When
+data is read from the remote this is buffered to disk as well.
-On Linux/macOS some characters are still interpreted inside `"`
-strings in the shell (notably `\` and `$` and `"`) so if your strings
-contain those you can swap the roles of `"` and `'` thus. (This syntax
-does not work on Windows.)
+In this mode the files in the cache will be sparse files and rclone
+will keep track of which bits of the files it has downloaded.
- rclone copy ':http,url="https://example.com":path/to/dir' /tmp/dir
+So if an application only reads the starts of each file, then rclone
+will only buffer the start of the file. These files will appear to be
+their full size in the cache, but they will be sparse files with only
+the data that has been downloaded present in them.
-#### Connection strings, config and logging
+This mode should support all normal file system operations and is
+otherwise identical to `--vfs-cache-mode` writes.
-If you supply extra configuration to a backend by command line flag,
-environment variable or connection string then rclone will add a
-suffix based on the hash of the config to the name of the remote, eg
+When reading a file rclone will read `--buffer-size` plus
+`--vfs-read-ahead` bytes ahead. The `--buffer-size` is buffered in memory
+whereas the `--vfs-read-ahead` is buffered on disk.
- rclone -vv lsf --s3-chunk-size 20M s3:
+When using this mode it is recommended that `--buffer-size` is not set
+too large and `--vfs-read-ahead` is set large if required.
-Has the log message
+**IMPORTANT** not all file systems support sparse files. In particular
+FAT/exFAT do not. Rclone will perform very badly if the cache
+directory is on a filesystem which doesn't support sparse files and it
+will log an ERROR message if one is detected.
- DEBUG : s3: detected overridden config - adding "{Srj1p}" suffix to name
+### Fingerprinting
-This is so rclone can tell the modified remote apart from the
-unmodified remote when caching the backends.
+Various parts of the VFS use fingerprinting to see if a local file
+copy has changed relative to a remote file. Fingerprints are made
+from:
-This should only be noticeable in the logs.
+- size
+- modification time
+- hash
-This means that on the fly backends such as
+where available on an object.
- rclone -vv lsf :s3,env_auth:
+On some backends some of these attributes are slow to read (they take
+an extra API call per object, or extra work per object).
-Will get their own names
+For example `hash` is slow with the `local` and `sftp` backends as
+they have to read the entire file and hash it, and `modtime` is slow
+with the `s3`, `swift`, `ftp` and `qinqstor` backends because they
+need to do an extra API call to fetch it.
- DEBUG : :s3: detected overridden config - adding "{YTu53}" suffix to name
+If you use the `--vfs-fast-fingerprint` flag then rclone will not
+include the slow operations in the fingerprint. This makes the
+fingerprinting less accurate but much faster and will improve the
+opening time of cached files.
-### Valid remote names
+If you are running a vfs cache over `local`, `s3` or `swift` backends
+then using this flag is recommended.
-Remote names are case sensitive, and must adhere to the following rules:
- - May contain number, letter, `_`, `-`, `.`, `+`, `@` and space.
- - May not start with `-` or space.
- - May not end with space.
+Note that if you change the value of this flag, the fingerprints of
+the files in the cache may be invalidated and the files will need to
+be downloaded again.
-Starting with rclone version 1.61, any Unicode numbers and letters are allowed,
-while in older versions it was limited to plain ASCII (0-9, A-Z, a-z). If you use
-the same rclone configuration from different shells, which may be configured with
-different character encoding, you must be cautious to use characters that are
-possible to write in all of them. This is mostly a problem on Windows, where
-the console traditionally uses a non-Unicode character set - defined
-by the so-called "code page".
+## VFS Chunked Reading
-Do not use single character names on Windows as it creates ambiguity with Windows
-drives' names, e.g.: remote called `C` is indistinguishable from `C` drive. Rclone
-will always assume that single letter name refers to a drive.
+When rclone reads files from a remote it reads them in chunks. This
+means that rather than requesting the whole file rclone reads the
+chunk specified. This can reduce the used download quota for some
+remotes by requesting only chunks from the remote that are actually
+read, at the cost of an increased number of requests.
-Quoting and the shell
----------------------
+These flags control the chunking:
-When you are typing commands to your computer you are using something
-called the command line shell. This interprets various characters in
-an OS specific way.
+ --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M)
+ --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off)
-Here are some gotchas which may help users unfamiliar with the shell rules
+Rclone will start reading a chunk of size `--vfs-read-chunk-size`,
+and then double the size for each read. When `--vfs-read-chunk-size-limit` is
+specified, and greater than `--vfs-read-chunk-size`, the chunk size for each
+open file will get doubled only until the specified value is reached. If the
+value is "off", which is the default, the limit is disabled and the chunk size
+will grow indefinitely.
-### Linux / OSX ###
+With `--vfs-read-chunk-size 100M` and `--vfs-read-chunk-size-limit 0`
+the following parts will be downloaded: 0-100M, 100M-200M, 200M-300M, 300M-400M and so on.
+When `--vfs-read-chunk-size-limit 500M` is specified, the result would be
+0-100M, 100M-300M, 300M-700M, 700M-1200M, 1200M-1700M and so on.
-If your names have spaces or shell metacharacters (e.g. `*`, `?`, `$`,
-`'`, `"`, etc.) then you must quote them. Use single quotes `'` by default.
+Setting `--vfs-read-chunk-size` to `0` or "off" disables chunked reading.
- rclone copy 'Important files?' remote:backup
+## VFS Performance
-If you want to send a `'` you will need to use `"`, e.g.
+These flags may be used to enable/disable features of the VFS for
+performance or other reasons. See also the [chunked reading](#vfs-chunked-reading)
+feature.
- rclone copy "O'Reilly Reviews" remote:backup
+In particular S3 and Swift benefit hugely from the `--no-modtime` flag
+(or use `--use-server-modtime` for a slightly different effect) as each
+read of the modification time takes a transaction.
-The rules for quoting metacharacters are complicated and if you want
-the full details you'll have to consult the manual page for your
-shell.
+ --no-checksum Don't compare checksums on up/download.
+ --no-modtime Don't read/write the modification time (can speed things up).
+ --no-seek Don't allow seeking in files.
+ --read-only Only allow read-only access.
-### Windows ###
+Sometimes rclone is delivered reads or writes out of order. Rather
+than seeking rclone will wait a short time for the in sequence read or
+write to come in. These flags only come into effect when not using an
+on disk cache file.
-If your names have spaces in you need to put them in `"`, e.g.
+ --vfs-read-wait duration Time to wait for in-sequence read before seeking (default 20ms)
+ --vfs-write-wait duration Time to wait for in-sequence write before giving error (default 1s)
- rclone copy "E:\folder name\folder name\folder name" remote:backup
+When using VFS write caching (`--vfs-cache-mode` with value writes or full),
+the global flag `--transfers` can be set to adjust the number of parallel uploads of
+modified files from the cache (the related global flag `--checkers` has no effect on the VFS).
-If you are using the root directory on its own then don't quote it
-(see [#464](https://github.com/rclone/rclone/issues/464) for why), e.g.
+ --transfers int Number of file transfers to run in parallel (default 4)
- rclone copy E:\ remote:backup
+## VFS Case Sensitivity
-Copying files or directories with `:` in the names
---------------------------------------------------
+Linux file systems are case-sensitive: two files can differ only
+by case, and the exact case must be used when opening a file.
-rclone uses `:` to mark a remote name. This is, however, a valid
-filename component in non-Windows OSes. The remote name parser will
-only search for a `:` up to the first `/` so if you need to act on a
-file or directory like this then use the full path starting with a
-`/`, or use `./` as a current directory prefix.
+File systems in modern Windows are case-insensitive but case-preserving:
+although existing files can be opened using any case, the exact case used
+to create the file is preserved and available for programs to query.
+It is not allowed for two files in the same directory to differ only by case.
-So to sync a directory called `sync:me` to a remote called `remote:` use
+Usually file systems on macOS are case-insensitive. It is possible to make macOS
+file systems case-sensitive but that is not the default.
- rclone sync --interactive ./sync:me remote:path
+The `--vfs-case-insensitive` VFS flag controls how rclone handles these
+two cases. If its value is "false", rclone passes file names to the remote
+as-is. If the flag is "true" (or appears without a value on the
+command line), rclone may perform a "fixup" as explained below.
-or
+The user may specify a file name to open/delete/rename/etc with a case
+different than what is stored on the remote. If an argument refers
+to an existing file with exactly the same name, then the case of the existing
+file on the disk will be used. However, if a file name with exactly the same
+name is not found but a name differing only by case exists, rclone will
+transparently fixup the name. This fixup happens only when an existing file
+is requested. Case sensitivity of file names created anew by rclone is
+controlled by the underlying remote.
- rclone sync --interactive /full/path/to/sync:me remote:path
+Note that case sensitivity of the operating system running rclone (the target)
+may differ from case sensitivity of a file system presented by rclone (the source).
+The flag controls whether "fixup" is performed to satisfy the target.
-Server Side Copy
-----------------
+If the flag is not provided on the command line, then its default value depends
+on the operating system where rclone runs: "true" on Windows and macOS, "false"
+otherwise. If the flag is provided without a value, then it is "true".
-Most remotes (but not all - see [the
-overview](https://rclone.org/overview/#optional-features)) support server-side copy.
+## VFS Disk Options
-This means if you want to copy one folder to another then rclone won't
-download all the files and re-upload them; it will instruct the server
-to copy them in place.
+This flag allows you to manually set the statistics about the filing system.
+It can be useful when those statistics cannot be read correctly automatically.
-Eg
+ --vfs-disk-space-total-size Manually set the total disk space size (example: 256G, default: -1)
- rclone copy s3:oldbucket s3:newbucket
+## Alternate report of used bytes
-Will copy the contents of `oldbucket` to `newbucket` without
-downloading and re-uploading.
+Some backends, most notably S3, do not report the amount of bytes used.
+If you need this information to be available when running `df` on the
+filesystem, then pass the flag `--vfs-used-is-size` to rclone.
+With this flag set, instead of relying on the backend to report this
+information, rclone will scan the whole remote similar to `rclone size`
+and compute the total used space itself.
-Remotes which don't support server-side copy **will** download and
-re-upload in this case.
+_WARNING._ Contrary to `rclone size`, this flag ignores filters so that the
+result is accurate. However, this is very inefficient and may cost lots of API
+calls resulting in extra charges. Use it as a last resort and only with caching.
-Server side copies are used with `sync` and `copy` and will be
-identified in the log when using the `-v` flag. The `move` command
-may also use them if remote doesn't support server-side move directly.
-This is done by issuing a server-side copy then a delete which is much
-quicker than a download and re-upload.
+## Auth Proxy
-Server side copies will only be attempted if the remote names are the
-same.
+If you supply the parameter `--auth-proxy /path/to/program` then
+rclone will use that program to generate backends on the fly which
+then are used to authenticate incoming requests. This uses a simple
+JSON based protocol with input on STDIN and output on STDOUT.
-This can be used when scripting to make aged backups efficiently, e.g.
+**PLEASE NOTE:** `--auth-proxy` and `--authorized-keys` cannot be used
+together, if `--auth-proxy` is set the authorized keys option will be
+ignored.
- rclone sync --interactive remote:current-backup remote:previous-backup
- rclone sync --interactive /path/to/files remote:current-backup
+There is an example program
+[bin/test_proxy.py](https://github.com/rclone/rclone/blob/master/test_proxy.py)
+in the rclone source code.
-## Metadata support {#metadata}
+The program's job is to take a `user` and `pass` on the input and turn
+those into the config for a backend on STDOUT in JSON format. This
+config will have any default parameters for the backend added, but it
+won't use configuration from environment variables or command line
+options - it is the job of the proxy program to make a complete
+config.
-Metadata is data about a file which isn't the contents of the file.
-Normally rclone only preserves the modification time and the content
-(MIME) type where possible.
+This config generated must have this extra parameter
+- `_root` - root to use for the backend
-Rclone supports preserving all the available metadata on files (not
-directories) when using the `--metadata` or `-M` flag.
+And it may have this parameter
+- `_obscure` - comma separated strings for parameters to obscure
-Exactly what metadata is supported and what that support means depends
-on the backend. Backends that support metadata have a metadata section
-in their docs and are listed in the [features table](https://rclone.org/overview/#features)
-(Eg [local](https://rclone.org/local/#metadata), [s3](/s3/#metadata))
+If password authentication was used by the client, input to the proxy
+process (on STDIN) would look similar to this:
-Rclone only supports a one-time sync of metadata. This means that
-metadata will be synced from the source object to the destination
-object only when the source object has changed and needs to be
-re-uploaded. If the metadata subsequently changes on the source object
-without changing the object itself then it won't be synced to the
-destination object. This is in line with the way rclone syncs
-`Content-Type` without the `--metadata` flag.
+```
+{
+ "user": "me",
+ "pass": "mypassword"
+}
+```
-Using `--metadata` when syncing from local to local will preserve file
-attributes such as file mode, owner, extended attributes (not
-Windows).
+If public-key authentication was used by the client, input to the
+proxy process (on STDIN) would look similar to this:
-Note that arbitrary metadata may be added to objects using the
-`--metadata-set key=value` flag when the object is first uploaded.
-This flag can be repeated as many times as necessary.
+```
+{
+ "user": "me",
+ "public_key": "AAAAB3NzaC1yc2EAAAADAQABAAABAQDuwESFdAe14hVS6omeyX7edc...JQdf"
+}
+```
-### Types of metadata
+And as an example return this on STDOUT
-Metadata is divided into two type. System metadata and User metadata.
+```
+{
+ "type": "sftp",
+ "_root": "",
+ "_obscure": "pass",
+ "user": "me",
+ "pass": "mypassword",
+ "host": "sftp.example.com"
+}
+```
-Metadata which the backend uses itself is called system metadata. For
-example on the local backend the system metadata `uid` will store the
-user ID of the file when used on a unix based platform.
+This would mean that an SFTP backend would be created on the fly for
+the `user` and `pass`/`public_key` returned in the output to the host given. Note
+that since `_obscure` is set to `pass`, rclone will obscure the `pass`
+parameter before creating the backend (which is required for sftp
+backends).
-Arbitrary metadata is called user metadata and this can be set however
-is desired.
+The program can manipulate the supplied `user` in any way, for example
+to make proxy to many different sftp backends, you could make the
+`user` be `user@example.com` and then set the `host` to `example.com`
+in the output and the user to `user`. For security you'd probably want
+to restrict the `host` to a limited list.
-When objects are copied from backend to backend, they will attempt to
-interpret system metadata if it is supplied. Metadata may change from
-being user metadata to system metadata as objects are copied between
-different backends. For example copying an object from s3 sets the
-`content-type` metadata. In a backend which understands this (like
-`azureblob`) this will become the Content-Type of the object. In a
-backend which doesn't understand this (like the `local` backend) this
-will become user metadata. However should the local object be copied
-back to s3, the Content-Type will be set correctly.
+Note that an internal cache is keyed on `user` so only use that for
+configuration, don't use `pass` or `public_key`. This also means that if a user's
+password or public-key is changed the cache will need to expire (which takes 5 mins)
+before it takes effect.
-### Metadata framework
+This can be used to build general purpose proxies to any kind of
+backend that rclone supports.
-Rclone implements a metadata framework which can read metadata from an
-object and write it to the object when (and only when) it is being
-uploaded.
-This metadata is stored as a dictionary with string keys and string
-values.
+```
+rclone serve webdav remote:path [flags]
+```
-There are some limits on the names of the keys (these may be clarified
-further in the future).
+## Options
-- must be lower case
-- may be `a-z` `0-9` containing `.` `-` or `_`
-- length is backend dependent
+```
+ --addr stringArray IPaddress:Port or :Port to bind server to (default [127.0.0.1:8080])
+ --allow-origin string Origin which cross-domain request (CORS) can be executed from
+ --auth-proxy string A program to use to create the backend from the auth
+ --baseurl string Prefix for URLs - leave blank for root
+ --cert string TLS PEM key (concatenation of certificate and CA certificate)
+ --client-ca string Client certificate authority to verify clients with
+ --dir-cache-time Duration Time to cache directory entries for (default 5m0s)
+ --dir-perms FileMode Directory permissions (default 0777)
+ --disable-dir-list Disable HTML directory list on GET request for a directory
+ --etag-hash string Which hash to use for the ETag, or auto or blank for off
+ --file-perms FileMode File permissions (default 0666)
+ --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000)
+ -h, --help help for webdav
+ --htpasswd string A htpasswd file - if not provided no authentication is done
+ --key string TLS PEM Private key
+ --max-header-bytes int Maximum size of request header (default 4096)
+ --min-tls-version string Minimum TLS version that is acceptable (default "tls1.0")
+ --no-checksum Don't compare checksums on up/download
+ --no-modtime Don't read/write the modification time (can speed things up)
+ --no-seek Don't allow seeking in files
+ --pass string Password for authentication
+ --poll-interval Duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s)
+ --read-only Only allow read-only access
+ --realm string Realm for authentication
+ --salt string Password hashing salt (default "dlPL2MqE")
+ --server-read-timeout Duration Timeout for server reading data (default 1h0m0s)
+ --server-write-timeout Duration Timeout for server writing data (default 1h0m0s)
+ --template string User-specified template
+ --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000)
+ --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2)
+ --user string User name for authentication
+ --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s)
+ --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off)
+ --vfs-cache-min-free-space SizeSuffix Target minimum free space on the disk containing the cache (default off)
+ --vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off)
+ --vfs-cache-poll-interval Duration Interval to poll the cache for stale objects (default 1m0s)
+ --vfs-case-insensitive If a file name not found, find a case insensitive match
+ --vfs-disk-space-total-size SizeSuffix Specify the total space of disk (default off)
+ --vfs-fast-fingerprint Use fast (less accurate) fingerprints for change detection
+ --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full
+ --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi)
+ --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off)
+ --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms)
+ --vfs-refresh Refreshes the directory cache recursively on start
+ --vfs-used-is-size rclone size Use the rclone size algorithm for Used size
+ --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s)
+ --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s)
+```
-Each backend can provide system metadata that it understands. Some
-backends can also store arbitrary user metadata.
-Where possible the key names are standardized, so, for example, it is
-possible to copy object metadata from s3 to azureblob for example and
-metadata will be translated appropriately.
+## Filter Options
-Some backends have limits on the size of the metadata and rclone will
-give errors on upload if they are exceeded.
+Flags for filtering directory listings.
-### Metadata preservation
+```
+ --delete-excluded Delete files on dest excluded from sync
+ --exclude stringArray Exclude files matching pattern
+ --exclude-from stringArray Read file exclude patterns from file (use - to read from stdin)
+ --exclude-if-present stringArray Exclude directories if filename is present
+ --files-from stringArray Read list of source-file names from file (use - to read from stdin)
+ --files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin)
+ -f, --filter stringArray Add a file filtering rule
+ --filter-from stringArray Read file filtering patterns from a file (use - to read from stdin)
+ --ignore-case Ignore case in filters (case insensitive)
+ --include stringArray Include files matching pattern
+ --include-from stringArray Read file include patterns from file (use - to read from stdin)
+ --max-age Duration Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off)
+ --max-depth int If set limits the recursion depth to this (default -1)
+ --max-size SizeSuffix Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off)
+ --metadata-exclude stringArray Exclude metadatas matching pattern
+ --metadata-exclude-from stringArray Read metadata exclude patterns from file (use - to read from stdin)
+ --metadata-filter stringArray Add a metadata filtering rule
+ --metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin)
+ --metadata-include stringArray Include metadatas matching pattern
+ --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin)
+ --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off)
+ --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off)
+```
-The goal of the implementation is to
+See the [global flags page](https://rclone.org/flags/) for global options not listed here.
-1. Preserve metadata if at all possible
-2. Interpret metadata if at all possible
+# SEE ALSO
-The consequences of 1 is that you can copy an S3 object to a local
-disk then back to S3 losslessly. Likewise you can copy a local file
-with file attributes and xattrs from local disk to s3 and back again
-losslessly.
+* [rclone serve](https://rclone.org/commands/rclone_serve/) - Serve a remote over a protocol.
-The consequence of 2 is that you can copy an S3 object with metadata
-to Azureblob (say) and have the metadata appear on the Azureblob
-object also.
+# rclone settier
-### Standard system metadata
+Changes storage class/tier of objects in remote.
-Here is a table of standard system metadata which, if appropriate, a
-backend may implement.
+## Synopsis
-| key | description | example |
-|---------------------|-------------|---------|
-| mode | File type and mode: octal, unix style | 0100664 |
-| uid | User ID of owner: decimal number | 500 |
-| gid | Group ID of owner: decimal number | 500 |
-| rdev | Device ID (if special file) => hexadecimal | 0 |
-| atime | Time of last access: RFC 3339 | 2006-01-02T15:04:05.999999999Z07:00 |
-| mtime | Time of last modification: RFC 3339 | 2006-01-02T15:04:05.999999999Z07:00 |
-| btime | Time of file creation (birth): RFC 3339 | 2006-01-02T15:04:05.999999999Z07:00 |
-| cache-control | Cache-Control header | no-cache |
-| content-disposition | Content-Disposition header | inline |
-| content-encoding | Content-Encoding header | gzip |
-| content-language | Content-Language header | en-US |
-| content-type | Content-Type header | text/plain |
-The metadata keys `mtime` and `content-type` will take precedence if
-supplied in the metadata over reading the `Content-Type` or
-modification time of the source object.
+rclone settier changes storage tier or class at remote if supported.
+Few cloud storage services provides different storage classes on objects,
+for example AWS S3 and Glacier, Azure Blob storage - Hot, Cool and Archive,
+Google Cloud Storage, Regional Storage, Nearline, Coldline etc.
-Hashes are not included in system metadata as there is a well defined
-way of reading those already.
+Note that, certain tier changes make objects not available to access immediately.
+For example tiering to archive in azure blob storage makes objects in frozen state,
+user can restore by setting tier to Hot/Cool, similarly S3 to Glacier makes object
+inaccessible.true
-Options
--------
+You can use it to tier single object
-Rclone has a number of options to control its behaviour.
+ rclone settier Cool remote:path/file
-Options that take parameters can have the values passed in two ways,
-`--option=value` or `--option value`. However boolean (true/false)
-options behave slightly differently to the other options in that
-`--boolean` sets the option to `true` and the absence of the flag sets
-it to `false`. It is also possible to specify `--boolean=false` or
-`--boolean=true`. Note that `--boolean false` is not valid - this is
-parsed as `--boolean` and the `false` is parsed as an extra command
-line argument for rclone.
+Or use rclone filters to set tier on only specific files
-### Time or duration options {#time-option}
+ rclone --include "*.txt" settier Hot remote:path/dir
-TIME or DURATION options can be specified as a duration string or a
-time string.
+Or just provide remote directory and all files in directory will be tiered
-A duration string is a possibly signed sequence of decimal numbers,
-each with optional fraction and a unit suffix, such as "300ms",
-"-1.5h" or "2h45m". Default units are seconds or the following
-abbreviations are valid:
+ rclone settier tier remote:path/dir
- * `ms` - Milliseconds
- * `s` - Seconds
- * `m` - Minutes
- * `h` - Hours
- * `d` - Days
- * `w` - Weeks
- * `M` - Months
- * `y` - Years
-These can also be specified as an absolute time in the following
-formats:
+```
+rclone settier tier remote:path [flags]
+```
-- RFC3339 - e.g. `2006-01-02T15:04:05Z` or `2006-01-02T15:04:05+07:00`
-- ISO8601 Date and time, local timezone - `2006-01-02T15:04:05`
-- ISO8601 Date and time, local timezone - `2006-01-02 15:04:05`
-- ISO8601 Date - `2006-01-02` (YYYY-MM-DD)
+## Options
-### Size options {#size-option}
+```
+ -h, --help help for settier
+```
-Options which use SIZE use KiB (multiples of 1024 bytes) by default.
-However, a suffix of `B` for Byte, `K` for KiB, `M` for MiB,
-`G` for GiB, `T` for TiB and `P` for PiB may be used. These are
-the binary units, e.g. 1, 2\*\*10, 2\*\*20, 2\*\*30 respectively.
-### --backup-dir=DIR ###
+See the [global flags page](https://rclone.org/flags/) for global options not listed here.
-When using `sync`, `copy` or `move` any files which would have been
-overwritten or deleted are moved in their original hierarchy into this
-directory.
+# SEE ALSO
-If `--suffix` is set, then the moved files will have the suffix added
-to them. If there is a file with the same path (after the suffix has
-been added) in DIR, then it will be overwritten.
+* [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends.
-The remote in use must support server-side move or copy and you must
-use the same remote as the destination of the sync. The backup
-directory must not overlap the destination directory without it being
-excluded by a filter rule.
+# rclone test
-For example
+Run a test command
- rclone sync --interactive /path/to/local remote:current --backup-dir remote:old
+## Synopsis
-will sync `/path/to/local` to `remote:current`, but for any files
-which would have been updated or deleted will be stored in
-`remote:old`.
+Rclone test is used to run test commands.
-If running rclone from a script you might want to use today's date as
-the directory name passed to `--backup-dir` to store the old files, or
-you might want to pass `--suffix` with today's date.
+Select which test command you want with the subcommand, eg
-See `--compare-dest` and `--copy-dest`.
+ rclone test memory remote:
-### --bind string ###
+Each subcommand has its own options which you can see in their help.
-Local address to bind to for outgoing connections. This can be an
-IPv4 address (1.2.3.4), an IPv6 address (1234::789A) or host name. If
-the host name doesn't resolve or resolves to more than one IP address
-it will give an error.
+**NB** Be careful running these commands, they may do strange things
+so reading their documentation first is recommended.
-You can use `--bind 0.0.0.0` to force rclone to use IPv4 addresses and
-`--bind ::0` to force rclone to use IPv6 addresses.
-### --bwlimit=BANDWIDTH_SPEC ###
+## Options
-This option controls the bandwidth limit. For example
+```
+ -h, --help help for test
+```
- --bwlimit 10M
-would mean limit the upload and download bandwidth to 10 MiB/s.
-**NB** this is **bytes** per second not **bits** per second. To use a
-single limit, specify the desired bandwidth in KiB/s, or use a
-suffix B|K|M|G|T|P. The default is `0` which means to not limit bandwidth.
+See the [global flags page](https://rclone.org/flags/) for global options not listed here.
-The upload and download bandwidth can be specified separately, as
-`--bwlimit UP:DOWN`, so
+# SEE ALSO
- --bwlimit 10M:100k
+* [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends.
+* [rclone test changenotify](https://rclone.org/commands/rclone_test_changenotify/) - Log any change notify requests for the remote passed in.
+* [rclone test histogram](https://rclone.org/commands/rclone_test_histogram/) - Makes a histogram of file name characters.
+* [rclone test info](https://rclone.org/commands/rclone_test_info/) - Discovers file name or other limitations for paths.
+* [rclone test makefile](https://rclone.org/commands/rclone_test_makefile/) - Make files with random contents of the size given
+* [rclone test makefiles](https://rclone.org/commands/rclone_test_makefiles/) - Make a random file hierarchy in a directory
+* [rclone test memory](https://rclone.org/commands/rclone_test_memory/) - Load all the objects at remote:path into memory and report memory stats.
-would mean limit the upload bandwidth to 10 MiB/s and the download
-bandwidth to 100 KiB/s. Either limit can be "off" meaning no limit, so
-to just limit the upload bandwidth you would use
+# rclone test changenotify
- --bwlimit 10M:off
+Log any change notify requests for the remote passed in.
-this would limit the upload bandwidth to 10 MiB/s but the download
-bandwidth would be unlimited.
+```
+rclone test changenotify remote: [flags]
+```
-When specified as above the bandwidth limits last for the duration of
-run of the rclone binary.
+## Options
-It is also possible to specify a "timetable" of limits, which will
-cause certain limits to be applied at certain times. To specify a
-timetable, format your entries as `WEEKDAY-HH:MM,BANDWIDTH
-WEEKDAY-HH:MM,BANDWIDTH...` where: `WEEKDAY` is optional element.
+```
+ -h, --help help for changenotify
+ --poll-interval Duration Time to wait between polling for changes (default 10s)
+```
-- `BANDWIDTH` can be a single number, e.g.`100k` or a pair of numbers
-for upload:download, e.g.`10M:1M`.
-- `WEEKDAY` can be written as the whole word or only using the first 3
- characters. It is optional.
-- `HH:MM` is an hour from 00:00 to 23:59.
-An example of a typical timetable to avoid link saturation during daytime
-working hours could be:
+See the [global flags page](https://rclone.org/flags/) for global options not listed here.
-`--bwlimit "08:00,512k 12:00,10M 13:00,512k 18:00,30M 23:00,off"`
+# SEE ALSO
-In this example, the transfer bandwidth will be set to 512 KiB/s
-at 8am every day. At noon, it will rise to 10 MiB/s, and drop back
-to 512 KiB/sec at 1pm. At 6pm, the bandwidth limit will be set to
-30 MiB/s, and at 11pm it will be completely disabled (full speed).
-Anything between 11pm and 8am will remain unlimited.
+* [rclone test](https://rclone.org/commands/rclone_test/) - Run a test command
-An example of timetable with `WEEKDAY` could be:
+# rclone test histogram
-`--bwlimit "Mon-00:00,512 Fri-23:59,10M Sat-10:00,1M Sun-20:00,off"`
+Makes a histogram of file name characters.
-It means that, the transfer bandwidth will be set to 512 KiB/s on
-Monday. It will rise to 10 MiB/s before the end of Friday. At 10:00
-on Saturday it will be set to 1 MiB/s. From 20:00 on Sunday it will
-be unlimited.
+## Synopsis
-Timeslots without `WEEKDAY` are extended to the whole week. So this
-example:
+This command outputs JSON which shows the histogram of characters used
+in filenames in the remote:path specified.
-`--bwlimit "Mon-00:00,512 12:00,1M Sun-20:00,off"`
+The data doesn't contain any identifying information but is useful for
+the rclone developers when developing filename compression.
-Is equivalent to this:
-`--bwlimit "Mon-00:00,512Mon-12:00,1M Tue-12:00,1M Wed-12:00,1M Thu-12:00,1M Fri-12:00,1M Sat-12:00,1M Sun-12:00,1M Sun-20:00,off"`
+```
+rclone test histogram [remote:path] [flags]
+```
-Bandwidth limit apply to the data transfer for all backends. For most
-backends the directory listing bandwidth is also included (exceptions
-being the non HTTP backends, `ftp`, `sftp` and `storj`).
+## Options
-Note that the units are **Byte/s**, not **bit/s**. Typically
-connections are measured in bit/s - to convert divide by 8. For
-example, let's say you have a 10 Mbit/s connection and you wish rclone
-to use half of it - 5 Mbit/s. This is 5/8 = 0.625 MiB/s so you would
-use a `--bwlimit 0.625M` parameter for rclone.
+```
+ -h, --help help for histogram
+```
-On Unix systems (Linux, macOS, …) the bandwidth limiter can be toggled by
-sending a `SIGUSR2` signal to rclone. This allows to remove the limitations
-of a long running rclone transfer and to restore it back to the value specified
-with `--bwlimit` quickly when needed. Assuming there is only one rclone instance
-running, you can toggle the limiter like this:
- kill -SIGUSR2 $(pidof rclone)
+See the [global flags page](https://rclone.org/flags/) for global options not listed here.
-If you configure rclone with a [remote control](/rc) then you can use
-change the bwlimit dynamically:
+# SEE ALSO
- rclone rc core/bwlimit rate=1M
+* [rclone test](https://rclone.org/commands/rclone_test/) - Run a test command
-### --bwlimit-file=BANDWIDTH_SPEC ###
+# rclone test info
-This option controls per file bandwidth limit. For the options see the
-`--bwlimit` flag.
+Discovers file name or other limitations for paths.
-For example use this to allow no transfers to be faster than 1 MiB/s
+## Synopsis
- --bwlimit-file 1M
+rclone info discovers what filenames and upload methods are possible
+to write to the paths passed in and how long they can be. It can take some
+time. It will write test files into the remote:path passed in. It outputs
+a bit of go code for each one.
-This can be used in conjunction with `--bwlimit`.
+**NB** this can create undeletable files and other hazards - use with care
-Note that if a schedule is provided the file will use the schedule in
-effect at the start of the transfer.
-### --buffer-size=SIZE ###
+```
+rclone test info [remote:path]+ [flags]
+```
-Use this sized buffer to speed up file transfers. Each `--transfer`
-will use this much memory for buffering.
+## Options
-When using `mount` or `cmount` each open file descriptor will use this much
-memory for buffering.
-See the [mount](https://rclone.org/commands/rclone_mount/#file-buffering) documentation for more details.
+```
+ --all Run all tests
+ --check-base32768 Check can store all possible base32768 characters
+ --check-control Check control characters
+ --check-length Check max filename length
+ --check-normalization Check UTF-8 Normalization
+ --check-streaming Check uploads with indeterminate file size
+ -h, --help help for info
+ --upload-wait Duration Wait after writing a file (default 0s)
+ --write-json string Write results to file
+```
-Set to `0` to disable the buffering for the minimum memory usage.
-Note that the memory allocation of the buffers is influenced by the
-[--use-mmap](#use-mmap) flag.
+See the [global flags page](https://rclone.org/flags/) for global options not listed here.
-### --cache-dir=DIR ###
+# SEE ALSO
-Specify the directory rclone will use for caching, to override
-the default.
+* [rclone test](https://rclone.org/commands/rclone_test/) - Run a test command
-Default value is depending on operating system:
-- Windows `%LocalAppData%\rclone`, if `LocalAppData` is defined.
-- macOS `$HOME/Library/Caches/rclone` if `HOME` is defined.
-- Unix `$XDG_CACHE_HOME/rclone` if `XDG_CACHE_HOME` is defined, else `$HOME/.cache/rclone` if `HOME` is defined.
-- Fallback (on all OS) to `$TMPDIR/rclone`, where `TMPDIR` is the value from [--temp-dir](#temp-dir-dir).
+# rclone test makefile
-You can use the [config paths](https://rclone.org/commands/rclone_config_paths/)
-command to see the current value.
+Make files with random contents of the size given
-Cache directory is heavily used by the [VFS File Caching](https://rclone.org/commands/rclone_mount/#vfs-file-caching)
-mount feature, but also by [serve](https://rclone.org/commands/rclone_serve/), [GUI](/gui) and other parts of rclone.
+```
+rclone test makefile []+ [flags]
+```
-### --check-first ###
+## Options
-If this flag is set then in a `sync`, `copy` or `move`, rclone will do
-all the checks to see whether files need to be transferred before
-doing any of the transfers. Normally rclone would start running
-transfers as soon as possible.
+```
+ --ascii Fill files with random ASCII printable bytes only
+ --chargen Fill files with a ASCII chargen pattern
+ -h, --help help for makefile
+ --pattern Fill files with a periodic pattern
+ --seed int Seed for the random number generator (0 for random) (default 1)
+ --sparse Make the files sparse (appear to be filled with ASCII 0x00)
+ --zero Fill files with ASCII 0x00
+```
-This flag can be useful on IO limited systems where transfers
-interfere with checking.
-It can also be useful to ensure perfect ordering when using
-`--order-by`.
+See the [global flags page](https://rclone.org/flags/) for global options not listed here.
-If both `--check-first` and `--order-by` are set when doing `rclone move`
-then rclone will use the transfer thread to delete source files which
-don't need transferring. This will enable perfect ordering of the
-transfers and deletes but will cause the transfer stats to have more
-items in than expected.
+# SEE ALSO
-Using this flag can use more memory as it effectively sets
-`--max-backlog` to infinite. This means that all the info on the
-objects to transfer is held in memory before the transfers start.
+* [rclone test](https://rclone.org/commands/rclone_test/) - Run a test command
-### --checkers=N ###
+# rclone test makefiles
-Originally controlling just the number of file checkers to run in parallel,
-e.g. by `rclone copy`. Now a fairly universal parallelism control
-used by `rclone` in several places.
+Make a random file hierarchy in a directory
-Note: checkers do the equality checking of files during a sync.
-For some storage systems (e.g. S3, Swift, Dropbox) this can take
-a significant amount of time so they are run in parallel.
+```
+rclone test makefiles [flags]
+```
-The default is to run 8 checkers in parallel. However, in case
-of slow-reacting backends you may need to lower (rather than increase)
-this default by setting `--checkers` to 4 or less threads. This is
-especially advised if you are experiencing backend server crashes
-during file checking phase (e.g. on subsequent or top-up backups
-where little or no file copying is done and checking takes up
-most of the time). Increase this setting only with utmost care,
-while monitoring your server health and file checking throughput.
+## Options
-### -c, --checksum ###
+```
+ --ascii Fill files with random ASCII printable bytes only
+ --chargen Fill files with a ASCII chargen pattern
+ --files int Number of files to create (default 1000)
+ --files-per-directory int Average number of files per directory (default 10)
+ -h, --help help for makefiles
+ --max-depth int Maximum depth of directory hierarchy (default 10)
+ --max-file-size SizeSuffix Maximum size of files to create (default 100)
+ --max-name-length int Maximum size of file names (default 12)
+ --min-file-size SizeSuffix Minimum size of file to create
+ --min-name-length int Minimum size of file names (default 4)
+ --pattern Fill files with a periodic pattern
+ --seed int Seed for the random number generator (0 for random) (default 1)
+ --sparse Make the files sparse (appear to be filled with ASCII 0x00)
+ --zero Fill files with ASCII 0x00
+```
-Normally rclone will look at modification time and size of files to
-see if they are equal. If you set this flag then rclone will check
-the file hash and size to determine if files are equal.
-This is useful when the remote doesn't support setting modified time
-and a more accurate sync is desired than just checking the file size.
+See the [global flags page](https://rclone.org/flags/) for global options not listed here.
-This is very useful when transferring between remotes which store the
-same hash type on the object, e.g. Drive and Swift. For details of which
-remotes support which hash type see the table in the [overview
-section](https://rclone.org/overview/).
+# SEE ALSO
-Eg `rclone --checksum sync s3:/bucket swift:/bucket` would run much
-quicker than without the `--checksum` flag.
+* [rclone test](https://rclone.org/commands/rclone_test/) - Run a test command
-When using this flag, rclone won't update mtimes of remote files if
-they are incorrect as it would normally.
+# rclone test memory
-### --color WHEN ###
+Load all the objects at remote:path into memory and report memory stats.
-Specify when colors (and other ANSI codes) should be added to the output.
+```
+rclone test memory remote:path [flags]
+```
-`AUTO` (default) only allows ANSI codes when the output is a terminal
+## Options
-`NEVER` never allow ANSI codes
+```
+ -h, --help help for memory
+```
-`ALWAYS` always add ANSI codes, regardless of the output format (terminal or file)
-### --compare-dest=DIR ###
+See the [global flags page](https://rclone.org/flags/) for global options not listed here.
-When using `sync`, `copy` or `move` DIR is checked in addition to the
-destination for files. If a file identical to the source is found that
-file is NOT copied from source. This is useful to copy just files that
-have changed since the last backup.
+# SEE ALSO
-You must use the same remote as the destination of the sync. The
-compare directory must not overlap the destination directory.
+* [rclone test](https://rclone.org/commands/rclone_test/) - Run a test command
-See `--copy-dest` and `--backup-dir`.
+# rclone touch
-### --config=CONFIG_FILE ###
+Create new file or change file modification time.
-Specify the location of the rclone configuration file, to override
-the default. E.g. `rclone config --config="rclone.conf"`.
+## Synopsis
-The exact default is a bit complex to describe, due to changes
-introduced through different versions of rclone while preserving
-backwards compatibility, but in most cases it is as simple as:
- - `%APPDATA%/rclone/rclone.conf` on Windows
- - `~/.config/rclone/rclone.conf` on other
+Set the modification time on file(s) as specified by remote:path to
+have the current time.
-The complete logic is as follows: Rclone will look for an existing
-configuration file in any of the following locations, in priority order:
+If remote:path does not exist then a zero sized file will be created,
+unless `--no-create` or `--recursive` is provided.
- 1. `rclone.conf` (in program directory, where rclone executable is)
- 2. `%APPDATA%/rclone/rclone.conf` (only on Windows)
- 3. `$XDG_CONFIG_HOME/rclone/rclone.conf` (on all systems, including Windows)
- 4. `~/.config/rclone/rclone.conf` (see below for explanation of ~ symbol)
- 5. `~/.rclone.conf`
+If `--recursive` is used then recursively sets the modification
+time on all existing files that is found under the path. Filters are supported,
+and you can test with the `--dry-run` or the `--interactive`/`-i` flag.
-If no existing configuration file is found, then a new one will be created
-in the following location:
+If `--timestamp` is used then sets the modification time to that
+time instead of the current time. Times may be specified as one of:
-- On Windows: Location 2 listed above, except in the unlikely event
- that `APPDATA` is not defined, then location 4 is used instead.
-- On Unix: Location 3 if `XDG_CONFIG_HOME` is defined, else location 4.
-- Fallback to location 5 (on all OS), when the rclone directory cannot be
- created, but if also a home directory was not found then path
- `.rclone.conf` relative to current working directory will be used as
- a final resort.
+- 'YYMMDD' - e.g. 17.10.30
+- 'YYYY-MM-DDTHH:MM:SS' - e.g. 2006-01-02T15:04:05
+- 'YYYY-MM-DDTHH:MM:SS.SSS' - e.g. 2006-01-02T15:04:05.123456789
-The `~` symbol in paths above represent the home directory of the current user
-on any OS, and the value is defined as following:
+Note that value of `--timestamp` is in UTC. If you want local time
+then add the `--localtime` flag.
- - On Windows: `%HOME%` if defined, else `%USERPROFILE%`, or else `%HOMEDRIVE%\%HOMEPATH%`.
- - On Unix: `$HOME` if defined, else by looking up current user in OS-specific user database
- (e.g. passwd file), or else use the result from shell command `cd && pwd`.
-If you run `rclone config file` you will see where the default
-location is for you.
+```
+rclone touch remote:path [flags]
+```
-The fact that an existing file `rclone.conf` in the same directory
-as the rclone executable is always preferred, means that it is easy
-to run in "portable" mode by downloading rclone executable to a
-writable directory and then create an empty file `rclone.conf` in the
-same directory.
+## Options
-If the location is set to empty string `""` or path to a file
-with name `notfound`, or the os null device represented by value `NUL` on
-Windows and `/dev/null` on Unix systems, then rclone will keep the
-config file in memory only.
+```
+ -h, --help help for touch
+ --localtime Use localtime for timestamp, not UTC
+ -C, --no-create Do not create the file if it does not exist (implied with --recursive)
+ -R, --recursive Recursively touch all files
+ -t, --timestamp string Use specified time instead of the current time of day
+```
-The file format is basic [INI](https://en.wikipedia.org/wiki/INI_file#Format):
-Sections of text, led by a `[section]` header and followed by
-`key=value` entries on separate lines. In rclone each remote is
-represented by its own section, where the section name defines the
-name of the remote. Options are specified as the `key=value` entries,
-where the key is the option name without the `--backend-` prefix,
-in lowercase and with `_` instead of `-`. E.g. option `--mega-hard-delete`
-corresponds to key `hard_delete`. Only backend options can be specified.
-A special, and required, key `type` identifies the [storage system](https://rclone.org/overview/),
-where the value is the internal lowercase name as returned by command
-`rclone help backends`. Comments are indicated by `;` or `#` at the
-beginning of a line.
-Example:
+## Important Options
- [megaremote]
- type = mega
- user = you@example.com
- pass = PDPcQVVjVtzFY-GTdDFozqBhTdsPg3qH
+Important flags useful for most commands.
-Note that passwords are in [obscured](https://rclone.org/commands/rclone_obscure/)
-form. Also, many storage systems uses token-based authentication instead
-of passwords, and this requires additional steps. It is easier, and safer,
-to use the interactive command `rclone config` instead of manually
-editing the configuration file.
+```
+ -n, --dry-run Do a trial run with no permanent changes
+ -i, --interactive Enable interactive mode
+ -v, --verbose count Print lots more stuff (repeat for more)
+```
-The configuration file will typically contain login information, and
-should therefore have restricted permissions so that only the current user
-can read it. Rclone tries to ensure this when it writes the file.
-You may also choose to [encrypt](#configuration-encryption) the file.
+## Filter Options
-When token-based authentication are used, the configuration file
-must be writable, because rclone needs to update the tokens inside it.
+Flags for filtering directory listings.
-To reduce risk of corrupting an existing configuration file, rclone
-will not write directly to it when saving changes. Instead it will
-first write to a new, temporary, file. If a configuration file already
-existed, it will (on Unix systems) try to mirror its permissions to
-the new file. Then it will rename the existing file to a temporary
-name as backup. Next, rclone will rename the new file to the correct name,
-before finally cleaning up by deleting the backup file.
+```
+ --delete-excluded Delete files on dest excluded from sync
+ --exclude stringArray Exclude files matching pattern
+ --exclude-from stringArray Read file exclude patterns from file (use - to read from stdin)
+ --exclude-if-present stringArray Exclude directories if filename is present
+ --files-from stringArray Read list of source-file names from file (use - to read from stdin)
+ --files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin)
+ -f, --filter stringArray Add a file filtering rule
+ --filter-from stringArray Read file filtering patterns from a file (use - to read from stdin)
+ --ignore-case Ignore case in filters (case insensitive)
+ --include stringArray Include files matching pattern
+ --include-from stringArray Read file include patterns from file (use - to read from stdin)
+ --max-age Duration Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off)
+ --max-depth int If set limits the recursion depth to this (default -1)
+ --max-size SizeSuffix Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off)
+ --metadata-exclude stringArray Exclude metadatas matching pattern
+ --metadata-exclude-from stringArray Read metadata exclude patterns from file (use - to read from stdin)
+ --metadata-filter stringArray Add a metadata filtering rule
+ --metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin)
+ --metadata-include stringArray Include metadatas matching pattern
+ --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin)
+ --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off)
+ --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off)
+```
-If the configuration file path used by rclone is a symbolic link, then
-this will be evaluated and rclone will write to the resolved path, instead
-of overwriting the symbolic link. Temporary files used in the process
-(described above) will be written to the same parent directory as that
-of the resolved configuration file, but if this directory is also a
-symbolic link it will not be resolved and the temporary files will be
-written to the location of the directory symbolic link.
+## Listing Options
-### --contimeout=TIME ###
+Flags for listing directories.
-Set the connection timeout. This should be in go time format which
-looks like `5s` for 5 seconds, `10m` for 10 minutes, or `3h30m`.
+```
+ --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z)
+ --fast-list Use recursive list if available; uses more memory but fewer transactions
+```
-The connection timeout is the amount of time rclone will wait for a
-connection to go through to a remote object storage system. It is
-`1m` by default.
+See the [global flags page](https://rclone.org/flags/) for global options not listed here.
-### --copy-dest=DIR ###
+# SEE ALSO
-When using `sync`, `copy` or `move` DIR is checked in addition to the
-destination for files. If a file identical to the source is found that
-file is server-side copied from DIR to the destination. This is useful
-for incremental backup.
+* [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends.
-The remote in use must support server-side copy and you must
-use the same remote as the destination of the sync. The compare
-directory must not overlap the destination directory.
+# rclone tree
-See `--compare-dest` and `--backup-dir`.
+List the contents of the remote in a tree like fashion.
-### --dedupe-mode MODE ###
+## Synopsis
-Mode to run dedupe command in. One of `interactive`, `skip`, `first`,
-`newest`, `oldest`, `rename`. The default is `interactive`.
-See the dedupe command for more information as to what these options mean.
-### --default-time TIME ###
+rclone tree lists the contents of a remote in a similar way to the
+unix tree command.
-If a file or directory does have a modification time rclone can read
-then rclone will display this fixed time instead.
+For example
-The default is `2000-01-01 00:00:00 UTC`. This can be configured in
-any of the ways shown in [the time or duration options](#time-option).
+ $ rclone tree remote:path
+ /
+ ├── file1
+ ├── file2
+ ├── file3
+ └── subdir
+ ├── file4
+ └── file5
-For example `--default-time 2020-06-01` to set the default time to the
-1st of June 2020 or `--default-time 0s` to set the default time to the
-time rclone started up.
+ 1 directories, 5 files
-### --disable FEATURE,FEATURE,... ###
+You can use any of the filtering options with the tree command (e.g.
+`--include` and `--exclude`. You can also use `--fast-list`.
-This disables a comma separated list of optional features. For example
-to disable server-side move and server-side copy use:
+The tree command has many options for controlling the listing which
+are compatible with the tree command, for example you can include file
+sizes with `--size`. Note that not all of them have
+short options as they conflict with rclone's short options.
- --disable move,copy
+For a more interactive navigation of the remote see the
+[ncdu](https://rclone.org/commands/rclone_ncdu/) command.
-The features can be put in any case.
-To see a list of which features can be disabled use:
+```
+rclone tree remote:path [flags]
+```
- --disable help
+## Options
-The features a remote has can be seen in JSON format with:
+```
+ -a, --all All files are listed (list . files too)
+ -d, --dirs-only List directories only
+ --dirsfirst List directories before files (-U disables)
+ --full-path Print the full path prefix for each file
+ -h, --help help for tree
+ --level int Descend only level directories deep
+ -D, --modtime Print the date of last modification.
+ --noindent Don't print indentation lines
+ --noreport Turn off file/directory count at end of tree listing
+ -o, --output string Output to file instead of stdout
+ -p, --protections Print the protections for each file.
+ -Q, --quote Quote filenames with double quotes.
+ -s, --size Print the size in bytes of each file.
+ --sort string Select sort: name,version,size,mtime,ctime
+ --sort-ctime Sort files by last status change time
+ -t, --sort-modtime Sort files by last modification time
+ -r, --sort-reverse Reverse the order of the sort
+ -U, --unsorted Leave files unsorted
+ --version Sort files alphanumerically by version
+```
- rclone backend features remote:
-See the overview [features](https://rclone.org/overview/#features) and
-[optional features](https://rclone.org/overview/#optional-features) to get an idea of
-which feature does what.
+## Filter Options
-Note that some features can be set to `true` if they are `true`/`false`
-feature flag features by prefixing them with `!`. For example the
-`CaseInsensitive` feature can be forced to `false` with `--disable CaseInsensitive`
-and forced to `true` with `--disable '!CaseInsensitive'`. In general
-it isn't a good idea doing this but it may be useful in extremis.
+Flags for filtering directory listings.
-(Note that `!` is a shell command which you will
-need to escape with single quotes or a backslash on unix like
-platforms.)
+```
+ --delete-excluded Delete files on dest excluded from sync
+ --exclude stringArray Exclude files matching pattern
+ --exclude-from stringArray Read file exclude patterns from file (use - to read from stdin)
+ --exclude-if-present stringArray Exclude directories if filename is present
+ --files-from stringArray Read list of source-file names from file (use - to read from stdin)
+ --files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin)
+ -f, --filter stringArray Add a file filtering rule
+ --filter-from stringArray Read file filtering patterns from a file (use - to read from stdin)
+ --ignore-case Ignore case in filters (case insensitive)
+ --include stringArray Include files matching pattern
+ --include-from stringArray Read file include patterns from file (use - to read from stdin)
+ --max-age Duration Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off)
+ --max-depth int If set limits the recursion depth to this (default -1)
+ --max-size SizeSuffix Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off)
+ --metadata-exclude stringArray Exclude metadatas matching pattern
+ --metadata-exclude-from stringArray Read metadata exclude patterns from file (use - to read from stdin)
+ --metadata-filter stringArray Add a metadata filtering rule
+ --metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin)
+ --metadata-include stringArray Include metadatas matching pattern
+ --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin)
+ --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off)
+ --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off)
+```
-This flag can be useful for debugging and in exceptional circumstances
-(e.g. Google Drive limiting the total volume of Server Side Copies to
-100 GiB/day).
+## Listing Options
-### --disable-http2
+Flags for listing directories.
-This stops rclone from trying to use HTTP/2 if available. This can
-sometimes speed up transfers due to a
-[problem in the Go standard library](https://github.com/golang/go/issues/37373).
+```
+ --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z)
+ --fast-list Use recursive list if available; uses more memory but fewer transactions
+```
-### --dscp VALUE ###
+See the [global flags page](https://rclone.org/flags/) for global options not listed here.
-Specify a DSCP value or name to use in connections. This could help QoS
-system to identify traffic class. BE, EF, DF, LE, CSx and AFxx are allowed.
+# SEE ALSO
-See the description of [differentiated services](https://en.wikipedia.org/wiki/Differentiated_services) to get an idea of
-this field. Setting this to 1 (LE) to identify the flow to SCAVENGER class
-can avoid occupying too much bandwidth in a network with DiffServ support ([RFC 8622](https://tools.ietf.org/html/rfc8622)).
-
-For example, if you configured QoS on router to handle LE properly. Running:
-```
-rclone copy --dscp LE from:/from to:/to
-```
-would make the priority lower than usual internet flows.
+* [rclone](https://rclone.org/commands/rclone/) - Show help for rclone commands, flags and backends.
-This option has no effect on Windows (see [golang/go#42728](https://github.com/golang/go/issues/42728)).
-### -n, --dry-run ###
+Copying single files
+--------------------
-Do a trial run with no permanent changes. Use this to see what rclone
-would do without actually doing it. Useful when setting up the `sync`
-command which deletes files in the destination.
+rclone normally syncs or copies directories. However, if the source
+remote points to a file, rclone will just copy that file. The
+destination remote must point to a directory - rclone will give the
+error `Failed to create file system for "remote:file": is a file not a
+directory` if it isn't.
-### --expect-continue-timeout=TIME ###
+For example, suppose you have a remote with a file in called
+`test.jpg`, then you could copy just that file like this
-This specifies the amount of time to wait for a server's first
-response headers after fully writing the request headers if the
-request has an "Expect: 100-continue" header. Not all backends support
-using this.
+ rclone copy remote:test.jpg /tmp/download
-Zero means no timeout and causes the body to be sent immediately,
-without waiting for the server to approve. This time does not include
-the time to send the request header.
+The file `test.jpg` will be placed inside `/tmp/download`.
-The default is `1s`. Set to `0` to disable.
+This is equivalent to specifying
-### --error-on-no-transfer ###
+ rclone copy --files-from /tmp/files remote: /tmp/download
-By default, rclone will exit with return code 0 if there were no errors.
+Where `/tmp/files` contains the single line
-This option allows rclone to return exit code 9 if no files were transferred
-between the source and destination. This allows using rclone in scripts, and
-triggering follow-on actions if data was copied, or skipping if not.
+ test.jpg
-NB: Enabling this option turns a usually non-fatal error into a potentially
-fatal one - please check and adjust your scripts accordingly!
+It is recommended to use `copy` when copying individual files, not `sync`.
+They have pretty much the same effect but `copy` will use a lot less
+memory.
-### --fs-cache-expire-duration=TIME
+Syntax of remote paths
+----------------------
-When using rclone via the API rclone caches created remotes for 5
-minutes by default in the "fs cache". This means that if you do
-repeated actions on the same remote then rclone won't have to build it
-again from scratch, which makes it more efficient.
+The syntax of the paths passed to the rclone command are as follows.
-This flag sets the time that the remotes are cached for. If you set it
-to `0` (or negative) then rclone won't cache the remotes at all.
+### /path/to/dir
-Note that if you use some flags, eg `--backup-dir` and if this is set
-to `0` rclone may build two remotes (one for the source or destination
-and one for the `--backup-dir` where it may have only built one
-before.
+This refers to the local file system.
-### --fs-cache-expire-interval=TIME
+On Windows `\` may be used instead of `/` in local paths **only**,
+non local paths must use `/`. See [local filesystem](https://rclone.org/local/#paths-on-windows)
+documentation for more about Windows-specific paths.
-This controls how often rclone checks for cached remotes to expire.
-See the `--fs-cache-expire-duration` documentation above for more
-info. The default is 60s, set to 0 to disable expiry.
+These paths needn't start with a leading `/` - if they don't then they
+will be relative to the current directory.
-### --header ###
+### remote:path/to/dir
-Add an HTTP header for all transactions. The flag can be repeated to
-add multiple headers.
+This refers to a directory `path/to/dir` on `remote:` as defined in
+the config file (configured with `rclone config`).
-If you want to add headers only for uploads use `--header-upload` and
-if you want to add headers only for downloads use `--header-download`.
+### remote:/path/to/dir
-This flag is supported for all HTTP based backends even those not
-supported by `--header-upload` and `--header-download` so may be used
-as a workaround for those with care.
+On most backends this is refers to the same directory as
+`remote:path/to/dir` and that format should be preferred. On a very
+small number of remotes (FTP, SFTP, Dropbox for business) this will
+refer to a different directory. On these, paths without a leading `/`
+will refer to your "home" directory and paths with a leading `/` will
+refer to the root.
-```
-rclone ls remote:test --header "X-Rclone: Foo" --header "X-LetMeIn: Yes"
-```
+### :backend:path/to/dir
-### --header-download ###
+This is an advanced form for creating remotes on the fly. `backend`
+should be the name or prefix of a backend (the `type` in the config
+file) and all the configuration for the backend should be provided on
+the command line (or in environment variables).
-Add an HTTP header for all download transactions. The flag can be repeated to
-add multiple headers.
+Here are some examples:
-```
-rclone sync --interactive s3:test/src ~/dst --header-download "X-Amz-Meta-Test: Foo" --header-download "X-Amz-Meta-Test2: Bar"
-```
+ rclone lsd --http-url https://pub.rclone.org :http:
-See the GitHub issue [here](https://github.com/rclone/rclone/issues/59) for
-currently supported backends.
+To list all the directories in the root of `https://pub.rclone.org/`.
-### --header-upload ###
+ rclone lsf --http-url https://example.com :http:path/to/dir
-Add an HTTP header for all upload transactions. The flag can be repeated to add
-multiple headers.
+To list files and directories in `https://example.com/path/to/dir/`
-```
-rclone sync --interactive ~/src s3:test/dst --header-upload "Content-Disposition: attachment; filename='cool.html'" --header-upload "X-Amz-Meta-Test: FooBar"
-```
+ rclone copy --http-url https://example.com :http:path/to/dir /tmp/dir
-See the GitHub issue [here](https://github.com/rclone/rclone/issues/59) for
-currently supported backends.
+To copy files and directories in `https://example.com/path/to/dir` to `/tmp/dir`.
-### --human-readable ###
+ rclone copy --sftp-host example.com :sftp:path/to/dir /tmp/dir
-Rclone commands output values for sizes (e.g. number of bytes) and
-counts (e.g. number of files) either as *raw* numbers, or
-in *human-readable* format.
+To copy files and directories from `example.com` in the relative
+directory `path/to/dir` to `/tmp/dir` using sftp.
-In human-readable format the values are scaled to larger units, indicated with
-a suffix shown after the value, and rounded to three decimals. Rclone consistently
-uses binary units (powers of 2) for sizes and decimal units (powers of 10) for counts.
-The unit prefix for size is according to IEC standard notation, e.g. `Ki` for kibi.
-Used with byte unit, `1 KiB` means 1024 Byte. In list type of output, only the
-unit prefix appended to the value (e.g. `9.762Ki`), while in more textual output
-the full unit is shown (e.g. `9.762 KiB`). For counts the SI standard notation is
-used, e.g. prefix `k` for kilo. Used with file counts, `1k` means 1000 files.
+### Connection strings {#connection-strings}
-The various [list](https://rclone.org/commands/rclone_ls/) commands output raw numbers by default.
-Option `--human-readable` will make them output values in human-readable format
-instead (with the short unit prefix).
+The above examples can also be written using a connection string
+syntax, so instead of providing the arguments as command line
+parameters `--http-url https://pub.rclone.org` they are provided as
+part of the remote specification as a kind of connection string.
-The [about](https://rclone.org/commands/rclone_about/) command outputs human-readable by default,
-with a command-specific option `--full` to output the raw numbers instead.
+ rclone lsd ":http,url='https://pub.rclone.org':"
+ rclone lsf ":http,url='https://example.com':path/to/dir"
+ rclone copy ":http,url='https://example.com':path/to/dir" /tmp/dir
+ rclone copy :sftp,host=example.com:path/to/dir /tmp/dir
-Command [size](https://rclone.org/commands/rclone_size/) outputs both human-readable and raw numbers
-in the same output.
+These can apply to modify existing remotes as well as create new
+remotes with the on the fly syntax. This example is equivalent to
+adding the `--drive-shared-with-me` parameter to the remote `gdrive:`.
-The [tree](https://rclone.org/commands/rclone_tree/) command also considers `--human-readable`, but
-it will not use the exact same notation as the other commands: It rounds to one
-decimal, and uses single letter suffix, e.g. `K` instead of `Ki`. The reason for
-this is that it relies on an external library.
+ rclone lsf "gdrive,shared_with_me:path/to/dir"
-The interactive command [ncdu](https://rclone.org/commands/rclone_ncdu/) shows human-readable by
-default, and responds to key `u` for toggling human-readable format.
+The major advantage to using the connection string style syntax is
+that it only applies to the remote, not to all the remotes of that
+type of the command line. A common confusion is this attempt to copy a
+file shared on google drive to the normal drive which **does not
+work** because the `--drive-shared-with-me` flag applies to both the
+source and the destination.
-### --ignore-case-sync ###
+ rclone copy --drive-shared-with-me gdrive:shared-file.txt gdrive:
-Using this option will cause rclone to ignore the case of the files
-when synchronizing so files will not be copied/synced when the
-existing filenames are the same, even if the casing is different.
+However using the connection string syntax, this does work.
-### --ignore-checksum ###
+ rclone copy "gdrive,shared_with_me:shared-file.txt" gdrive:
-Normally rclone will check that the checksums of transferred files
-match, and give an error "corrupted on transfer" if they don't.
+Note that the connection string only affects the options of the immediate
+backend. If for example gdriveCrypt is a crypt based on gdrive, then the
+following command **will not work** as intended, because
+`shared_with_me` is ignored by the crypt backend:
-You can use this option to skip that check. You should only use it if
-you have had the "corrupted on transfer" error message and you are
-sure you might want to transfer potentially corrupted data.
+ rclone copy "gdriveCrypt,shared_with_me:shared-file.txt" gdriveCrypt:
-### --ignore-existing ###
+The connection strings have the following syntax
-Using this option will make rclone unconditionally skip all files
-that exist on the destination, no matter the content of these files.
+ remote,parameter=value,parameter2=value2:path/to/dir
+ :backend,parameter=value,parameter2=value2:path/to/dir
-While this isn't a generally recommended option, it can be useful
-in cases where your files change due to encryption. However, it cannot
-correct partial transfers in case a transfer was interrupted.
+If the `parameter` has a `:` or `,` then it must be placed in quotes `"` or
+`'`, so
-When performing a `move`/`moveto` command, this flag will leave skipped
-files in the source location unchanged when a file with the same name
-exists on the destination.
+ remote,parameter="colon:value",parameter2="comma,value":path/to/dir
+ :backend,parameter='colon:value',parameter2='comma,value':path/to/dir
-### --ignore-size ###
+If a quoted value needs to include that quote, then it should be
+doubled, so
-Normally rclone will look at modification time and size of files to
-see if they are equal. If you set this flag then rclone will check
-only the modification time. If `--checksum` is set then it only
-checks the checksum.
+ remote,parameter="with""quote",parameter2='with''quote':path/to/dir
-It will also cause rclone to skip verifying the sizes are the same
-after transfer.
+This will make `parameter` be `with"quote` and `parameter2` be
+`with'quote`.
-This can be useful for transferring files to and from OneDrive which
-occasionally misreports the size of image files (see
-[#399](https://github.com/rclone/rclone/issues/399) for more info).
+If you leave off the `=parameter` then rclone will substitute `=true`
+which works very well with flags. For example, to use s3 configured in
+the environment you could use:
-### -I, --ignore-times ###
+ rclone lsd :s3,env_auth:
-Using this option will cause rclone to unconditionally upload all
-files regardless of the state of files on the destination.
+Which is equivalent to
-Normally rclone would skip any files that have the same
-modification time and are the same size (or have the same checksum if
-using `--checksum`).
+ rclone lsd :s3,env_auth=true:
-### --immutable ###
+Note that on the command line you might need to surround these
+connection strings with `"` or `'` to stop the shell interpreting any
+special characters within them.
-Treat source and destination files as immutable and disallow
-modification.
+If you are a shell master then you'll know which strings are OK and
+which aren't, but if you aren't sure then enclose them in `"` and use
+`'` as the inside quote. This syntax works on all OSes.
-With this option set, files will be created and deleted as requested,
-but existing files will never be updated. If an existing file does
-not match between the source and destination, rclone will give the error
-`Source and destination exist but do not match: immutable file modified`.
+ rclone copy ":http,url='https://example.com':path/to/dir" /tmp/dir
-Note that only commands which transfer files (e.g. `sync`, `copy`,
-`move`) are affected by this behavior, and only modification is
-disallowed. Files may still be deleted explicitly (e.g. `delete`,
-`purge`) or implicitly (e.g. `sync`, `move`). Use `copy --immutable`
-if it is desired to avoid deletion as well as modification.
+On Linux/macOS some characters are still interpreted inside `"`
+strings in the shell (notably `\` and `$` and `"`) so if your strings
+contain those you can swap the roles of `"` and `'` thus. (This syntax
+does not work on Windows.)
-This can be useful as an additional layer of protection for immutable
-or append-only data sets (notably backup archives), where modification
-implies corruption and should not be propagated.
+ rclone copy ':http,url="https://example.com":path/to/dir' /tmp/dir
-### --inplace {#inplace}
+#### Connection strings, config and logging
-The `--inplace` flag changes the behaviour of rclone when uploading
-files to some backends (backends with the `PartialUploads` feature
-flag set) such as:
+If you supply extra configuration to a backend by command line flag,
+environment variable or connection string then rclone will add a
+suffix based on the hash of the config to the name of the remote, eg
-- local
-- ftp
-- sftp
+ rclone -vv lsf --s3-chunk-size 20M s3:
-Without `--inplace` (the default) rclone will first upload to a
-temporary file with an extension like this where `XXXXXX` represents a
-random string.
+Has the log message
- original-file-name.XXXXXX.partial
+ DEBUG : s3: detected overridden config - adding "{Srj1p}" suffix to name
-(rclone will make sure the final name is no longer than 100 characters
-by truncating the `original-file-name` part if necessary).
+This is so rclone can tell the modified remote apart from the
+unmodified remote when caching the backends.
-When the upload is complete, rclone will rename the `.partial` file to
-the correct name, overwriting any existing file at that point. If the
-upload fails then the `.partial` file will be deleted.
+This should only be noticeable in the logs.
-This prevents other users of the backend from seeing partially
-uploaded files in their new names and prevents overwriting the old
-file until the new one is completely uploaded.
+This means that on the fly backends such as
-If the `--inplace` flag is supplied, rclone will upload directly to
-the final name without creating a `.partial` file.
+ rclone -vv lsf :s3,env_auth:
-This means that an incomplete file will be visible in the directory
-listings while the upload is in progress and any existing files will
-be overwritten as soon as the upload starts. If the transfer fails
-then the file will be deleted. This can cause data loss of the
-existing file if the transfer fails.
+Will get their own names
-Note that on the local file system if you don't use `--inplace` hard
-links (Unix only) will be broken. And if you do use `--inplace` you
-won't be able to update in use executables.
+ DEBUG : :s3: detected overridden config - adding "{YTu53}" suffix to name
-Note also that versions of rclone prior to v1.63.0 behave as if the
-`--inplace` flag is always supplied.
+### Valid remote names
-### -i, --interactive {#interactive}
+Remote names are case sensitive, and must adhere to the following rules:
+ - May contain number, letter, `_`, `-`, `.`, `+`, `@` and space.
+ - May not start with `-` or space.
+ - May not end with space.
-This flag can be used to tell rclone that you wish a manual
-confirmation before destructive operations.
+Starting with rclone version 1.61, any Unicode numbers and letters are allowed,
+while in older versions it was limited to plain ASCII (0-9, A-Z, a-z). If you use
+the same rclone configuration from different shells, which may be configured with
+different character encoding, you must be cautious to use characters that are
+possible to write in all of them. This is mostly a problem on Windows, where
+the console traditionally uses a non-Unicode character set - defined
+by the so-called "code page".
-It is **recommended** that you use this flag while learning rclone
-especially with `rclone sync`.
+Do not use single character names on Windows as it creates ambiguity with Windows
+drives' names, e.g.: remote called `C` is indistinguishable from `C` drive. Rclone
+will always assume that single letter name refers to a drive.
-For example
+Quoting and the shell
+---------------------
-```
-$ rclone delete --interactive /tmp/dir
-rclone: delete "important-file.txt"?
-y) Yes, this is OK (default)
-n) No, skip this
-s) Skip all delete operations with no more questions
-!) Do all delete operations with no more questions
-q) Exit rclone now.
-y/n/s/!/q> n
-```
+When you are typing commands to your computer you are using something
+called the command line shell. This interprets various characters in
+an OS specific way.
-The options mean
+Here are some gotchas which may help users unfamiliar with the shell rules
-- `y`: **Yes**, this operation should go ahead. You can also press Return
- for this to happen. You'll be asked every time unless you choose `s`
- or `!`.
-- `n`: **No**, do not do this operation. You'll be asked every time unless
- you choose `s` or `!`.
-- `s`: **Skip** all the following operations of this type with no more
- questions. This takes effect until rclone exits. If there are any
- different kind of operations you'll be prompted for them.
-- `!`: **Do all** the following operations with no more
- questions. Useful if you've decided that you don't mind rclone doing
- that kind of operation. This takes effect until rclone exits . If
- there are any different kind of operations you'll be prompted for
- them.
-- `q`: **Quit** rclone now, just in case!
+### Linux / OSX ###
-### --leave-root ####
+If your names have spaces or shell metacharacters (e.g. `*`, `?`, `$`,
+`'`, `"`, etc.) then you must quote them. Use single quotes `'` by default.
-During rmdirs it will not remove root directory, even if it's empty.
+ rclone copy 'Important files?' remote:backup
-### --log-file=FILE ###
+If you want to send a `'` you will need to use `"`, e.g.
-Log all of rclone's output to FILE. This is not active by default.
-This can be useful for tracking down problems with syncs in
-combination with the `-v` flag. See the [Logging section](#logging)
-for more info.
+ rclone copy "O'Reilly Reviews" remote:backup
-If FILE exists then rclone will append to it.
+The rules for quoting metacharacters are complicated and if you want
+the full details you'll have to consult the manual page for your
+shell.
-Note that if you are using the `logrotate` program to manage rclone's
-logs, then you should use the `copytruncate` option as rclone doesn't
-have a signal to rotate logs.
+### Windows ###
-### --log-format LIST ###
+If your names have spaces in you need to put them in `"`, e.g.
-Comma separated list of log format options. Accepted options are `date`,
-`time`, `microseconds`, `pid`, `longfile`, `shortfile`, `UTC`. Any other
-keywords will be silently ignored. `pid` will tag log messages with process
-identifier which useful with `rclone mount --daemon`. Other accepted
-options are explained in the [go documentation](https://pkg.go.dev/log#pkg-constants).
-The default log format is "`date`,`time`".
+ rclone copy "E:\folder name\folder name\folder name" remote:backup
-### --log-level LEVEL ###
+If you are using the root directory on its own then don't quote it
+(see [#464](https://github.com/rclone/rclone/issues/464) for why), e.g.
-This sets the log level for rclone. The default log level is `NOTICE`.
+ rclone copy E:\ remote:backup
-`DEBUG` is equivalent to `-vv`. It outputs lots of debug info - useful
-for bug reports and really finding out what rclone is doing.
+Copying files or directories with `:` in the names
+--------------------------------------------------
-`INFO` is equivalent to `-v`. It outputs information about each transfer
-and prints stats once a minute by default.
+rclone uses `:` to mark a remote name. This is, however, a valid
+filename component in non-Windows OSes. The remote name parser will
+only search for a `:` up to the first `/` so if you need to act on a
+file or directory like this then use the full path starting with a
+`/`, or use `./` as a current directory prefix.
-`NOTICE` is the default log level if no logging flags are supplied. It
-outputs very little when things are working normally. It outputs
-warnings and significant events.
+So to sync a directory called `sync:me` to a remote called `remote:` use
-`ERROR` is equivalent to `-q`. It only outputs error messages.
+ rclone sync --interactive ./sync:me remote:path
-### --use-json-log ###
+or
-This switches the log format to JSON for rclone. The fields of json log
-are level, msg, source, time.
+ rclone sync --interactive /full/path/to/sync:me remote:path
-### --low-level-retries NUMBER ###
+Server Side Copy
+----------------
-This controls the number of low level retries rclone does.
+Most remotes (but not all - see [the
+overview](https://rclone.org/overview/#optional-features)) support server-side copy.
-A low level retry is used to retry a failing operation - typically one
-HTTP request. This might be uploading a chunk of a big file for
-example. You will see low level retries in the log with the `-v`
-flag.
+This means if you want to copy one folder to another then rclone won't
+download all the files and re-upload them; it will instruct the server
+to copy them in place.
-This shouldn't need to be changed from the default in normal operations.
-However, if you get a lot of low level retries you may wish
-to reduce the value so rclone moves on to a high level retry (see the
-`--retries` flag) quicker.
+Eg
-Disable low level retries with `--low-level-retries 1`.
+ rclone copy s3:oldbucket s3:newbucket
-### --max-backlog=N ###
+Will copy the contents of `oldbucket` to `newbucket` without
+downloading and re-uploading.
-This is the maximum allowable backlog of files in a sync/copy/move
-queued for being checked or transferred.
+Remotes which don't support server-side copy **will** download and
+re-upload in this case.
-This can be set arbitrarily large. It will only use memory when the
-queue is in use. Note that it will use in the order of N KiB of memory
-when the backlog is in use.
+Server side copies are used with `sync` and `copy` and will be
+identified in the log when using the `-v` flag. The `move` command
+may also use them if remote doesn't support server-side move directly.
+This is done by issuing a server-side copy then a delete which is much
+quicker than a download and re-upload.
-Setting this large allows rclone to calculate how many files are
-pending more accurately, give a more accurate estimated finish
-time and make `--order-by` work more accurately.
+Server side copies will only be attempted if the remote names are the
+same.
-Setting this small will make rclone more synchronous to the listings
-of the remote which may be desirable.
+This can be used when scripting to make aged backups efficiently, e.g.
-Setting this to a negative number will make the backlog as large as
-possible.
+ rclone sync --interactive remote:current-backup remote:previous-backup
+ rclone sync --interactive /path/to/files remote:current-backup
-### --max-delete=N ###
+## Metadata support {#metadata}
-This tells rclone not to delete more than N files. If that limit is
-exceeded then a fatal error will be generated and rclone will stop the
-operation in progress.
+Metadata is data about a file which isn't the contents of the file.
+Normally rclone only preserves the modification time and the content
+(MIME) type where possible.
-### --max-delete-size=SIZE ###
+Rclone supports preserving all the available metadata on files (not
+directories) when using the `--metadata` or `-M` flag.
-Rclone will stop deleting files when the total size of deletions has
-reached the size specified. It defaults to off.
+Exactly what metadata is supported and what that support means depends
+on the backend. Backends that support metadata have a metadata section
+in their docs and are listed in the [features table](https://rclone.org/overview/#features)
+(Eg [local](https://rclone.org/local/#metadata), [s3](/s3/#metadata))
-If that limit is exceeded then a fatal error will be generated and
-rclone will stop the operation in progress.
+Rclone only supports a one-time sync of metadata. This means that
+metadata will be synced from the source object to the destination
+object only when the source object has changed and needs to be
+re-uploaded. If the metadata subsequently changes on the source object
+without changing the object itself then it won't be synced to the
+destination object. This is in line with the way rclone syncs
+`Content-Type` without the `--metadata` flag.
-### --max-depth=N ###
+Using `--metadata` when syncing from local to local will preserve file
+attributes such as file mode, owner, extended attributes (not
+Windows).
-This modifies the recursion depth for all the commands except purge.
+Note that arbitrary metadata may be added to objects using the
+`--metadata-set key=value` flag when the object is first uploaded.
+This flag can be repeated as many times as necessary.
-So if you do `rclone --max-depth 1 ls remote:path` you will see only
-the files in the top level directory. Using `--max-depth 2` means you
-will see all the files in first two directory levels and so on.
+The [--metadata-mapper](#metadata-mapper) flag can be used to pass the
+name of a program in which can transform metadata when it is being
+copied from source to destination.
-For historical reasons the `lsd` command defaults to using a
-`--max-depth` of 1 - you can override this with the command line flag.
+### Types of metadata
-You can use this command to disable recursion (with `--max-depth 1`).
+Metadata is divided into two type. System metadata and User metadata.
-Note that if you use this with `sync` and `--delete-excluded` the
-files not recursed through are considered excluded and will be deleted
-on the destination. Test first with `--dry-run` if you are not sure
-what will happen.
+Metadata which the backend uses itself is called system metadata. For
+example on the local backend the system metadata `uid` will store the
+user ID of the file when used on a unix based platform.
-### --max-duration=TIME ###
+Arbitrary metadata is called user metadata and this can be set however
+is desired.
-Rclone will stop transferring when it has run for the
-duration specified.
-Defaults to off.
-
-When the limit is reached all transfers will stop immediately.
-Use `--cutoff-mode` to modify this behaviour.
-
-Rclone will exit with exit code 10 if the duration limit is reached.
+When objects are copied from backend to backend, they will attempt to
+interpret system metadata if it is supplied. Metadata may change from
+being user metadata to system metadata as objects are copied between
+different backends. For example copying an object from s3 sets the
+`content-type` metadata. In a backend which understands this (like
+`azureblob`) this will become the Content-Type of the object. In a
+backend which doesn't understand this (like the `local` backend) this
+will become user metadata. However should the local object be copied
+back to s3, the Content-Type will be set correctly.
-### --max-transfer=SIZE ###
+### Metadata framework
-Rclone will stop transferring when it has reached the size specified.
-Defaults to off.
+Rclone implements a metadata framework which can read metadata from an
+object and write it to the object when (and only when) it is being
+uploaded.
-When the limit is reached all transfers will stop immediately.
-Use `--cutoff-mode` to modify this behaviour.
+This metadata is stored as a dictionary with string keys and string
+values.
-Rclone will exit with exit code 8 if the transfer limit is reached.
+There are some limits on the names of the keys (these may be clarified
+further in the future).
-### --cutoff-mode=hard|soft|cautious ###
+- must be lower case
+- may be `a-z` `0-9` containing `.` `-` or `_`
+- length is backend dependent
-This modifies the behavior of `--max-transfer` and `--max-duration`
-Defaults to `--cutoff-mode=hard`.
+Each backend can provide system metadata that it understands. Some
+backends can also store arbitrary user metadata.
-Specifying `--cutoff-mode=hard` will stop transferring immediately
-when Rclone reaches the limit.
+Where possible the key names are standardized, so, for example, it is
+possible to copy object metadata from s3 to azureblob for example and
+metadata will be translated appropriately.
-Specifying `--cutoff-mode=soft` will stop starting new transfers
-when Rclone reaches the limit.
+Some backends have limits on the size of the metadata and rclone will
+give errors on upload if they are exceeded.
-Specifying `--cutoff-mode=cautious` will try to prevent Rclone
-from reaching the limit. Only applicable for `--max-transfer`
+### Metadata preservation
-## -M, --metadata
+The goal of the implementation is to
-Setting this flag enables rclone to copy the metadata from the source
-to the destination. For local backends this is ownership, permissions,
-xattr etc. See the [#metadata](metadata section) for more info.
+1. Preserve metadata if at all possible
+2. Interpret metadata if at all possible
-### --metadata-set key=value
+The consequences of 1 is that you can copy an S3 object to a local
+disk then back to S3 losslessly. Likewise you can copy a local file
+with file attributes and xattrs from local disk to s3 and back again
+losslessly.
-Add metadata `key` = `value` when uploading. This can be repeated as
-many times as required. See the [#metadata](metadata section) for more
-info.
+The consequence of 2 is that you can copy an S3 object with metadata
+to Azureblob (say) and have the metadata appear on the Azureblob
+object also.
-### --modify-window=TIME ###
+### Standard system metadata
-When checking whether a file has been modified, this is the maximum
-allowed time difference that a file can have and still be considered
-equivalent.
+Here is a table of standard system metadata which, if appropriate, a
+backend may implement.
-The default is `1ns` unless this is overridden by a remote. For
-example OS X only stores modification times to the nearest second so
-if you are reading and writing to an OS X filing system this will be
-`1s` by default.
+| key | description | example |
+|---------------------|-------------|---------|
+| mode | File type and mode: octal, unix style | 0100664 |
+| uid | User ID of owner: decimal number | 500 |
+| gid | Group ID of owner: decimal number | 500 |
+| rdev | Device ID (if special file) => hexadecimal | 0 |
+| atime | Time of last access: RFC 3339 | 2006-01-02T15:04:05.999999999Z07:00 |
+| mtime | Time of last modification: RFC 3339 | 2006-01-02T15:04:05.999999999Z07:00 |
+| btime | Time of file creation (birth): RFC 3339 | 2006-01-02T15:04:05.999999999Z07:00 |
+| utime | Time of file upload: RFC 3339 | 2006-01-02T15:04:05.999999999Z07:00 |
+| cache-control | Cache-Control header | no-cache |
+| content-disposition | Content-Disposition header | inline |
+| content-encoding | Content-Encoding header | gzip |
+| content-language | Content-Language header | en-US |
+| content-type | Content-Type header | text/plain |
-This command line flag allows you to override that computed default.
+The metadata keys `mtime` and `content-type` will take precedence if
+supplied in the metadata over reading the `Content-Type` or
+modification time of the source object.
-### --multi-thread-write-buffer-size=SIZE ###
+Hashes are not included in system metadata as there is a well defined
+way of reading those already.
-When transferring with multiple threads, rclone will buffer SIZE bytes
-in memory before writing to disk for each thread.
+Options
+-------
-This can improve performance if the underlying filesystem does not deal
-well with a lot of small writes in different positions of the file, so
-if you see transfers being limited by disk write speed, you might want
-to experiment with different values. Specially for magnetic drives and
-remote file systems a higher value can be useful.
+Rclone has a number of options to control its behaviour.
-Nevertheless, the default of `128k` should be fine for almost all use
-cases, so before changing it ensure that network is not really your
-bottleneck.
+Options that take parameters can have the values passed in two ways,
+`--option=value` or `--option value`. However boolean (true/false)
+options behave slightly differently to the other options in that
+`--boolean` sets the option to `true` and the absence of the flag sets
+it to `false`. It is also possible to specify `--boolean=false` or
+`--boolean=true`. Note that `--boolean false` is not valid - this is
+parsed as `--boolean` and the `false` is parsed as an extra command
+line argument for rclone.
-As a final hint, size is not the only factor: block size (or similar
-concept) can have an impact. In one case, we observed that exact
-multiples of 16k performed much better than other values.
+### Time or duration options {#time-option}
-### --multi-thread-chunk-size=SizeSuffix ###
+TIME or DURATION options can be specified as a duration string or a
+time string.
-Normally the chunk size for multi thread transfers is set by the backend.
-However some backends such as `local` and `smb` (which implement `OpenWriterAt`
-but not `OpenChunkWriter`) don't have a natural chunk size.
+A duration string is a possibly signed sequence of decimal numbers,
+each with optional fraction and a unit suffix, such as "300ms",
+"-1.5h" or "2h45m". Default units are seconds or the following
+abbreviations are valid:
-In this case the value of this option is used (default 64Mi).
+ * `ms` - Milliseconds
+ * `s` - Seconds
+ * `m` - Minutes
+ * `h` - Hours
+ * `d` - Days
+ * `w` - Weeks
+ * `M` - Months
+ * `y` - Years
-### --multi-thread-cutoff=SIZE {#multi-thread-cutoff}
+These can also be specified as an absolute time in the following
+formats:
-When transferring files above SIZE to capable backends, rclone will
-use multiple threads to transfer the file (default 256M).
+- RFC3339 - e.g. `2006-01-02T15:04:05Z` or `2006-01-02T15:04:05+07:00`
+- ISO8601 Date and time, local timezone - `2006-01-02T15:04:05`
+- ISO8601 Date and time, local timezone - `2006-01-02 15:04:05`
+- ISO8601 Date - `2006-01-02` (YYYY-MM-DD)
-Capable backends are marked in the
-[overview](https://rclone.org/overview/#optional-features) as `MultithreadUpload`. (They
-need to implement either the `OpenWriterAt` or `OpenChunkedWriter`
-internal interfaces). These include include, `local`, `s3`,
-`azureblob`, `b2`, `oracleobjectstorage` and `smb` at the time of
-writing.
+### Size options {#size-option}
-On the local disk, rclone preallocates the file (using
-`fallocate(FALLOC_FL_KEEP_SIZE)` on unix or `NTSetInformationFile` on
-Windows both of which takes no time) then each thread writes directly
-into the file at the correct place. This means that rclone won't
-create fragmented or sparse files and there won't be any assembly time
-at the end of the transfer.
+Options which use SIZE use KiB (multiples of 1024 bytes) by default.
+However, a suffix of `B` for Byte, `K` for KiB, `M` for MiB,
+`G` for GiB, `T` for TiB and `P` for PiB may be used. These are
+the binary units, e.g. 1, 2\*\*10, 2\*\*20, 2\*\*30 respectively.
-The number of threads used to transfer is controlled by
-`--multi-thread-streams`.
+### --backup-dir=DIR ###
-Use `-vv` if you wish to see info about the threads.
+When using `sync`, `copy` or `move` any files which would have been
+overwritten or deleted are moved in their original hierarchy into this
+directory.
-This will work with the `sync`/`copy`/`move` commands and friends
-`copyto`/`moveto`. Multi thread transfers will be used with `rclone
-mount` and `rclone serve` if `--vfs-cache-mode` is set to `writes` or
-above.
+If `--suffix` is set, then the moved files will have the suffix added
+to them. If there is a file with the same path (after the suffix has
+been added) in DIR, then it will be overwritten.
-**NB** that this **only** works with supported backends as the
-destination but will work with any backend as the source.
+The remote in use must support server-side move or copy and you must
+use the same remote as the destination of the sync. The backup
+directory must not overlap the destination directory without it being
+excluded by a filter rule.
-**NB** that multi-thread copies are disabled for local to local copies
-as they are faster without unless `--multi-thread-streams` is set
-explicitly.
+For example
-**NB** on Windows using multi-thread transfers to the local disk will
-cause the resulting files to be [sparse](https://en.wikipedia.org/wiki/Sparse_file).
-Use `--local-no-sparse` to disable sparse files (which may cause long
-delays at the start of transfers) or disable multi-thread transfers
-with `--multi-thread-streams 0`
+ rclone sync --interactive /path/to/local remote:current --backup-dir remote:old
-### --multi-thread-streams=N ###
+will sync `/path/to/local` to `remote:current`, but for any files
+which would have been updated or deleted will be stored in
+`remote:old`.
-When using multi thread transfers (see above `--multi-thread-cutoff`)
-this sets the number of streams to use. Set to `0` to disable multi
-thread transfers (Default 4).
+If running rclone from a script you might want to use today's date as
+the directory name passed to `--backup-dir` to store the old files, or
+you might want to pass `--suffix` with today's date.
-If the backend has a `--backend-upload-concurrency` setting (eg
-`--s3-upload-concurrency`) then this setting will be used as the
-number of transfers instead if it is larger than the value of
-`--multi-thread-streams` or `--multi-thread-streams` isn't set.
+See `--compare-dest` and `--copy-dest`.
-### --no-check-dest ###
+### --bind string ###
-The `--no-check-dest` can be used with `move` or `copy` and it causes
-rclone not to check the destination at all when copying files.
+Local address to bind to for outgoing connections. This can be an
+IPv4 address (1.2.3.4), an IPv6 address (1234::789A) or host name. If
+the host name doesn't resolve or resolves to more than one IP address
+it will give an error.
-This means that:
+You can use `--bind 0.0.0.0` to force rclone to use IPv4 addresses and
+`--bind ::0` to force rclone to use IPv6 addresses.
-- the destination is not listed minimising the API calls
-- files are always transferred
-- this can cause duplicates on remotes which allow it (e.g. Google Drive)
-- `--retries 1` is recommended otherwise you'll transfer everything again on a retry
+### --bwlimit=BANDWIDTH_SPEC ###
-This flag is useful to minimise the transactions if you know that none
-of the files are on the destination.
+This option controls the bandwidth limit. For example
-This is a specialized flag which should be ignored by most users!
+ --bwlimit 10M
-### --no-gzip-encoding ###
+would mean limit the upload and download bandwidth to 10 MiB/s.
+**NB** this is **bytes** per second not **bits** per second. To use a
+single limit, specify the desired bandwidth in KiB/s, or use a
+suffix B|K|M|G|T|P. The default is `0` which means to not limit bandwidth.
-Don't set `Accept-Encoding: gzip`. This means that rclone won't ask
-the server for compressed files automatically. Useful if you've set
-the server to return files with `Content-Encoding: gzip` but you
-uploaded compressed files.
+The upload and download bandwidth can be specified separately, as
+`--bwlimit UP:DOWN`, so
-There is no need to set this in normal operation, and doing so will
-decrease the network transfer efficiency of rclone.
+ --bwlimit 10M:100k
-### --no-traverse ###
+would mean limit the upload bandwidth to 10 MiB/s and the download
+bandwidth to 100 KiB/s. Either limit can be "off" meaning no limit, so
+to just limit the upload bandwidth you would use
-The `--no-traverse` flag controls whether the destination file system
-is traversed when using the `copy` or `move` commands.
-`--no-traverse` is not compatible with `sync` and will be ignored if
-you supply it with `sync`.
+ --bwlimit 10M:off
-If you are only copying a small number of files (or are filtering most
-of the files) and/or have a large number of files on the destination
-then `--no-traverse` will stop rclone listing the destination and save
-time.
+this would limit the upload bandwidth to 10 MiB/s but the download
+bandwidth would be unlimited.
-However, if you are copying a large number of files, especially if you
-are doing a copy where lots of the files under consideration haven't
-changed and won't need copying then you shouldn't use `--no-traverse`.
+When specified as above the bandwidth limits last for the duration of
+run of the rclone binary.
-See [rclone copy](https://rclone.org/commands/rclone_copy/) for an example of how to use it.
+It is also possible to specify a "timetable" of limits, which will
+cause certain limits to be applied at certain times. To specify a
+timetable, format your entries as `WEEKDAY-HH:MM,BANDWIDTH
+WEEKDAY-HH:MM,BANDWIDTH...` where: `WEEKDAY` is optional element.
-### --no-unicode-normalization ###
+- `BANDWIDTH` can be a single number, e.g.`100k` or a pair of numbers
+for upload:download, e.g.`10M:1M`.
+- `WEEKDAY` can be written as the whole word or only using the first 3
+ characters. It is optional.
+- `HH:MM` is an hour from 00:00 to 23:59.
-Don't normalize unicode characters in filenames during the sync routine.
+An example of a typical timetable to avoid link saturation during daytime
+working hours could be:
-Sometimes, an operating system will store filenames containing unicode
-parts in their decomposed form (particularly macOS). Some cloud storage
-systems will then recompose the unicode, resulting in duplicate files if
-the data is ever copied back to a local filesystem.
+`--bwlimit "08:00,512k 12:00,10M 13:00,512k 18:00,30M 23:00,off"`
-Using this flag will disable that functionality, treating each unicode
-character as unique. For example, by default eÌ and é will be normalized
-into the same character. With `--no-unicode-normalization` they will be
-treated as unique characters.
+In this example, the transfer bandwidth will be set to 512 KiB/s
+at 8am every day. At noon, it will rise to 10 MiB/s, and drop back
+to 512 KiB/sec at 1pm. At 6pm, the bandwidth limit will be set to
+30 MiB/s, and at 11pm it will be completely disabled (full speed).
+Anything between 11pm and 8am will remain unlimited.
-### --no-update-modtime ###
+An example of timetable with `WEEKDAY` could be:
-When using this flag, rclone won't update modification times of remote
-files if they are incorrect as it would normally.
+`--bwlimit "Mon-00:00,512 Fri-23:59,10M Sat-10:00,1M Sun-20:00,off"`
-This can be used if the remote is being synced with another tool also
-(e.g. the Google Drive client).
+It means that, the transfer bandwidth will be set to 512 KiB/s on
+Monday. It will rise to 10 MiB/s before the end of Friday. At 10:00
+on Saturday it will be set to 1 MiB/s. From 20:00 on Sunday it will
+be unlimited.
-### --order-by string ###
+Timeslots without `WEEKDAY` are extended to the whole week. So this
+example:
-The `--order-by` flag controls the order in which files in the backlog
-are processed in `rclone sync`, `rclone copy` and `rclone move`.
+`--bwlimit "Mon-00:00,512 12:00,1M Sun-20:00,off"`
-The order by string is constructed like this. The first part
-describes what aspect is being measured:
+Is equivalent to this:
-- `size` - order by the size of the files
-- `name` - order by the full path of the files
-- `modtime` - order by the modification date of the files
+`--bwlimit "Mon-00:00,512Mon-12:00,1M Tue-12:00,1M Wed-12:00,1M Thu-12:00,1M Fri-12:00,1M Sat-12:00,1M Sun-12:00,1M Sun-20:00,off"`
-This can have a modifier appended with a comma:
+Bandwidth limit apply to the data transfer for all backends. For most
+backends the directory listing bandwidth is also included (exceptions
+being the non HTTP backends, `ftp`, `sftp` and `storj`).
-- `ascending` or `asc` - order so that the smallest (or oldest) is processed first
-- `descending` or `desc` - order so that the largest (or newest) is processed first
-- `mixed` - order so that the smallest is processed first for some threads and the largest for others
+Note that the units are **Byte/s**, not **bit/s**. Typically
+connections are measured in bit/s - to convert divide by 8. For
+example, let's say you have a 10 Mbit/s connection and you wish rclone
+to use half of it - 5 Mbit/s. This is 5/8 = 0.625 MiB/s so you would
+use a `--bwlimit 0.625M` parameter for rclone.
-If the modifier is `mixed` then it can have an optional percentage
-(which defaults to `50`), e.g. `size,mixed,25` which means that 25% of
-the threads should be taking the smallest items and 75% the
-largest. The threads which take the smallest first will always take
-the smallest first and likewise the largest first threads. The `mixed`
-mode can be useful to minimise the transfer time when you are
-transferring a mixture of large and small files - the large files are
-guaranteed upload threads and bandwidth and the small files will be
-processed continuously.
+On Unix systems (Linux, macOS, …) the bandwidth limiter can be toggled by
+sending a `SIGUSR2` signal to rclone. This allows to remove the limitations
+of a long running rclone transfer and to restore it back to the value specified
+with `--bwlimit` quickly when needed. Assuming there is only one rclone instance
+running, you can toggle the limiter like this:
-If no modifier is supplied then the order is `ascending`.
+ kill -SIGUSR2 $(pidof rclone)
-For example
+If you configure rclone with a [remote control](/rc) then you can use
+change the bwlimit dynamically:
-- `--order-by size,desc` - send the largest files first
-- `--order-by modtime,ascending` - send the oldest files first
-- `--order-by name` - send the files with alphabetically by path first
+ rclone rc core/bwlimit rate=1M
-If the `--order-by` flag is not supplied or it is supplied with an
-empty string then the default ordering will be used which is as
-scanned. With `--checkers 1` this is mostly alphabetical, however
-with the default `--checkers 8` it is somewhat random.
+### --bwlimit-file=BANDWIDTH_SPEC ###
-#### Limitations
+This option controls per file bandwidth limit. For the options see the
+`--bwlimit` flag.
-The `--order-by` flag does not do a separate pass over the data. This
-means that it may transfer some files out of the order specified if
+For example use this to allow no transfers to be faster than 1 MiB/s
-- there are no files in the backlog or the source has not been fully scanned yet
-- there are more than [--max-backlog](#max-backlog-n) files in the backlog
+ --bwlimit-file 1M
-Rclone will do its best to transfer the best file it has so in
-practice this should not cause a problem. Think of `--order-by` as
-being more of a best efforts flag rather than a perfect ordering.
+This can be used in conjunction with `--bwlimit`.
-If you want perfect ordering then you will need to specify
-[--check-first](#check-first) which will find all the files which need
-transferring first before transferring any.
+Note that if a schedule is provided the file will use the schedule in
+effect at the start of the transfer.
-### --password-command SpaceSepList ###
+### --buffer-size=SIZE ###
-This flag supplies a program which should supply the config password
-when run. This is an alternative to rclone prompting for the password
-or setting the `RCLONE_CONFIG_PASS` variable.
+Use this sized buffer to speed up file transfers. Each `--transfer`
+will use this much memory for buffering.
-The argument to this should be a command with a space separated list
-of arguments. If one of the arguments has a space in then enclose it
-in `"`, if you want a literal `"` in an argument then enclose the
-argument in `"` and double the `"`. See [CSV encoding](https://godoc.org/encoding/csv)
-for more info.
+When using `mount` or `cmount` each open file descriptor will use this much
+memory for buffering.
+See the [mount](https://rclone.org/commands/rclone_mount/#file-buffering) documentation for more details.
-Eg
+Set to `0` to disable the buffering for the minimum memory usage.
- --password-command echo hello
- --password-command echo "hello with space"
- --password-command echo "hello with ""quotes"" and space"
+Note that the memory allocation of the buffers is influenced by the
+[--use-mmap](#use-mmap) flag.
-See the [Configuration Encryption](#configuration-encryption) for more info.
+### --cache-dir=DIR ###
-See a [Windows PowerShell example on the Wiki](https://github.com/rclone/rclone/wiki/Windows-Powershell-use-rclone-password-command-for-Config-file-password).
+Specify the directory rclone will use for caching, to override
+the default.
-### -P, --progress ###
+Default value is depending on operating system:
+- Windows `%LocalAppData%\rclone`, if `LocalAppData` is defined.
+- macOS `$HOME/Library/Caches/rclone` if `HOME` is defined.
+- Unix `$XDG_CACHE_HOME/rclone` if `XDG_CACHE_HOME` is defined, else `$HOME/.cache/rclone` if `HOME` is defined.
+- Fallback (on all OS) to `$TMPDIR/rclone`, where `TMPDIR` is the value from [--temp-dir](#temp-dir-dir).
-This flag makes rclone update the stats in a static block in the
-terminal providing a realtime overview of the transfer.
+You can use the [config paths](https://rclone.org/commands/rclone_config_paths/)
+command to see the current value.
-Any log messages will scroll above the static block. Log messages
-will push the static block down to the bottom of the terminal where it
-will stay.
+Cache directory is heavily used by the [VFS File Caching](https://rclone.org/commands/rclone_mount/#vfs-file-caching)
+mount feature, but also by [serve](https://rclone.org/commands/rclone_serve/), [GUI](/gui) and other parts of rclone.
-Normally this is updated every 500mS but this period can be overridden
-with the `--stats` flag.
+### --check-first ###
-This can be used with the `--stats-one-line` flag for a simpler
-display.
+If this flag is set then in a `sync`, `copy` or `move`, rclone will do
+all the checks to see whether files need to be transferred before
+doing any of the transfers. Normally rclone would start running
+transfers as soon as possible.
-Note: On Windows until [this bug](https://github.com/Azure/go-ansiterm/issues/26)
-is fixed all non-ASCII characters will be replaced with `.` when
-`--progress` is in use.
+This flag can be useful on IO limited systems where transfers
+interfere with checking.
-### --progress-terminal-title ###
+It can also be useful to ensure perfect ordering when using
+`--order-by`.
-This flag, when used with `-P/--progress`, will print the string `ETA: %s`
-to the terminal title.
+If both `--check-first` and `--order-by` are set when doing `rclone move`
+then rclone will use the transfer thread to delete source files which
+don't need transferring. This will enable perfect ordering of the
+transfers and deletes but will cause the transfer stats to have more
+items in than expected.
-### -q, --quiet ###
+Using this flag can use more memory as it effectively sets
+`--max-backlog` to infinite. This means that all the info on the
+objects to transfer is held in memory before the transfers start.
-This flag will limit rclone's output to error messages only.
+### --checkers=N ###
-### --refresh-times ###
+Originally controlling just the number of file checkers to run in parallel,
+e.g. by `rclone copy`. Now a fairly universal parallelism control
+used by `rclone` in several places.
-The `--refresh-times` flag can be used to update modification times of
-existing files when they are out of sync on backends which don't
-support hashes.
+Note: checkers do the equality checking of files during a sync.
+For some storage systems (e.g. S3, Swift, Dropbox) this can take
+a significant amount of time so they are run in parallel.
-This is useful if you uploaded files with the incorrect timestamps and
-you now wish to correct them.
+The default is to run 8 checkers in parallel. However, in case
+of slow-reacting backends you may need to lower (rather than increase)
+this default by setting `--checkers` to 4 or less threads. This is
+especially advised if you are experiencing backend server crashes
+during file checking phase (e.g. on subsequent or top-up backups
+where little or no file copying is done and checking takes up
+most of the time). Increase this setting only with utmost care,
+while monitoring your server health and file checking throughput.
-This flag is **only** useful for destinations which don't support
-hashes (e.g. `crypt`).
+### -c, --checksum ###
-This can be used any of the sync commands `sync`, `copy` or `move`.
+Normally rclone will look at modification time and size of files to
+see if they are equal. If you set this flag then rclone will check
+the file hash and size to determine if files are equal.
-To use this flag you will need to be doing a modification time sync
-(so not using `--size-only` or `--checksum`). The flag will have no
-effect when using `--size-only` or `--checksum`.
+This is useful when the remote doesn't support setting modified time
+and a more accurate sync is desired than just checking the file size.
-If this flag is used when rclone comes to upload a file it will check
-to see if there is an existing file on the destination. If this file
-matches the source with size (and checksum if available) but has a
-differing timestamp then instead of re-uploading it, rclone will
-update the timestamp on the destination file. If the checksum does not
-match rclone will upload the new file. If the checksum is absent (e.g.
-on a `crypt` backend) then rclone will update the timestamp.
+This is very useful when transferring between remotes which store the
+same hash type on the object, e.g. Drive and Swift. For details of which
+remotes support which hash type see the table in the [overview
+section](https://rclone.org/overview/).
-Note that some remotes can't set the modification time without
-re-uploading the file so this flag is less useful on them.
+Eg `rclone --checksum sync s3:/bucket swift:/bucket` would run much
+quicker than without the `--checksum` flag.
-Normally if you are doing a modification time sync rclone will update
-modification times without `--refresh-times` provided that the remote
-supports checksums **and** the checksums match on the file. However if the
-checksums are absent then rclone will upload the file rather than
-setting the timestamp as this is the safe behaviour.
+When using this flag, rclone won't update mtimes of remote files if
+they are incorrect as it would normally.
-### --retries int ###
+### --color WHEN ###
-Retry the entire sync if it fails this many times it fails (default 3).
+Specify when colors (and other ANSI codes) should be added to the output.
-Some remotes can be unreliable and a few retries help pick up the
-files which didn't get transferred because of errors.
+`AUTO` (default) only allows ANSI codes when the output is a terminal
-Disable retries with `--retries 1`.
+`NEVER` never allow ANSI codes
-### --retries-sleep=TIME ###
+`ALWAYS` always add ANSI codes, regardless of the output format (terminal or file)
-This sets the interval between each retry specified by `--retries`
+### --compare-dest=DIR ###
-The default is `0`. Use `0` to disable.
+When using `sync`, `copy` or `move` DIR is checked in addition to the
+destination for files. If a file identical to the source is found that
+file is NOT copied from source. This is useful to copy just files that
+have changed since the last backup.
-### --server-side-across-configs ###
+You must use the same remote as the destination of the sync. The
+compare directory must not overlap the destination directory.
-Allow server-side operations (e.g. copy or move) to work across
-different configurations.
+See `--copy-dest` and `--backup-dir`.
-This can be useful if you wish to do a server-side copy or move
-between two remotes which use the same backend but are configured
-differently.
+### --config=CONFIG_FILE ###
-Note that this isn't enabled by default because it isn't easy for
-rclone to tell if it will work between any two configurations.
+Specify the location of the rclone configuration file, to override
+the default. E.g. `rclone config --config="rclone.conf"`.
-### --size-only ###
+The exact default is a bit complex to describe, due to changes
+introduced through different versions of rclone while preserving
+backwards compatibility, but in most cases it is as simple as:
-Normally rclone will look at modification time and size of files to
-see if they are equal. If you set this flag then rclone will check
-only the size.
+ - `%APPDATA%/rclone/rclone.conf` on Windows
+ - `~/.config/rclone/rclone.conf` on other
-This can be useful transferring files from Dropbox which have been
-modified by the desktop sync client which doesn't set checksums of
-modification times in the same way as rclone.
+The complete logic is as follows: Rclone will look for an existing
+configuration file in any of the following locations, in priority order:
-### --stats=TIME ###
+ 1. `rclone.conf` (in program directory, where rclone executable is)
+ 2. `%APPDATA%/rclone/rclone.conf` (only on Windows)
+ 3. `$XDG_CONFIG_HOME/rclone/rclone.conf` (on all systems, including Windows)
+ 4. `~/.config/rclone/rclone.conf` (see below for explanation of ~ symbol)
+ 5. `~/.rclone.conf`
-Commands which transfer data (`sync`, `copy`, `copyto`, `move`,
-`moveto`) will print data transfer stats at regular intervals to show
-their progress.
+If no existing configuration file is found, then a new one will be created
+in the following location:
-This sets the interval.
+- On Windows: Location 2 listed above, except in the unlikely event
+ that `APPDATA` is not defined, then location 4 is used instead.
+- On Unix: Location 3 if `XDG_CONFIG_HOME` is defined, else location 4.
+- Fallback to location 5 (on all OS), when the rclone directory cannot be
+ created, but if also a home directory was not found then path
+ `.rclone.conf` relative to current working directory will be used as
+ a final resort.
-The default is `1m`. Use `0` to disable.
+The `~` symbol in paths above represent the home directory of the current user
+on any OS, and the value is defined as following:
-If you set the stats interval then all commands can show stats. This
-can be useful when running other commands, `check` or `mount` for
-example.
+ - On Windows: `%HOME%` if defined, else `%USERPROFILE%`, or else `%HOMEDRIVE%\%HOMEPATH%`.
+ - On Unix: `$HOME` if defined, else by looking up current user in OS-specific user database
+ (e.g. passwd file), or else use the result from shell command `cd && pwd`.
-Stats are logged at `INFO` level by default which means they won't
-show at default log level `NOTICE`. Use `--stats-log-level NOTICE` or
-`-v` to make them show. See the [Logging section](#logging) for more
-info on log levels.
+If you run `rclone config file` you will see where the default
+location is for you.
-Note that on macOS you can send a SIGINFO (which is normally ctrl-T in
-the terminal) to make the stats print immediately.
+The fact that an existing file `rclone.conf` in the same directory
+as the rclone executable is always preferred, means that it is easy
+to run in "portable" mode by downloading rclone executable to a
+writable directory and then create an empty file `rclone.conf` in the
+same directory.
-### --stats-file-name-length integer ###
-By default, the `--stats` output will truncate file names and paths longer
-than 40 characters. This is equivalent to providing
-`--stats-file-name-length 40`. Use `--stats-file-name-length 0` to disable
-any truncation of file names printed by stats.
+If the location is set to empty string `""` or path to a file
+with name `notfound`, or the os null device represented by value `NUL` on
+Windows and `/dev/null` on Unix systems, then rclone will keep the
+config file in memory only.
-### --stats-log-level string ###
+The file format is basic [INI](https://en.wikipedia.org/wiki/INI_file#Format):
+Sections of text, led by a `[section]` header and followed by
+`key=value` entries on separate lines. In rclone each remote is
+represented by its own section, where the section name defines the
+name of the remote. Options are specified as the `key=value` entries,
+where the key is the option name without the `--backend-` prefix,
+in lowercase and with `_` instead of `-`. E.g. option `--mega-hard-delete`
+corresponds to key `hard_delete`. Only backend options can be specified.
+A special, and required, key `type` identifies the [storage system](https://rclone.org/overview/),
+where the value is the internal lowercase name as returned by command
+`rclone help backends`. Comments are indicated by `;` or `#` at the
+beginning of a line.
-Log level to show `--stats` output at. This can be `DEBUG`, `INFO`,
-`NOTICE`, or `ERROR`. The default is `INFO`. This means at the
-default level of logging which is `NOTICE` the stats won't show - if
-you want them to then use `--stats-log-level NOTICE`. See the [Logging
-section](#logging) for more info on log levels.
+Example:
-### --stats-one-line ###
+ [megaremote]
+ type = mega
+ user = you@example.com
+ pass = PDPcQVVjVtzFY-GTdDFozqBhTdsPg3qH
-When this is specified, rclone condenses the stats into a single line
-showing the most important stats only.
+Note that passwords are in [obscured](https://rclone.org/commands/rclone_obscure/)
+form. Also, many storage systems uses token-based authentication instead
+of passwords, and this requires additional steps. It is easier, and safer,
+to use the interactive command `rclone config` instead of manually
+editing the configuration file.
-### --stats-one-line-date ###
+The configuration file will typically contain login information, and
+should therefore have restricted permissions so that only the current user
+can read it. Rclone tries to ensure this when it writes the file.
+You may also choose to [encrypt](#configuration-encryption) the file.
-When this is specified, rclone enables the single-line stats and prepends
-the display with a date string. The default is `2006/01/02 15:04:05 - `
+When token-based authentication are used, the configuration file
+must be writable, because rclone needs to update the tokens inside it.
-### --stats-one-line-date-format ###
+To reduce risk of corrupting an existing configuration file, rclone
+will not write directly to it when saving changes. Instead it will
+first write to a new, temporary, file. If a configuration file already
+existed, it will (on Unix systems) try to mirror its permissions to
+the new file. Then it will rename the existing file to a temporary
+name as backup. Next, rclone will rename the new file to the correct name,
+before finally cleaning up by deleting the backup file.
-When this is specified, rclone enables the single-line stats and prepends
-the display with a user-supplied date string. The date string MUST be
-enclosed in quotes. Follow [golang specs](https://golang.org/pkg/time/#Time.Format) for
-date formatting syntax.
+If the configuration file path used by rclone is a symbolic link, then
+this will be evaluated and rclone will write to the resolved path, instead
+of overwriting the symbolic link. Temporary files used in the process
+(described above) will be written to the same parent directory as that
+of the resolved configuration file, but if this directory is also a
+symbolic link it will not be resolved and the temporary files will be
+written to the location of the directory symbolic link.
-### --stats-unit=bits|bytes ###
+### --contimeout=TIME ###
-By default, data transfer rates will be printed in bytes per second.
+Set the connection timeout. This should be in go time format which
+looks like `5s` for 5 seconds, `10m` for 10 minutes, or `3h30m`.
-This option allows the data rate to be printed in bits per second.
+The connection timeout is the amount of time rclone will wait for a
+connection to go through to a remote object storage system. It is
+`1m` by default.
-Data transfer volume will still be reported in bytes.
+### --copy-dest=DIR ###
-The rate is reported as a binary unit, not SI unit. So 1 Mbit/s
-equals 1,048,576 bit/s and not 1,000,000 bit/s.
+When using `sync`, `copy` or `move` DIR is checked in addition to the
+destination for files. If a file identical to the source is found that
+file is server-side copied from DIR to the destination. This is useful
+for incremental backup.
-The default is `bytes`.
+The remote in use must support server-side copy and you must
+use the same remote as the destination of the sync. The compare
+directory must not overlap the destination directory.
-### --suffix=SUFFIX ###
+See `--compare-dest` and `--backup-dir`.
-When using `sync`, `copy` or `move` any files which would have been
-overwritten or deleted will have the suffix added to them. If there
-is a file with the same path (after the suffix has been added), then
-it will be overwritten.
+### --dedupe-mode MODE ###
-The remote in use must support server-side move or copy and you must
-use the same remote as the destination of the sync.
+Mode to run dedupe command in. One of `interactive`, `skip`, `first`,
+`newest`, `oldest`, `rename`. The default is `interactive`.
+See the dedupe command for more information as to what these options mean.
-This is for use with files to add the suffix in the current directory
-or with `--backup-dir`. See `--backup-dir` for more info.
+### --default-time TIME ###
-For example
+If a file or directory does have a modification time rclone can read
+then rclone will display this fixed time instead.
- rclone copy --interactive /path/to/local/file remote:current --suffix .bak
+The default is `2000-01-01 00:00:00 UTC`. This can be configured in
+any of the ways shown in [the time or duration options](#time-option).
-will copy `/path/to/local` to `remote:current`, but for any files
-which would have been updated or deleted have .bak added.
+For example `--default-time 2020-06-01` to set the default time to the
+1st of June 2020 or `--default-time 0s` to set the default time to the
+time rclone started up.
-If using `rclone sync` with `--suffix` and without `--backup-dir` then
-it is recommended to put a filter rule in excluding the suffix
-otherwise the `sync` will delete the backup files.
+### --disable FEATURE,FEATURE,... ###
- rclone sync --interactive /path/to/local/file remote:current --suffix .bak --exclude "*.bak"
+This disables a comma separated list of optional features. For example
+to disable server-side move and server-side copy use:
-### --suffix-keep-extension ###
+ --disable move,copy
-When using `--suffix`, setting this causes rclone put the SUFFIX
-before the extension of the files that it backs up rather than after.
+The features can be put in any case.
-So let's say we had `--suffix -2019-01-01`, without the flag `file.txt`
-would be backed up to `file.txt-2019-01-01` and with the flag it would
-be backed up to `file-2019-01-01.txt`. This can be helpful to make
-sure the suffixed files can still be opened.
+To see a list of which features can be disabled use:
-If a file has two (or more) extensions and the second (or subsequent)
-extension is recognised as a valid mime type, then the suffix will go
-before that extension. So `file.tar.gz` would be backed up to
-`file-2019-01-01.tar.gz` whereas `file.badextension.gz` would be
-backed up to `file.badextension-2019-01-01.gz`.
+ --disable help
-### --syslog ###
+The features a remote has can be seen in JSON format with:
-On capable OSes (not Windows or Plan9) send all log output to syslog.
+ rclone backend features remote:
-This can be useful for running rclone in a script or `rclone mount`.
+See the overview [features](https://rclone.org/overview/#features) and
+[optional features](https://rclone.org/overview/#optional-features) to get an idea of
+which feature does what.
-### --syslog-facility string ###
+Note that some features can be set to `true` if they are `true`/`false`
+feature flag features by prefixing them with `!`. For example the
+`CaseInsensitive` feature can be forced to `false` with `--disable CaseInsensitive`
+and forced to `true` with `--disable '!CaseInsensitive'`. In general
+it isn't a good idea doing this but it may be useful in extremis.
-If using `--syslog` this sets the syslog facility (e.g. `KERN`, `USER`).
-See `man syslog` for a list of possible facilities. The default
-facility is `DAEMON`.
+(Note that `!` is a shell command which you will
+need to escape with single quotes or a backslash on unix like
+platforms.)
-### --temp-dir=DIR ###
+This flag can be useful for debugging and in exceptional circumstances
+(e.g. Google Drive limiting the total volume of Server Side Copies to
+100 GiB/day).
-Specify the directory rclone will use for temporary files, to override
-the default. Make sure the directory exists and have accessible permissions.
+### --disable-http2
-By default the operating system's temp directory will be used:
-- On Unix systems, `$TMPDIR` if non-empty, else `/tmp`.
-- On Windows, the first non-empty value from `%TMP%`, `%TEMP%`, `%USERPROFILE%`, or the Windows directory.
+This stops rclone from trying to use HTTP/2 if available. This can
+sometimes speed up transfers due to a
+[problem in the Go standard library](https://github.com/golang/go/issues/37373).
-When overriding the default with this option, the specified path will be
-set as value of environment variable `TMPDIR` on Unix systems
-and `TMP` and `TEMP` on Windows.
+### --dscp VALUE ###
-You can use the [config paths](https://rclone.org/commands/rclone_config_paths/)
-command to see the current value.
+Specify a DSCP value or name to use in connections. This could help QoS
+system to identify traffic class. BE, EF, DF, LE, CSx and AFxx are allowed.
-### --tpslimit float ###
+See the description of [differentiated services](https://en.wikipedia.org/wiki/Differentiated_services) to get an idea of
+this field. Setting this to 1 (LE) to identify the flow to SCAVENGER class
+can avoid occupying too much bandwidth in a network with DiffServ support ([RFC 8622](https://tools.ietf.org/html/rfc8622)).
-Limit transactions per second to this number. Default is 0 which is
-used to mean unlimited transactions per second.
+For example, if you configured QoS on router to handle LE properly. Running:
+```
+rclone copy --dscp LE from:/from to:/to
+```
+would make the priority lower than usual internet flows.
-A transaction is roughly defined as an API call; its exact meaning
-will depend on the backend. For HTTP based backends it is an HTTP
-PUT/GET/POST/etc and its response. For FTP/SFTP it is a round trip
-transaction over TCP.
+This option has no effect on Windows (see [golang/go#42728](https://github.com/golang/go/issues/42728)).
-For example, to limit rclone to 10 transactions per second use
-`--tpslimit 10`, or to 1 transaction every 2 seconds use `--tpslimit
-0.5`.
+### -n, --dry-run ###
-Use this when the number of transactions per second from rclone is
-causing a problem with the cloud storage provider (e.g. getting you
-banned or rate limited).
+Do a trial run with no permanent changes. Use this to see what rclone
+would do without actually doing it. Useful when setting up the `sync`
+command which deletes files in the destination.
-This can be very useful for `rclone mount` to control the behaviour of
-applications using it.
+### --expect-continue-timeout=TIME ###
-This limit applies to all HTTP based backends and to the FTP and SFTP
-backends. It does not apply to the local backend or the Storj backend.
+This specifies the amount of time to wait for a server's first
+response headers after fully writing the request headers if the
+request has an "Expect: 100-continue" header. Not all backends support
+using this.
-See also `--tpslimit-burst`.
+Zero means no timeout and causes the body to be sent immediately,
+without waiting for the server to approve. This time does not include
+the time to send the request header.
-### --tpslimit-burst int ###
+The default is `1s`. Set to `0` to disable.
-Max burst of transactions for `--tpslimit` (default `1`).
+### --error-on-no-transfer ###
-Normally `--tpslimit` will do exactly the number of transaction per
-second specified. However if you supply `--tps-burst` then rclone can
-save up some transactions from when it was idle giving a burst of up
-to the parameter supplied.
+By default, rclone will exit with return code 0 if there were no errors.
-For example if you provide `--tpslimit-burst 10` then if rclone has
-been idle for more than 10*`--tpslimit` then it can do 10 transactions
-very quickly before they are limited again.
+This option allows rclone to return exit code 9 if no files were transferred
+between the source and destination. This allows using rclone in scripts, and
+triggering follow-on actions if data was copied, or skipping if not.
-This may be used to increase performance of `--tpslimit` without
-changing the long term average number of transactions per second.
+NB: Enabling this option turns a usually non-fatal error into a potentially
+fatal one - please check and adjust your scripts accordingly!
-### --track-renames ###
+### --fs-cache-expire-duration=TIME
-By default, rclone doesn't keep track of renamed files, so if you
-rename a file locally then sync it to a remote, rclone will delete the
-old file on the remote and upload a new copy.
+When using rclone via the API rclone caches created remotes for 5
+minutes by default in the "fs cache". This means that if you do
+repeated actions on the same remote then rclone won't have to build it
+again from scratch, which makes it more efficient.
-An rclone sync with `--track-renames` runs like a normal sync, but keeps
-track of objects which exist in the destination but not in the source
-(which would normally be deleted), and which objects exist in the
-source but not the destination (which would normally be transferred).
-These objects are then candidates for renaming.
+This flag sets the time that the remotes are cached for. If you set it
+to `0` (or negative) then rclone won't cache the remotes at all.
-After the sync, rclone matches up the source only and destination only
-objects using the `--track-renames-strategy` specified and either
-renames the destination object or transfers the source and deletes the
-destination object. `--track-renames` is stateless like all of
-rclone's syncs.
+Note that if you use some flags, eg `--backup-dir` and if this is set
+to `0` rclone may build two remotes (one for the source or destination
+and one for the `--backup-dir` where it may have only built one
+before.
-To use this flag the destination must support server-side copy or
-server-side move, and to use a hash based `--track-renames-strategy`
-(the default) the source and the destination must have a compatible
-hash.
+### --fs-cache-expire-interval=TIME
-If the destination does not support server-side copy or move, rclone
-will fall back to the default behaviour and log an error level message
-to the console.
+This controls how often rclone checks for cached remotes to expire.
+See the `--fs-cache-expire-duration` documentation above for more
+info. The default is 60s, set to 0 to disable expiry.
-Encrypted destinations are not currently supported by `--track-renames`
-if `--track-renames-strategy` includes `hash`.
+### --header ###
-Note that `--track-renames` is incompatible with `--no-traverse` and
-that it uses extra memory to keep track of all the rename candidates.
+Add an HTTP header for all transactions. The flag can be repeated to
+add multiple headers.
-Note also that `--track-renames` is incompatible with
-`--delete-before` and will select `--delete-after` instead of
-`--delete-during`.
+If you want to add headers only for uploads use `--header-upload` and
+if you want to add headers only for downloads use `--header-download`.
-### --track-renames-strategy (hash,modtime,leaf,size) ###
+This flag is supported for all HTTP based backends even those not
+supported by `--header-upload` and `--header-download` so may be used
+as a workaround for those with care.
-This option changes the file matching criteria for `--track-renames`.
+```
+rclone ls remote:test --header "X-Rclone: Foo" --header "X-LetMeIn: Yes"
+```
-The matching is controlled by a comma separated selection of these tokens:
+### --header-download ###
-- `modtime` - the modification time of the file - not supported on all backends
-- `hash` - the hash of the file contents - not supported on all backends
-- `leaf` - the name of the file not including its directory name
-- `size` - the size of the file (this is always enabled)
+Add an HTTP header for all download transactions. The flag can be repeated to
+add multiple headers.
-The default option is `hash`.
+```
+rclone sync --interactive s3:test/src ~/dst --header-download "X-Amz-Meta-Test: Foo" --header-download "X-Amz-Meta-Test2: Bar"
+```
-Using `--track-renames-strategy modtime,leaf` would match files
-based on modification time, the leaf of the file name and the size
-only.
+See the GitHub issue [here](https://github.com/rclone/rclone/issues/59) for
+currently supported backends.
-Using `--track-renames-strategy modtime` or `leaf` can enable
-`--track-renames` support for encrypted destinations.
+### --header-upload ###
-Note that the `hash` strategy is not supported with encrypted destinations.
+Add an HTTP header for all upload transactions. The flag can be repeated to add
+multiple headers.
-### --delete-(before,during,after) ###
+```
+rclone sync --interactive ~/src s3:test/dst --header-upload "Content-Disposition: attachment; filename='cool.html'" --header-upload "X-Amz-Meta-Test: FooBar"
+```
-This option allows you to specify when files on your destination are
-deleted when you sync folders.
+See the GitHub issue [here](https://github.com/rclone/rclone/issues/59) for
+currently supported backends.
-Specifying the value `--delete-before` will delete all files present
-on the destination, but not on the source *before* starting the
-transfer of any new or updated files. This uses two passes through the
-file systems, one for the deletions and one for the copies.
+### --human-readable ###
-Specifying `--delete-during` will delete files while checking and
-uploading files. This is the fastest option and uses the least memory.
+Rclone commands output values for sizes (e.g. number of bytes) and
+counts (e.g. number of files) either as *raw* numbers, or
+in *human-readable* format.
-Specifying `--delete-after` (the default value) will delay deletion of
-files until all new/updated files have been successfully transferred.
-The files to be deleted are collected in the copy pass then deleted
-after the copy pass has completed successfully. The files to be
-deleted are held in memory so this mode may use more memory. This is
-the safest mode as it will only delete files if there have been no
-errors subsequent to that. If there have been errors before the
-deletions start then you will get the message `not deleting files as
-there were IO errors`.
+In human-readable format the values are scaled to larger units, indicated with
+a suffix shown after the value, and rounded to three decimals. Rclone consistently
+uses binary units (powers of 2) for sizes and decimal units (powers of 10) for counts.
+The unit prefix for size is according to IEC standard notation, e.g. `Ki` for kibi.
+Used with byte unit, `1 KiB` means 1024 Byte. In list type of output, only the
+unit prefix appended to the value (e.g. `9.762Ki`), while in more textual output
+the full unit is shown (e.g. `9.762 KiB`). For counts the SI standard notation is
+used, e.g. prefix `k` for kilo. Used with file counts, `1k` means 1000 files.
-### --fast-list ###
+The various [list](https://rclone.org/commands/rclone_ls/) commands output raw numbers by default.
+Option `--human-readable` will make them output values in human-readable format
+instead (with the short unit prefix).
-When doing anything which involves a directory listing (e.g. `sync`,
-`copy`, `ls` - in fact nearly every command), rclone normally lists a
-directory and processes it before using more directory lists to
-process any subdirectories. This can be parallelised and works very
-quickly using the least amount of memory.
-
-However, some remotes have a way of listing all files beneath a
-directory in one (or a small number) of transactions. These tend to
-be the bucket-based remotes (e.g. S3, B2, GCS, Swift).
-
-If you use the `--fast-list` flag then rclone will use this method for
-listing directories. This will have the following consequences for
-the listing:
-
- * It **will** use fewer transactions (important if you pay for them)
- * It **will** use more memory. Rclone has to load the whole listing into memory.
- * It *may* be faster because it uses fewer transactions
- * It *may* be slower because it can't be parallelized
-
-rclone should always give identical results with and without
-`--fast-list`.
-
-If you pay for transactions and can fit your entire sync listing into
-memory then `--fast-list` is recommended. If you have a very big sync
-to do then don't use `--fast-list` otherwise you will run out of
-memory.
+The [about](https://rclone.org/commands/rclone_about/) command outputs human-readable by default,
+with a command-specific option `--full` to output the raw numbers instead.
-If you use `--fast-list` on a remote which doesn't support it, then
-rclone will just ignore it.
+Command [size](https://rclone.org/commands/rclone_size/) outputs both human-readable and raw numbers
+in the same output.
-### --timeout=TIME ###
+The [tree](https://rclone.org/commands/rclone_tree/) command also considers `--human-readable`, but
+it will not use the exact same notation as the other commands: It rounds to one
+decimal, and uses single letter suffix, e.g. `K` instead of `Ki`. The reason for
+this is that it relies on an external library.
-This sets the IO idle timeout. If a transfer has started but then
-becomes idle for this long it is considered broken and disconnected.
+The interactive command [ncdu](https://rclone.org/commands/rclone_ncdu/) shows human-readable by
+default, and responds to key `u` for toggling human-readable format.
-The default is `5m`. Set to `0` to disable.
+### --ignore-case-sync ###
-### --transfers=N ###
+Using this option will cause rclone to ignore the case of the files
+when synchronizing so files will not be copied/synced when the
+existing filenames are the same, even if the casing is different.
-The number of file transfers to run in parallel. It can sometimes be
-useful to set this to a smaller number if the remote is giving a lot
-of timeouts or bigger if you have lots of bandwidth and a fast remote.
+### --ignore-checksum ###
-The default is to run 4 file transfers in parallel.
+Normally rclone will check that the checksums of transferred files
+match, and give an error "corrupted on transfer" if they don't.
-Look at --multi-thread-streams if you would like to control single file transfers.
+You can use this option to skip that check. You should only use it if
+you have had the "corrupted on transfer" error message and you are
+sure you might want to transfer potentially corrupted data.
-### -u, --update ###
+### --ignore-existing ###
-This forces rclone to skip any files which exist on the destination
-and have a modified time that is newer than the source file.
+Using this option will make rclone unconditionally skip all files
+that exist on the destination, no matter the content of these files.
-This can be useful in avoiding needless transfers when transferring to
-a remote which doesn't support modification times directly (or when
-using `--use-server-modtime` to avoid extra API calls) as it is more
-accurate than a `--size-only` check and faster than using
-`--checksum`. On such remotes (or when using `--use-server-modtime`)
-the time checked will be the uploaded time.
+While this isn't a generally recommended option, it can be useful
+in cases where your files change due to encryption. However, it cannot
+correct partial transfers in case a transfer was interrupted.
-If an existing destination file has a modification time older than the
-source file's, it will be updated if the sizes are different. If the
-sizes are the same, it will be updated if the checksum is different or
-not available.
+When performing a `move`/`moveto` command, this flag will leave skipped
+files in the source location unchanged when a file with the same name
+exists on the destination.
-If an existing destination file has a modification time equal (within
-the computed modify window) to the source file's, it will be updated
-if the sizes are different. The checksum will not be checked in this
-case unless the `--checksum` flag is provided.
+### --ignore-size ###
-In all other cases the file will not be updated.
+Normally rclone will look at modification time and size of files to
+see if they are equal. If you set this flag then rclone will check
+only the modification time. If `--checksum` is set then it only
+checks the checksum.
-Consider using the `--modify-window` flag to compensate for time skews
-between the source and the backend, for backends that do not support
-mod times, and instead use uploaded times. However, if the backend
-does not support checksums, note that syncing or copying within the
-time skew window may still result in additional transfers for safety.
+It will also cause rclone to skip verifying the sizes are the same
+after transfer.
-### --use-mmap ###
+This can be useful for transferring files to and from OneDrive which
+occasionally misreports the size of image files (see
+[#399](https://github.com/rclone/rclone/issues/399) for more info).
-If this flag is set then rclone will use anonymous memory allocated by
-mmap on Unix based platforms and VirtualAlloc on Windows for its
-transfer buffers (size controlled by `--buffer-size`). Memory
-allocated like this does not go on the Go heap and can be returned to
-the OS immediately when it is finished with.
+### -I, --ignore-times ###
-If this flag is not set then rclone will allocate and free the buffers
-using the Go memory allocator which may use more memory as memory
-pages are returned less aggressively to the OS.
+Using this option will cause rclone to unconditionally upload all
+files regardless of the state of files on the destination.
-It is possible this does not work well on all platforms so it is
-disabled by default; in the future it may be enabled by default.
+Normally rclone would skip any files that have the same
+modification time and are the same size (or have the same checksum if
+using `--checksum`).
-### --use-server-modtime ###
+### --immutable ###
-Some object-store backends (e.g, Swift, S3) do not preserve file modification
-times (modtime). On these backends, rclone stores the original modtime as
-additional metadata on the object. By default it will make an API call to
-retrieve the metadata when the modtime is needed by an operation.
+Treat source and destination files as immutable and disallow
+modification.
-Use this flag to disable the extra API call and rely instead on the server's
-modified time. In cases such as a local to remote sync using `--update`,
-knowing the local file is newer than the time it was last uploaded to the
-remote is sufficient. In those cases, this flag can speed up the process and
-reduce the number of API calls necessary.
+With this option set, files will be created and deleted as requested,
+but existing files will never be updated. If an existing file does
+not match between the source and destination, rclone will give the error
+`Source and destination exist but do not match: immutable file modified`.
-Using this flag on a sync operation without also using `--update` would cause
-all files modified at any time other than the last upload time to be uploaded
-again, which is probably not what you want.
+Note that only commands which transfer files (e.g. `sync`, `copy`,
+`move`) are affected by this behavior, and only modification is
+disallowed. Files may still be deleted explicitly (e.g. `delete`,
+`purge`) or implicitly (e.g. `sync`, `move`). Use `copy --immutable`
+if it is desired to avoid deletion as well as modification.
-### -v, -vv, --verbose ###
+This can be useful as an additional layer of protection for immutable
+or append-only data sets (notably backup archives), where modification
+implies corruption and should not be propagated.
-With `-v` rclone will tell you about each file that is transferred and
-a small number of significant events.
+### --inplace {#inplace}
-With `-vv` rclone will become very verbose telling you about every
-file it considers and transfers. Please send bug reports with a log
-with this setting.
+The `--inplace` flag changes the behaviour of rclone when uploading
+files to some backends (backends with the `PartialUploads` feature
+flag set) such as:
-When setting verbosity as an environment variable, use
-`RCLONE_VERBOSE=1` or `RCLONE_VERBOSE=2` for `-v` and `-vv` respectively.
+- local
+- ftp
+- sftp
-### -V, --version ###
+Without `--inplace` (the default) rclone will first upload to a
+temporary file with an extension like this, where `XXXXXX` represents a
+random string and `.partial` is [--partial-suffix](#partial-suffix) value
+(`.partial` by default).
-Prints the version number
+ original-file-name.XXXXXX.partial
-SSL/TLS options
----------------
+(rclone will make sure the final name is no longer than 100 characters
+by truncating the `original-file-name` part if necessary).
-The outgoing SSL/TLS connections rclone makes can be controlled with
-these options. For example this can be very useful with the HTTP or
-WebDAV backends. Rclone HTTP servers have their own set of
-configuration for SSL/TLS which you can find in their documentation.
+When the upload is complete, rclone will rename the `.partial` file to
+the correct name, overwriting any existing file at that point. If the
+upload fails then the `.partial` file will be deleted.
-### --ca-cert stringArray
+This prevents other users of the backend from seeing partially
+uploaded files in their new names and prevents overwriting the old
+file until the new one is completely uploaded.
-This loads the PEM encoded certificate authority certificates and uses
-it to verify the certificates of the servers rclone connects to.
+If the `--inplace` flag is supplied, rclone will upload directly to
+the final name without creating a `.partial` file.
-If you have generated certificates signed with a local CA then you
-will need this flag to connect to servers using those certificates.
+This means that an incomplete file will be visible in the directory
+listings while the upload is in progress and any existing files will
+be overwritten as soon as the upload starts. If the transfer fails
+then the file will be deleted. This can cause data loss of the
+existing file if the transfer fails.
-### --client-cert string
+Note that on the local file system if you don't use `--inplace` hard
+links (Unix only) will be broken. And if you do use `--inplace` you
+won't be able to update in use executables.
-This loads the PEM encoded client side certificate.
+Note also that versions of rclone prior to v1.63.0 behave as if the
+`--inplace` flag is always supplied.
-This is used for [mutual TLS authentication](https://en.wikipedia.org/wiki/Mutual_authentication).
+### -i, --interactive {#interactive}
-The `--client-key` flag is required too when using this.
+This flag can be used to tell rclone that you wish a manual
+confirmation before destructive operations.
-### --client-key string
+It is **recommended** that you use this flag while learning rclone
+especially with `rclone sync`.
-This loads the PEM encoded client side private key used for mutual TLS
-authentication. Used in conjunction with `--client-cert`.
+For example
-### --no-check-certificate=true/false ###
+```
+$ rclone delete --interactive /tmp/dir
+rclone: delete "important-file.txt"?
+y) Yes, this is OK (default)
+n) No, skip this
+s) Skip all delete operations with no more questions
+!) Do all delete operations with no more questions
+q) Exit rclone now.
+y/n/s/!/q> n
+```
-`--no-check-certificate` controls whether a client verifies the
-server's certificate chain and host name.
-If `--no-check-certificate` is true, TLS accepts any certificate
-presented by the server and any host name in that certificate.
-In this mode, TLS is susceptible to man-in-the-middle attacks.
+The options mean
-This option defaults to `false`.
+- `y`: **Yes**, this operation should go ahead. You can also press Return
+ for this to happen. You'll be asked every time unless you choose `s`
+ or `!`.
+- `n`: **No**, do not do this operation. You'll be asked every time unless
+ you choose `s` or `!`.
+- `s`: **Skip** all the following operations of this type with no more
+ questions. This takes effect until rclone exits. If there are any
+ different kind of operations you'll be prompted for them.
+- `!`: **Do all** the following operations with no more
+ questions. Useful if you've decided that you don't mind rclone doing
+ that kind of operation. This takes effect until rclone exits . If
+ there are any different kind of operations you'll be prompted for
+ them.
+- `q`: **Quit** rclone now, just in case!
-**This should be used only for testing.**
+### --leave-root ####
-Configuration Encryption
-------------------------
-Your configuration file contains information for logging in to
-your cloud services. This means that you should keep your
-`rclone.conf` file in a secure location.
+During rmdirs it will not remove root directory, even if it's empty.
-If you are in an environment where that isn't possible, you can
-add a password to your configuration. This means that you will
-have to supply the password every time you start rclone.
+### --log-file=FILE ###
-To add a password to your rclone configuration, execute `rclone config`.
+Log all of rclone's output to FILE. This is not active by default.
+This can be useful for tracking down problems with syncs in
+combination with the `-v` flag. See the [Logging section](#logging)
+for more info.
-```
->rclone config
-Current remotes:
+If FILE exists then rclone will append to it.
-e) Edit existing remote
-n) New remote
-d) Delete remote
-s) Set configuration password
-q) Quit config
-e/n/d/s/q>
-```
+Note that if you are using the `logrotate` program to manage rclone's
+logs, then you should use the `copytruncate` option as rclone doesn't
+have a signal to rotate logs.
-Go into `s`, Set configuration password:
-```
-e/n/d/s/q> s
-Your configuration is not encrypted.
-If you add a password, you will protect your login information to cloud services.
-a) Add Password
-q) Quit to main menu
-a/q> a
-Enter NEW configuration password:
-password:
-Confirm NEW password:
-password:
-Password set
-Your configuration is encrypted.
-c) Change Password
-u) Unencrypt configuration
-q) Quit to main menu
-c/u/q>
-```
+### --log-format LIST ###
-Your configuration is now encrypted, and every time you start rclone
-you will have to supply the password. See below for details.
-In the same menu, you can change the password or completely remove
-encryption from your configuration.
+Comma separated list of log format options. Accepted options are `date`,
+`time`, `microseconds`, `pid`, `longfile`, `shortfile`, `UTC`. Any other
+keywords will be silently ignored. `pid` will tag log messages with process
+identifier which useful with `rclone mount --daemon`. Other accepted
+options are explained in the [go documentation](https://pkg.go.dev/log#pkg-constants).
+The default log format is "`date`,`time`".
-There is no way to recover the configuration if you lose your password.
+### --log-level LEVEL ###
-rclone uses [nacl secretbox](https://godoc.org/golang.org/x/crypto/nacl/secretbox)
-which in turn uses XSalsa20 and Poly1305 to encrypt and authenticate
-your configuration with secret-key cryptography.
-The password is SHA-256 hashed, which produces the key for secretbox.
-The hashed password is not stored.
+This sets the log level for rclone. The default log level is `NOTICE`.
-While this provides very good security, we do not recommend storing
-your encrypted rclone configuration in public if it contains sensitive
-information, maybe except if you use a very strong password.
+`DEBUG` is equivalent to `-vv`. It outputs lots of debug info - useful
+for bug reports and really finding out what rclone is doing.
-If it is safe in your environment, you can set the `RCLONE_CONFIG_PASS`
-environment variable to contain your password, in which case it will be
-used for decrypting the configuration.
+`INFO` is equivalent to `-v`. It outputs information about each transfer
+and prints stats once a minute by default.
-You can set this for a session from a script. For unix like systems
-save this to a file called `set-rclone-password`:
+`NOTICE` is the default log level if no logging flags are supplied. It
+outputs very little when things are working normally. It outputs
+warnings and significant events.
-```
-#!/bin/echo Source this file don't run it
+`ERROR` is equivalent to `-q`. It only outputs error messages.
-read -s RCLONE_CONFIG_PASS
-export RCLONE_CONFIG_PASS
-```
+### --use-json-log ###
-Then source the file when you want to use it. From the shell you
-would do `source set-rclone-password`. It will then ask you for the
-password and set it in the environment variable.
+This switches the log format to JSON for rclone. The fields of json log
+are level, msg, source, time.
-An alternate means of supplying the password is to provide a script
-which will retrieve the password and print on standard output. This
-script should have a fully specified path name and not rely on any
-environment variables. The script is supplied either via
-`--password-command="..."` command line argument or via the
-`RCLONE_PASSWORD_COMMAND` environment variable.
+### --low-level-retries NUMBER ###
-One useful example of this is using the `passwordstore` application
-to retrieve the password:
+This controls the number of low level retries rclone does.
-```
-export RCLONE_PASSWORD_COMMAND="pass rclone/config"
-```
+A low level retry is used to retry a failing operation - typically one
+HTTP request. This might be uploading a chunk of a big file for
+example. You will see low level retries in the log with the `-v`
+flag.
-If the `passwordstore` password manager holds the password for the
-rclone configuration, using the script method means the password
-is primarily protected by the `passwordstore` system, and is never
-embedded in the clear in scripts, nor available for examination
-using the standard commands available. It is quite possible with
-long running rclone sessions for copies of passwords to be innocently
-captured in log files or terminal scroll buffers, etc. Using the
-script method of supplying the password enhances the security of
-the config password considerably.
+This shouldn't need to be changed from the default in normal operations.
+However, if you get a lot of low level retries you may wish
+to reduce the value so rclone moves on to a high level retry (see the
+`--retries` flag) quicker.
-If you are running rclone inside a script, unless you are using the
-`--password-command` method, you might want to disable
-password prompts. To do that, pass the parameter
-`--ask-password=false` to rclone. This will make rclone fail instead
-of asking for a password if `RCLONE_CONFIG_PASS` doesn't contain
-a valid password, and `--password-command` has not been supplied.
+Disable low level retries with `--low-level-retries 1`.
-Whenever running commands that may be affected by options in a
-configuration file, rclone will look for an existing file according
-to the rules described [above](#config-config-file), and load any it
-finds. If an encrypted file is found, this includes decrypting it,
-with the possible consequence of a password prompt. When executing
-a command line that you know are not actually using anything from such
-a configuration file, you can avoid it being loaded by overriding the
-location, e.g. with one of the documented special values for
-memory-only configuration. Since only backend options can be stored
-in configuration files, this is normally unnecessary for commands
-that do not operate on backends, e.g. `genautocomplete`. However,
-it will be relevant for commands that do operate on backends in
-general, but are used without referencing a stored remote, e.g.
-listing local filesystem paths, or
-[connection strings](#connection-strings): `rclone --config="" ls .`
+### --max-backlog=N ###
-Developer options
------------------
+This is the maximum allowable backlog of files in a sync/copy/move
+queued for being checked or transferred.
-These options are useful when developing or debugging rclone. There
-are also some more remote specific options which aren't documented
-here which are used for testing. These start with remote name e.g.
-`--drive-test-option` - see the docs for the remote in question.
+This can be set arbitrarily large. It will only use memory when the
+queue is in use. Note that it will use in the order of N KiB of memory
+when the backlog is in use.
-### --cpuprofile=FILE ###
+Setting this large allows rclone to calculate how many files are
+pending more accurately, give a more accurate estimated finish
+time and make `--order-by` work more accurately.
-Write CPU profile to file. This can be analysed with `go tool pprof`.
+Setting this small will make rclone more synchronous to the listings
+of the remote which may be desirable.
-#### --dump flag,flag,flag ####
+Setting this to a negative number will make the backlog as large as
+possible.
-The `--dump` flag takes a comma separated list of flags to dump info
-about.
+### --max-delete=N ###
-Note that some headers including `Accept-Encoding` as shown may not
-be correct in the request and the response may not show `Content-Encoding`
-if the go standard libraries auto gzip encoding was in effect. In this case
-the body of the request will be gunzipped before showing it.
+This tells rclone not to delete more than N files. If that limit is
+exceeded then a fatal error will be generated and rclone will stop the
+operation in progress.
-The available flags are:
+### --max-delete-size=SIZE ###
-#### --dump headers ####
+Rclone will stop deleting files when the total size of deletions has
+reached the size specified. It defaults to off.
-Dump HTTP headers with `Authorization:` lines removed. May still
-contain sensitive info. Can be very verbose. Useful for debugging
-only.
+If that limit is exceeded then a fatal error will be generated and
+rclone will stop the operation in progress.
-Use `--dump auth` if you do want the `Authorization:` headers.
+### --max-depth=N ###
-#### --dump bodies ####
+This modifies the recursion depth for all the commands except purge.
-Dump HTTP headers and bodies - may contain sensitive info. Can be
-very verbose. Useful for debugging only.
+So if you do `rclone --max-depth 1 ls remote:path` you will see only
+the files in the top level directory. Using `--max-depth 2` means you
+will see all the files in first two directory levels and so on.
-Note that the bodies are buffered in memory so don't use this for
-enormous files.
+For historical reasons the `lsd` command defaults to using a
+`--max-depth` of 1 - you can override this with the command line flag.
-#### --dump requests ####
+You can use this command to disable recursion (with `--max-depth 1`).
-Like `--dump bodies` but dumps the request bodies and the response
-headers. Useful for debugging download problems.
+Note that if you use this with `sync` and `--delete-excluded` the
+files not recursed through are considered excluded and will be deleted
+on the destination. Test first with `--dry-run` if you are not sure
+what will happen.
-#### --dump responses ####
+### --max-duration=TIME ###
-Like `--dump bodies` but dumps the response bodies and the request
-headers. Useful for debugging upload problems.
+Rclone will stop transferring when it has run for the
+duration specified.
+Defaults to off.
-#### --dump auth ####
+When the limit is reached all transfers will stop immediately.
+Use `--cutoff-mode` to modify this behaviour.
-Dump HTTP headers - will contain sensitive info such as
-`Authorization:` headers - use `--dump headers` to dump without
-`Authorization:` headers. Can be very verbose. Useful for debugging
-only.
+Rclone will exit with exit code 10 if the duration limit is reached.
-#### --dump filters ####
+### --max-transfer=SIZE ###
-Dump the filters to the output. Useful to see exactly what include
-and exclude options are filtering on.
+Rclone will stop transferring when it has reached the size specified.
+Defaults to off.
-#### --dump goroutines ####
+When the limit is reached all transfers will stop immediately.
+Use `--cutoff-mode` to modify this behaviour.
-This dumps a list of the running go-routines at the end of the command
-to standard output.
+Rclone will exit with exit code 8 if the transfer limit is reached.
-#### --dump openfiles ####
+### --cutoff-mode=hard|soft|cautious ###
-This dumps a list of the open files at the end of the command. It
-uses the `lsof` command to do that so you'll need that installed to
-use it.
+This modifies the behavior of `--max-transfer` and `--max-duration`
+Defaults to `--cutoff-mode=hard`.
-### --memprofile=FILE ###
+Specifying `--cutoff-mode=hard` will stop transferring immediately
+when Rclone reaches the limit.
-Write memory profile to file. This can be analysed with `go tool pprof`.
+Specifying `--cutoff-mode=soft` will stop starting new transfers
+when Rclone reaches the limit.
-Filtering
----------
+Specifying `--cutoff-mode=cautious` will try to prevent Rclone
+from reaching the limit. Only applicable for `--max-transfer`
-For the filtering options
+## -M, --metadata
- * `--delete-excluded`
- * `--filter`
- * `--filter-from`
- * `--exclude`
- * `--exclude-from`
- * `--exclude-if-present`
- * `--include`
- * `--include-from`
- * `--files-from`
- * `--files-from-raw`
- * `--min-size`
- * `--max-size`
- * `--min-age`
- * `--max-age`
- * `--dump filters`
- * `--metadata-include`
- * `--metadata-include-from`
- * `--metadata-exclude`
- * `--metadata-exclude-from`
- * `--metadata-filter`
- * `--metadata-filter-from`
+Setting this flag enables rclone to copy the metadata from the source
+to the destination. For local backends this is ownership, permissions,
+xattr etc. See the [metadata section](#metadata) for more info.
-See the [filtering section](https://rclone.org/filtering/).
+### --metadata-mapper SpaceSepList {#metadata-mapper}
-Remote control
---------------
+If you supply the parameter `--metadata-mapper /path/to/program` then
+rclone will use that program to map metadata from source object to
+destination object.
-For the remote control options and for instructions on how to remote control rclone
+The argument to this flag should be a command with an optional space separated
+list of arguments. If one of the arguments has a space in then enclose
+it in `"`, if you want a literal `"` in an argument then enclose the
+argument in `"` and double the `"`. See [CSV encoding](https://godoc.org/encoding/csv)
+for more info.
- * `--rc`
- * and anything starting with `--rc-`
+ --metadata-mapper "python bin/test_metadata_mapper.py"
+ --metadata-mapper 'python bin/test_metadata_mapper.py "argument with a space"'
+ --metadata-mapper 'python bin/test_metadata_mapper.py "argument with ""two"" quotes"'
-See [the remote control section](https://rclone.org/rc/).
+This uses a simple JSON based protocol with input on STDIN and output
+on STDOUT. This will be called for every file and directory copied and
+may be called concurrently.
-Logging
--------
+The program's job is to take a metadata blob on the input and turn it
+into a metadata blob on the output suitable for the destination
+backend.
-rclone has 4 levels of logging, `ERROR`, `NOTICE`, `INFO` and `DEBUG`.
+Input to the program (via STDIN) might look like this. This provides
+some context for the `Metadata` which may be important.
-By default, rclone logs to standard error. This means you can redirect
-standard error and still see the normal output of rclone commands (e.g.
-`rclone ls`).
+- `SrcFs` is the config string for the remote that the object is currently on.
+- `SrcFsType` is the name of the source backend.
+- `DstFs` is the config string for the remote that the object is being copied to
+- `DstFsType` is the name of the destination backend.
+- `Remote` is the path of the file relative to the root.
+- `Size`, `MimeType`, `ModTime` are attributes of the file.
+- `IsDir` is `true` if this is a directory (not yet implemented).
+- `ID` is the source `ID` of the file if known.
+- `Metadata` is the backend specific metadata as described in the backend docs.
-By default, rclone will produce `Error` and `Notice` level messages.
+```json
+{
+ "SrcFs": "gdrive:",
+ "SrcFsType": "drive",
+ "DstFs": "newdrive:user",
+ "DstFsType": "onedrive",
+ "Remote": "test.txt",
+ "Size": 6,
+ "MimeType": "text/plain; charset=utf-8",
+ "ModTime": "2022-10-11T17:53:10.286745272+01:00",
+ "IsDir": false,
+ "ID": "xyz",
+ "Metadata": {
+ "btime": "2022-10-11T16:53:11Z",
+ "content-type": "text/plain; charset=utf-8",
+ "mtime": "2022-10-11T17:53:10.286745272+01:00",
+ "owner": "user1@domain1.com",
+ "permissions": "...",
+ "description": "my nice file",
+ "starred": "false"
+ }
+}
+```
-If you use the `-q` flag, rclone will only produce `Error` messages.
+The program should then modify the input as desired and send it to
+STDOUT. The returned `Metadata` field will be used in its entirety for
+the destination object. Any other fields will be ignored. Note in this
+example we translate user names and permissions and add something to
+the description:
-If you use the `-v` flag, rclone will produce `Error`, `Notice` and
-`Info` messages.
+```json
+{
+ "Metadata": {
+ "btime": "2022-10-11T16:53:11Z",
+ "content-type": "text/plain; charset=utf-8",
+ "mtime": "2022-10-11T17:53:10.286745272+01:00",
+ "owner": "user1@domain2.com",
+ "permissions": "...",
+ "description": "my nice file [migrated from domain1]",
+ "starred": "false"
+ }
+}
+```
-If you use the `-vv` flag, rclone will produce `Error`, `Notice`,
-`Info` and `Debug` messages.
+Metadata can be removed here too.
-You can also control the log levels with the `--log-level` flag.
+An example python program might look something like this to implement
+the above transformations.
-If you use the `--log-file=FILE` option, rclone will redirect `Error`,
-`Info` and `Debug` messages along with standard error to FILE.
+```python
+import sys, json
-If you use the `--syslog` flag then rclone will log to syslog and the
-`--syslog-facility` control which facility it uses.
+i = json.load(sys.stdin)
+metadata = i["Metadata"]
+# Add tag to description
+if "description" in metadata:
+ metadata["description"] += " [migrated from domain1]"
+else:
+ metadata["description"] = "[migrated from domain1]"
+# Modify owner
+if "owner" in metadata:
+ metadata["owner"] = metadata["owner"].replace("domain1.com", "domain2.com")
+o = { "Metadata": metadata }
+json.dump(o, sys.stdout, indent="\t")
+```
-Rclone prefixes all log messages with their level in capitals, e.g. INFO
-which makes it easy to grep the log file for different kinds of
-information.
+You can find this example (slightly expanded) in the rclone source code at
+[bin/test_metadata_mapper.py](https://github.com/rclone/rclone/blob/master/test_metadata_mapper.py).
-Exit Code
----------
+If you want to see the input to the metadata mapper and the output
+returned from it in the log you can use `-vv --dump mapper`.
-If any errors occur during the command execution, rclone will exit with a
-non-zero exit code. This allows scripts to detect when rclone
-operations have failed.
+See the [metadata section](#metadata) for more info.
-During the startup phase, rclone will exit immediately if an error is
-detected in the configuration. There will always be a log message
-immediately before exiting.
+### --metadata-set key=value
-When rclone is running it will accumulate errors as it goes along, and
-only exit with a non-zero exit code if (after retries) there were
-still failed transfers. For every error counted there will be a high
-priority log message (visible with `-q`) showing the message and
-which file caused the problem. A high priority message is also shown
-when starting a retry so the user can see that any previous error
-messages may not be valid after the retry. If rclone has done a retry
-it will log a high priority message if the retry was successful.
+Add metadata `key` = `value` when uploading. This can be repeated as
+many times as required. See the [metadata section](#metadata) for more
+info.
-### List of exit codes ###
- * `0` - success
- * `1` - Syntax or usage error
- * `2` - Error not otherwise categorised
- * `3` - Directory not found
- * `4` - File not found
- * `5` - Temporary error (one that more retries might fix) (Retry errors)
- * `6` - Less serious errors (like 461 errors from dropbox) (NoRetry errors)
- * `7` - Fatal error (one that more retries won't fix, like account suspended) (Fatal errors)
- * `8` - Transfer exceeded - limit set by --max-transfer reached
- * `9` - Operation successful, but no files transferred
- * `10` - Duration exceeded - limit set by --max-duration reached
+### --modify-window=TIME ###
-Environment Variables
----------------------
+When checking whether a file has been modified, this is the maximum
+allowed time difference that a file can have and still be considered
+equivalent.
-Rclone can be configured entirely using environment variables. These
-can be used to set defaults for options or config file entries.
+The default is `1ns` unless this is overridden by a remote. For
+example OS X only stores modification times to the nearest second so
+if you are reading and writing to an OS X filing system this will be
+`1s` by default.
-### Options ###
+This command line flag allows you to override that computed default.
-Every option in rclone can have its default set by environment
-variable.
+### --multi-thread-write-buffer-size=SIZE ###
-To find the name of the environment variable, first, take the long
-option name, strip the leading `--`, change `-` to `_`, make
-upper case and prepend `RCLONE_`.
+When transferring with multiple threads, rclone will buffer SIZE bytes
+in memory before writing to disk for each thread.
-For example, to always set `--stats 5s`, set the environment variable
-`RCLONE_STATS=5s`. If you set stats on the command line this will
-override the environment variable setting.
+This can improve performance if the underlying filesystem does not deal
+well with a lot of small writes in different positions of the file, so
+if you see transfers being limited by disk write speed, you might want
+to experiment with different values. Specially for magnetic drives and
+remote file systems a higher value can be useful.
-Or to always use the trash in drive `--drive-use-trash`, set
-`RCLONE_DRIVE_USE_TRASH=true`.
+Nevertheless, the default of `128k` should be fine for almost all use
+cases, so before changing it ensure that network is not really your
+bottleneck.
-Verbosity is slightly different, the environment variable
-equivalent of `--verbose` or `-v` is `RCLONE_VERBOSE=1`,
-or for `-vv`, `RCLONE_VERBOSE=2`.
+As a final hint, size is not the only factor: block size (or similar
+concept) can have an impact. In one case, we observed that exact
+multiples of 16k performed much better than other values.
-The same parser is used for the options and the environment variables
-so they take exactly the same form.
+### --multi-thread-chunk-size=SizeSuffix ###
-The options set by environment variables can be seen with the `-vv` flag, e.g. `rclone version -vv`.
+Normally the chunk size for multi thread transfers is set by the backend.
+However some backends such as `local` and `smb` (which implement `OpenWriterAt`
+but not `OpenChunkWriter`) don't have a natural chunk size.
-### Config file ###
+In this case the value of this option is used (default 64Mi).
-You can set defaults for values in the config file on an individual
-remote basis. The names of the config items are documented in the page
-for each backend.
+### --multi-thread-cutoff=SIZE {#multi-thread-cutoff}
-To find the name of the environment variable, you need to set, take
-`RCLONE_CONFIG_` + name of remote + `_` + name of config file option
-and make it all uppercase.
-Note one implication here is the remote's name must be
-convertible into a valid environment variable name,
-so it can only contain letters, digits, or the `_` (underscore) character.
+When transferring files above SIZE to capable backends, rclone will
+use multiple threads to transfer the file (default 256M).
-For example, to configure an S3 remote named `mys3:` without a config
-file (using unix ways of setting environment variables):
+Capable backends are marked in the
+[overview](https://rclone.org/overview/#optional-features) as `MultithreadUpload`. (They
+need to implement either the `OpenWriterAt` or `OpenChunkedWriter`
+internal interfaces). These include include, `local`, `s3`,
+`azureblob`, `b2`, `oracleobjectstorage` and `smb` at the time of
+writing.
-```
-$ export RCLONE_CONFIG_MYS3_TYPE=s3
-$ export RCLONE_CONFIG_MYS3_ACCESS_KEY_ID=XXX
-$ export RCLONE_CONFIG_MYS3_SECRET_ACCESS_KEY=XXX
-$ rclone lsd mys3:
- -1 2016-09-21 12:54:21 -1 my-bucket
-$ rclone listremotes | grep mys3
-mys3:
-```
+On the local disk, rclone preallocates the file (using
+`fallocate(FALLOC_FL_KEEP_SIZE)` on unix or `NTSetInformationFile` on
+Windows both of which takes no time) then each thread writes directly
+into the file at the correct place. This means that rclone won't
+create fragmented or sparse files and there won't be any assembly time
+at the end of the transfer.
-Note that if you want to create a remote using environment variables
-you must create the `..._TYPE` variable as above.
+The number of threads used to transfer is controlled by
+`--multi-thread-streams`.
-Note that the name of a remote created using environment variable is
-case insensitive, in contrast to regular remotes stored in config
-file as documented [above](#valid-remote-names).
-You must write the name in uppercase in the environment variable, but
-as seen from example above it will be listed and can be accessed in
-lowercase, while you can also refer to the same remote in uppercase:
-```
-$ rclone lsd mys3:
- -1 2016-09-21 12:54:21 -1 my-bucket
-$ rclone lsd MYS3:
- -1 2016-09-21 12:54:21 -1 my-bucket
-```
+Use `-vv` if you wish to see info about the threads.
+This will work with the `sync`/`copy`/`move` commands and friends
+`copyto`/`moveto`. Multi thread transfers will be used with `rclone
+mount` and `rclone serve` if `--vfs-cache-mode` is set to `writes` or
+above.
-Note that you can only set the options of the immediate backend,
-so RCLONE_CONFIG_MYS3CRYPT_ACCESS_KEY_ID has no effect, if myS3Crypt is
-a crypt remote based on an S3 remote. However RCLONE_S3_ACCESS_KEY_ID will
-set the access key of all remotes using S3, including myS3Crypt.
+**NB** that this **only** works with supported backends as the
+destination but will work with any backend as the source.
-Note also that now rclone has [connection strings](#connection-strings),
-it is probably easier to use those instead which makes the above example
+**NB** that multi-thread copies are disabled for local to local copies
+as they are faster without unless `--multi-thread-streams` is set
+explicitly.
- rclone lsd :s3,access_key_id=XXX,secret_access_key=XXX:
+**NB** on Windows using multi-thread transfers to the local disk will
+cause the resulting files to be [sparse](https://en.wikipedia.org/wiki/Sparse_file).
+Use `--local-no-sparse` to disable sparse files (which may cause long
+delays at the start of transfers) or disable multi-thread transfers
+with `--multi-thread-streams 0`
-### Precedence
+### --multi-thread-streams=N ###
-The various different methods of backend configuration are read in
-this order and the first one with a value is used.
+When using multi thread transfers (see above `--multi-thread-cutoff`)
+this sets the number of streams to use. Set to `0` to disable multi
+thread transfers (Default 4).
-- Parameters in connection strings, e.g. `myRemote,skip_links:`
-- Flag values as supplied on the command line, e.g. `--skip-links`
-- Remote specific environment vars, e.g. `RCLONE_CONFIG_MYREMOTE_SKIP_LINKS` (see above).
-- Backend-specific environment vars, e.g. `RCLONE_LOCAL_SKIP_LINKS`.
-- Backend generic environment vars, e.g. `RCLONE_SKIP_LINKS`.
-- Config file, e.g. `skip_links = true`.
-- Default values, e.g. `false` - these can't be changed.
+If the backend has a `--backend-upload-concurrency` setting (eg
+`--s3-upload-concurrency`) then this setting will be used as the
+number of transfers instead if it is larger than the value of
+`--multi-thread-streams` or `--multi-thread-streams` isn't set.
-So if both `--skip-links` is supplied on the command line and an
-environment variable `RCLONE_LOCAL_SKIP_LINKS` is set, the command line
-flag will take preference.
+### --no-check-dest ###
-The backend configurations set by environment variables can be seen with the `-vv` flag, e.g. `rclone about myRemote: -vv`.
+The `--no-check-dest` can be used with `move` or `copy` and it causes
+rclone not to check the destination at all when copying files.
-For non backend configuration the order is as follows:
+This means that:
-- Flag values as supplied on the command line, e.g. `--stats 5s`.
-- Environment vars, e.g. `RCLONE_STATS=5s`.
-- Default values, e.g. `1m` - these can't be changed.
+- the destination is not listed minimising the API calls
+- files are always transferred
+- this can cause duplicates on remotes which allow it (e.g. Google Drive)
+- `--retries 1` is recommended otherwise you'll transfer everything again on a retry
-### Other environment variables ###
+This flag is useful to minimise the transactions if you know that none
+of the files are on the destination.
-- `RCLONE_CONFIG_PASS` set to contain your config file password (see [Configuration Encryption](#configuration-encryption) section)
-- `HTTP_PROXY`, `HTTPS_PROXY` and `NO_PROXY` (or the lowercase versions thereof).
- - `HTTPS_PROXY` takes precedence over `HTTP_PROXY` for https requests.
- - The environment values may be either a complete URL or a "host[:port]" for, in which case the "http" scheme is assumed.
-- `USER` and `LOGNAME` values are used as fallbacks for current username. The primary method for looking up username is OS-specific: Windows API on Windows, real user ID in /etc/passwd on Unix systems. In the documentation the current username is simply referred to as `$USER`.
-- `RCLONE_CONFIG_DIR` - rclone **sets** this variable for use in config files and sub processes to point to the directory holding the config file.
+This is a specialized flag which should be ignored by most users!
-The options set by environment variables can be seen with the `-vv` and `--log-level=DEBUG` flags, e.g. `rclone version -vv`.
+### --no-gzip-encoding ###
-# Configuring rclone on a remote / headless machine #
+Don't set `Accept-Encoding: gzip`. This means that rclone won't ask
+the server for compressed files automatically. Useful if you've set
+the server to return files with `Content-Encoding: gzip` but you
+uploaded compressed files.
-Some of the configurations (those involving oauth2) require an
-Internet connected web browser.
+There is no need to set this in normal operation, and doing so will
+decrease the network transfer efficiency of rclone.
-If you are trying to set rclone up on a remote or headless box with no
-browser available on it (e.g. a NAS or a server in a datacenter) then
-you will need to use an alternative means of configuration. There are
-two ways of doing it, described below.
+### --no-traverse ###
-## Configuring using rclone authorize ##
+The `--no-traverse` flag controls whether the destination file system
+is traversed when using the `copy` or `move` commands.
+`--no-traverse` is not compatible with `sync` and will be ignored if
+you supply it with `sync`.
-On the headless box run `rclone` config but answer `N` to the `Use web browser
-to automatically authenticate?` question.
+If you are only copying a small number of files (or are filtering most
+of the files) and/or have a large number of files on the destination
+then `--no-traverse` will stop rclone listing the destination and save
+time.
-```
-...
-Remote config
-Use web browser to automatically authenticate rclone with remote?
- * Say Y if the machine running rclone has a web browser you can use
- * Say N if running rclone on a (remote) machine without web browser access
-If not sure try Y. If Y failed, try N.
-y) Yes (default)
-n) No
-y/n> n
-For this to work, you will need rclone available on a machine that has
-a web browser available.
+However, if you are copying a large number of files, especially if you
+are doing a copy where lots of the files under consideration haven't
+changed and won't need copying then you shouldn't use `--no-traverse`.
-For more help and alternate methods see: https://rclone.org/remote_setup/
+See [rclone copy](https://rclone.org/commands/rclone_copy/) for an example of how to use it.
-Execute the following on the machine with the web browser (same rclone
-version recommended):
+### --no-unicode-normalization ###
- rclone authorize "amazon cloud drive"
+Don't normalize unicode characters in filenames during the sync routine.
-Then paste the result below:
-result>
-```
+Sometimes, an operating system will store filenames containing unicode
+parts in their decomposed form (particularly macOS). Some cloud storage
+systems will then recompose the unicode, resulting in duplicate files if
+the data is ever copied back to a local filesystem.
-Then on your main desktop machine
+Using this flag will disable that functionality, treating each unicode
+character as unique. For example, by default eÌ and é will be normalized
+into the same character. With `--no-unicode-normalization` they will be
+treated as unique characters.
-```
-rclone authorize "amazon cloud drive"
-If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth
-Log in and authorize rclone for access
-Waiting for code...
-Got code
-Paste the following into your remote machine --->
-SECRET_TOKEN
-<---End paste
-```
+### --no-update-modtime ###
-Then back to the headless box, paste in the code
+When using this flag, rclone won't update modification times of remote
+files if they are incorrect as it would normally.
-```
-result> SECRET_TOKEN
---------------------
-[acd12]
-client_id =
-client_secret =
-token = SECRET_TOKEN
---------------------
-y) Yes this is OK
-e) Edit this remote
-d) Delete this remote
-y/e/d>
-```
+This can be used if the remote is being synced with another tool also
+(e.g. the Google Drive client).
-## Configuring by copying the config file ##
+### --order-by string ###
-Rclone stores all of its config in a single configuration file. This
-can easily be copied to configure a remote rclone.
+The `--order-by` flag controls the order in which files in the backlog
+are processed in `rclone sync`, `rclone copy` and `rclone move`.
-So first configure rclone on your desktop machine with
+The order by string is constructed like this. The first part
+describes what aspect is being measured:
- rclone config
+- `size` - order by the size of the files
+- `name` - order by the full path of the files
+- `modtime` - order by the modification date of the files
-to set up the config file.
+This can have a modifier appended with a comma:
-Find the config file by running `rclone config file`, for example
+- `ascending` or `asc` - order so that the smallest (or oldest) is processed first
+- `descending` or `desc` - order so that the largest (or newest) is processed first
+- `mixed` - order so that the smallest is processed first for some threads and the largest for others
-```
-$ rclone config file
-Configuration file is stored at:
-/home/user/.rclone.conf
-```
+If the modifier is `mixed` then it can have an optional percentage
+(which defaults to `50`), e.g. `size,mixed,25` which means that 25% of
+the threads should be taking the smallest items and 75% the
+largest. The threads which take the smallest first will always take
+the smallest first and likewise the largest first threads. The `mixed`
+mode can be useful to minimise the transfer time when you are
+transferring a mixture of large and small files - the large files are
+guaranteed upload threads and bandwidth and the small files will be
+processed continuously.
-Now transfer it to the remote box (scp, cut paste, ftp, sftp, etc.) and
-place it in the correct place (use `rclone config file` on the remote
-box to find out where).
+If no modifier is supplied then the order is `ascending`.
-## Configuring using SSH Tunnel ##
+For example
-Linux and MacOS users can utilize SSH Tunnel to redirect the headless box port 53682 to local machine by using the following command:
-```
-ssh -L localhost:53682:localhost:53682 username@remote_server
-```
-Then on the headless box run `rclone` config and answer `Y` to the `Use web
-browser to automatically authenticate?` question.
+- `--order-by size,desc` - send the largest files first
+- `--order-by modtime,ascending` - send the oldest files first
+- `--order-by name` - send the files with alphabetically by path first
-```
-...
-Remote config
-Use web browser to automatically authenticate rclone with remote?
- * Say Y if the machine running rclone has a web browser you can use
- * Say N if running rclone on a (remote) machine without web browser access
-If not sure try Y. If Y failed, try N.
-y) Yes (default)
-n) No
-y/n> y
-```
-Then copy and paste the auth url `http://127.0.0.1:53682/auth?state=xxxxxxxxxxxx` to the browser on your local machine, complete the auth and it is done.
+If the `--order-by` flag is not supplied or it is supplied with an
+empty string then the default ordering will be used which is as
+scanned. With `--checkers 1` this is mostly alphabetical, however
+with the default `--checkers 8` it is somewhat random.
-# Filtering, includes and excludes
+#### Limitations
-Filter flags determine which files rclone `sync`, `move`, `ls`, `lsl`,
-`md5sum`, `sha1sum`, `size`, `delete`, `check` and similar commands
-apply to.
+The `--order-by` flag does not do a separate pass over the data. This
+means that it may transfer some files out of the order specified if
-They are specified in terms of path/file name patterns; path/file
-lists; file age and size, or presence of a file in a directory. Bucket
-based remotes without the concept of directory apply filters to object
-key, age and size in an analogous way.
+- there are no files in the backlog or the source has not been fully scanned yet
+- there are more than [--max-backlog](#max-backlog-n) files in the backlog
-Rclone `purge` does not obey filters.
+Rclone will do its best to transfer the best file it has so in
+practice this should not cause a problem. Think of `--order-by` as
+being more of a best efforts flag rather than a perfect ordering.
-To test filters without risk of damage to data, apply them to `rclone
-ls`, or with the `--dry-run` and `-vv` flags.
+If you want perfect ordering then you will need to specify
+[--check-first](#check-first) which will find all the files which need
+transferring first before transferring any.
-Rclone filter patterns can only be used in filter command line options, not
-in the specification of a remote.
+### --partial-suffix {#partial-suffix}
-E.g. `rclone copy "remote:dir*.jpg" /path/to/dir` does not have a filter effect.
-`rclone copy remote:dir /path/to/dir --include "*.jpg"` does.
+When [--inplace](#inplace) is not used, it causes rclone to use
+the `--partial-suffix` as suffix for temporary files.
-**Important** Avoid mixing any two of `--include...`, `--exclude...` or
-`--filter...` flags in an rclone command. The results might not be what
-you expect. Instead use a `--filter...` flag.
+Suffix length limit is 16 characters.
-## Patterns for matching path/file names
+The default is `.partial`.
-### Pattern syntax {#patterns}
+### --password-command SpaceSepList ###
-Here is a formal definition of the pattern syntax,
-[examples](#examples) are below.
+This flag supplies a program which should supply the config password
+when run. This is an alternative to rclone prompting for the password
+or setting the `RCLONE_CONFIG_PASS` variable.
-Rclone matching rules follow a glob style:
+The argument to this should be a command with a space separated list
+of arguments. If one of the arguments has a space in then enclose it
+in `"`, if you want a literal `"` in an argument then enclose the
+argument in `"` and double the `"`. See [CSV encoding](https://godoc.org/encoding/csv)
+for more info.
- * matches any sequence of non-separator (/) characters
- ** matches any sequence of characters including / separators
- ? matches any single non-separator (/) character
- [ [ ! ] { character-range } ]
- character class (must be non-empty)
- { pattern-list }
- pattern alternatives
- {{ regexp }}
- regular expression to match
- c matches character c (c != *, **, ?, \, [, {, })
- \c matches reserved character c (c = *, **, ?, \, [, {, }) or character class
+Eg
-character-range:
+ --password-command "echo hello"
+ --password-command 'echo "hello with space"'
+ --password-command 'echo "hello with ""quotes"" and space"'
- c matches character c (c != \, -, ])
- \c matches reserved character c (c = \, -, ])
- lo - hi matches character c for lo <= c <= hi
+See the [Configuration Encryption](#configuration-encryption) for more info.
-pattern-list:
+See a [Windows PowerShell example on the Wiki](https://github.com/rclone/rclone/wiki/Windows-Powershell-use-rclone-password-command-for-Config-file-password).
- pattern { , pattern }
- comma-separated (without spaces) patterns
+### -P, --progress ###
-character classes (see [Go regular expression reference](https://golang.org/pkg/regexp/syntax/)) include:
+This flag makes rclone update the stats in a static block in the
+terminal providing a realtime overview of the transfer.
- Named character classes (e.g. [\d], [^\d], [\D], [^\D])
- Perl character classes (e.g. \s, \S, \w, \W)
- ASCII character classes (e.g. [[:alnum:]], [[:alpha:]], [[:punct:]], [[:xdigit:]])
+Any log messages will scroll above the static block. Log messages
+will push the static block down to the bottom of the terminal where it
+will stay.
-regexp for advanced users to insert a regular expression - see [below](#regexp) for more info:
+Normally this is updated every 500mS but this period can be overridden
+with the `--stats` flag.
- Any re2 regular expression not containing `}}`
+This can be used with the `--stats-one-line` flag for a simpler
+display.
-If the filter pattern starts with a `/` then it only matches
-at the top level of the directory tree,
-**relative to the root of the remote** (not necessarily the root
-of the drive). If it does not start with `/` then it is matched
-starting at the **end of the path/file name** but it only matches
-a complete path element - it must match from a `/`
-separator or the beginning of the path/file.
+Note: On Windows until [this bug](https://github.com/Azure/go-ansiterm/issues/26)
+is fixed all non-ASCII characters will be replaced with `.` when
+`--progress` is in use.
- file.jpg - matches "file.jpg"
- - matches "directory/file.jpg"
- - doesn't match "afile.jpg"
- - doesn't match "directory/afile.jpg"
- /file.jpg - matches "file.jpg" in the root directory of the remote
- - doesn't match "afile.jpg"
- - doesn't match "directory/file.jpg"
+### --progress-terminal-title ###
-The top level of the remote might not be the top level of the drive.
+This flag, when used with `-P/--progress`, will print the string `ETA: %s`
+to the terminal title.
-E.g. for a Microsoft Windows local directory structure
+### -q, --quiet ###
- F:
- ├── bkp
- ├── data
- │ ├── excl
- │ │ ├── 123.jpg
- │ │ └── 456.jpg
- │ ├── incl
- │ │ └── document.pdf
+This flag will limit rclone's output to error messages only.
-To copy the contents of folder `data` into folder `bkp` excluding the contents of subfolder
-`excl`the following command treats `F:\data` and `F:\bkp` as top level for filtering.
+### --refresh-times ###
-`rclone copy F:\data\ F:\bkp\ --exclude=/excl/**`
+The `--refresh-times` flag can be used to update modification times of
+existing files when they are out of sync on backends which don't
+support hashes.
-**Important** Use `/` in path/file name patterns and not `\` even if
-running on Microsoft Windows.
+This is useful if you uploaded files with the incorrect timestamps and
+you now wish to correct them.
-Simple patterns are case sensitive unless the `--ignore-case` flag is used.
+This flag is **only** useful for destinations which don't support
+hashes (e.g. `crypt`).
-Without `--ignore-case` (default)
+This can be used any of the sync commands `sync`, `copy` or `move`.
- potato - matches "potato"
- - doesn't match "POTATO"
+To use this flag you will need to be doing a modification time sync
+(so not using `--size-only` or `--checksum`). The flag will have no
+effect when using `--size-only` or `--checksum`.
-With `--ignore-case`
+If this flag is used when rclone comes to upload a file it will check
+to see if there is an existing file on the destination. If this file
+matches the source with size (and checksum if available) but has a
+differing timestamp then instead of re-uploading it, rclone will
+update the timestamp on the destination file. If the checksum does not
+match rclone will upload the new file. If the checksum is absent (e.g.
+on a `crypt` backend) then rclone will update the timestamp.
- potato - matches "potato"
- - matches "POTATO"
+Note that some remotes can't set the modification time without
+re-uploading the file so this flag is less useful on them.
-## Using regular expressions in filter patterns {#regexp}
+Normally if you are doing a modification time sync rclone will update
+modification times without `--refresh-times` provided that the remote
+supports checksums **and** the checksums match on the file. However if the
+checksums are absent then rclone will upload the file rather than
+setting the timestamp as this is the safe behaviour.
-The syntax of filter patterns is glob style matching (like `bash`
-uses) to make things easy for users. However this does not provide
-absolute control over the matching, so for advanced users rclone also
-provides a regular expression syntax.
+### --retries int ###
-The regular expressions used are as defined in the [Go regular
-expression reference](https://golang.org/pkg/regexp/syntax/). Regular
-expressions should be enclosed in `{{` `}}`. They will match only the
-last path segment if the glob doesn't start with `/` or the whole path
-name if it does. Note that rclone does not attempt to parse the
-supplied regular expression, meaning that using any regular expression
-filter will prevent rclone from using [directory filter rules](#directory_filter),
-as it will instead check every path against
-the supplied regular expression(s).
+Retry the entire sync if it fails this many times it fails (default 3).
-Here is how the `{{regexp}}` is transformed into an full regular
-expression to match the entire path:
+Some remotes can be unreliable and a few retries help pick up the
+files which didn't get transferred because of errors.
- {{regexp}} becomes (^|/)(regexp)$
- /{{regexp}} becomes ^(regexp)$
+Disable retries with `--retries 1`.
-Regexp syntax can be mixed with glob syntax, for example
+### --retries-sleep=TIME ###
- *.{{jpe?g}} to match file.jpg, file.jpeg but not file.png
+This sets the interval between each retry specified by `--retries`
-You can also use regexp flags - to set case insensitive, for example
+The default is `0`. Use `0` to disable.
- *.{{(?i)jpg}} to match file.jpg, file.JPG but not file.png
+### --server-side-across-configs ###
-Be careful with wildcards in regular expressions - you don't want them
-to match path separators normally. To match any file name starting
-with `start` and ending with `end` write
+Allow server-side operations (e.g. copy or move) to work across
+different configurations.
- {{start[^/]*end\.jpg}}
+This can be useful if you wish to do a server-side copy or move
+between two remotes which use the same backend but are configured
+differently.
-Not
+Note that this isn't enabled by default because it isn't easy for
+rclone to tell if it will work between any two configurations.
- {{start.*end\.jpg}}
+### --size-only ###
-Which will match a directory called `start` with a file called
-`end.jpg` in it as the `.*` will match `/` characters.
+Normally rclone will look at modification time and size of files to
+see if they are equal. If you set this flag then rclone will check
+only the size.
-Note that you can use `-vv --dump filters` to show the filter patterns
-in regexp format - rclone implements the glob patterns by transforming
-them into regular expressions.
+This can be useful transferring files from Dropbox which have been
+modified by the desktop sync client which doesn't set checksums of
+modification times in the same way as rclone.
-## Filter pattern examples {#examples}
+### --stats=TIME ###
-| Description | Pattern | Matches | Does not match |
-| ----------- |-------- | ------- | -------------- |
-| Wildcard | `*.jpg` | `/file.jpg` | `/file.png` |
-| | | `/dir/file.jpg` | `/dir/file.png` |
-| Rooted | `/*.jpg` | `/file.jpg` | `/file.png` |
-| | | `/file2.jpg` | `/dir/file.jpg` |
-| Alternates | `*.{jpg,png}` | `/file.jpg` | `/file.gif` |
-| | | `/dir/file.png` | `/dir/file.gif` |
-| Path Wildcard | `dir/**` | `/dir/anyfile` | `file.png` |
-| | | `/subdir/dir/subsubdir/anyfile` | `/subdir/file.png` |
-| Any Char | `*.t?t` | `/file.txt` | `/file.qxt` |
-| | | `/dir/file.tzt` | `/dir/file.png` |
-| Range | `*.[a-z]` | `/file.a` | `/file.0` |
-| | | `/dir/file.b` | `/dir/file.1` |
-| Escape | `*.\?\?\?` | `/file.???` | `/file.abc` |
-| | | `/dir/file.???` | `/dir/file.def` |
-| Class | `*.\d\d\d` | `/file.012` | `/file.abc` |
-| | | `/dir/file.345` | `/dir/file.def` |
-| Regexp | `*.{{jpe?g}}` | `/file.jpeg` | `/file.png` |
-| | | `/dir/file.jpg` | `/dir/file.jpeeg` |
-| Rooted Regexp | `/{{.*\.jpe?g}}` | `/file.jpeg` | `/file.png` |
-| | | `/file.jpg` | `/dir/file.jpg` |
-
-## How filter rules are applied to files {#how-filter-rules-work}
+Commands which transfer data (`sync`, `copy`, `copyto`, `move`,
+`moveto`) will print data transfer stats at regular intervals to show
+their progress.
-Rclone path/file name filters are made up of one or more of the following flags:
+This sets the interval.
- * `--include`
- * `--include-from`
- * `--exclude`
- * `--exclude-from`
- * `--filter`
- * `--filter-from`
+The default is `1m`. Use `0` to disable.
-There can be more than one instance of individual flags.
+If you set the stats interval then all commands can show stats. This
+can be useful when running other commands, `check` or `mount` for
+example.
-Rclone internally uses a combined list of all the include and exclude
-rules. The order in which rules are processed can influence the result
-of the filter.
+Stats are logged at `INFO` level by default which means they won't
+show at default log level `NOTICE`. Use `--stats-log-level NOTICE` or
+`-v` to make them show. See the [Logging section](#logging) for more
+info on log levels.
-All flags of the same type are processed together in the order
-above, regardless of what order the different types of flags are
-included on the command line.
+Note that on macOS you can send a SIGINFO (which is normally ctrl-T in
+the terminal) to make the stats print immediately.
-Multiple instances of the same flag are processed from left
-to right according to their position in the command line.
+### --stats-file-name-length integer ###
+By default, the `--stats` output will truncate file names and paths longer
+than 40 characters. This is equivalent to providing
+`--stats-file-name-length 40`. Use `--stats-file-name-length 0` to disable
+any truncation of file names printed by stats.
-To mix up the order of processing includes and excludes use `--filter...`
-flags.
+### --stats-log-level string ###
-Within `--include-from`, `--exclude-from` and `--filter-from` flags
-rules are processed from top to bottom of the referenced file.
+Log level to show `--stats` output at. This can be `DEBUG`, `INFO`,
+`NOTICE`, or `ERROR`. The default is `INFO`. This means at the
+default level of logging which is `NOTICE` the stats won't show - if
+you want them to then use `--stats-log-level NOTICE`. See the [Logging
+section](#logging) for more info on log levels.
-If there is an `--include` or `--include-from` flag specified, rclone
-implies a `- **` rule which it adds to the bottom of the internal rule
-list. Specifying a `+` rule with a `--filter...` flag does not imply
-that rule.
+### --stats-one-line ###
-Each path/file name passed through rclone is matched against the
-combined filter list. At first match to a rule the path/file name
-is included or excluded and no further filter rules are processed for
-that path/file.
+When this is specified, rclone condenses the stats into a single line
+showing the most important stats only.
-If rclone does not find a match, after testing against all rules
-(including the implied rule if appropriate), the path/file name
-is included.
+### --stats-one-line-date ###
-Any path/file included at that stage is processed by the rclone
-command.
+When this is specified, rclone enables the single-line stats and prepends
+the display with a date string. The default is `2006/01/02 15:04:05 - `
-`--files-from` and `--files-from-raw` flags over-ride and cannot be
-combined with other filter options.
+### --stats-one-line-date-format ###
-To see the internal combined rule list, in regular expression form,
-for a command add the `--dump filters` flag. Running an rclone command
-with `--dump filters` and `-vv` flags lists the internal filter elements
-and shows how they are applied to each source path/file. There is not
-currently a means provided to pass regular expression filter options into
-rclone directly though character class filter rules contain character
-classes. [Go regular expression reference](https://golang.org/pkg/regexp/syntax/)
+When this is specified, rclone enables the single-line stats and prepends
+the display with a user-supplied date string. The date string MUST be
+enclosed in quotes. Follow [golang specs](https://golang.org/pkg/time/#Time.Format) for
+date formatting syntax.
-### How filter rules are applied to directories {#directory_filter}
+### --stats-unit=bits|bytes ###
-Rclone commands are applied to path/file names not
-directories. The entire contents of a directory can be matched
-to a filter by the pattern `directory/*` or recursively by
-`directory/**`.
+By default, data transfer rates will be printed in bytes per second.
-Directory filter rules are defined with a closing `/` separator.
+This option allows the data rate to be printed in bits per second.
-E.g. `/directory/subdirectory/` is an rclone directory filter rule.
+Data transfer volume will still be reported in bytes.
-Rclone commands can use directory filter rules to determine whether they
-recurse into subdirectories. This potentially optimises access to a remote
-by avoiding listing unnecessary directories. Whether optimisation is
-desirable depends on the specific filter rules and source remote content.
+The rate is reported as a binary unit, not SI unit. So 1 Mbit/s
+equals 1,048,576 bit/s and not 1,000,000 bit/s.
-If any [regular expression filters](#regexp) are in use, then no
-directory recursion optimisation is possible, as rclone must check
-every path against the supplied regular expression(s).
+The default is `bytes`.
-Directory recursion optimisation occurs if either:
+### --suffix=SUFFIX ###
-* A source remote does not support the rclone `ListR` primitive. local,
-sftp, Microsoft OneDrive and WebDAV do not support `ListR`. Google
-Drive and most bucket type storage do. [Full list](https://rclone.org/overview/#optional-features)
+When using `sync`, `copy` or `move` any files which would have been
+overwritten or deleted will have the suffix added to them. If there
+is a file with the same path (after the suffix has been added), then
+it will be overwritten.
-* On other remotes (those that support `ListR`), if the rclone command is not naturally recursive, and
-provided it is not run with the `--fast-list` flag. `ls`, `lsf -R` and
-`size` are naturally recursive but `sync`, `copy` and `move` are not.
+The remote in use must support server-side move or copy and you must
+use the same remote as the destination of the sync.
-* Whenever the `--disable ListR` flag is applied to an rclone command.
+This is for use with files to add the suffix in the current directory
+or with `--backup-dir`. See `--backup-dir` for more info.
-Rclone commands imply directory filter rules from path/file filter
-rules. To view the directory filter rules rclone has implied for a
-command specify the `--dump filters` flag.
+For example
-E.g. for an include rule
+ rclone copy --interactive /path/to/local/file remote:current --suffix .bak
- /a/*.jpg
+will copy `/path/to/local` to `remote:current`, but for any files
+which would have been updated or deleted have .bak added.
-Rclone implies the directory include rule
+If using `rclone sync` with `--suffix` and without `--backup-dir` then
+it is recommended to put a filter rule in excluding the suffix
+otherwise the `sync` will delete the backup files.
- /a/
+ rclone sync --interactive /path/to/local/file remote:current --suffix .bak --exclude "*.bak"
-Directory filter rules specified in an rclone command can limit
-the scope of an rclone command but path/file filters still have
-to be specified.
+### --suffix-keep-extension ###
-E.g. `rclone ls remote: --include /directory/` will not match any
-files. Because it is an `--include` option the `--exclude **` rule
-is implied, and the `/directory/` pattern serves only to optimise
-access to the remote by ignoring everything outside of that directory.
+When using `--suffix`, setting this causes rclone put the SUFFIX
+before the extension of the files that it backs up rather than after.
-E.g. `rclone ls remote: --filter-from filter-list.txt` with a file
-`filter-list.txt`:
+So let's say we had `--suffix -2019-01-01`, without the flag `file.txt`
+would be backed up to `file.txt-2019-01-01` and with the flag it would
+be backed up to `file-2019-01-01.txt`. This can be helpful to make
+sure the suffixed files can still be opened.
- - /dir1/
- - /dir2/
- + *.pdf
- - **
+If a file has two (or more) extensions and the second (or subsequent)
+extension is recognised as a valid mime type, then the suffix will go
+before that extension. So `file.tar.gz` would be backed up to
+`file-2019-01-01.tar.gz` whereas `file.badextension.gz` would be
+backed up to `file.badextension-2019-01-01.gz`.
-All files in directories `dir1` or `dir2` or their subdirectories
-are completely excluded from the listing. Only files of suffix
-`pdf` in the root of `remote:` or its subdirectories are listed.
-The `- **` rule prevents listing of any path/files not previously
-matched by the rules above.
+### --syslog ###
-Option `exclude-if-present` creates a directory exclude rule based
-on the presence of a file in a directory and takes precedence over
-other rclone directory filter rules.
+On capable OSes (not Windows or Plan9) send all log output to syslog.
-When using pattern list syntax, if a pattern item contains either
-`/` or `**`, then rclone will not able to imply a directory filter rule
-from this pattern list.
+This can be useful for running rclone in a script or `rclone mount`.
-E.g. for an include rule
+### --syslog-facility string ###
- {dir1/**,dir2/**}
+If using `--syslog` this sets the syslog facility (e.g. `KERN`, `USER`).
+See `man syslog` for a list of possible facilities. The default
+facility is `DAEMON`.
-Rclone will match files below directories `dir1` or `dir2` only,
-but will not be able to use this filter to exclude a directory `dir3`
-from being traversed.
+### --temp-dir=DIR ###
-Directory recursion optimisation may affect performance, but normally
-not the result. One exception to this is sync operations with option
-`--create-empty-src-dirs`, where any traversed empty directories will
-be created. With the pattern list example `{dir1/**,dir2/**}` above,
-this would create an empty directory `dir3` on destination (when it exists
-on source). Changing the filter to `{dir1,dir2}/**`, or splitting it into
-two include rules `--include dir1/** --include dir2/**`, will match the
-same files while also filtering directories, with the result that an empty
-directory `dir3` will no longer be created.
+Specify the directory rclone will use for temporary files, to override
+the default. Make sure the directory exists and have accessible permissions.
-### `--exclude` - Exclude files matching pattern
+By default the operating system's temp directory will be used:
+- On Unix systems, `$TMPDIR` if non-empty, else `/tmp`.
+- On Windows, the first non-empty value from `%TMP%`, `%TEMP%`, `%USERPROFILE%`, or the Windows directory.
-Excludes path/file names from an rclone command based on a single exclude
-rule.
+When overriding the default with this option, the specified path will be
+set as value of environment variable `TMPDIR` on Unix systems
+and `TMP` and `TEMP` on Windows.
-This flag can be repeated. See above for the order filter flags are
-processed in.
+You can use the [config paths](https://rclone.org/commands/rclone_config_paths/)
+command to see the current value.
-`--exclude` should not be used with `--include`, `--include-from`,
-`--filter` or `--filter-from` flags.
+### --tpslimit float ###
-`--exclude` has no effect when combined with `--files-from` or
-`--files-from-raw` flags.
+Limit transactions per second to this number. Default is 0 which is
+used to mean unlimited transactions per second.
-E.g. `rclone ls remote: --exclude *.bak` excludes all .bak files
-from listing.
+A transaction is roughly defined as an API call; its exact meaning
+will depend on the backend. For HTTP based backends it is an HTTP
+PUT/GET/POST/etc and its response. For FTP/SFTP it is a round trip
+transaction over TCP.
-E.g. `rclone size remote: "--exclude /dir/**"` returns the total size of
-all files on `remote:` excluding those in root directory `dir` and sub
-directories.
+For example, to limit rclone to 10 transactions per second use
+`--tpslimit 10`, or to 1 transaction every 2 seconds use `--tpslimit
+0.5`.
-E.g. on Microsoft Windows `rclone ls remote: --exclude "*\[{JP,KR,HK}\]*"`
-lists the files in `remote:` without `[JP]` or `[KR]` or `[HK]` in
-their name. Quotes prevent the shell from interpreting the `\`
-characters.`\` characters escape the `[` and `]` so an rclone filter
-treats them literally rather than as a character-range. The `{` and `}`
-define an rclone pattern list. For other operating systems single quotes are
-required ie `rclone ls remote: --exclude '*\[{JP,KR,HK}\]*'`
+Use this when the number of transactions per second from rclone is
+causing a problem with the cloud storage provider (e.g. getting you
+banned or rate limited).
-### `--exclude-from` - Read exclude patterns from file
+This can be very useful for `rclone mount` to control the behaviour of
+applications using it.
-Excludes path/file names from an rclone command based on rules in a
-named file. The file contains a list of remarks and pattern rules.
+This limit applies to all HTTP based backends and to the FTP and SFTP
+backends. It does not apply to the local backend or the Storj backend.
-For an example `exclude-file.txt`:
+See also `--tpslimit-burst`.
- # a sample exclude rule file
- *.bak
- file2.jpg
+### --tpslimit-burst int ###
-`rclone ls remote: --exclude-from exclude-file.txt` lists the files on
-`remote:` except those named `file2.jpg` or with a suffix `.bak`. That is
-equivalent to `rclone ls remote: --exclude file2.jpg --exclude "*.bak"`.
+Max burst of transactions for `--tpslimit` (default `1`).
-This flag can be repeated. See above for the order filter flags are
-processed in.
+Normally `--tpslimit` will do exactly the number of transaction per
+second specified. However if you supply `--tps-burst` then rclone can
+save up some transactions from when it was idle giving a burst of up
+to the parameter supplied.
-The `--exclude-from` flag is useful where multiple exclude filter rules
-are applied to an rclone command.
+For example if you provide `--tpslimit-burst 10` then if rclone has
+been idle for more than 10*`--tpslimit` then it can do 10 transactions
+very quickly before they are limited again.
-`--exclude-from` should not be used with `--include`, `--include-from`,
-`--filter` or `--filter-from` flags.
+This may be used to increase performance of `--tpslimit` without
+changing the long term average number of transactions per second.
-`--exclude-from` has no effect when combined with `--files-from` or
-`--files-from-raw` flags.
+### --track-renames ###
-`--exclude-from` followed by `-` reads filter rules from standard input.
+By default, rclone doesn't keep track of renamed files, so if you
+rename a file locally then sync it to a remote, rclone will delete the
+old file on the remote and upload a new copy.
-### `--include` - Include files matching pattern
+An rclone sync with `--track-renames` runs like a normal sync, but keeps
+track of objects which exist in the destination but not in the source
+(which would normally be deleted), and which objects exist in the
+source but not the destination (which would normally be transferred).
+These objects are then candidates for renaming.
-Adds a single include rule based on path/file names to an rclone
-command.
+After the sync, rclone matches up the source only and destination only
+objects using the `--track-renames-strategy` specified and either
+renames the destination object or transfers the source and deletes the
+destination object. `--track-renames` is stateless like all of
+rclone's syncs.
-This flag can be repeated. See above for the order filter flags are
-processed in.
+To use this flag the destination must support server-side copy or
+server-side move, and to use a hash based `--track-renames-strategy`
+(the default) the source and the destination must have a compatible
+hash.
-`--include` has no effect when combined with `--files-from` or
-`--files-from-raw` flags.
+If the destination does not support server-side copy or move, rclone
+will fall back to the default behaviour and log an error level message
+to the console.
-`--include` implies `--exclude **` at the end of an rclone internal
-filter list. Therefore if you mix `--include` and `--include-from`
-flags with `--exclude`, `--exclude-from`, `--filter` or `--filter-from`,
-you must use include rules for all the files you want in the include
-statement. For more flexibility use the `--filter-from` flag.
+Encrypted destinations are not currently supported by `--track-renames`
+if `--track-renames-strategy` includes `hash`.
-E.g. `rclone ls remote: --include "*.{png,jpg}"` lists the files on
-`remote:` with suffix `.png` and `.jpg`. All other files are excluded.
+Note that `--track-renames` is incompatible with `--no-traverse` and
+that it uses extra memory to keep track of all the rename candidates.
-E.g. multiple rclone copy commands can be combined with `--include` and a
-pattern-list.
+Note also that `--track-renames` is incompatible with
+`--delete-before` and will select `--delete-after` instead of
+`--delete-during`.
- rclone copy /vol1/A remote:A
- rclone copy /vol1/B remote:B
+### --track-renames-strategy (hash,modtime,leaf,size) ###
-is equivalent to:
+This option changes the file matching criteria for `--track-renames`.
- rclone copy /vol1 remote: --include "{A,B}/**"
+The matching is controlled by a comma separated selection of these tokens:
-E.g. `rclone ls remote:/wheat --include "??[^[:punct:]]*"` lists the
-files `remote:` directory `wheat` (and subdirectories) whose third
-character is not punctuation. This example uses
-an [ASCII character class](https://golang.org/pkg/regexp/syntax/).
+- `modtime` - the modification time of the file - not supported on all backends
+- `hash` - the hash of the file contents - not supported on all backends
+- `leaf` - the name of the file not including its directory name
+- `size` - the size of the file (this is always enabled)
-### `--include-from` - Read include patterns from file
+The default option is `hash`.
-Adds path/file names to an rclone command based on rules in a
-named file. The file contains a list of remarks and pattern rules.
+Using `--track-renames-strategy modtime,leaf` would match files
+based on modification time, the leaf of the file name and the size
+only.
-For an example `include-file.txt`:
+Using `--track-renames-strategy modtime` or `leaf` can enable
+`--track-renames` support for encrypted destinations.
- # a sample include rule file
- *.jpg
- file2.avi
+Note that the `hash` strategy is not supported with encrypted destinations.
-`rclone ls remote: --include-from include-file.txt` lists the files on
-`remote:` with name `file2.avi` or suffix `.jpg`. That is equivalent to
-`rclone ls remote: --include file2.avi --include "*.jpg"`.
+### --delete-(before,during,after) ###
-This flag can be repeated. See above for the order filter flags are
-processed in.
+This option allows you to specify when files on your destination are
+deleted when you sync folders.
-The `--include-from` flag is useful where multiple include filter rules
-are applied to an rclone command.
+Specifying the value `--delete-before` will delete all files present
+on the destination, but not on the source *before* starting the
+transfer of any new or updated files. This uses two passes through the
+file systems, one for the deletions and one for the copies.
-`--include-from` implies `--exclude **` at the end of an rclone internal
-filter list. Therefore if you mix `--include` and `--include-from`
-flags with `--exclude`, `--exclude-from`, `--filter` or `--filter-from`,
-you must use include rules for all the files you want in the include
-statement. For more flexibility use the `--filter-from` flag.
+Specifying `--delete-during` will delete files while checking and
+uploading files. This is the fastest option and uses the least memory.
-`--exclude-from` has no effect when combined with `--files-from` or
-`--files-from-raw` flags.
+Specifying `--delete-after` (the default value) will delay deletion of
+files until all new/updated files have been successfully transferred.
+The files to be deleted are collected in the copy pass then deleted
+after the copy pass has completed successfully. The files to be
+deleted are held in memory so this mode may use more memory. This is
+the safest mode as it will only delete files if there have been no
+errors subsequent to that. If there have been errors before the
+deletions start then you will get the message `not deleting files as
+there were IO errors`.
-`--exclude-from` followed by `-` reads filter rules from standard input.
+### --fast-list ###
-### `--filter` - Add a file-filtering rule
+When doing anything which involves a directory listing (e.g. `sync`,
+`copy`, `ls` - in fact nearly every command), rclone has different
+strategies to choose from.
+
+The basic strategy is to list one directory and processes it before using
+more directory lists to process any subdirectories. This is a mandatory
+backend feature, called `List`, which means it is supported by all backends.
+This strategy uses small amount of memory, and because it can be parallelised
+it is fast for operations involving processing of the list results.
+
+Some backends provide the support for an alternative strategy, where all
+files beneath a directory can be listed in one (or a small number) of
+transactions. Rclone supports this alternative strategy through an optional
+backend feature called [`ListR`](https://rclone.org/overview/#listr). You can see in the storage
+system overview documentation's [optional features](https://rclone.org/overview/#optional-features)
+section which backends it is enabled for (these tend to be the bucket-based
+ones, e.g. S3, B2, GCS, Swift). This strategy requires fewer transactions
+for highly recursive operations, which is important on backends where this
+is charged or heavily rate limited. It may be faster (due to fewer transactions)
+or slower (because it can't be parallelized) depending on different parameters,
+and may require more memory if rclone has to keep the whole listing in memory.
+
+Which listing strategy rclone picks for a given operation is complicated, but
+in general it tries to choose the best possible. It will prefer `ListR` in
+situations where it doesn't need to store the listed files in memory, e.g.
+for unlimited recursive `ls` command variants. In other situations it will
+prefer `List`, e.g. for `sync` and `copy`, where it needs to keep the listed
+files in memory, and is performing operations on them where parallelization
+may be a huge advantage.
+
+Rclone is not able to take all relevant parameters into account for deciding
+the best strategy, and therefore allows you to influence the choice in two ways:
+You can stop rclone from using `ListR` by disabling the feature, using the
+[--disable](#disable-feature-feature) option (`--disable ListR`), or you can
+allow rclone to use `ListR` where it would normally choose not to do so due to
+higher memory usage, using the `--fast-list` option. Rclone should always
+produce identical results either way. Using `--disable ListR` or `--fast-list`
+on a remote which doesn't support `ListR` does nothing, rclone will just ignore
+it.
-Specifies path/file names to an rclone command, based on a single
-include or exclude rule, in `+` or `-` format.
+A rule of thumb is that if you pay for transactions and can fit your entire
+sync listing into memory, then `--fast-list` is recommended. If you have a
+very big sync to do, then don't use `--fast-list`, otherwise you will run out
+of memory. Run some tests and compare before you decide, and if in doubt then
+just leave the default, let rclone decide, i.e. not use `--fast-list`.
-This flag can be repeated. See above for the order filter flags are
-processed in.
+### --timeout=TIME ###
-`--filter +` differs from `--include`. In the case of `--include` rclone
-implies an `--exclude *` rule which it adds to the bottom of the internal rule
-list. `--filter...+` does not imply
-that rule.
+This sets the IO idle timeout. If a transfer has started but then
+becomes idle for this long it is considered broken and disconnected.
-`--filter` has no effect when combined with `--files-from` or
-`--files-from-raw` flags.
+The default is `5m`. Set to `0` to disable.
-`--filter` should not be used with `--include`, `--include-from`,
-`--exclude` or `--exclude-from` flags.
+### --transfers=N ###
-E.g. `rclone ls remote: --filter "- *.bak"` excludes all `.bak` files
-from a list of `remote:`.
+The number of file transfers to run in parallel. It can sometimes be
+useful to set this to a smaller number if the remote is giving a lot
+of timeouts or bigger if you have lots of bandwidth and a fast remote.
-### `--filter-from` - Read filtering patterns from a file
+The default is to run 4 file transfers in parallel.
-Adds path/file names to an rclone command based on rules in a
-named file. The file contains a list of remarks and pattern rules. Include
-rules start with `+ ` and exclude rules with `- `. `!` clears existing
-rules. Rules are processed in the order they are defined.
+Look at --multi-thread-streams if you would like to control single file transfers.
-This flag can be repeated. See above for the order filter flags are
-processed in.
+### -u, --update ###
-Arrange the order of filter rules with the most restrictive first and
-work down.
+This forces rclone to skip any files which exist on the destination
+and have a modified time that is newer than the source file.
-E.g. for `filter-file.txt`:
+This can be useful in avoiding needless transfers when transferring to
+a remote which doesn't support modification times directly (or when
+using `--use-server-modtime` to avoid extra API calls) as it is more
+accurate than a `--size-only` check and faster than using
+`--checksum`. On such remotes (or when using `--use-server-modtime`)
+the time checked will be the uploaded time.
- # a sample filter rule file
- - secret*.jpg
- + *.jpg
- + *.png
- + file2.avi
- - /dir/Trash/**
- + /dir/**
- # exclude everything else
- - *
+If an existing destination file has a modification time older than the
+source file's, it will be updated if the sizes are different. If the
+sizes are the same, it will be updated if the checksum is different or
+not available.
-`rclone ls remote: --filter-from filter-file.txt` lists the path/files on
-`remote:` including all `jpg` and `png` files, excluding any
-matching `secret*.jpg` and including `file2.avi`. It also includes
-everything in the directory `dir` at the root of `remote`, except
-`remote:dir/Trash` which it excludes. Everything else is excluded.
+If an existing destination file has a modification time equal (within
+the computed modify window) to the source file's, it will be updated
+if the sizes are different. The checksum will not be checked in this
+case unless the `--checksum` flag is provided.
+In all other cases the file will not be updated.
-E.g. for an alternative `filter-file.txt`:
+Consider using the `--modify-window` flag to compensate for time skews
+between the source and the backend, for backends that do not support
+mod times, and instead use uploaded times. However, if the backend
+does not support checksums, note that syncing or copying within the
+time skew window may still result in additional transfers for safety.
- - secret*.jpg
- + *.jpg
- + *.png
- + file2.avi
- - *
+### --use-mmap ###
-Files `file1.jpg`, `file3.png` and `file2.avi` are listed whilst
-`secret17.jpg` and files without the suffix .jpg` or `.png` are excluded.
+If this flag is set then rclone will use anonymous memory allocated by
+mmap on Unix based platforms and VirtualAlloc on Windows for its
+transfer buffers (size controlled by `--buffer-size`). Memory
+allocated like this does not go on the Go heap and can be returned to
+the OS immediately when it is finished with.
-E.g. for an alternative `filter-file.txt`:
+If this flag is not set then rclone will allocate and free the buffers
+using the Go memory allocator which may use more memory as memory
+pages are returned less aggressively to the OS.
- + *.jpg
- + *.gif
- !
- + 42.doc
- - *
+It is possible this does not work well on all platforms so it is
+disabled by default; in the future it may be enabled by default.
-Only file 42.doc is listed. Prior rules are cleared by the `!`.
+### --use-server-modtime ###
-### `--files-from` - Read list of source-file names
+Some object-store backends (e.g, Swift, S3) do not preserve file modification
+times (modtime). On these backends, rclone stores the original modtime as
+additional metadata on the object. By default it will make an API call to
+retrieve the metadata when the modtime is needed by an operation.
-Adds path/files to an rclone command from a list in a named file.
-Rclone processes the path/file names in the order of the list, and
-no others.
+Use this flag to disable the extra API call and rely instead on the server's
+modified time. In cases such as a local to remote sync using `--update`,
+knowing the local file is newer than the time it was last uploaded to the
+remote is sufficient. In those cases, this flag can speed up the process and
+reduce the number of API calls necessary.
-Other filter flags (`--include`, `--include-from`, `--exclude`,
-`--exclude-from`, `--filter` and `--filter-from`) are ignored when
-`--files-from` is used.
+Using this flag on a sync operation without also using `--update` would cause
+all files modified at any time other than the last upload time to be uploaded
+again, which is probably not what you want.
-`--files-from` expects a list of files as its input. Leading or
-trailing whitespace is stripped from the input lines. Lines starting
-with `#` or `;` are ignored.
+### -v, -vv, --verbose ###
-Rclone commands with a `--files-from` flag traverse the remote,
-treating the names in `--files-from` as a set of filters.
+With `-v` rclone will tell you about each file that is transferred and
+a small number of significant events.
-If the `--no-traverse` and `--files-from` flags are used together
-an rclone command does not traverse the remote. Instead it addresses
-each path/file named in the file individually. For each path/file name, that
-requires typically 1 API call. This can be efficient for a short `--files-from`
-list and a remote containing many files.
-
-Rclone commands do not error if any names in the `--files-from` file are
-missing from the source remote.
+With `-vv` rclone will become very verbose telling you about every
+file it considers and transfers. Please send bug reports with a log
+with this setting.
-The `--files-from` flag can be repeated in a single rclone command to
-read path/file names from more than one file. The files are read from left
-to right along the command line.
+When setting verbosity as an environment variable, use
+`RCLONE_VERBOSE=1` or `RCLONE_VERBOSE=2` for `-v` and `-vv` respectively.
-Paths within the `--files-from` file are interpreted as starting
-with the root specified in the rclone command. Leading `/` separators are
-ignored. See [--files-from-raw](#files-from-raw-read-list-of-source-file-names-without-any-processing) if
-you need the input to be processed in a raw manner.
+### -V, --version ###
-E.g. for a file `files-from.txt`:
+Prints the version number
- # comment
- file1.jpg
- subdir/file2.jpg
+SSL/TLS options
+---------------
-`rclone copy --files-from files-from.txt /home/me/pics remote:pics`
-copies the following, if they exist, and only those files.
+The outgoing SSL/TLS connections rclone makes can be controlled with
+these options. For example this can be very useful with the HTTP or
+WebDAV backends. Rclone HTTP servers have their own set of
+configuration for SSL/TLS which you can find in their documentation.
- /home/me/pics/file1.jpg → remote:pics/file1.jpg
- /home/me/pics/subdir/file2.jpg → remote:pics/subdir/file2.jpg
+### --ca-cert stringArray
-E.g. to copy the following files referenced by their absolute paths:
+This loads the PEM encoded certificate authority certificates and uses
+it to verify the certificates of the servers rclone connects to.
- /home/user1/42
- /home/user1/dir/ford
- /home/user2/prefect
+If you have generated certificates signed with a local CA then you
+will need this flag to connect to servers using those certificates.
-First find a common subdirectory - in this case `/home`
-and put the remaining files in `files-from.txt` with or without
-leading `/`, e.g.
+### --client-cert string
- user1/42
- user1/dir/ford
- user2/prefect
+This loads the PEM encoded client side certificate.
-Then copy these to a remote:
+This is used for [mutual TLS authentication](https://en.wikipedia.org/wiki/Mutual_authentication).
- rclone copy --files-from files-from.txt /home remote:backup
+The `--client-key` flag is required too when using this.
-The three files are transferred as follows:
+### --client-key string
- /home/user1/42 → remote:backup/user1/important
- /home/user1/dir/ford → remote:backup/user1/dir/file
- /home/user2/prefect → remote:backup/user2/stuff
+This loads the PEM encoded client side private key used for mutual TLS
+authentication. Used in conjunction with `--client-cert`.
-Alternatively if `/` is chosen as root `files-from.txt` will be:
+### --no-check-certificate=true/false ###
- /home/user1/42
- /home/user1/dir/ford
- /home/user2/prefect
+`--no-check-certificate` controls whether a client verifies the
+server's certificate chain and host name.
+If `--no-check-certificate` is true, TLS accepts any certificate
+presented by the server and any host name in that certificate.
+In this mode, TLS is susceptible to man-in-the-middle attacks.
-The copy command will be:
+This option defaults to `false`.
- rclone copy --files-from files-from.txt / remote:backup
+**This should be used only for testing.**
-Then there will be an extra `home` directory on the remote:
+Configuration Encryption
+------------------------
+Your configuration file contains information for logging in to
+your cloud services. This means that you should keep your
+`rclone.conf` file in a secure location.
- /home/user1/42 → remote:backup/home/user1/42
- /home/user1/dir/ford → remote:backup/home/user1/dir/ford
- /home/user2/prefect → remote:backup/home/user2/prefect
+If you are in an environment where that isn't possible, you can
+add a password to your configuration. This means that you will
+have to supply the password every time you start rclone.
-### `--files-from-raw` - Read list of source-file names without any processing
+To add a password to your rclone configuration, execute `rclone config`.
-This flag is the same as `--files-from` except that input is read in a
-raw manner. Lines with leading / trailing whitespace, and lines starting
-with `;` or `#` are read without any processing. [rclone lsf](https://rclone.org/commands/rclone_lsf/) has
-a compatible format that can be used to export file lists from remotes for
-input to `--files-from-raw`.
+```
+>rclone config
+Current remotes:
-### `--ignore-case` - make searches case insensitive
+e) Edit existing remote
+n) New remote
+d) Delete remote
+s) Set configuration password
+q) Quit config
+e/n/d/s/q>
+```
-By default, rclone filter patterns are case sensitive. The `--ignore-case`
-flag makes all of the filters patterns on the command line case
-insensitive.
+Go into `s`, Set configuration password:
+```
+e/n/d/s/q> s
+Your configuration is not encrypted.
+If you add a password, you will protect your login information to cloud services.
+a) Add Password
+q) Quit to main menu
+a/q> a
+Enter NEW configuration password:
+password:
+Confirm NEW password:
+password:
+Password set
+Your configuration is encrypted.
+c) Change Password
+u) Unencrypt configuration
+q) Quit to main menu
+c/u/q>
+```
-E.g. `--include "zaphod.txt"` does not match a file `Zaphod.txt`. With
-`--ignore-case` a match is made.
+Your configuration is now encrypted, and every time you start rclone
+you will have to supply the password. See below for details.
+In the same menu, you can change the password or completely remove
+encryption from your configuration.
-## Quoting shell metacharacters
+There is no way to recover the configuration if you lose your password.
-Rclone commands with filter patterns containing shell metacharacters may
-not as work as expected in your shell and may require quoting.
+rclone uses [nacl secretbox](https://godoc.org/golang.org/x/crypto/nacl/secretbox)
+which in turn uses XSalsa20 and Poly1305 to encrypt and authenticate
+your configuration with secret-key cryptography.
+The password is SHA-256 hashed, which produces the key for secretbox.
+The hashed password is not stored.
-E.g. linux, OSX (`*` metacharacter)
+While this provides very good security, we do not recommend storing
+your encrypted rclone configuration in public if it contains sensitive
+information, maybe except if you use a very strong password.
- * `--include \*.jpg`
- * `--include '*.jpg'`
- * `--include='*.jpg'`
+If it is safe in your environment, you can set the `RCLONE_CONFIG_PASS`
+environment variable to contain your password, in which case it will be
+used for decrypting the configuration.
-Microsoft Windows expansion is done by the command, not shell, so
-`--include *.jpg` does not require quoting.
+You can set this for a session from a script. For unix like systems
+save this to a file called `set-rclone-password`:
-If the rclone error
-`Command .... needs .... arguments maximum: you provided .... non flag arguments:`
-is encountered, the cause is commonly spaces within the name of a
-remote or flag value. The fix then is to quote values containing spaces.
+```
+#!/bin/echo Source this file don't run it
-## Other filters
+read -s RCLONE_CONFIG_PASS
+export RCLONE_CONFIG_PASS
+```
-### `--min-size` - Don't transfer any file smaller than this
+Then source the file when you want to use it. From the shell you
+would do `source set-rclone-password`. It will then ask you for the
+password and set it in the environment variable.
-Controls the minimum size file within the scope of an rclone command.
-Default units are `KiB` but abbreviations `K`, `M`, `G`, `T` or `P` are valid.
+An alternate means of supplying the password is to provide a script
+which will retrieve the password and print on standard output. This
+script should have a fully specified path name and not rely on any
+environment variables. The script is supplied either via
+`--password-command="..."` command line argument or via the
+`RCLONE_PASSWORD_COMMAND` environment variable.
-E.g. `rclone ls remote: --min-size 50k` lists files on `remote:` of 50 KiB
-size or larger.
+One useful example of this is using the `passwordstore` application
+to retrieve the password:
-See [the size option docs](https://rclone.org/docs/#size-option) for more info.
+```
+export RCLONE_PASSWORD_COMMAND="pass rclone/config"
+```
-### `--max-size` - Don't transfer any file larger than this
+If the `passwordstore` password manager holds the password for the
+rclone configuration, using the script method means the password
+is primarily protected by the `passwordstore` system, and is never
+embedded in the clear in scripts, nor available for examination
+using the standard commands available. It is quite possible with
+long running rclone sessions for copies of passwords to be innocently
+captured in log files or terminal scroll buffers, etc. Using the
+script method of supplying the password enhances the security of
+the config password considerably.
-Controls the maximum size file within the scope of an rclone command.
-Default units are `KiB` but abbreviations `K`, `M`, `G`, `T` or `P` are valid.
+If you are running rclone inside a script, unless you are using the
+`--password-command` method, you might want to disable
+password prompts. To do that, pass the parameter
+`--ask-password=false` to rclone. This will make rclone fail instead
+of asking for a password if `RCLONE_CONFIG_PASS` doesn't contain
+a valid password, and `--password-command` has not been supplied.
-E.g. `rclone ls remote: --max-size 1G` lists files on `remote:` of 1 GiB
-size or smaller.
+Whenever running commands that may be affected by options in a
+configuration file, rclone will look for an existing file according
+to the rules described [above](#config-config-file), and load any it
+finds. If an encrypted file is found, this includes decrypting it,
+with the possible consequence of a password prompt. When executing
+a command line that you know are not actually using anything from such
+a configuration file, you can avoid it being loaded by overriding the
+location, e.g. with one of the documented special values for
+memory-only configuration. Since only backend options can be stored
+in configuration files, this is normally unnecessary for commands
+that do not operate on backends, e.g. `genautocomplete`. However,
+it will be relevant for commands that do operate on backends in
+general, but are used without referencing a stored remote, e.g.
+listing local filesystem paths, or
+[connection strings](#connection-strings): `rclone --config="" ls .`
-See [the size option docs](https://rclone.org/docs/#size-option) for more info.
+Developer options
+-----------------
-### `--max-age` - Don't transfer any file older than this
+These options are useful when developing or debugging rclone. There
+are also some more remote specific options which aren't documented
+here which are used for testing. These start with remote name e.g.
+`--drive-test-option` - see the docs for the remote in question.
-Controls the maximum age of files within the scope of an rclone command.
+### --cpuprofile=FILE ###
-`--max-age` applies only to files and not to directories.
+Write CPU profile to file. This can be analysed with `go tool pprof`.
-E.g. `rclone ls remote: --max-age 2d` lists files on `remote:` of 2 days
-old or less.
+#### --dump flag,flag,flag ####
-See [the time option docs](https://rclone.org/docs/#time-option) for valid formats.
+The `--dump` flag takes a comma separated list of flags to dump info
+about.
-### `--min-age` - Don't transfer any file younger than this
+Note that some headers including `Accept-Encoding` as shown may not
+be correct in the request and the response may not show `Content-Encoding`
+if the go standard libraries auto gzip encoding was in effect. In this case
+the body of the request will be gunzipped before showing it.
-Controls the minimum age of files within the scope of an rclone command.
-(see `--max-age` for valid formats)
+The available flags are:
-`--min-age` applies only to files and not to directories.
+#### --dump headers ####
-E.g. `rclone ls remote: --min-age 2d` lists files on `remote:` of 2 days
-old or more.
+Dump HTTP headers with `Authorization:` lines removed. May still
+contain sensitive info. Can be very verbose. Useful for debugging
+only.
-See [the time option docs](https://rclone.org/docs/#time-option) for valid formats.
+Use `--dump auth` if you do want the `Authorization:` headers.
-## Other flags
+#### --dump bodies ####
-### `--delete-excluded` - Delete files on dest excluded from sync
+Dump HTTP headers and bodies - may contain sensitive info. Can be
+very verbose. Useful for debugging only.
-**Important** this flag is dangerous to your data - use with `--dry-run`
-and `-v` first.
+Note that the bodies are buffered in memory so don't use this for
+enormous files.
-In conjunction with `rclone sync`, `--delete-excluded` deletes any files
-on the destination which are excluded from the command.
+#### --dump requests ####
-E.g. the scope of `rclone sync --interactive A: B:` can be restricted:
+Like `--dump bodies` but dumps the request bodies and the response
+headers. Useful for debugging download problems.
- rclone --min-size 50k --delete-excluded sync A: B:
+#### --dump responses ####
-All files on `B:` which are less than 50 KiB are deleted
-because they are excluded from the rclone sync command.
+Like `--dump bodies` but dumps the response bodies and the request
+headers. Useful for debugging upload problems.
-### `--dump filters` - dump the filters to the output
+#### --dump auth ####
-Dumps the defined filters to standard output in regular expression
-format.
+Dump HTTP headers - will contain sensitive info such as
+`Authorization:` headers - use `--dump headers` to dump without
+`Authorization:` headers. Can be very verbose. Useful for debugging
+only.
-Useful for debugging.
+#### --dump filters ####
-## Exclude directory based on a file
+Dump the filters to the output. Useful to see exactly what include
+and exclude options are filtering on.
-The `--exclude-if-present` flag controls whether a directory is
-within the scope of an rclone command based on the presence of a
-named file within it. The flag can be repeated to check for
-multiple file names, presence of any of them will exclude the
-directory.
+#### --dump goroutines ####
-This flag has a priority over other filter flags.
+This dumps a list of the running go-routines at the end of the command
+to standard output.
-E.g. for the following directory structure:
+#### --dump openfiles ####
- dir1/file1
- dir1/dir2/file2
- dir1/dir2/dir3/file3
- dir1/dir2/dir3/.ignore
+This dumps a list of the open files at the end of the command. It
+uses the `lsof` command to do that so you'll need that installed to
+use it.
-The command `rclone ls --exclude-if-present .ignore dir1` does
-not list `dir3`, `file3` or `.ignore`.
+#### --dump mapper ####
-## Metadata filters {#metadata}
+This shows the JSON blobs being sent to the program supplied with
+`--metadata-mapper` and received from it. It can be useful for
+debugging the metadata mapper interface.
-The metadata filters work in a very similar way to the normal file
-name filters, except they match [metadata](https://rclone.org/docs/#metadata) on the
-object.
+### --memprofile=FILE ###
-The metadata should be specified as `key=value` patterns. This may be
-wildcarded using the normal [filter patterns](#patterns) or [regular
-expressions](#regexp).
+Write memory profile to file. This can be analysed with `go tool pprof`.
-For example if you wished to list only local files with a mode of
-`100664` you could do that with:
+Filtering
+---------
- rclone lsf -M --files-only --metadata-include "mode=100664" .
+For the filtering options
-Or if you wished to show files with an `atime`, `mtime` or `btime` at a given date:
+ * `--delete-excluded`
+ * `--filter`
+ * `--filter-from`
+ * `--exclude`
+ * `--exclude-from`
+ * `--exclude-if-present`
+ * `--include`
+ * `--include-from`
+ * `--files-from`
+ * `--files-from-raw`
+ * `--min-size`
+ * `--max-size`
+ * `--min-age`
+ * `--max-age`
+ * `--dump filters`
+ * `--metadata-include`
+ * `--metadata-include-from`
+ * `--metadata-exclude`
+ * `--metadata-exclude-from`
+ * `--metadata-filter`
+ * `--metadata-filter-from`
- rclone lsf -M --files-only --metadata-include "[abm]time=2022-12-16*" .
+See the [filtering section](https://rclone.org/filtering/).
-Like file filtering, metadata filtering only applies to files not to
-directories.
+Remote control
+--------------
-The filters can be applied using these flags.
+For the remote control options and for instructions on how to remote control rclone
-- `--metadata-include` - Include metadatas matching pattern
-- `--metadata-include-from` - Read metadata include patterns from file (use - to read from stdin)
-- `--metadata-exclude` - Exclude metadatas matching pattern
-- `--metadata-exclude-from` - Read metadata exclude patterns from file (use - to read from stdin)
-- `--metadata-filter` - Add a metadata filtering rule
-- `--metadata-filter-from` - Read metadata filtering patterns from a file (use - to read from stdin)
+ * `--rc`
+ * and anything starting with `--rc-`
-Each flag can be repeated. See the section on [how filter rules are
-applied](#how-filter-rules-work) for more details - these flags work
-in an identical way to the file name filtering flags, but instead of
-file name patterns have metadata patterns.
+See [the remote control section](https://rclone.org/rc/).
+Logging
+-------
-## Common pitfalls
+rclone has 4 levels of logging, `ERROR`, `NOTICE`, `INFO` and `DEBUG`.
-The most frequent filter support issues on
-the [rclone forum](https://forum.rclone.org/) are:
+By default, rclone logs to standard error. This means you can redirect
+standard error and still see the normal output of rclone commands (e.g.
+`rclone ls`).
-* Not using paths relative to the root of the remote
-* Not using `/` to match from the root of a remote
-* Not using `**` to match the contents of a directory
+By default, rclone will produce `Error` and `Notice` level messages.
-# GUI (Experimental)
+If you use the `-q` flag, rclone will only produce `Error` messages.
-Rclone can serve a web based GUI (graphical user interface). This is
-somewhat experimental at the moment so things may be subject to
-change.
+If you use the `-v` flag, rclone will produce `Error`, `Notice` and
+`Info` messages.
-Run this command in a terminal and rclone will download and then
-display the GUI in a web browser.
+If you use the `-vv` flag, rclone will produce `Error`, `Notice`,
+`Info` and `Debug` messages.
-```
-rclone rcd --rc-web-gui
-```
+You can also control the log levels with the `--log-level` flag.
-This will produce logs like this and rclone needs to continue to run to serve the GUI:
+If you use the `--log-file=FILE` option, rclone will redirect `Error`,
+`Info` and `Debug` messages along with standard error to FILE.
-```
-2019/08/25 11:40:14 NOTICE: A new release for gui is present at https://github.com/rclone/rclone-webui-react/releases/download/v0.0.6/currentbuild.zip
-2019/08/25 11:40:14 NOTICE: Downloading webgui binary. Please wait. [Size: 3813937, Path : /home/USER/.cache/rclone/webgui/v0.0.6.zip]
-2019/08/25 11:40:16 NOTICE: Unzipping
-2019/08/25 11:40:16 NOTICE: Serving remote control on http://127.0.0.1:5572/
-```
+If you use the `--syslog` flag then rclone will log to syslog and the
+`--syslog-facility` control which facility it uses.
-This assumes you are running rclone locally on your machine. It is
-possible to separate the rclone and the GUI - see below for details.
+Rclone prefixes all log messages with their level in capitals, e.g. INFO
+which makes it easy to grep the log file for different kinds of
+information.
-If you wish to check for updates then you can add `--rc-web-gui-update`
-to the command line.
+Exit Code
+---------
-If you find your GUI broken, you may force it to update by add `--rc-web-gui-force-update`.
+If any errors occur during the command execution, rclone will exit with a
+non-zero exit code. This allows scripts to detect when rclone
+operations have failed.
-By default, rclone will open your browser. Add `--rc-web-gui-no-open-browser`
-to disable this feature.
+During the startup phase, rclone will exit immediately if an error is
+detected in the configuration. There will always be a log message
+immediately before exiting.
-## Using the GUI
+When rclone is running it will accumulate errors as it goes along, and
+only exit with a non-zero exit code if (after retries) there were
+still failed transfers. For every error counted there will be a high
+priority log message (visible with `-q`) showing the message and
+which file caused the problem. A high priority message is also shown
+when starting a retry so the user can see that any previous error
+messages may not be valid after the retry. If rclone has done a retry
+it will log a high priority message if the retry was successful.
-Once the GUI opens, you will be looking at the dashboard which has an overall overview.
+### List of exit codes ###
+ * `0` - success
+ * `1` - Syntax or usage error
+ * `2` - Error not otherwise categorised
+ * `3` - Directory not found
+ * `4` - File not found
+ * `5` - Temporary error (one that more retries might fix) (Retry errors)
+ * `6` - Less serious errors (like 461 errors from dropbox) (NoRetry errors)
+ * `7` - Fatal error (one that more retries won't fix, like account suspended) (Fatal errors)
+ * `8` - Transfer exceeded - limit set by --max-transfer reached
+ * `9` - Operation successful, but no files transferred
+ * `10` - Duration exceeded - limit set by --max-duration reached
-On the left hand side you will see a series of view buttons you can click on:
+Environment Variables
+---------------------
-- Dashboard - main overview
-- Configs - examine and create new configurations
-- Explorer - view, download and upload files to the cloud storage systems
-- Backend - view or alter the backend config
-- Log out
+Rclone can be configured entirely using environment variables. These
+can be used to set defaults for options or config file entries.
-(More docs and walkthrough video to come!)
+### Options ###
-## How it works
+Every option in rclone can have its default set by environment
+variable.
-When you run the `rclone rcd --rc-web-gui` this is what happens
+To find the name of the environment variable, first, take the long
+option name, strip the leading `--`, change `-` to `_`, make
+upper case and prepend `RCLONE_`.
-- Rclone starts but only runs the remote control API ("rc").
-- The API is bound to localhost with an auto-generated username and password.
-- If the API bundle is missing then rclone will download it.
-- rclone will start serving the files from the API bundle over the same port as the API
-- rclone will open the browser with a `login_token` so it can log straight in.
+For example, to always set `--stats 5s`, set the environment variable
+`RCLONE_STATS=5s`. If you set stats on the command line this will
+override the environment variable setting.
-## Advanced use
+Or to always use the trash in drive `--drive-use-trash`, set
+`RCLONE_DRIVE_USE_TRASH=true`.
-The `rclone rcd` may use any of the [flags documented on the rc page](https://rclone.org/rc/#supported-parameters).
+Verbosity is slightly different, the environment variable
+equivalent of `--verbose` or `-v` is `RCLONE_VERBOSE=1`,
+or for `-vv`, `RCLONE_VERBOSE=2`.
-The flag `--rc-web-gui` is shorthand for
+The same parser is used for the options and the environment variables
+so they take exactly the same form.
-- Download the web GUI if necessary
-- Check we are using some authentication
-- `--rc-user gui`
-- `--rc-pass `
-- `--rc-serve`
+The options set by environment variables can be seen with the `-vv` flag, e.g. `rclone version -vv`.
-These flags can be overridden as desired.
+### Config file ###
-See also the [rclone rcd documentation](https://rclone.org/commands/rclone_rcd/).
+You can set defaults for values in the config file on an individual
+remote basis. The names of the config items are documented in the page
+for each backend.
-### Example: Running a public GUI
+To find the name of the environment variable, you need to set, take
+`RCLONE_CONFIG_` + name of remote + `_` + name of config file option
+and make it all uppercase.
+Note one implication here is the remote's name must be
+convertible into a valid environment variable name,
+so it can only contain letters, digits, or the `_` (underscore) character.
-For example the GUI could be served on a public port over SSL using an htpasswd file using the following flags:
+For example, to configure an S3 remote named `mys3:` without a config
+file (using unix ways of setting environment variables):
-- `--rc-web-gui`
-- `--rc-addr :443`
-- `--rc-htpasswd /path/to/htpasswd`
-- `--rc-cert /path/to/ssl.crt`
-- `--rc-key /path/to/ssl.key`
+```
+$ export RCLONE_CONFIG_MYS3_TYPE=s3
+$ export RCLONE_CONFIG_MYS3_ACCESS_KEY_ID=XXX
+$ export RCLONE_CONFIG_MYS3_SECRET_ACCESS_KEY=XXX
+$ rclone lsd mys3:
+ -1 2016-09-21 12:54:21 -1 my-bucket
+$ rclone listremotes | grep mys3
+mys3:
+```
-### Example: Running a GUI behind a proxy
+Note that if you want to create a remote using environment variables
+you must create the `..._TYPE` variable as above.
-If you want to run the GUI behind a proxy at `/rclone` you could use these flags:
+Note that the name of a remote created using environment variable is
+case insensitive, in contrast to regular remotes stored in config
+file as documented [above](#valid-remote-names).
+You must write the name in uppercase in the environment variable, but
+as seen from example above it will be listed and can be accessed in
+lowercase, while you can also refer to the same remote in uppercase:
+```
+$ rclone lsd mys3:
+ -1 2016-09-21 12:54:21 -1 my-bucket
+$ rclone lsd MYS3:
+ -1 2016-09-21 12:54:21 -1 my-bucket
+```
-- `--rc-web-gui`
-- `--rc-baseurl rclone`
-- `--rc-htpasswd /path/to/htpasswd`
-Or instead of htpasswd if you just want a single user and password:
+Note that you can only set the options of the immediate backend,
+so RCLONE_CONFIG_MYS3CRYPT_ACCESS_KEY_ID has no effect, if myS3Crypt is
+a crypt remote based on an S3 remote. However RCLONE_S3_ACCESS_KEY_ID will
+set the access key of all remotes using S3, including myS3Crypt.
-- `--rc-user me`
-- `--rc-pass mypassword`
+Note also that now rclone has [connection strings](#connection-strings),
+it is probably easier to use those instead which makes the above example
-## Project
+ rclone lsd :s3,access_key_id=XXX,secret_access_key=XXX:
-The GUI is being developed in the: [rclone/rclone-webui-react repository](https://github.com/rclone/rclone-webui-react).
+### Precedence
-Bug reports and contributions are very welcome :-)
+The various different methods of backend configuration are read in
+this order and the first one with a value is used.
-If you have questions then please ask them on the [rclone forum](https://forum.rclone.org/).
+- Parameters in connection strings, e.g. `myRemote,skip_links:`
+- Flag values as supplied on the command line, e.g. `--skip-links`
+- Remote specific environment vars, e.g. `RCLONE_CONFIG_MYREMOTE_SKIP_LINKS` (see above).
+- Backend-specific environment vars, e.g. `RCLONE_LOCAL_SKIP_LINKS`.
+- Backend generic environment vars, e.g. `RCLONE_SKIP_LINKS`.
+- Config file, e.g. `skip_links = true`.
+- Default values, e.g. `false` - these can't be changed.
-# Remote controlling rclone with its API
+So if both `--skip-links` is supplied on the command line and an
+environment variable `RCLONE_LOCAL_SKIP_LINKS` is set, the command line
+flag will take preference.
-If rclone is run with the `--rc` flag then it starts an HTTP server
-which can be used to remote control rclone using its API.
+The backend configurations set by environment variables can be seen with the `-vv` flag, e.g. `rclone about myRemote: -vv`.
-You can either use the [rc](#api-rc) command to access the API
-or [use HTTP directly](#api-http).
+For non backend configuration the order is as follows:
-If you just want to run a remote control then see the [rcd](https://rclone.org/commands/rclone_rcd/) command.
+- Flag values as supplied on the command line, e.g. `--stats 5s`.
+- Environment vars, e.g. `RCLONE_STATS=5s`.
+- Default values, e.g. `1m` - these can't be changed.
-## Supported parameters
+### Other environment variables ###
-### --rc
+- `RCLONE_CONFIG_PASS` set to contain your config file password (see [Configuration Encryption](#configuration-encryption) section)
+- `HTTP_PROXY`, `HTTPS_PROXY` and `NO_PROXY` (or the lowercase versions thereof).
+ - `HTTPS_PROXY` takes precedence over `HTTP_PROXY` for https requests.
+ - The environment values may be either a complete URL or a "host[:port]" for, in which case the "http" scheme is assumed.
+- `USER` and `LOGNAME` values are used as fallbacks for current username. The primary method for looking up username is OS-specific: Windows API on Windows, real user ID in /etc/passwd on Unix systems. In the documentation the current username is simply referred to as `$USER`.
+- `RCLONE_CONFIG_DIR` - rclone **sets** this variable for use in config files and sub processes to point to the directory holding the config file.
-Flag to start the http server listen on remote requests
-
-### --rc-addr=IP
+The options set by environment variables can be seen with the `-vv` and `--log-level=DEBUG` flags, e.g. `rclone version -vv`.
-IPaddress:Port or :Port to bind server to. (default "localhost:5572")
+# Configuring rclone on a remote / headless machine #
-### --rc-cert=KEY
-SSL PEM key (concatenation of certificate and CA certificate)
+Some of the configurations (those involving oauth2) require an
+Internet connected web browser.
-### --rc-client-ca=PATH
-Client certificate authority to verify clients with
+If you are trying to set rclone up on a remote or headless box with no
+browser available on it (e.g. a NAS or a server in a datacenter) then
+you will need to use an alternative means of configuration. There are
+two ways of doing it, described below.
-### --rc-htpasswd=PATH
+## Configuring using rclone authorize ##
-htpasswd file - if not provided no authentication is done
+On the headless box run `rclone` config but answer `N` to the `Use web browser
+to automatically authenticate?` question.
-### --rc-key=PATH
+```
+...
+Remote config
+Use web browser to automatically authenticate rclone with remote?
+ * Say Y if the machine running rclone has a web browser you can use
+ * Say N if running rclone on a (remote) machine without web browser access
+If not sure try Y. If Y failed, try N.
+y) Yes (default)
+n) No
+y/n> n
+For this to work, you will need rclone available on a machine that has
+a web browser available.
-SSL PEM Private key
+For more help and alternate methods see: https://rclone.org/remote_setup/
-### --rc-max-header-bytes=VALUE
+Execute the following on the machine with the web browser (same rclone
+version recommended):
-Maximum size of request header (default 4096)
+ rclone authorize "amazon cloud drive"
-### --rc-min-tls-version=VALUE
+Then paste the result below:
+result>
+```
-The minimum TLS version that is acceptable. Valid values are "tls1.0",
-"tls1.1", "tls1.2" and "tls1.3" (default "tls1.0").
+Then on your main desktop machine
-### --rc-user=VALUE
+```
+rclone authorize "amazon cloud drive"
+If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth
+Log in and authorize rclone for access
+Waiting for code...
+Got code
+Paste the following into your remote machine --->
+SECRET_TOKEN
+<---End paste
+```
-User name for authentication.
+Then back to the headless box, paste in the code
-### --rc-pass=VALUE
+```
+result> SECRET_TOKEN
+--------------------
+[acd12]
+client_id =
+client_secret =
+token = SECRET_TOKEN
+--------------------
+y) Yes this is OK
+e) Edit this remote
+d) Delete this remote
+y/e/d>
+```
-Password for authentication.
+## Configuring by copying the config file ##
-### --rc-realm=VALUE
+Rclone stores all of its config in a single configuration file. This
+can easily be copied to configure a remote rclone.
-Realm for authentication (default "rclone")
+So first configure rclone on your desktop machine with
-### --rc-server-read-timeout=DURATION
+ rclone config
-Timeout for server reading data (default 1h0m0s)
+to set up the config file.
-### --rc-server-write-timeout=DURATION
+Find the config file by running `rclone config file`, for example
-Timeout for server writing data (default 1h0m0s)
+```
+$ rclone config file
+Configuration file is stored at:
+/home/user/.rclone.conf
+```
-### --rc-serve
+Now transfer it to the remote box (scp, cut paste, ftp, sftp, etc.) and
+place it in the correct place (use `rclone config file` on the remote
+box to find out where).
-Enable the serving of remote objects via the HTTP interface. This
-means objects will be accessible at http://127.0.0.1:5572/ by default,
-so you can browse to http://127.0.0.1:5572/ or http://127.0.0.1:5572/*
-to see a listing of the remotes. Objects may be requested from
-remotes using this syntax http://127.0.0.1:5572/[remote:path]/path/to/object
+## Configuring using SSH Tunnel ##
-Default Off.
+Linux and MacOS users can utilize SSH Tunnel to redirect the headless box port 53682 to local machine by using the following command:
+```
+ssh -L localhost:53682:localhost:53682 username@remote_server
+```
+Then on the headless box run `rclone` config and answer `Y` to the `Use web
+browser to automatically authenticate?` question.
-### --rc-files /path/to/directory
+```
+...
+Remote config
+Use web browser to automatically authenticate rclone with remote?
+ * Say Y if the machine running rclone has a web browser you can use
+ * Say N if running rclone on a (remote) machine without web browser access
+If not sure try Y. If Y failed, try N.
+y) Yes (default)
+n) No
+y/n> y
+```
+Then copy and paste the auth url `http://127.0.0.1:53682/auth?state=xxxxxxxxxxxx` to the browser on your local machine, complete the auth and it is done.
-Path to local files to serve on the HTTP server.
+# Filtering, includes and excludes
-If this is set then rclone will serve the files in that directory. It
-will also open the root in the web browser if specified. This is for
-implementing browser based GUIs for rclone functions.
+Filter flags determine which files rclone `sync`, `move`, `ls`, `lsl`,
+`md5sum`, `sha1sum`, `size`, `delete`, `check` and similar commands
+apply to.
-If `--rc-user` or `--rc-pass` is set then the URL that is opened will
-have the authorization in the URL in the `http://user:pass@localhost/`
-style.
+They are specified in terms of path/file name patterns; path/file
+lists; file age and size, or presence of a file in a directory. Bucket
+based remotes without the concept of directory apply filters to object
+key, age and size in an analogous way.
-Default Off.
+Rclone `purge` does not obey filters.
-### --rc-enable-metrics
+To test filters without risk of damage to data, apply them to `rclone
+ls`, or with the `--dry-run` and `-vv` flags.
-Enable OpenMetrics/Prometheus compatible endpoint at `/metrics`.
+Rclone filter patterns can only be used in filter command line options, not
+in the specification of a remote.
-Default Off.
+E.g. `rclone copy "remote:dir*.jpg" /path/to/dir` does not have a filter effect.
+`rclone copy remote:dir /path/to/dir --include "*.jpg"` does.
-### --rc-web-gui
+**Important** Avoid mixing any two of `--include...`, `--exclude...` or
+`--filter...` flags in an rclone command. The results might not be what
+you expect. Instead use a `--filter...` flag.
-Set this flag to serve the default web gui on the same port as rclone.
+## Patterns for matching path/file names
-Default Off.
+### Pattern syntax {#patterns}
-### --rc-allow-origin
+Here is a formal definition of the pattern syntax,
+[examples](#examples) are below.
-Set the allowed Access-Control-Allow-Origin for rc requests.
+Rclone matching rules follow a glob style:
-Can be used with --rc-web-gui if the rclone is running on different IP than the web-gui.
+ * matches any sequence of non-separator (/) characters
+ ** matches any sequence of characters including / separators
+ ? matches any single non-separator (/) character
+ [ [ ! ] { character-range } ]
+ character class (must be non-empty)
+ { pattern-list }
+ pattern alternatives
+ {{ regexp }}
+ regular expression to match
+ c matches character c (c != *, **, ?, \, [, {, })
+ \c matches reserved character c (c = *, **, ?, \, [, {, }) or character class
-Default is IP address on which rc is running.
+character-range:
-### --rc-web-fetch-url
+ c matches character c (c != \, -, ])
+ \c matches reserved character c (c = \, -, ])
+ lo - hi matches character c for lo <= c <= hi
-Set the URL to fetch the rclone-web-gui files from.
+pattern-list:
-Default https://api.github.com/repos/rclone/rclone-webui-react/releases/latest.
+ pattern { , pattern }
+ comma-separated (without spaces) patterns
-### --rc-web-gui-update
+character classes (see [Go regular expression reference](https://golang.org/pkg/regexp/syntax/)) include:
-Set this flag to check and update rclone-webui-react from the rc-web-fetch-url.
+ Named character classes (e.g. [\d], [^\d], [\D], [^\D])
+ Perl character classes (e.g. \s, \S, \w, \W)
+ ASCII character classes (e.g. [[:alnum:]], [[:alpha:]], [[:punct:]], [[:xdigit:]])
-Default Off.
+regexp for advanced users to insert a regular expression - see [below](#regexp) for more info:
-### --rc-web-gui-force-update
+ Any re2 regular expression not containing `}}`
-Set this flag to force update rclone-webui-react from the rc-web-fetch-url.
+If the filter pattern starts with a `/` then it only matches
+at the top level of the directory tree,
+**relative to the root of the remote** (not necessarily the root
+of the drive). If it does not start with `/` then it is matched
+starting at the **end of the path/file name** but it only matches
+a complete path element - it must match from a `/`
+separator or the beginning of the path/file.
-Default Off.
+ file.jpg - matches "file.jpg"
+ - matches "directory/file.jpg"
+ - doesn't match "afile.jpg"
+ - doesn't match "directory/afile.jpg"
+ /file.jpg - matches "file.jpg" in the root directory of the remote
+ - doesn't match "afile.jpg"
+ - doesn't match "directory/file.jpg"
-### --rc-web-gui-no-open-browser
+The top level of the remote might not be the top level of the drive.
-Set this flag to disable opening browser automatically when using web-gui.
+E.g. for a Microsoft Windows local directory structure
-Default Off.
+ F:
+ ├── bkp
+ ├── data
+ │ ├── excl
+ │ │ ├── 123.jpg
+ │ │ └── 456.jpg
+ │ ├── incl
+ │ │ └── document.pdf
-### --rc-job-expire-duration=DURATION
+To copy the contents of folder `data` into folder `bkp` excluding the contents of subfolder
+`excl`the following command treats `F:\data` and `F:\bkp` as top level for filtering.
-Expire finished async jobs older than DURATION (default 60s).
+`rclone copy F:\data\ F:\bkp\ --exclude=/excl/**`
-### --rc-job-expire-interval=DURATION
+**Important** Use `/` in path/file name patterns and not `\` even if
+running on Microsoft Windows.
-Interval duration to check for expired async jobs (default 10s).
+Simple patterns are case sensitive unless the `--ignore-case` flag is used.
-### --rc-no-auth
+Without `--ignore-case` (default)
-By default rclone will require authorisation to have been set up on
-the rc interface in order to use any methods which access any rclone
-remotes. Eg `operations/list` is denied as it involved creating a
-remote as is `sync/copy`.
+ potato - matches "potato"
+ - doesn't match "POTATO"
-If this is set then no authorisation will be required on the server to
-use these methods. The alternative is to use `--rc-user` and
-`--rc-pass` and use these credentials in the request.
+With `--ignore-case`
-Default Off.
+ potato - matches "potato"
+ - matches "POTATO"
-### --rc-baseurl
+## Using regular expressions in filter patterns {#regexp}
-Prefix for URLs.
+The syntax of filter patterns is glob style matching (like `bash`
+uses) to make things easy for users. However this does not provide
+absolute control over the matching, so for advanced users rclone also
+provides a regular expression syntax.
-Default is root
+The regular expressions used are as defined in the [Go regular
+expression reference](https://golang.org/pkg/regexp/syntax/). Regular
+expressions should be enclosed in `{{` `}}`. They will match only the
+last path segment if the glob doesn't start with `/` or the whole path
+name if it does. Note that rclone does not attempt to parse the
+supplied regular expression, meaning that using any regular expression
+filter will prevent rclone from using [directory filter rules](#directory_filter),
+as it will instead check every path against
+the supplied regular expression(s).
-### --rc-template
+Here is how the `{{regexp}}` is transformed into an full regular
+expression to match the entire path:
-User-specified template.
+ {{regexp}} becomes (^|/)(regexp)$
+ /{{regexp}} becomes ^(regexp)$
-## Accessing the remote control via the rclone rc command {#api-rc}
+Regexp syntax can be mixed with glob syntax, for example
-Rclone itself implements the remote control protocol in its `rclone
-rc` command.
+ *.{{jpe?g}} to match file.jpg, file.jpeg but not file.png
-You can use it like this
+You can also use regexp flags - to set case insensitive, for example
-```
-$ rclone rc rc/noop param1=one param2=two
-{
- "param1": "one",
- "param2": "two"
-}
-```
+ *.{{(?i)jpg}} to match file.jpg, file.JPG but not file.png
-Run `rclone rc` on its own to see the help for the installed remote
-control commands.
+Be careful with wildcards in regular expressions - you don't want them
+to match path separators normally. To match any file name starting
+with `start` and ending with `end` write
-## JSON input
+ {{start[^/]*end\.jpg}}
-`rclone rc` also supports a `--json` flag which can be used to send
-more complicated input parameters.
+Not
-```
-$ rclone rc --json '{ "p1": [1,"2",null,4], "p2": { "a":1, "b":2 } }' rc/noop
-{
- "p1": [
- 1,
- "2",
- null,
- 4
- ],
- "p2": {
- "a": 1,
- "b": 2
- }
-}
-```
+ {{start.*end\.jpg}}
-If the parameter being passed is an object then it can be passed as a
-JSON string rather than using the `--json` flag which simplifies the
-command line.
+Which will match a directory called `start` with a file called
+`end.jpg` in it as the `.*` will match `/` characters.
-```
-rclone rc operations/list fs=/tmp remote=test opt='{"showHash": true}'
-```
+Note that you can use `-vv --dump filters` to show the filter patterns
+in regexp format - rclone implements the glob patterns by transforming
+them into regular expressions.
-Rather than
+## Filter pattern examples {#examples}
-```
-rclone rc operations/list --json '{"fs": "/tmp", "remote": "test", "opt": {"showHash": true}}'
-```
+| Description | Pattern | Matches | Does not match |
+| ----------- |-------- | ------- | -------------- |
+| Wildcard | `*.jpg` | `/file.jpg` | `/file.png` |
+| | | `/dir/file.jpg` | `/dir/file.png` |
+| Rooted | `/*.jpg` | `/file.jpg` | `/file.png` |
+| | | `/file2.jpg` | `/dir/file.jpg` |
+| Alternates | `*.{jpg,png}` | `/file.jpg` | `/file.gif` |
+| | | `/dir/file.png` | `/dir/file.gif` |
+| Path Wildcard | `dir/**` | `/dir/anyfile` | `file.png` |
+| | | `/subdir/dir/subsubdir/anyfile` | `/subdir/file.png` |
+| Any Char | `*.t?t` | `/file.txt` | `/file.qxt` |
+| | | `/dir/file.tzt` | `/dir/file.png` |
+| Range | `*.[a-z]` | `/file.a` | `/file.0` |
+| | | `/dir/file.b` | `/dir/file.1` |
+| Escape | `*.\?\?\?` | `/file.???` | `/file.abc` |
+| | | `/dir/file.???` | `/dir/file.def` |
+| Class | `*.\d\d\d` | `/file.012` | `/file.abc` |
+| | | `/dir/file.345` | `/dir/file.def` |
+| Regexp | `*.{{jpe?g}}` | `/file.jpeg` | `/file.png` |
+| | | `/dir/file.jpg` | `/dir/file.jpeeg` |
+| Rooted Regexp | `/{{.*\.jpe?g}}` | `/file.jpeg` | `/file.png` |
+| | | `/file.jpg` | `/dir/file.jpg` |
-## Special parameters
+## How filter rules are applied to files {#how-filter-rules-work}
-The rc interface supports some special parameters which apply to
-**all** commands. These start with `_` to show they are different.
+Rclone path/file name filters are made up of one or more of the following flags:
-### Running asynchronous jobs with _async = true
+ * `--include`
+ * `--include-from`
+ * `--exclude`
+ * `--exclude-from`
+ * `--filter`
+ * `--filter-from`
-Each rc call is classified as a job and it is assigned its own id. By default
-jobs are executed immediately as they are created or synchronously.
+There can be more than one instance of individual flags.
-If `_async` has a true value when supplied to an rc call then it will
-return immediately with a job id and the task will be run in the
-background. The `job/status` call can be used to get information of
-the background job. The job can be queried for up to 1 minute after
-it has finished.
+Rclone internally uses a combined list of all the include and exclude
+rules. The order in which rules are processed can influence the result
+of the filter.
-It is recommended that potentially long running jobs, e.g. `sync/sync`,
-`sync/copy`, `sync/move`, `operations/purge` are run with the `_async`
-flag to avoid any potential problems with the HTTP request and
-response timing out.
+All flags of the same type are processed together in the order
+above, regardless of what order the different types of flags are
+included on the command line.
-Starting a job with the `_async` flag:
+Multiple instances of the same flag are processed from left
+to right according to their position in the command line.
-```
-$ rclone rc --json '{ "p1": [1,"2",null,4], "p2": { "a":1, "b":2 }, "_async": true }' rc/noop
-{
- "jobid": 2
-}
-```
+To mix up the order of processing includes and excludes use `--filter...`
+flags.
-Query the status to see if the job has finished. For more information
-on the meaning of these return parameters see the `job/status` call.
+Within `--include-from`, `--exclude-from` and `--filter-from` flags
+rules are processed from top to bottom of the referenced file.
-```
-$ rclone rc --json '{ "jobid":2 }' job/status
-{
- "duration": 0.000124163,
- "endTime": "2018-10-27T11:38:07.911245881+01:00",
- "error": "",
- "finished": true,
- "id": 2,
- "output": {
- "_async": true,
- "p1": [
- 1,
- "2",
- null,
- 4
- ],
- "p2": {
- "a": 1,
- "b": 2
- }
- },
- "startTime": "2018-10-27T11:38:07.911121728+01:00",
- "success": true
-}
-```
+If there is an `--include` or `--include-from` flag specified, rclone
+implies a `- **` rule which it adds to the bottom of the internal rule
+list. Specifying a `+` rule with a `--filter...` flag does not imply
+that rule.
-`job/list` can be used to show the running or recently completed jobs
+Each path/file name passed through rclone is matched against the
+combined filter list. At first match to a rule the path/file name
+is included or excluded and no further filter rules are processed for
+that path/file.
-```
-$ rclone rc job/list
-{
- "jobids": [
- 2
- ]
-}
-```
+If rclone does not find a match, after testing against all rules
+(including the implied rule if appropriate), the path/file name
+is included.
-### Setting config flags with _config
+Any path/file included at that stage is processed by the rclone
+command.
-If you wish to set config (the equivalent of the global flags) for the
-duration of an rc call only then pass in the `_config` parameter.
+`--files-from` and `--files-from-raw` flags over-ride and cannot be
+combined with other filter options.
-This should be in the same format as the `config` key returned by
-[options/get](#options-get).
+To see the internal combined rule list, in regular expression form,
+for a command add the `--dump filters` flag. Running an rclone command
+with `--dump filters` and `-vv` flags lists the internal filter elements
+and shows how they are applied to each source path/file. There is not
+currently a means provided to pass regular expression filter options into
+rclone directly though character class filter rules contain character
+classes. [Go regular expression reference](https://golang.org/pkg/regexp/syntax/)
-For example, if you wished to run a sync with the `--checksum`
-parameter, you would pass this parameter in your JSON blob.
+### How filter rules are applied to directories {#directory_filter}
- "_config":{"CheckSum": true}
+Rclone commands are applied to path/file names not
+directories. The entire contents of a directory can be matched
+to a filter by the pattern `directory/*` or recursively by
+`directory/**`.
-If using `rclone rc` this could be passed as
+Directory filter rules are defined with a closing `/` separator.
- rclone rc sync/sync ... _config='{"CheckSum": true}'
+E.g. `/directory/subdirectory/` is an rclone directory filter rule.
-Any config parameters you don't set will inherit the global defaults
-which were set with command line flags or environment variables.
+Rclone commands can use directory filter rules to determine whether they
+recurse into subdirectories. This potentially optimises access to a remote
+by avoiding listing unnecessary directories. Whether optimisation is
+desirable depends on the specific filter rules and source remote content.
-Note that it is possible to set some values as strings or integers -
-see [data types](#data-types) for more info. Here is an example
-setting the equivalent of `--buffer-size` in string or integer format.
+If any [regular expression filters](#regexp) are in use, then no
+directory recursion optimisation is possible, as rclone must check
+every path against the supplied regular expression(s).
- "_config":{"BufferSize": "42M"}
- "_config":{"BufferSize": 44040192}
+Directory recursion optimisation occurs if either:
-If you wish to check the `_config` assignment has worked properly then
-calling `options/local` will show what the value got set to.
+* A source remote does not support the rclone `ListR` primitive. local,
+sftp, Microsoft OneDrive and WebDAV do not support `ListR`. Google
+Drive and most bucket type storage do. [Full list](https://rclone.org/overview/#optional-features)
-### Setting filter flags with _filter
+* On other remotes (those that support `ListR`), if the rclone command is not naturally recursive, and
+provided it is not run with the `--fast-list` flag. `ls`, `lsf -R` and
+`size` are naturally recursive but `sync`, `copy` and `move` are not.
-If you wish to set filters for the duration of an rc call only then
-pass in the `_filter` parameter.
+* Whenever the `--disable ListR` flag is applied to an rclone command.
-This should be in the same format as the `filter` key returned by
-[options/get](#options-get).
+Rclone commands imply directory filter rules from path/file filter
+rules. To view the directory filter rules rclone has implied for a
+command specify the `--dump filters` flag.
-For example, if you wished to run a sync with these flags
+E.g. for an include rule
- --max-size 1M --max-age 42s --include "a" --include "b"
+ /a/*.jpg
-you would pass this parameter in your JSON blob.
+Rclone implies the directory include rule
- "_filter":{"MaxSize":"1M", "IncludeRule":["a","b"], "MaxAge":"42s"}
+ /a/
-If using `rclone rc` this could be passed as
+Directory filter rules specified in an rclone command can limit
+the scope of an rclone command but path/file filters still have
+to be specified.
- rclone rc ... _filter='{"MaxSize":"1M", "IncludeRule":["a","b"], "MaxAge":"42s"}'
+E.g. `rclone ls remote: --include /directory/` will not match any
+files. Because it is an `--include` option the `--exclude **` rule
+is implied, and the `/directory/` pattern serves only to optimise
+access to the remote by ignoring everything outside of that directory.
-Any filter parameters you don't set will inherit the global defaults
-which were set with command line flags or environment variables.
+E.g. `rclone ls remote: --filter-from filter-list.txt` with a file
+`filter-list.txt`:
-Note that it is possible to set some values as strings or integers -
-see [data types](#data-types) for more info. Here is an example
-setting the equivalent of `--buffer-size` in string or integer format.
+ - /dir1/
+ - /dir2/
+ + *.pdf
+ - **
- "_filter":{"MinSize": "42M"}
- "_filter":{"MinSize": 44040192}
+All files in directories `dir1` or `dir2` or their subdirectories
+are completely excluded from the listing. Only files of suffix
+`pdf` in the root of `remote:` or its subdirectories are listed.
+The `- **` rule prevents listing of any path/files not previously
+matched by the rules above.
-If you wish to check the `_filter` assignment has worked properly then
-calling `options/local` will show what the value got set to.
+Option `exclude-if-present` creates a directory exclude rule based
+on the presence of a file in a directory and takes precedence over
+other rclone directory filter rules.
-### Assigning operations to groups with _group = value
+When using pattern list syntax, if a pattern item contains either
+`/` or `**`, then rclone will not able to imply a directory filter rule
+from this pattern list.
-Each rc call has its own stats group for tracking its metrics. By default
-grouping is done by the composite group name from prefix `job/` and id of the
-job like so `job/1`.
+E.g. for an include rule
-If `_group` has a value then stats for that request will be grouped under that
-value. This allows caller to group stats under their own name.
+ {dir1/**,dir2/**}
-Stats for specific group can be accessed by passing `group` to `core/stats`:
+Rclone will match files below directories `dir1` or `dir2` only,
+but will not be able to use this filter to exclude a directory `dir3`
+from being traversed.
-```
-$ rclone rc --json '{ "group": "job/1" }' core/stats
-{
- "speed": 12345
- ...
-}
-```
+Directory recursion optimisation may affect performance, but normally
+not the result. One exception to this is sync operations with option
+`--create-empty-src-dirs`, where any traversed empty directories will
+be created. With the pattern list example `{dir1/**,dir2/**}` above,
+this would create an empty directory `dir3` on destination (when it exists
+on source). Changing the filter to `{dir1,dir2}/**`, or splitting it into
+two include rules `--include dir1/** --include dir2/**`, will match the
+same files while also filtering directories, with the result that an empty
+directory `dir3` will no longer be created.
-## Data types {#data-types}
+### `--exclude` - Exclude files matching pattern
-When the API returns types, these will mostly be straight forward
-integer, string or boolean types.
+Excludes path/file names from an rclone command based on a single exclude
+rule.
-However some of the types returned by the [options/get](#options-get)
-call and taken by the [options/set](#options-set) calls as well as the
-`vfsOpt`, `mountOpt` and the `_config` parameters.
+This flag can be repeated. See above for the order filter flags are
+processed in.
-- `Duration` - these are returned as an integer duration in
- nanoseconds. They may be set as an integer, or they may be set with
- time string, eg "5s". See the [options section](https://rclone.org/docs/#options) for
- more info.
-- `Size` - these are returned as an integer number of bytes. They may
- be set as an integer or they may be set with a size suffix string,
- eg "10M". See the [options section](https://rclone.org/docs/#options) for more info.
-- Enumerated type (such as `CutoffMode`, `DumpFlags`, `LogLevel`,
- `VfsCacheMode` - these will be returned as an integer and may be set
- as an integer but more conveniently they can be set as a string, eg
- "HARD" for `CutoffMode` or `DEBUG` for `LogLevel`.
-- `BandwidthSpec` - this will be set and returned as a string, eg
- "1M".
+`--exclude` should not be used with `--include`, `--include-from`,
+`--filter` or `--filter-from` flags.
-## Specifying remotes to work on
+`--exclude` has no effect when combined with `--files-from` or
+`--files-from-raw` flags.
-Remotes are specified with the `fs=`, `srcFs=`, `dstFs=`
-parameters depending on the command being used.
+E.g. `rclone ls remote: --exclude *.bak` excludes all .bak files
+from listing.
-The parameters can be a string as per the rest of rclone, eg
-`s3:bucket/path` or `:sftp:/my/dir`. They can also be specified as
-JSON blobs.
+E.g. `rclone size remote: "--exclude /dir/**"` returns the total size of
+all files on `remote:` excluding those in root directory `dir` and sub
+directories.
-If specifying a JSON blob it should be a object mapping strings to
-strings. These values will be used to configure the remote. There are
-3 special values which may be set:
+E.g. on Microsoft Windows `rclone ls remote: --exclude "*\[{JP,KR,HK}\]*"`
+lists the files in `remote:` without `[JP]` or `[KR]` or `[HK]` in
+their name. Quotes prevent the shell from interpreting the `\`
+characters.`\` characters escape the `[` and `]` so an rclone filter
+treats them literally rather than as a character-range. The `{` and `}`
+define an rclone pattern list. For other operating systems single quotes are
+required ie `rclone ls remote: --exclude '*\[{JP,KR,HK}\]*'`
-- `type` - set to `type` to specify a remote called `:type:`
-- `_name` - set to `name` to specify a remote called `name:`
-- `_root` - sets the root of the remote - may be empty
+### `--exclude-from` - Read exclude patterns from file
-One of `_name` or `type` should normally be set. If the `local`
-backend is desired then `type` should be set to `local`. If `_root`
-isn't specified then it defaults to the root of the remote.
+Excludes path/file names from an rclone command based on rules in a
+named file. The file contains a list of remarks and pattern rules.
-For example this JSON is equivalent to `remote:/tmp`
+For an example `exclude-file.txt`:
-```
-{
- "_name": "remote",
- "_path": "/tmp"
-}
-```
+ # a sample exclude rule file
+ *.bak
+ file2.jpg
-And this is equivalent to `:sftp,host='example.com':/tmp`
+`rclone ls remote: --exclude-from exclude-file.txt` lists the files on
+`remote:` except those named `file2.jpg` or with a suffix `.bak`. That is
+equivalent to `rclone ls remote: --exclude file2.jpg --exclude "*.bak"`.
-```
-{
- "type": "sftp",
- "host": "example.com",
- "_path": "/tmp"
-}
-```
+This flag can be repeated. See above for the order filter flags are
+processed in.
-And this is equivalent to `/tmp/dir`
+The `--exclude-from` flag is useful where multiple exclude filter rules
+are applied to an rclone command.
-```
-{
- type = "local",
- _ path = "/tmp/dir"
-}
-```
+`--exclude-from` should not be used with `--include`, `--include-from`,
+`--filter` or `--filter-from` flags.
-## Supported commands
+`--exclude-from` has no effect when combined with `--files-from` or
+`--files-from-raw` flags.
-### backend/command: Runs a backend command. {#backend-command}
+`--exclude-from` followed by `-` reads filter rules from standard input.
-This takes the following parameters:
+### `--include` - Include files matching pattern
-- command - a string with the command name
-- fs - a remote name string e.g. "drive:"
-- arg - a list of arguments for the backend command
-- opt - a map of string to string of options
+Adds a single include rule based on path/file names to an rclone
+command.
-Returns:
+This flag can be repeated. See above for the order filter flags are
+processed in.
-- result - result from the backend command
+`--include` has no effect when combined with `--files-from` or
+`--files-from-raw` flags.
-Example:
+`--include` implies `--exclude **` at the end of an rclone internal
+filter list. Therefore if you mix `--include` and `--include-from`
+flags with `--exclude`, `--exclude-from`, `--filter` or `--filter-from`,
+you must use include rules for all the files you want in the include
+statement. For more flexibility use the `--filter-from` flag.
- rclone rc backend/command command=noop fs=. -o echo=yes -o blue -a path1 -a path2
+E.g. `rclone ls remote: --include "*.{png,jpg}"` lists the files on
+`remote:` with suffix `.png` and `.jpg`. All other files are excluded.
-Returns
+E.g. multiple rclone copy commands can be combined with `--include` and a
+pattern-list.
-```
-{
- "result": {
- "arg": [
- "path1",
- "path2"
- ],
- "name": "noop",
- "opt": {
- "blue": "",
- "echo": "yes"
- }
- }
-}
-```
+ rclone copy /vol1/A remote:A
+ rclone copy /vol1/B remote:B
-Note that this is the direct equivalent of using this "backend"
-command:
+is equivalent to:
- rclone backend noop . -o echo=yes -o blue path1 path2
+ rclone copy /vol1 remote: --include "{A,B}/**"
-Note that arguments must be preceded by the "-a" flag
+E.g. `rclone ls remote:/wheat --include "??[^[:punct:]]*"` lists the
+files `remote:` directory `wheat` (and subdirectories) whose third
+character is not punctuation. This example uses
+an [ASCII character class](https://golang.org/pkg/regexp/syntax/).
-See the [backend](https://rclone.org/commands/rclone_backend/) command for more information.
+### `--include-from` - Read include patterns from file
-**Authentication is required for this call.**
+Adds path/file names to an rclone command based on rules in a
+named file. The file contains a list of remarks and pattern rules.
-### cache/expire: Purge a remote from cache {#cache-expire}
+For an example `include-file.txt`:
-Purge a remote from the cache backend. Supports either a directory or a file.
-Params:
- - remote = path to remote (required)
- - withData = true/false to delete cached data (chunks) as well (optional)
+ # a sample include rule file
+ *.jpg
+ file2.avi
-Eg
+`rclone ls remote: --include-from include-file.txt` lists the files on
+`remote:` with name `file2.avi` or suffix `.jpg`. That is equivalent to
+`rclone ls remote: --include file2.avi --include "*.jpg"`.
- rclone rc cache/expire remote=path/to/sub/folder/
- rclone rc cache/expire remote=/ withData=true
+This flag can be repeated. See above for the order filter flags are
+processed in.
-### cache/fetch: Fetch file chunks {#cache-fetch}
+The `--include-from` flag is useful where multiple include filter rules
+are applied to an rclone command.
-Ensure the specified file chunks are cached on disk.
+`--include-from` implies `--exclude **` at the end of an rclone internal
+filter list. Therefore if you mix `--include` and `--include-from`
+flags with `--exclude`, `--exclude-from`, `--filter` or `--filter-from`,
+you must use include rules for all the files you want in the include
+statement. For more flexibility use the `--filter-from` flag.
-The chunks= parameter specifies the file chunks to check.
-It takes a comma separated list of array slice indices.
-The slice indices are similar to Python slices: start[:end]
+`--exclude-from` has no effect when combined with `--files-from` or
+`--files-from-raw` flags.
-start is the 0 based chunk number from the beginning of the file
-to fetch inclusive. end is 0 based chunk number from the beginning
-of the file to fetch exclusive.
-Both values can be negative, in which case they count from the back
-of the file. The value "-5:" represents the last 5 chunks of a file.
+`--exclude-from` followed by `-` reads filter rules from standard input.
-Some valid examples are:
-":5,-5:" -> the first and last five chunks
-"0,-2" -> the first and the second last chunk
-"0:10" -> the first ten chunks
+### `--filter` - Add a file-filtering rule
-Any parameter with a key that starts with "file" can be used to
-specify files to fetch, e.g.
+Specifies path/file names to an rclone command, based on a single
+include or exclude rule, in `+` or `-` format.
- rclone rc cache/fetch chunks=0 file=hello file2=home/goodbye
+This flag can be repeated. See above for the order filter flags are
+processed in.
-File names will automatically be encrypted when the a crypt remote
-is used on top of the cache.
+`--filter +` differs from `--include`. In the case of `--include` rclone
+implies an `--exclude *` rule which it adds to the bottom of the internal rule
+list. `--filter...+` does not imply
+that rule.
-### cache/stats: Get cache stats {#cache-stats}
+`--filter` has no effect when combined with `--files-from` or
+`--files-from-raw` flags.
-Show statistics for the cache remote.
+`--filter` should not be used with `--include`, `--include-from`,
+`--exclude` or `--exclude-from` flags.
-### config/create: create the config for a remote. {#config-create}
+E.g. `rclone ls remote: --filter "- *.bak"` excludes all `.bak` files
+from a list of `remote:`.
-This takes the following parameters:
+### `--filter-from` - Read filtering patterns from a file
-- name - name of remote
-- parameters - a map of \{ "key": "value" \} pairs
-- type - type of the new remote
-- opt - a dictionary of options to control the configuration
- - obscure - declare passwords are plain and need obscuring
- - noObscure - declare passwords are already obscured and don't need obscuring
- - nonInteractive - don't interact with a user, return questions
- - continue - continue the config process with an answer
- - all - ask all the config questions not just the post config ones
- - state - state to restart with - used with continue
- - result - result to restart with - used with continue
+Adds path/file names to an rclone command based on rules in a
+named file. The file contains a list of remarks and pattern rules. Include
+rules start with `+ ` and exclude rules with `- `. `!` clears existing
+rules. Rules are processed in the order they are defined.
+This flag can be repeated. See above for the order filter flags are
+processed in.
-See the [config create](https://rclone.org/commands/rclone_config_create/) command for more information on the above.
+Arrange the order of filter rules with the most restrictive first and
+work down.
-**Authentication is required for this call.**
+E.g. for `filter-file.txt`:
-### config/delete: Delete a remote in the config file. {#config-delete}
+ # a sample filter rule file
+ - secret*.jpg
+ + *.jpg
+ + *.png
+ + file2.avi
+ - /dir/Trash/**
+ + /dir/**
+ # exclude everything else
+ - *
-Parameters:
+`rclone ls remote: --filter-from filter-file.txt` lists the path/files on
+`remote:` including all `jpg` and `png` files, excluding any
+matching `secret*.jpg` and including `file2.avi`. It also includes
+everything in the directory `dir` at the root of `remote`, except
+`remote:dir/Trash` which it excludes. Everything else is excluded.
-- name - name of remote to delete
-See the [config delete](https://rclone.org/commands/rclone_config_delete/) command for more information on the above.
+E.g. for an alternative `filter-file.txt`:
-**Authentication is required for this call.**
+ - secret*.jpg
+ + *.jpg
+ + *.png
+ + file2.avi
+ - *
-### config/dump: Dumps the config file. {#config-dump}
+Files `file1.jpg`, `file3.png` and `file2.avi` are listed whilst
+`secret17.jpg` and files without the suffix .jpg` or `.png` are excluded.
-Returns a JSON object:
-- key: value
+E.g. for an alternative `filter-file.txt`:
-Where keys are remote names and values are the config parameters.
+ + *.jpg
+ + *.gif
+ !
+ + 42.doc
+ - *
-See the [config dump](https://rclone.org/commands/rclone_config_dump/) command for more information on the above.
+Only file 42.doc is listed. Prior rules are cleared by the `!`.
-**Authentication is required for this call.**
+### `--files-from` - Read list of source-file names
-### config/get: Get a remote in the config file. {#config-get}
+Adds path/files to an rclone command from a list in a named file.
+Rclone processes the path/file names in the order of the list, and
+no others.
-Parameters:
+Other filter flags (`--include`, `--include-from`, `--exclude`,
+`--exclude-from`, `--filter` and `--filter-from`) are ignored when
+`--files-from` is used.
-- name - name of remote to get
+`--files-from` expects a list of files as its input. Leading or
+trailing whitespace is stripped from the input lines. Lines starting
+with `#` or `;` are ignored.
-See the [config dump](https://rclone.org/commands/rclone_config_dump/) command for more information on the above.
+Rclone commands with a `--files-from` flag traverse the remote,
+treating the names in `--files-from` as a set of filters.
-**Authentication is required for this call.**
+If the `--no-traverse` and `--files-from` flags are used together
+an rclone command does not traverse the remote. Instead it addresses
+each path/file named in the file individually. For each path/file name, that
+requires typically 1 API call. This can be efficient for a short `--files-from`
+list and a remote containing many files.
-### config/listremotes: Lists the remotes in the config file and defined in environment variables. {#config-listremotes}
+Rclone commands do not error if any names in the `--files-from` file are
+missing from the source remote.
-Returns
-- remotes - array of remote names
+The `--files-from` flag can be repeated in a single rclone command to
+read path/file names from more than one file. The files are read from left
+to right along the command line.
-See the [listremotes](https://rclone.org/commands/rclone_listremotes/) command for more information on the above.
+Paths within the `--files-from` file are interpreted as starting
+with the root specified in the rclone command. Leading `/` separators are
+ignored. See [--files-from-raw](#files-from-raw-read-list-of-source-file-names-without-any-processing) if
+you need the input to be processed in a raw manner.
-**Authentication is required for this call.**
+E.g. for a file `files-from.txt`:
-### config/password: password the config for a remote. {#config-password}
+ # comment
+ file1.jpg
+ subdir/file2.jpg
-This takes the following parameters:
+`rclone copy --files-from files-from.txt /home/me/pics remote:pics`
+copies the following, if they exist, and only those files.
-- name - name of remote
-- parameters - a map of \{ "key": "value" \} pairs
+ /home/me/pics/file1.jpg → remote:pics/file1.jpg
+ /home/me/pics/subdir/file2.jpg → remote:pics/subdir/file2.jpg
+E.g. to copy the following files referenced by their absolute paths:
-See the [config password](https://rclone.org/commands/rclone_config_password/) command for more information on the above.
+ /home/user1/42
+ /home/user1/dir/ford
+ /home/user2/prefect
-**Authentication is required for this call.**
+First find a common subdirectory - in this case `/home`
+and put the remaining files in `files-from.txt` with or without
+leading `/`, e.g.
-### config/providers: Shows how providers are configured in the config file. {#config-providers}
+ user1/42
+ user1/dir/ford
+ user2/prefect
-Returns a JSON object:
-- providers - array of objects
+Then copy these to a remote:
-See the [config providers](https://rclone.org/commands/rclone_config_providers/) command for more information on the above.
+ rclone copy --files-from files-from.txt /home remote:backup
-**Authentication is required for this call.**
+The three files are transferred as follows:
-### config/setpath: Set the path of the config file {#config-setpath}
+ /home/user1/42 → remote:backup/user1/important
+ /home/user1/dir/ford → remote:backup/user1/dir/file
+ /home/user2/prefect → remote:backup/user2/stuff
-Parameters:
+Alternatively if `/` is chosen as root `files-from.txt` will be:
-- path - path to the config file to use
+ /home/user1/42
+ /home/user1/dir/ford
+ /home/user2/prefect
-**Authentication is required for this call.**
+The copy command will be:
-### config/update: update the config for a remote. {#config-update}
+ rclone copy --files-from files-from.txt / remote:backup
-This takes the following parameters:
+Then there will be an extra `home` directory on the remote:
-- name - name of remote
-- parameters - a map of \{ "key": "value" \} pairs
-- opt - a dictionary of options to control the configuration
- - obscure - declare passwords are plain and need obscuring
- - noObscure - declare passwords are already obscured and don't need obscuring
- - nonInteractive - don't interact with a user, return questions
- - continue - continue the config process with an answer
- - all - ask all the config questions not just the post config ones
- - state - state to restart with - used with continue
- - result - result to restart with - used with continue
-
-
-See the [config update](https://rclone.org/commands/rclone_config_update/) command for more information on the above.
-
-**Authentication is required for this call.**
+ /home/user1/42 → remote:backup/home/user1/42
+ /home/user1/dir/ford → remote:backup/home/user1/dir/ford
+ /home/user2/prefect → remote:backup/home/user2/prefect
-### core/bwlimit: Set the bandwidth limit. {#core-bwlimit}
+### `--files-from-raw` - Read list of source-file names without any processing
-This sets the bandwidth limit to the string passed in. This should be
-a single bandwidth limit entry or a pair of upload:download bandwidth.
+This flag is the same as `--files-from` except that input is read in a
+raw manner. Lines with leading / trailing whitespace, and lines starting
+with `;` or `#` are read without any processing. [rclone lsf](https://rclone.org/commands/rclone_lsf/) has
+a compatible format that can be used to export file lists from remotes for
+input to `--files-from-raw`.
-Eg
+### `--ignore-case` - make searches case insensitive
- rclone rc core/bwlimit rate=off
- {
- "bytesPerSecond": -1,
- "bytesPerSecondTx": -1,
- "bytesPerSecondRx": -1,
- "rate": "off"
- }
- rclone rc core/bwlimit rate=1M
- {
- "bytesPerSecond": 1048576,
- "bytesPerSecondTx": 1048576,
- "bytesPerSecondRx": 1048576,
- "rate": "1M"
- }
- rclone rc core/bwlimit rate=1M:100k
- {
- "bytesPerSecond": 1048576,
- "bytesPerSecondTx": 1048576,
- "bytesPerSecondRx": 131072,
- "rate": "1M"
- }
+By default, rclone filter patterns are case sensitive. The `--ignore-case`
+flag makes all of the filters patterns on the command line case
+insensitive.
+E.g. `--include "zaphod.txt"` does not match a file `Zaphod.txt`. With
+`--ignore-case` a match is made.
-If the rate parameter is not supplied then the bandwidth is queried
+## Quoting shell metacharacters
- rclone rc core/bwlimit
- {
- "bytesPerSecond": 1048576,
- "bytesPerSecondTx": 1048576,
- "bytesPerSecondRx": 1048576,
- "rate": "1M"
- }
+Rclone commands with filter patterns containing shell metacharacters may
+not as work as expected in your shell and may require quoting.
-The format of the parameter is exactly the same as passed to --bwlimit
-except only one bandwidth may be specified.
+E.g. linux, OSX (`*` metacharacter)
-In either case "rate" is returned as a human-readable string, and
-"bytesPerSecond" is returned as a number.
+ * `--include \*.jpg`
+ * `--include '*.jpg'`
+ * `--include='*.jpg'`
-### core/command: Run a rclone terminal command over rc. {#core-command}
+Microsoft Windows expansion is done by the command, not shell, so
+`--include *.jpg` does not require quoting.
-This takes the following parameters:
+If the rclone error
+`Command .... needs .... arguments maximum: you provided .... non flag arguments:`
+is encountered, the cause is commonly spaces within the name of a
+remote or flag value. The fix then is to quote values containing spaces.
-- command - a string with the command name.
-- arg - a list of arguments for the backend command.
-- opt - a map of string to string of options.
-- returnType - one of ("COMBINED_OUTPUT", "STREAM", "STREAM_ONLY_STDOUT", "STREAM_ONLY_STDERR").
- - Defaults to "COMBINED_OUTPUT" if not set.
- - The STREAM returnTypes will write the output to the body of the HTTP message.
- - The COMBINED_OUTPUT will write the output to the "result" parameter.
+## Other filters
-Returns:
+### `--min-size` - Don't transfer any file smaller than this
-- result - result from the backend command.
- - Only set when using returnType "COMBINED_OUTPUT".
-- error - set if rclone exits with an error code.
-- returnType - one of ("COMBINED_OUTPUT", "STREAM", "STREAM_ONLY_STDOUT", "STREAM_ONLY_STDERR").
+Controls the minimum size file within the scope of an rclone command.
+Default units are `KiB` but abbreviations `K`, `M`, `G`, `T` or `P` are valid.
-Example:
+E.g. `rclone ls remote: --min-size 50k` lists files on `remote:` of 50 KiB
+size or larger.
- rclone rc core/command command=ls -a mydrive:/ -o max-depth=1
- rclone rc core/command -a ls -a mydrive:/ -o max-depth=1
+See [the size option docs](https://rclone.org/docs/#size-option) for more info.
-Returns:
+### `--max-size` - Don't transfer any file larger than this
-```
-{
- "error": false,
- "result": ""
-}
+Controls the maximum size file within the scope of an rclone command.
+Default units are `KiB` but abbreviations `K`, `M`, `G`, `T` or `P` are valid.
-OR
-{
- "error": true,
- "result": ""
-}
+E.g. `rclone ls remote: --max-size 1G` lists files on `remote:` of 1 GiB
+size or smaller.
-```
+See [the size option docs](https://rclone.org/docs/#size-option) for more info.
-**Authentication is required for this call.**
+### `--max-age` - Don't transfer any file older than this
-### core/du: Returns disk usage of a locally attached disk. {#core-du}
+Controls the maximum age of files within the scope of an rclone command.
-This returns the disk usage for the local directory passed in as dir.
+`--max-age` applies only to files and not to directories.
-If the directory is not passed in, it defaults to the directory
-pointed to by --cache-dir.
+E.g. `rclone ls remote: --max-age 2d` lists files on `remote:` of 2 days
+old or less.
-- dir - string (optional)
+See [the time option docs](https://rclone.org/docs/#time-option) for valid formats.
-Returns:
+### `--min-age` - Don't transfer any file younger than this
-```
-{
- "dir": "/",
- "info": {
- "Available": 361769115648,
- "Free": 361785892864,
- "Total": 982141468672
- }
-}
-```
+Controls the minimum age of files within the scope of an rclone command.
+(see `--max-age` for valid formats)
-### core/gc: Runs a garbage collection. {#core-gc}
+`--min-age` applies only to files and not to directories.
-This tells the go runtime to do a garbage collection run. It isn't
-necessary to call this normally, but it can be useful for debugging
-memory problems.
+E.g. `rclone ls remote: --min-age 2d` lists files on `remote:` of 2 days
+old or more.
-### core/group-list: Returns list of stats. {#core-group-list}
+See [the time option docs](https://rclone.org/docs/#time-option) for valid formats.
-This returns list of stats groups currently in memory.
+## Other flags
-Returns the following values:
-```
-{
- "groups": an array of group names:
- [
- "group1",
- "group2",
- ...
- ]
-}
-```
+### `--delete-excluded` - Delete files on dest excluded from sync
-### core/memstats: Returns the memory statistics {#core-memstats}
+**Important** this flag is dangerous to your data - use with `--dry-run`
+and `-v` first.
-This returns the memory statistics of the running program. What the values mean
-are explained in the go docs: https://golang.org/pkg/runtime/#MemStats
+In conjunction with `rclone sync`, `--delete-excluded` deletes any files
+on the destination which are excluded from the command.
-The most interesting values for most people are:
+E.g. the scope of `rclone sync --interactive A: B:` can be restricted:
-- HeapAlloc - this is the amount of memory rclone is actually using
-- HeapSys - this is the amount of memory rclone has obtained from the OS
-- Sys - this is the total amount of memory requested from the OS
- - It is virtual memory so may include unused memory
+ rclone --min-size 50k --delete-excluded sync A: B:
-### core/obscure: Obscures a string passed in. {#core-obscure}
+All files on `B:` which are less than 50 KiB are deleted
+because they are excluded from the rclone sync command.
-Pass a clear string and rclone will obscure it for the config file:
-- clear - string
+### `--dump filters` - dump the filters to the output
-Returns:
-- obscured - string
+Dumps the defined filters to standard output in regular expression
+format.
-### core/pid: Return PID of current process {#core-pid}
+Useful for debugging.
-This returns PID of current process.
-Useful for stopping rclone process.
+## Exclude directory based on a file
-### core/quit: Terminates the app. {#core-quit}
+The `--exclude-if-present` flag controls whether a directory is
+within the scope of an rclone command based on the presence of a
+named file within it. The flag can be repeated to check for
+multiple file names, presence of any of them will exclude the
+directory.
-(Optional) Pass an exit code to be used for terminating the app:
-- exitCode - int
+This flag has a priority over other filter flags.
-### core/stats: Returns stats about current transfers. {#core-stats}
+E.g. for the following directory structure:
-This returns all available stats:
+ dir1/file1
+ dir1/dir2/file2
+ dir1/dir2/dir3/file3
+ dir1/dir2/dir3/.ignore
- rclone rc core/stats
+The command `rclone ls --exclude-if-present .ignore dir1` does
+not list `dir3`, `file3` or `.ignore`.
-If group is not provided then summed up stats for all groups will be
-returned.
+## Metadata filters {#metadata}
-Parameters
+The metadata filters work in a very similar way to the normal file
+name filters, except they match [metadata](https://rclone.org/docs/#metadata) on the
+object.
-- group - name of the stats group (string)
+The metadata should be specified as `key=value` patterns. This may be
+wildcarded using the normal [filter patterns](#patterns) or [regular
+expressions](#regexp).
-Returns the following values:
+For example if you wished to list only local files with a mode of
+`100664` you could do that with:
-```
-{
- "bytes": total transferred bytes since the start of the group,
- "checks": number of files checked,
- "deletes" : number of files deleted,
- "elapsedTime": time in floating point seconds since rclone was started,
- "errors": number of errors,
- "eta": estimated time in seconds until the group completes,
- "fatalError": boolean whether there has been at least one fatal error,
- "lastError": last error string,
- "renames" : number of files renamed,
- "retryError": boolean showing whether there has been at least one non-NoRetryError,
- "serverSideCopies": number of server side copies done,
- "serverSideCopyBytes": number bytes server side copied,
- "serverSideMoves": number of server side moves done,
- "serverSideMoveBytes": number bytes server side moved,
- "speed": average speed in bytes per second since start of the group,
- "totalBytes": total number of bytes in the group,
- "totalChecks": total number of checks in the group,
- "totalTransfers": total number of transfers in the group,
- "transferTime" : total time spent on running jobs,
- "transfers": number of transferred files,
- "transferring": an array of currently active file transfers:
- [
- {
- "bytes": total transferred bytes for this file,
- "eta": estimated time in seconds until file transfer completion
- "name": name of the file,
- "percentage": progress of the file transfer in percent,
- "speed": average speed over the whole transfer in bytes per second,
- "speedAvg": current speed in bytes per second as an exponentially weighted moving average,
- "size": size of the file in bytes
- }
- ],
- "checking": an array of names of currently active file checks
- []
-}
-```
-Values for "transferring", "checking" and "lastError" are only assigned if data is available.
-The value for "eta" is null if an eta cannot be determined.
+ rclone lsf -M --files-only --metadata-include "mode=100664" .
-### core/stats-delete: Delete stats group. {#core-stats-delete}
+Or if you wished to show files with an `atime`, `mtime` or `btime` at a given date:
-This deletes entire stats group.
+ rclone lsf -M --files-only --metadata-include "[abm]time=2022-12-16*" .
-Parameters
+Like file filtering, metadata filtering only applies to files not to
+directories.
-- group - name of the stats group (string)
+The filters can be applied using these flags.
-### core/stats-reset: Reset stats. {#core-stats-reset}
+- `--metadata-include` - Include metadatas matching pattern
+- `--metadata-include-from` - Read metadata include patterns from file (use - to read from stdin)
+- `--metadata-exclude` - Exclude metadatas matching pattern
+- `--metadata-exclude-from` - Read metadata exclude patterns from file (use - to read from stdin)
+- `--metadata-filter` - Add a metadata filtering rule
+- `--metadata-filter-from` - Read metadata filtering patterns from a file (use - to read from stdin)
-This clears counters, errors and finished transfers for all stats or specific
-stats group if group is provided.
+Each flag can be repeated. See the section on [how filter rules are
+applied](#how-filter-rules-work) for more details - these flags work
+in an identical way to the file name filtering flags, but instead of
+file name patterns have metadata patterns.
-Parameters
-- group - name of the stats group (string)
+## Common pitfalls
-### core/transferred: Returns stats about completed transfers. {#core-transferred}
+The most frequent filter support issues on
+the [rclone forum](https://forum.rclone.org/) are:
-This returns stats about completed transfers:
+* Not using paths relative to the root of the remote
+* Not using `/` to match from the root of a remote
+* Not using `**` to match the contents of a directory
- rclone rc core/transferred
+# GUI (Experimental)
-If group is not provided then completed transfers for all groups will be
-returned.
+Rclone can serve a web based GUI (graphical user interface). This is
+somewhat experimental at the moment so things may be subject to
+change.
-Note only the last 100 completed transfers are returned.
+Run this command in a terminal and rclone will download and then
+display the GUI in a web browser.
-Parameters
+```
+rclone rcd --rc-web-gui
+```
-- group - name of the stats group (string)
+This will produce logs like this and rclone needs to continue to run to serve the GUI:
-Returns the following values:
```
-{
- "transferred": an array of completed transfers (including failed ones):
- [
- {
- "name": name of the file,
- "size": size of the file in bytes,
- "bytes": total transferred bytes for this file,
- "checked": if the transfer is only checked (skipped, deleted),
- "timestamp": integer representing millisecond unix epoch,
- "error": string description of the error (empty if successful),
- "jobid": id of the job that this transfer belongs to
- }
- ]
-}
+2019/08/25 11:40:14 NOTICE: A new release for gui is present at https://github.com/rclone/rclone-webui-react/releases/download/v0.0.6/currentbuild.zip
+2019/08/25 11:40:14 NOTICE: Downloading webgui binary. Please wait. [Size: 3813937, Path : /home/USER/.cache/rclone/webgui/v0.0.6.zip]
+2019/08/25 11:40:16 NOTICE: Unzipping
+2019/08/25 11:40:16 NOTICE: Serving remote control on http://127.0.0.1:5572/
```
-### core/version: Shows the current version of rclone and the go runtime. {#core-version}
+This assumes you are running rclone locally on your machine. It is
+possible to separate the rclone and the GUI - see below for details.
-This shows the current version of go and the go runtime:
+If you wish to check for updates then you can add `--rc-web-gui-update`
+to the command line.
-- version - rclone version, e.g. "v1.53.0"
-- decomposed - version number as [major, minor, patch]
-- isGit - boolean - true if this was compiled from the git version
-- isBeta - boolean - true if this is a beta version
-- os - OS in use as according to Go
-- arch - cpu architecture in use according to Go
-- goVersion - version of Go runtime in use
-- linking - type of rclone executable (static or dynamic)
-- goTags - space separated build tags or "none"
+If you find your GUI broken, you may force it to update by add `--rc-web-gui-force-update`.
-### debug/set-block-profile-rate: Set runtime.SetBlockProfileRate for blocking profiling. {#debug-set-block-profile-rate}
+By default, rclone will open your browser. Add `--rc-web-gui-no-open-browser`
+to disable this feature.
-SetBlockProfileRate controls the fraction of goroutine blocking events
-that are reported in the blocking profile. The profiler aims to sample
-an average of one blocking event per rate nanoseconds spent blocked.
+## Using the GUI
-To include every blocking event in the profile, pass rate = 1. To turn
-off profiling entirely, pass rate <= 0.
+Once the GUI opens, you will be looking at the dashboard which has an overall overview.
-After calling this you can use this to see the blocking profile:
+On the left hand side you will see a series of view buttons you can click on:
- go tool pprof http://localhost:5572/debug/pprof/block
+- Dashboard - main overview
+- Configs - examine and create new configurations
+- Explorer - view, download and upload files to the cloud storage systems
+- Backend - view or alter the backend config
+- Log out
-Parameters:
+(More docs and walkthrough video to come!)
-- rate - int
+## How it works
-### debug/set-gc-percent: Call runtime/debug.SetGCPercent for setting the garbage collection target percentage. {#debug-set-gc-percent}
+When you run the `rclone rcd --rc-web-gui` this is what happens
-SetGCPercent sets the garbage collection target percentage: a collection is triggered
-when the ratio of freshly allocated data to live data remaining after the previous collection
-reaches this percentage. SetGCPercent returns the previous setting. The initial setting is the
-value of the GOGC environment variable at startup, or 100 if the variable is not set.
+- Rclone starts but only runs the remote control API ("rc").
+- The API is bound to localhost with an auto-generated username and password.
+- If the API bundle is missing then rclone will download it.
+- rclone will start serving the files from the API bundle over the same port as the API
+- rclone will open the browser with a `login_token` so it can log straight in.
-This setting may be effectively reduced in order to maintain a memory limit.
-A negative percentage effectively disables garbage collection, unless the memory limit is reached.
+## Advanced use
-See https://pkg.go.dev/runtime/debug#SetMemoryLimit for more details.
+The `rclone rcd` may use any of the [flags documented on the rc page](https://rclone.org/rc/#supported-parameters).
-Parameters:
+The flag `--rc-web-gui` is shorthand for
-- gc-percent - int
+- Download the web GUI if necessary
+- Check we are using some authentication
+- `--rc-user gui`
+- `--rc-pass `
+- `--rc-serve`
-### debug/set-mutex-profile-fraction: Set runtime.SetMutexProfileFraction for mutex profiling. {#debug-set-mutex-profile-fraction}
+These flags can be overridden as desired.
-SetMutexProfileFraction controls the fraction of mutex contention
-events that are reported in the mutex profile. On average 1/rate
-events are reported. The previous rate is returned.
+See also the [rclone rcd documentation](https://rclone.org/commands/rclone_rcd/).
-To turn off profiling entirely, pass rate 0. To just read the current
-rate, pass rate < 0. (For n>1 the details of sampling may change.)
+### Example: Running a public GUI
-Once this is set you can look use this to profile the mutex contention:
+For example the GUI could be served on a public port over SSL using an htpasswd file using the following flags:
- go tool pprof http://localhost:5572/debug/pprof/mutex
+- `--rc-web-gui`
+- `--rc-addr :443`
+- `--rc-htpasswd /path/to/htpasswd`
+- `--rc-cert /path/to/ssl.crt`
+- `--rc-key /path/to/ssl.key`
-Parameters:
+### Example: Running a GUI behind a proxy
-- rate - int
+If you want to run the GUI behind a proxy at `/rclone` you could use these flags:
-Results:
+- `--rc-web-gui`
+- `--rc-baseurl rclone`
+- `--rc-htpasswd /path/to/htpasswd`
-- previousRate - int
+Or instead of htpasswd if you just want a single user and password:
-### debug/set-soft-memory-limit: Call runtime/debug.SetMemoryLimit for setting a soft memory limit for the runtime. {#debug-set-soft-memory-limit}
+- `--rc-user me`
+- `--rc-pass mypassword`
-SetMemoryLimit provides the runtime with a soft memory limit.
+## Project
-The runtime undertakes several processes to try to respect this memory limit, including
-adjustments to the frequency of garbage collections and returning memory to the underlying
-system more aggressively. This limit will be respected even if GOGC=off (or, if SetGCPercent(-1) is executed).
+The GUI is being developed in the: [rclone/rclone-webui-react repository](https://github.com/rclone/rclone-webui-react).
-The input limit is provided as bytes, and includes all memory mapped, managed, and not
-released by the Go runtime. Notably, it does not account for space used by the Go binary
-and memory external to Go, such as memory managed by the underlying system on behalf of
-the process, or memory managed by non-Go code inside the same process.
-Examples of excluded memory sources include: OS kernel memory held on behalf of the process,
-memory allocated by C code, and memory mapped by syscall.Mmap (because it is not managed by the Go runtime).
+Bug reports and contributions are very welcome :-)
-A zero limit or a limit that's lower than the amount of memory used by the Go runtime may cause
-the garbage collector to run nearly continuously. However, the application may still make progress.
+If you have questions then please ask them on the [rclone forum](https://forum.rclone.org/).
-The memory limit is always respected by the Go runtime, so to effectively disable this behavior,
-set the limit very high. math.MaxInt64 is the canonical value for disabling the limit, but values
-much greater than the available memory on the underlying system work just as well.
+# Remote controlling rclone with its API
-See https://go.dev/doc/gc-guide for a detailed guide explaining the soft memory limit in more detail,
-as well as a variety of common use-cases and scenarios.
+If rclone is run with the `--rc` flag then it starts an HTTP server
+which can be used to remote control rclone using its API.
-SetMemoryLimit returns the previously set memory limit. A negative input does not adjust the limit,
-and allows for retrieval of the currently set memory limit.
+You can either use the [rc](#api-rc) command to access the API
+or [use HTTP directly](#api-http).
-Parameters:
+If you just want to run a remote control then see the [rcd](https://rclone.org/commands/rclone_rcd/) command.
-- mem-limit - int
+## Supported parameters
-### fscache/clear: Clear the Fs cache. {#fscache-clear}
+### --rc
-This clears the fs cache. This is where remotes created from backends
-are cached for a short while to make repeated rc calls more efficient.
+Flag to start the http server listen on remote requests
+
+### --rc-addr=IP
-If you change the parameters of a backend then you may want to call
-this to clear an existing remote out of the cache before re-creating
-it.
+IPaddress:Port or :Port to bind server to. (default "localhost:5572")
-**Authentication is required for this call.**
+### --rc-cert=KEY
+SSL PEM key (concatenation of certificate and CA certificate)
-### fscache/entries: Returns the number of entries in the fs cache. {#fscache-entries}
+### --rc-client-ca=PATH
+Client certificate authority to verify clients with
-This returns the number of entries in the fs cache.
+### --rc-htpasswd=PATH
-Returns
-- entries - number of items in the cache
+htpasswd file - if not provided no authentication is done
-**Authentication is required for this call.**
+### --rc-key=PATH
-### job/list: Lists the IDs of the running jobs {#job-list}
+SSL PEM Private key
-Parameters: None.
+### --rc-max-header-bytes=VALUE
-Results:
+Maximum size of request header (default 4096)
-- executeId - string id of rclone executing (change after restart)
-- jobids - array of integer job ids (starting at 1 on each restart)
+### --rc-min-tls-version=VALUE
-### job/status: Reads the status of the job ID {#job-status}
+The minimum TLS version that is acceptable. Valid values are "tls1.0",
+"tls1.1", "tls1.2" and "tls1.3" (default "tls1.0").
-Parameters:
+### --rc-user=VALUE
-- jobid - id of the job (integer).
+User name for authentication.
-Results:
+### --rc-pass=VALUE
-- finished - boolean
-- duration - time in seconds that the job ran for
-- endTime - time the job finished (e.g. "2018-10-26T18:50:20.528746884+01:00")
-- error - error from the job or empty string for no error
-- finished - boolean whether the job has finished or not
-- id - as passed in above
-- startTime - time the job started (e.g. "2018-10-26T18:50:20.528336039+01:00")
-- success - boolean - true for success false otherwise
-- output - output of the job as would have been returned if called synchronously
-- progress - output of the progress related to the underlying job
+Password for authentication.
-### job/stop: Stop the running job {#job-stop}
-
-Parameters:
-
-- jobid - id of the job (integer).
-
-### job/stopgroup: Stop all running jobs in a group {#job-stopgroup}
-
-Parameters:
-
-- group - name of the group (string).
-
-### mount/listmounts: Show current mount points {#mount-listmounts}
+### --rc-realm=VALUE
-This shows currently mounted points, which can be used for performing an unmount.
+Realm for authentication (default "rclone")
-This takes no parameters and returns
+### --rc-server-read-timeout=DURATION
-- mountPoints: list of current mount points
+Timeout for server reading data (default 1h0m0s)
-Eg
+### --rc-server-write-timeout=DURATION
- rclone rc mount/listmounts
+Timeout for server writing data (default 1h0m0s)
-**Authentication is required for this call.**
+### --rc-serve
-### mount/mount: Create a new mount point {#mount-mount}
+Enable the serving of remote objects via the HTTP interface. This
+means objects will be accessible at http://127.0.0.1:5572/ by default,
+so you can browse to http://127.0.0.1:5572/ or http://127.0.0.1:5572/*
+to see a listing of the remotes. Objects may be requested from
+remotes using this syntax http://127.0.0.1:5572/[remote:path]/path/to/object
-rclone allows Linux, FreeBSD, macOS and Windows to mount any of
-Rclone's cloud storage systems as a file system with FUSE.
+Default Off.
-If no mountType is provided, the priority is given as follows: 1. mount 2.cmount 3.mount2
+### --rc-files /path/to/directory
-This takes the following parameters:
+Path to local files to serve on the HTTP server.
-- fs - a remote path to be mounted (required)
-- mountPoint: valid path on the local machine (required)
-- mountType: one of the values (mount, cmount, mount2) specifies the mount implementation to use
-- mountOpt: a JSON object with Mount options in.
-- vfsOpt: a JSON object with VFS options in.
+If this is set then rclone will serve the files in that directory. It
+will also open the root in the web browser if specified. This is for
+implementing browser based GUIs for rclone functions.
-Example:
+If `--rc-user` or `--rc-pass` is set then the URL that is opened will
+have the authorization in the URL in the `http://user:pass@localhost/`
+style.
- rclone rc mount/mount fs=mydrive: mountPoint=/home//mountPoint
- rclone rc mount/mount fs=mydrive: mountPoint=/home//mountPoint mountType=mount
- rclone rc mount/mount fs=TestDrive: mountPoint=/mnt/tmp vfsOpt='{"CacheMode": 2}' mountOpt='{"AllowOther": true}'
+Default Off.
-The vfsOpt are as described in options/get and can be seen in the the
-"vfs" section when running and the mountOpt can be seen in the "mount" section:
+### --rc-enable-metrics
- rclone rc options/get
+Enable OpenMetrics/Prometheus compatible endpoint at `/metrics`.
-**Authentication is required for this call.**
+Default Off.
-### mount/types: Show all possible mount types {#mount-types}
+### --rc-web-gui
-This shows all possible mount types and returns them as a list.
+Set this flag to serve the default web gui on the same port as rclone.
-This takes no parameters and returns
+Default Off.
-- mountTypes: list of mount types
+### --rc-allow-origin
-The mount types are strings like "mount", "mount2", "cmount" and can
-be passed to mount/mount as the mountType parameter.
+Set the allowed Access-Control-Allow-Origin for rc requests.
-Eg
+Can be used with --rc-web-gui if the rclone is running on different IP than the web-gui.
- rclone rc mount/types
+Default is IP address on which rc is running.
-**Authentication is required for this call.**
+### --rc-web-fetch-url
-### mount/unmount: Unmount selected active mount {#mount-unmount}
+Set the URL to fetch the rclone-web-gui files from.
-rclone allows Linux, FreeBSD, macOS and Windows to
-mount any of Rclone's cloud storage systems as a file system with
-FUSE.
+Default https://api.github.com/repos/rclone/rclone-webui-react/releases/latest.
-This takes the following parameters:
+### --rc-web-gui-update
-- mountPoint: valid path on the local machine where the mount was created (required)
+Set this flag to check and update rclone-webui-react from the rc-web-fetch-url.
-Example:
+Default Off.
- rclone rc mount/unmount mountPoint=/home//mountPoint
+### --rc-web-gui-force-update
-**Authentication is required for this call.**
+Set this flag to force update rclone-webui-react from the rc-web-fetch-url.
-### mount/unmountall: Unmount all active mounts {#mount-unmountall}
+Default Off.
-rclone allows Linux, FreeBSD, macOS and Windows to
-mount any of Rclone's cloud storage systems as a file system with
-FUSE.
+### --rc-web-gui-no-open-browser
-This takes no parameters and returns error if unmount does not succeed.
+Set this flag to disable opening browser automatically when using web-gui.
-Eg
+Default Off.
- rclone rc mount/unmountall
+### --rc-job-expire-duration=DURATION
-**Authentication is required for this call.**
+Expire finished async jobs older than DURATION (default 60s).
-### operations/about: Return the space used on the remote {#operations-about}
+### --rc-job-expire-interval=DURATION
-This takes the following parameters:
+Interval duration to check for expired async jobs (default 10s).
-- fs - a remote name string e.g. "drive:"
+### --rc-no-auth
-The result is as returned from rclone about --json
+By default rclone will require authorisation to have been set up on
+the rc interface in order to use any methods which access any rclone
+remotes. Eg `operations/list` is denied as it involved creating a
+remote as is `sync/copy`.
-See the [about](https://rclone.org/commands/rclone_about/) command for more information on the above.
+If this is set then no authorisation will be required on the server to
+use these methods. The alternative is to use `--rc-user` and
+`--rc-pass` and use these credentials in the request.
-**Authentication is required for this call.**
+Default Off.
-### operations/cleanup: Remove trashed files in the remote or path {#operations-cleanup}
+### --rc-baseurl
-This takes the following parameters:
+Prefix for URLs.
-- fs - a remote name string e.g. "drive:"
+Default is root
-See the [cleanup](https://rclone.org/commands/rclone_cleanup/) command for more information on the above.
+### --rc-template
-**Authentication is required for this call.**
+User-specified template.
-### operations/copyfile: Copy a file from source remote to destination remote {#operations-copyfile}
+## Accessing the remote control via the rclone rc command {#api-rc}
-This takes the following parameters:
+Rclone itself implements the remote control protocol in its `rclone
+rc` command.
-- srcFs - a remote name string e.g. "drive:" for the source, "/" for local filesystem
-- srcRemote - a path within that remote e.g. "file.txt" for the source
-- dstFs - a remote name string e.g. "drive2:" for the destination, "/" for local filesystem
-- dstRemote - a path within that remote e.g. "file2.txt" for the destination
+You can use it like this
-**Authentication is required for this call.**
+```
+$ rclone rc rc/noop param1=one param2=two
+{
+ "param1": "one",
+ "param2": "two"
+}
+```
-### operations/copyurl: Copy the URL to the object {#operations-copyurl}
+Run `rclone rc` on its own to see the help for the installed remote
+control commands.
-This takes the following parameters:
+## JSON input
-- fs - a remote name string e.g. "drive:"
-- remote - a path within that remote e.g. "dir"
-- url - string, URL to read from
- - autoFilename - boolean, set to true to retrieve destination file name from url
+`rclone rc` also supports a `--json` flag which can be used to send
+more complicated input parameters.
-See the [copyurl](https://rclone.org/commands/rclone_copyurl/) command for more information on the above.
+```
+$ rclone rc --json '{ "p1": [1,"2",null,4], "p2": { "a":1, "b":2 } }' rc/noop
+{
+ "p1": [
+ 1,
+ "2",
+ null,
+ 4
+ ],
+ "p2": {
+ "a": 1,
+ "b": 2
+ }
+}
+```
-**Authentication is required for this call.**
+If the parameter being passed is an object then it can be passed as a
+JSON string rather than using the `--json` flag which simplifies the
+command line.
-### operations/delete: Remove files in the path {#operations-delete}
+```
+rclone rc operations/list fs=/tmp remote=test opt='{"showHash": true}'
+```
-This takes the following parameters:
+Rather than
-- fs - a remote name string e.g. "drive:"
+```
+rclone rc operations/list --json '{"fs": "/tmp", "remote": "test", "opt": {"showHash": true}}'
+```
-See the [delete](https://rclone.org/commands/rclone_delete/) command for more information on the above.
+## Special parameters
-**Authentication is required for this call.**
+The rc interface supports some special parameters which apply to
+**all** commands. These start with `_` to show they are different.
-### operations/deletefile: Remove the single file pointed to {#operations-deletefile}
+### Running asynchronous jobs with _async = true
-This takes the following parameters:
+Each rc call is classified as a job and it is assigned its own id. By default
+jobs are executed immediately as they are created or synchronously.
-- fs - a remote name string e.g. "drive:"
-- remote - a path within that remote e.g. "dir"
+If `_async` has a true value when supplied to an rc call then it will
+return immediately with a job id and the task will be run in the
+background. The `job/status` call can be used to get information of
+the background job. The job can be queried for up to 1 minute after
+it has finished.
-See the [deletefile](https://rclone.org/commands/rclone_deletefile/) command for more information on the above.
+It is recommended that potentially long running jobs, e.g. `sync/sync`,
+`sync/copy`, `sync/move`, `operations/purge` are run with the `_async`
+flag to avoid any potential problems with the HTTP request and
+response timing out.
-**Authentication is required for this call.**
+Starting a job with the `_async` flag:
-### operations/fsinfo: Return information about the remote {#operations-fsinfo}
+```
+$ rclone rc --json '{ "p1": [1,"2",null,4], "p2": { "a":1, "b":2 }, "_async": true }' rc/noop
+{
+ "jobid": 2
+}
+```
-This takes the following parameters:
+Query the status to see if the job has finished. For more information
+on the meaning of these return parameters see the `job/status` call.
-- fs - a remote name string e.g. "drive:"
+```
+$ rclone rc --json '{ "jobid":2 }' job/status
+{
+ "duration": 0.000124163,
+ "endTime": "2018-10-27T11:38:07.911245881+01:00",
+ "error": "",
+ "finished": true,
+ "id": 2,
+ "output": {
+ "_async": true,
+ "p1": [
+ 1,
+ "2",
+ null,
+ 4
+ ],
+ "p2": {
+ "a": 1,
+ "b": 2
+ }
+ },
+ "startTime": "2018-10-27T11:38:07.911121728+01:00",
+ "success": true
+}
+```
-This returns info about the remote passed in;
+`job/list` can be used to show the running or recently completed jobs
```
+$ rclone rc job/list
{
- // optional features and whether they are available or not
- "Features": {
- "About": true,
- "BucketBased": false,
- "BucketBasedRootOK": false,
- "CanHaveEmptyDirectories": true,
- "CaseInsensitive": false,
- "ChangeNotify": false,
- "CleanUp": false,
- "Command": true,
- "Copy": false,
- "DirCacheFlush": false,
- "DirMove": true,
- "Disconnect": false,
- "DuplicateFiles": false,
- "GetTier": false,
- "IsLocal": true,
- "ListR": false,
- "MergeDirs": false,
- "MetadataInfo": true,
- "Move": true,
- "OpenWriterAt": true,
- "PublicLink": false,
- "Purge": true,
- "PutStream": true,
- "PutUnchecked": false,
- "ReadMetadata": true,
- "ReadMimeType": false,
- "ServerSideAcrossConfigs": false,
- "SetTier": false,
- "SetWrapper": false,
- "Shutdown": false,
- "SlowHash": true,
- "SlowModTime": false,
- "UnWrap": false,
- "UserInfo": false,
- "UserMetadata": true,
- "WrapFs": false,
- "WriteMetadata": true,
- "WriteMimeType": false
- },
- // Names of hashes available
- "Hashes": [
- "md5",
- "sha1",
- "whirlpool",
- "crc32",
- "sha256",
- "dropbox",
- "mailru",
- "quickxor"
- ],
- "Name": "local", // Name as created
- "Precision": 1, // Precision of timestamps in ns
- "Root": "/", // Path as created
- "String": "Local file system at /", // how the remote will appear in logs
- // Information about the system metadata for this backend
- "MetadataInfo": {
- "System": {
- "atime": {
- "Help": "Time of last access",
- "Type": "RFC 3339",
- "Example": "2006-01-02T15:04:05.999999999Z07:00"
- },
- "btime": {
- "Help": "Time of file birth (creation)",
- "Type": "RFC 3339",
- "Example": "2006-01-02T15:04:05.999999999Z07:00"
- },
- "gid": {
- "Help": "Group ID of owner",
- "Type": "decimal number",
- "Example": "500"
- },
- "mode": {
- "Help": "File type and mode",
- "Type": "octal, unix style",
- "Example": "0100664"
- },
- "mtime": {
- "Help": "Time of last modification",
- "Type": "RFC 3339",
- "Example": "2006-01-02T15:04:05.999999999Z07:00"
- },
- "rdev": {
- "Help": "Device ID (if special file)",
- "Type": "hexadecimal",
- "Example": "1abc"
- },
- "uid": {
- "Help": "User ID of owner",
- "Type": "decimal number",
- "Example": "500"
- }
- },
- "Help": "Textual help string\n"
- }
+ "jobids": [
+ 2
+ ]
}
```
-This command does not have a command line equivalent so use this instead:
-
- rclone rc --loopback operations/fsinfo fs=remote:
+### Setting config flags with _config
-### operations/list: List the given remote and path in JSON format {#operations-list}
+If you wish to set config (the equivalent of the global flags) for the
+duration of an rc call only then pass in the `_config` parameter.
-This takes the following parameters:
+This should be in the same format as the `config` key returned by
+[options/get](#options-get).
-- fs - a remote name string e.g. "drive:"
-- remote - a path within that remote e.g. "dir"
-- opt - a dictionary of options to control the listing (optional)
- - recurse - If set recurse directories
- - noModTime - If set return modification time
- - showEncrypted - If set show decrypted names
- - showOrigIDs - If set show the IDs for each item if known
- - showHash - If set return a dictionary of hashes
- - noMimeType - If set don't show mime types
- - dirsOnly - If set only show directories
- - filesOnly - If set only show files
- - metadata - If set return metadata of objects also
- - hashTypes - array of strings of hash types to show if showHash set
+For example, if you wished to run a sync with the `--checksum`
+parameter, you would pass this parameter in your JSON blob.
-Returns:
+ "_config":{"CheckSum": true}
-- list
- - This is an array of objects as described in the lsjson command
+If using `rclone rc` this could be passed as
-See the [lsjson](https://rclone.org/commands/rclone_lsjson/) command for more information on the above and examples.
+ rclone rc sync/sync ... _config='{"CheckSum": true}'
-**Authentication is required for this call.**
+Any config parameters you don't set will inherit the global defaults
+which were set with command line flags or environment variables.
-### operations/mkdir: Make a destination directory or container {#operations-mkdir}
+Note that it is possible to set some values as strings or integers -
+see [data types](#data-types) for more info. Here is an example
+setting the equivalent of `--buffer-size` in string or integer format.
-This takes the following parameters:
+ "_config":{"BufferSize": "42M"}
+ "_config":{"BufferSize": 44040192}
-- fs - a remote name string e.g. "drive:"
-- remote - a path within that remote e.g. "dir"
+If you wish to check the `_config` assignment has worked properly then
+calling `options/local` will show what the value got set to.
-See the [mkdir](https://rclone.org/commands/rclone_mkdir/) command for more information on the above.
+### Setting filter flags with _filter
-**Authentication is required for this call.**
+If you wish to set filters for the duration of an rc call only then
+pass in the `_filter` parameter.
-### operations/movefile: Move a file from source remote to destination remote {#operations-movefile}
+This should be in the same format as the `filter` key returned by
+[options/get](#options-get).
-This takes the following parameters:
+For example, if you wished to run a sync with these flags
-- srcFs - a remote name string e.g. "drive:" for the source, "/" for local filesystem
-- srcRemote - a path within that remote e.g. "file.txt" for the source
-- dstFs - a remote name string e.g. "drive2:" for the destination, "/" for local filesystem
-- dstRemote - a path within that remote e.g. "file2.txt" for the destination
+ --max-size 1M --max-age 42s --include "a" --include "b"
-**Authentication is required for this call.**
+you would pass this parameter in your JSON blob.
-### operations/publiclink: Create or retrieve a public link to the given file or folder. {#operations-publiclink}
+ "_filter":{"MaxSize":"1M", "IncludeRule":["a","b"], "MaxAge":"42s"}
-This takes the following parameters:
+If using `rclone rc` this could be passed as
-- fs - a remote name string e.g. "drive:"
-- remote - a path within that remote e.g. "dir"
-- unlink - boolean - if set removes the link rather than adding it (optional)
-- expire - string - the expiry time of the link e.g. "1d" (optional)
+ rclone rc ... _filter='{"MaxSize":"1M", "IncludeRule":["a","b"], "MaxAge":"42s"}'
-Returns:
+Any filter parameters you don't set will inherit the global defaults
+which were set with command line flags or environment variables.
-- url - URL of the resource
+Note that it is possible to set some values as strings or integers -
+see [data types](#data-types) for more info. Here is an example
+setting the equivalent of `--buffer-size` in string or integer format.
-See the [link](https://rclone.org/commands/rclone_link/) command for more information on the above.
+ "_filter":{"MinSize": "42M"}
+ "_filter":{"MinSize": 44040192}
-**Authentication is required for this call.**
+If you wish to check the `_filter` assignment has worked properly then
+calling `options/local` will show what the value got set to.
-### operations/purge: Remove a directory or container and all of its contents {#operations-purge}
+### Assigning operations to groups with _group = value
-This takes the following parameters:
+Each rc call has its own stats group for tracking its metrics. By default
+grouping is done by the composite group name from prefix `job/` and id of the
+job like so `job/1`.
-- fs - a remote name string e.g. "drive:"
-- remote - a path within that remote e.g. "dir"
+If `_group` has a value then stats for that request will be grouped under that
+value. This allows caller to group stats under their own name.
-See the [purge](https://rclone.org/commands/rclone_purge/) command for more information on the above.
+Stats for specific group can be accessed by passing `group` to `core/stats`:
-**Authentication is required for this call.**
+```
+$ rclone rc --json '{ "group": "job/1" }' core/stats
+{
+ "speed": 12345
+ ...
+}
+```
-### operations/rmdir: Remove an empty directory or container {#operations-rmdir}
+## Data types {#data-types}
-This takes the following parameters:
+When the API returns types, these will mostly be straight forward
+integer, string or boolean types.
-- fs - a remote name string e.g. "drive:"
-- remote - a path within that remote e.g. "dir"
+However some of the types returned by the [options/get](#options-get)
+call and taken by the [options/set](#options-set) calls as well as the
+`vfsOpt`, `mountOpt` and the `_config` parameters.
-See the [rmdir](https://rclone.org/commands/rclone_rmdir/) command for more information on the above.
+- `Duration` - these are returned as an integer duration in
+ nanoseconds. They may be set as an integer, or they may be set with
+ time string, eg "5s". See the [options section](https://rclone.org/docs/#options) for
+ more info.
+- `Size` - these are returned as an integer number of bytes. They may
+ be set as an integer or they may be set with a size suffix string,
+ eg "10M". See the [options section](https://rclone.org/docs/#options) for more info.
+- Enumerated type (such as `CutoffMode`, `DumpFlags`, `LogLevel`,
+ `VfsCacheMode` - these will be returned as an integer and may be set
+ as an integer but more conveniently they can be set as a string, eg
+ "HARD" for `CutoffMode` or `DEBUG` for `LogLevel`.
+- `BandwidthSpec` - this will be set and returned as a string, eg
+ "1M".
-**Authentication is required for this call.**
+## Specifying remotes to work on
-### operations/rmdirs: Remove all the empty directories in the path {#operations-rmdirs}
+Remotes are specified with the `fs=`, `srcFs=`, `dstFs=`
+parameters depending on the command being used.
-This takes the following parameters:
+The parameters can be a string as per the rest of rclone, eg
+`s3:bucket/path` or `:sftp:/my/dir`. They can also be specified as
+JSON blobs.
-- fs - a remote name string e.g. "drive:"
-- remote - a path within that remote e.g. "dir"
-- leaveRoot - boolean, set to true not to delete the root
+If specifying a JSON blob it should be a object mapping strings to
+strings. These values will be used to configure the remote. There are
+3 special values which may be set:
-See the [rmdirs](https://rclone.org/commands/rclone_rmdirs/) command for more information on the above.
+- `type` - set to `type` to specify a remote called `:type:`
+- `_name` - set to `name` to specify a remote called `name:`
+- `_root` - sets the root of the remote - may be empty
-**Authentication is required for this call.**
+One of `_name` or `type` should normally be set. If the `local`
+backend is desired then `type` should be set to `local`. If `_root`
+isn't specified then it defaults to the root of the remote.
-### operations/settier: Changes storage tier or class on all files in the path {#operations-settier}
+For example this JSON is equivalent to `remote:/tmp`
-This takes the following parameters:
-
-- fs - a remote name string e.g. "drive:"
-
-See the [settier](https://rclone.org/commands/rclone_settier/) command for more information on the above.
-
-**Authentication is required for this call.**
+```
+{
+ "_name": "remote",
+ "_path": "/tmp"
+}
+```
-### operations/settierfile: Changes storage tier or class on the single file pointed to {#operations-settierfile}
+And this is equivalent to `:sftp,host='example.com':/tmp`
-This takes the following parameters:
+```
+{
+ "type": "sftp",
+ "host": "example.com",
+ "_path": "/tmp"
+}
+```
-- fs - a remote name string e.g. "drive:"
-- remote - a path within that remote e.g. "dir"
+And this is equivalent to `/tmp/dir`
-See the [settierfile](https://rclone.org/commands/rclone_settierfile/) command for more information on the above.
+```
+{
+ type = "local",
+ _ path = "/tmp/dir"
+}
+```
-**Authentication is required for this call.**
+## Supported commands
-### operations/size: Count the number of bytes and files in remote {#operations-size}
+### backend/command: Runs a backend command. {#backend-command}
This takes the following parameters:
-- fs - a remote name string e.g. "drive:path/to/dir"
+- command - a string with the command name
+- fs - a remote name string e.g. "drive:"
+- arg - a list of arguments for the backend command
+- opt - a map of string to string of options
Returns:
-- count - number of files
-- bytes - number of bytes in those files
-
-See the [size](https://rclone.org/commands/rclone_size/) command for more information on the above.
-
-**Authentication is required for this call.**
-
-### operations/stat: Give information about the supplied file or directory {#operations-stat}
-
-This takes the following parameters
-
-- fs - a remote name string eg "drive:"
-- remote - a path within that remote eg "dir"
-- opt - a dictionary of options to control the listing (optional)
- - see operations/list for the options
-
-The result is
+- result - result from the backend command
-- item - an object as described in the lsjson command. Will be null if not found.
+Example:
-Note that if you are only interested in files then it is much more
-efficient to set the filesOnly flag in the options.
+ rclone rc backend/command command=noop fs=. -o echo=yes -o blue -a path1 -a path2
-See the [lsjson](https://rclone.org/commands/rclone_lsjson/) command for more information on the above and examples.
+Returns
-**Authentication is required for this call.**
+```
+{
+ "result": {
+ "arg": [
+ "path1",
+ "path2"
+ ],
+ "name": "noop",
+ "opt": {
+ "blue": "",
+ "echo": "yes"
+ }
+ }
+}
+```
-### operations/uploadfile: Upload file using multiform/form-data {#operations-uploadfile}
+Note that this is the direct equivalent of using this "backend"
+command:
-This takes the following parameters:
+ rclone backend noop . -o echo=yes -o blue path1 path2
-- fs - a remote name string e.g. "drive:"
-- remote - a path within that remote e.g. "dir"
-- each part in body represents a file to be uploaded
+Note that arguments must be preceded by the "-a" flag
-See the [uploadfile](https://rclone.org/commands/rclone_uploadfile/) command for more information on the above.
+See the [backend](https://rclone.org/commands/rclone_backend/) command for more information.
**Authentication is required for this call.**
-### options/blocks: List all the option blocks {#options-blocks}
+### cache/expire: Purge a remote from cache {#cache-expire}
-Returns:
-- options - a list of the options block names
+Purge a remote from the cache backend. Supports either a directory or a file.
+Params:
+ - remote = path to remote (required)
+ - withData = true/false to delete cached data (chunks) as well (optional)
-### options/get: Get all the global options {#options-get}
+Eg
-Returns an object where keys are option block names and values are an
-object with the current option values in.
+ rclone rc cache/expire remote=path/to/sub/folder/
+ rclone rc cache/expire remote=/ withData=true
-Note that these are the global options which are unaffected by use of
-the _config and _filter parameters. If you wish to read the parameters
-set in _config then use options/config and for _filter use options/filter.
+### cache/fetch: Fetch file chunks {#cache-fetch}
-This shows the internal names of the option within rclone which should
-map to the external options very easily with a few exceptions.
+Ensure the specified file chunks are cached on disk.
-### options/local: Get the currently active config for this call {#options-local}
+The chunks= parameter specifies the file chunks to check.
+It takes a comma separated list of array slice indices.
+The slice indices are similar to Python slices: start[:end]
-Returns an object with the keys "config" and "filter".
-The "config" key contains the local config and the "filter" key contains
-the local filters.
+start is the 0 based chunk number from the beginning of the file
+to fetch inclusive. end is 0 based chunk number from the beginning
+of the file to fetch exclusive.
+Both values can be negative, in which case they count from the back
+of the file. The value "-5:" represents the last 5 chunks of a file.
-Note that these are the local options specific to this rc call. If
-_config was not supplied then they will be the global options.
-Likewise with "_filter".
+Some valid examples are:
+":5,-5:" -> the first and last five chunks
+"0,-2" -> the first and the second last chunk
+"0:10" -> the first ten chunks
-This call is mostly useful for seeing if _config and _filter passing
-is working.
+Any parameter with a key that starts with "file" can be used to
+specify files to fetch, e.g.
-This shows the internal names of the option within rclone which should
-map to the external options very easily with a few exceptions.
+ rclone rc cache/fetch chunks=0 file=hello file2=home/goodbye
-### options/set: Set an option {#options-set}
+File names will automatically be encrypted when the a crypt remote
+is used on top of the cache.
-Parameters:
+### cache/stats: Get cache stats {#cache-stats}
-- option block name containing an object with
- - key: value
+Show statistics for the cache remote.
-Repeated as often as required.
+### config/create: create the config for a remote. {#config-create}
-Only supply the options you wish to change. If an option is unknown
-it will be silently ignored. Not all options will have an effect when
-changed like this.
+This takes the following parameters:
-For example:
+- name - name of remote
+- parameters - a map of \{ "key": "value" \} pairs
+- type - type of the new remote
+- opt - a dictionary of options to control the configuration
+ - obscure - declare passwords are plain and need obscuring
+ - noObscure - declare passwords are already obscured and don't need obscuring
+ - nonInteractive - don't interact with a user, return questions
+ - continue - continue the config process with an answer
+ - all - ask all the config questions not just the post config ones
+ - state - state to restart with - used with continue
+ - result - result to restart with - used with continue
-This sets DEBUG level logs (-vv) (these can be set by number or string)
- rclone rc options/set --json '{"main": {"LogLevel": "DEBUG"}}'
- rclone rc options/set --json '{"main": {"LogLevel": 8}}'
+See the [config create](https://rclone.org/commands/rclone_config_create/) command for more information on the above.
-And this sets INFO level logs (-v)
+**Authentication is required for this call.**
- rclone rc options/set --json '{"main": {"LogLevel": "INFO"}}'
+### config/delete: Delete a remote in the config file. {#config-delete}
-And this sets NOTICE level logs (normal without -v)
+Parameters:
- rclone rc options/set --json '{"main": {"LogLevel": "NOTICE"}}'
+- name - name of remote to delete
-### pluginsctl/addPlugin: Add a plugin using url {#pluginsctl-addPlugin}
+See the [config delete](https://rclone.org/commands/rclone_config_delete/) command for more information on the above.
-Used for adding a plugin to the webgui.
+**Authentication is required for this call.**
-This takes the following parameters:
+### config/dump: Dumps the config file. {#config-dump}
-- url - http url of the github repo where the plugin is hosted (http://github.com/rclone/rclone-webui-react).
+Returns a JSON object:
+- key: value
-Example:
+Where keys are remote names and values are the config parameters.
- rclone rc pluginsctl/addPlugin
+See the [config dump](https://rclone.org/commands/rclone_config_dump/) command for more information on the above.
**Authentication is required for this call.**
-### pluginsctl/getPluginsForType: Get plugins with type criteria {#pluginsctl-getPluginsForType}
+### config/get: Get a remote in the config file. {#config-get}
-This shows all possible plugins by a mime type.
+Parameters:
-This takes the following parameters:
+- name - name of remote to get
-- type - supported mime type by a loaded plugin e.g. (video/mp4, audio/mp3).
-- pluginType - filter plugins based on their type e.g. (DASHBOARD, FILE_HANDLER, TERMINAL).
+See the [config dump](https://rclone.org/commands/rclone_config_dump/) command for more information on the above.
-Returns:
+**Authentication is required for this call.**
-- loadedPlugins - list of current production plugins.
-- testPlugins - list of temporarily loaded development plugins, usually running on a different server.
+### config/listremotes: Lists the remotes in the config file and defined in environment variables. {#config-listremotes}
-Example:
+Returns
+- remotes - array of remote names
- rclone rc pluginsctl/getPluginsForType type=video/mp4
+See the [listremotes](https://rclone.org/commands/rclone_listremotes/) command for more information on the above.
**Authentication is required for this call.**
-### pluginsctl/listPlugins: Get the list of currently loaded plugins {#pluginsctl-listPlugins}
-
-This allows you to get the currently enabled plugins and their details.
+### config/password: password the config for a remote. {#config-password}
-This takes no parameters and returns:
+This takes the following parameters:
-- loadedPlugins - list of current production plugins.
-- testPlugins - list of temporarily loaded development plugins, usually running on a different server.
+- name - name of remote
+- parameters - a map of \{ "key": "value" \} pairs
-E.g.
- rclone rc pluginsctl/listPlugins
+See the [config password](https://rclone.org/commands/rclone_config_password/) command for more information on the above.
**Authentication is required for this call.**
-### pluginsctl/listTestPlugins: Show currently loaded test plugins {#pluginsctl-listTestPlugins}
-
-Allows listing of test plugins with the rclone.test set to true in package.json of the plugin.
-
-This takes no parameters and returns:
-
-- loadedTestPlugins - list of currently available test plugins.
+### config/providers: Shows how providers are configured in the config file. {#config-providers}
-E.g.
+Returns a JSON object:
+- providers - array of objects
- rclone rc pluginsctl/listTestPlugins
+See the [config providers](https://rclone.org/commands/rclone_config_providers/) command for more information on the above.
**Authentication is required for this call.**
-### pluginsctl/removePlugin: Remove a loaded plugin {#pluginsctl-removePlugin}
-
-This allows you to remove a plugin using it's name.
-
-This takes parameters:
-
-- name - name of the plugin in the format `author`/`plugin_name`.
+### config/setpath: Set the path of the config file {#config-setpath}
-E.g.
+Parameters:
- rclone rc pluginsctl/removePlugin name=rclone/video-plugin
+- path - path to the config file to use
**Authentication is required for this call.**
-### pluginsctl/removeTestPlugin: Remove a test plugin {#pluginsctl-removeTestPlugin}
-
-This allows you to remove a plugin using it's name.
+### config/update: update the config for a remote. {#config-update}
This takes the following parameters:
-- name - name of the plugin in the format `author`/`plugin_name`.
+- name - name of remote
+- parameters - a map of \{ "key": "value" \} pairs
+- opt - a dictionary of options to control the configuration
+ - obscure - declare passwords are plain and need obscuring
+ - noObscure - declare passwords are already obscured and don't need obscuring
+ - nonInteractive - don't interact with a user, return questions
+ - continue - continue the config process with an answer
+ - all - ask all the config questions not just the post config ones
+ - state - state to restart with - used with continue
+ - result - result to restart with - used with continue
-Example:
- rclone rc pluginsctl/removeTestPlugin name=rclone/rclone-webui-react
+See the [config update](https://rclone.org/commands/rclone_config_update/) command for more information on the above.
**Authentication is required for this call.**
-### rc/error: This returns an error {#rc-error}
-
-This returns an error with the input as part of its error string.
-Useful for testing error handling.
-
-### rc/list: List all the registered remote control commands {#rc-list}
-
-This lists all the registered remote control commands as a JSON map in
-the commands response.
-
-### rc/noop: Echo the input to the output parameters {#rc-noop}
-
-This echoes the input parameters to the output parameters for testing
-purposes. It can be used to check that rclone is still alive and to
-check that parameter passing is working properly.
+### core/bwlimit: Set the bandwidth limit. {#core-bwlimit}
-### rc/noopauth: Echo the input to the output parameters requiring auth {#rc-noopauth}
+This sets the bandwidth limit to the string passed in. This should be
+a single bandwidth limit entry or a pair of upload:download bandwidth.
-This echoes the input parameters to the output parameters for testing
-purposes. It can be used to check that rclone is still alive and to
-check that parameter passing is working properly.
+Eg
-**Authentication is required for this call.**
+ rclone rc core/bwlimit rate=off
+ {
+ "bytesPerSecond": -1,
+ "bytesPerSecondTx": -1,
+ "bytesPerSecondRx": -1,
+ "rate": "off"
+ }
+ rclone rc core/bwlimit rate=1M
+ {
+ "bytesPerSecond": 1048576,
+ "bytesPerSecondTx": 1048576,
+ "bytesPerSecondRx": 1048576,
+ "rate": "1M"
+ }
+ rclone rc core/bwlimit rate=1M:100k
+ {
+ "bytesPerSecond": 1048576,
+ "bytesPerSecondTx": 1048576,
+ "bytesPerSecondRx": 131072,
+ "rate": "1M"
+ }
-### sync/bisync: Perform bidirectional synchronization between two paths. {#sync-bisync}
-This takes the following parameters
+If the rate parameter is not supplied then the bandwidth is queried
-- path1 - a remote directory string e.g. `drive:path1`
-- path2 - a remote directory string e.g. `drive:path2`
-- dryRun - dry-run mode
-- resync - performs the resync run
-- checkAccess - abort if RCLONE_TEST files are not found on both filesystems
-- checkFilename - file name for checkAccess (default: RCLONE_TEST)
-- maxDelete - abort sync if percentage of deleted files is above
- this threshold (default: 50)
-- force - Bypass maxDelete safety check and run the sync
-- checkSync - `true` by default, `false` disables comparison of final listings,
- `only` will skip sync, only compare listings from the last run
-- createEmptySrcDirs - Sync creation and deletion of empty directories.
- (Not compatible with --remove-empty-dirs)
-- removeEmptyDirs - remove empty directories at the final cleanup step
-- filtersFile - read filtering patterns from a file
-- ignoreListingChecksum - Do not use checksums for listings
-- resilient - Allow future runs to retry after certain less-serious errors, instead of requiring resync.
- Use at your own risk!
-- workdir - server directory for history files (default: /home/ncw/.cache/rclone/bisync)
-- noCleanup - retain working files
+ rclone rc core/bwlimit
+ {
+ "bytesPerSecond": 1048576,
+ "bytesPerSecondTx": 1048576,
+ "bytesPerSecondRx": 1048576,
+ "rate": "1M"
+ }
-See [bisync command help](https://rclone.org/commands/rclone_bisync/)
-and [full bisync description](https://rclone.org/bisync/)
-for more information.
+The format of the parameter is exactly the same as passed to --bwlimit
+except only one bandwidth may be specified.
-**Authentication is required for this call.**
+In either case "rate" is returned as a human-readable string, and
+"bytesPerSecond" is returned as a number.
-### sync/copy: copy a directory from source remote to destination remote {#sync-copy}
+### core/command: Run a rclone terminal command over rc. {#core-command}
This takes the following parameters:
-- srcFs - a remote name string e.g. "drive:src" for the source
-- dstFs - a remote name string e.g. "drive:dst" for the destination
-- createEmptySrcDirs - create empty src directories on destination if set
+- command - a string with the command name.
+- arg - a list of arguments for the backend command.
+- opt - a map of string to string of options.
+- returnType - one of ("COMBINED_OUTPUT", "STREAM", "STREAM_ONLY_STDOUT", "STREAM_ONLY_STDERR").
+ - Defaults to "COMBINED_OUTPUT" if not set.
+ - The STREAM returnTypes will write the output to the body of the HTTP message.
+ - The COMBINED_OUTPUT will write the output to the "result" parameter.
+Returns:
-See the [copy](https://rclone.org/commands/rclone_copy/) command for more information on the above.
+- result - result from the backend command.
+ - Only set when using returnType "COMBINED_OUTPUT".
+- error - set if rclone exits with an error code.
+- returnType - one of ("COMBINED_OUTPUT", "STREAM", "STREAM_ONLY_STDOUT", "STREAM_ONLY_STDERR").
-**Authentication is required for this call.**
+Example:
-### sync/move: move a directory from source remote to destination remote {#sync-move}
+ rclone rc core/command command=ls -a mydrive:/ -o max-depth=1
+ rclone rc core/command -a ls -a mydrive:/ -o max-depth=1
-This takes the following parameters:
+Returns:
-- srcFs - a remote name string e.g. "drive:src" for the source
-- dstFs - a remote name string e.g. "drive:dst" for the destination
-- createEmptySrcDirs - create empty src directories on destination if set
-- deleteEmptySrcDirs - delete empty src directories if set
+```
+{
+ "error": false,
+ "result": ""
+}
+OR
+{
+ "error": true,
+ "result": ""
+}
-See the [move](https://rclone.org/commands/rclone_move/) command for more information on the above.
+```
**Authentication is required for this call.**
-### sync/sync: sync a directory from source remote to destination remote {#sync-sync}
+### core/du: Returns disk usage of a locally attached disk. {#core-du}
-This takes the following parameters:
+This returns the disk usage for the local directory passed in as dir.
-- srcFs - a remote name string e.g. "drive:src" for the source
-- dstFs - a remote name string e.g. "drive:dst" for the destination
-- createEmptySrcDirs - create empty src directories on destination if set
+If the directory is not passed in, it defaults to the directory
+pointed to by --cache-dir.
+- dir - string (optional)
-See the [sync](https://rclone.org/commands/rclone_sync/) command for more information on the above.
+Returns:
-**Authentication is required for this call.**
+```
+{
+ "dir": "/",
+ "info": {
+ "Available": 361769115648,
+ "Free": 361785892864,
+ "Total": 982141468672
+ }
+}
+```
-### vfs/forget: Forget files or directories in the directory cache. {#vfs-forget}
+### core/gc: Runs a garbage collection. {#core-gc}
-This forgets the paths in the directory cache causing them to be
-re-read from the remote when needed.
+This tells the go runtime to do a garbage collection run. It isn't
+necessary to call this normally, but it can be useful for debugging
+memory problems.
-If no paths are passed in then it will forget all the paths in the
-directory cache.
+### core/group-list: Returns list of stats. {#core-group-list}
- rclone rc vfs/forget
+This returns list of stats groups currently in memory.
-Otherwise pass files or dirs in as file=path or dir=path. Any
-parameter key starting with file will forget that file and any
-starting with dir will forget that dir, e.g.
+Returns the following values:
+```
+{
+ "groups": an array of group names:
+ [
+ "group1",
+ "group2",
+ ...
+ ]
+}
+```
- rclone rc vfs/forget file=hello file2=goodbye dir=home/junk
-
-This command takes an "fs" parameter. If this parameter is not
-supplied and if there is only one VFS in use then that VFS will be
-used. If there is more than one VFS in use then the "fs" parameter
-must be supplied.
+### core/memstats: Returns the memory statistics {#core-memstats}
-### vfs/list: List active VFSes. {#vfs-list}
+This returns the memory statistics of the running program. What the values mean
+are explained in the go docs: https://golang.org/pkg/runtime/#MemStats
-This lists the active VFSes.
+The most interesting values for most people are:
-It returns a list under the key "vfses" where the values are the VFS
-names that could be passed to the other VFS commands in the "fs"
-parameter.
+- HeapAlloc - this is the amount of memory rclone is actually using
+- HeapSys - this is the amount of memory rclone has obtained from the OS
+- Sys - this is the total amount of memory requested from the OS
+ - It is virtual memory so may include unused memory
-### vfs/poll-interval: Get the status or update the value of the poll-interval option. {#vfs-poll-interval}
+### core/obscure: Obscures a string passed in. {#core-obscure}
-Without any parameter given this returns the current status of the
-poll-interval setting.
+Pass a clear string and rclone will obscure it for the config file:
+- clear - string
-When the interval=duration parameter is set, the poll-interval value
-is updated and the polling function is notified.
-Setting interval=0 disables poll-interval.
+Returns:
+- obscured - string
- rclone rc vfs/poll-interval interval=5m
+### core/pid: Return PID of current process {#core-pid}
-The timeout=duration parameter can be used to specify a time to wait
-for the current poll function to apply the new value.
-If timeout is less or equal 0, which is the default, wait indefinitely.
+This returns PID of current process.
+Useful for stopping rclone process.
-The new poll-interval value will only be active when the timeout is
-not reached.
+### core/quit: Terminates the app. {#core-quit}
-If poll-interval is updated or disabled temporarily, some changes
-might not get picked up by the polling function, depending on the
-used remote.
-
-This command takes an "fs" parameter. If this parameter is not
-supplied and if there is only one VFS in use then that VFS will be
-used. If there is more than one VFS in use then the "fs" parameter
-must be supplied.
+(Optional) Pass an exit code to be used for terminating the app:
+- exitCode - int
-### vfs/refresh: Refresh the directory cache. {#vfs-refresh}
+### core/stats: Returns stats about current transfers. {#core-stats}
-This reads the directories for the specified paths and freshens the
-directory cache.
+This returns all available stats:
-If no paths are passed in then it will refresh the root directory.
+ rclone rc core/stats
- rclone rc vfs/refresh
+If group is not provided then summed up stats for all groups will be
+returned.
-Otherwise pass directories in as dir=path. Any parameter key
-starting with dir will refresh that directory, e.g.
+Parameters
- rclone rc vfs/refresh dir=home/junk dir2=data/misc
+- group - name of the stats group (string)
-If the parameter recursive=true is given the whole directory tree
-will get refreshed. This refresh will use --fast-list if enabled.
-
-This command takes an "fs" parameter. If this parameter is not
-supplied and if there is only one VFS in use then that VFS will be
-used. If there is more than one VFS in use then the "fs" parameter
-must be supplied.
+Returns the following values:
-### vfs/stats: Stats for a VFS. {#vfs-stats}
+```
+{
+ "bytes": total transferred bytes since the start of the group,
+ "checks": number of files checked,
+ "deletes" : number of files deleted,
+ "elapsedTime": time in floating point seconds since rclone was started,
+ "errors": number of errors,
+ "eta": estimated time in seconds until the group completes,
+ "fatalError": boolean whether there has been at least one fatal error,
+ "lastError": last error string,
+ "renames" : number of files renamed,
+ "retryError": boolean showing whether there has been at least one non-NoRetryError,
+ "serverSideCopies": number of server side copies done,
+ "serverSideCopyBytes": number bytes server side copied,
+ "serverSideMoves": number of server side moves done,
+ "serverSideMoveBytes": number bytes server side moved,
+ "speed": average speed in bytes per second since start of the group,
+ "totalBytes": total number of bytes in the group,
+ "totalChecks": total number of checks in the group,
+ "totalTransfers": total number of transfers in the group,
+ "transferTime" : total time spent on running jobs,
+ "transfers": number of transferred files,
+ "transferring": an array of currently active file transfers:
+ [
+ {
+ "bytes": total transferred bytes for this file,
+ "eta": estimated time in seconds until file transfer completion
+ "name": name of the file,
+ "percentage": progress of the file transfer in percent,
+ "speed": average speed over the whole transfer in bytes per second,
+ "speedAvg": current speed in bytes per second as an exponentially weighted moving average,
+ "size": size of the file in bytes
+ }
+ ],
+ "checking": an array of names of currently active file checks
+ []
+}
+```
+Values for "transferring", "checking" and "lastError" are only assigned if data is available.
+The value for "eta" is null if an eta cannot be determined.
-This returns stats for the selected VFS.
+### core/stats-delete: Delete stats group. {#core-stats-delete}
- {
- // Status of the disk cache - only present if --vfs-cache-mode > off
- "diskCache": {
- "bytesUsed": 0,
- "erroredFiles": 0,
- "files": 0,
- "hashType": 1,
- "outOfSpace": false,
- "path": "/home/user/.cache/rclone/vfs/local/mnt/a",
- "pathMeta": "/home/user/.cache/rclone/vfsMeta/local/mnt/a",
- "uploadsInProgress": 0,
- "uploadsQueued": 0
- },
- "fs": "/mnt/a",
- "inUse": 1,
- // Status of the in memory metadata cache
- "metadataCache": {
- "dirs": 1,
- "files": 0
- },
- // Options as returned by options/get
- "opt": {
- "CacheMaxAge": 3600000000000,
- // ...
- "WriteWait": 1000000000
- }
- }
+This deletes entire stats group.
-
-This command takes an "fs" parameter. If this parameter is not
-supplied and if there is only one VFS in use then that VFS will be
-used. If there is more than one VFS in use then the "fs" parameter
-must be supplied.
+Parameters
+- group - name of the stats group (string)
+### core/stats-reset: Reset stats. {#core-stats-reset}
-## Accessing the remote control via HTTP {#api-http}
+This clears counters, errors and finished transfers for all stats or specific
+stats group if group is provided.
-Rclone implements a simple HTTP based protocol.
+Parameters
-Each endpoint takes an JSON object and returns a JSON object or an
-error. The JSON objects are essentially a map of string names to
-values.
+- group - name of the stats group (string)
-All calls must made using POST.
+### core/transferred: Returns stats about completed transfers. {#core-transferred}
-The input objects can be supplied using URL parameters, POST
-parameters or by supplying "Content-Type: application/json" and a JSON
-blob in the body. There are examples of these below using `curl`.
+This returns stats about completed transfers:
-The response will be a JSON blob in the body of the response. This is
-formatted to be reasonably human-readable.
+ rclone rc core/transferred
-### Error returns
+If group is not provided then completed transfers for all groups will be
+returned.
-If an error occurs then there will be an HTTP error status (e.g. 500)
-and the body of the response will contain a JSON encoded error object,
-e.g.
+Note only the last 100 completed transfers are returned.
+
+Parameters
+
+- group - name of the stats group (string)
+Returns the following values:
```
{
- "error": "Expecting string value for key \"remote\" (was float64)",
- "input": {
- "fs": "/tmp",
- "remote": 3
- },
- "status": 400
- "path": "operations/rmdir",
+ "transferred": an array of completed transfers (including failed ones):
+ [
+ {
+ "name": name of the file,
+ "size": size of the file in bytes,
+ "bytes": total transferred bytes for this file,
+ "checked": if the transfer is only checked (skipped, deleted),
+ "timestamp": integer representing millisecond unix epoch,
+ "error": string description of the error (empty if successful),
+ "jobid": id of the job that this transfer belongs to
+ }
+ ]
}
```
-The keys in the error response are
-- error - error string
-- input - the input parameters to the call
-- status - the HTTP status code
-- path - the path of the call
+### core/version: Shows the current version of rclone and the go runtime. {#core-version}
-### CORS
+This shows the current version of go and the go runtime:
-The sever implements basic CORS support and allows all origins for that.
-The response to a preflight OPTIONS request will echo the requested "Access-Control-Request-Headers" back.
+- version - rclone version, e.g. "v1.53.0"
+- decomposed - version number as [major, minor, patch]
+- isGit - boolean - true if this was compiled from the git version
+- isBeta - boolean - true if this is a beta version
+- os - OS in use as according to Go
+- arch - cpu architecture in use according to Go
+- goVersion - version of Go runtime in use
+- linking - type of rclone executable (static or dynamic)
+- goTags - space separated build tags or "none"
-### Using POST with URL parameters only
+### debug/set-block-profile-rate: Set runtime.SetBlockProfileRate for blocking profiling. {#debug-set-block-profile-rate}
-```
-curl -X POST 'http://localhost:5572/rc/noop?potato=1&sausage=2'
-```
+SetBlockProfileRate controls the fraction of goroutine blocking events
+that are reported in the blocking profile. The profiler aims to sample
+an average of one blocking event per rate nanoseconds spent blocked.
-Response
+To include every blocking event in the profile, pass rate = 1. To turn
+off profiling entirely, pass rate <= 0.
-```
-{
- "potato": "1",
- "sausage": "2"
-}
-```
+After calling this you can use this to see the blocking profile:
-Here is what an error response looks like:
+ go tool pprof http://localhost:5572/debug/pprof/block
-```
-curl -X POST 'http://localhost:5572/rc/error?potato=1&sausage=2'
-```
+Parameters:
-```
-{
- "error": "arbitrary error on input map[potato:1 sausage:2]",
- "input": {
- "potato": "1",
- "sausage": "2"
- }
-}
-```
+- rate - int
-Note that curl doesn't return errors to the shell unless you use the `-f` option
+### debug/set-gc-percent: Call runtime/debug.SetGCPercent for setting the garbage collection target percentage. {#debug-set-gc-percent}
-```
-$ curl -f -X POST 'http://localhost:5572/rc/error?potato=1&sausage=2'
-curl: (22) The requested URL returned error: 400 Bad Request
-$ echo $?
-22
-```
+SetGCPercent sets the garbage collection target percentage: a collection is triggered
+when the ratio of freshly allocated data to live data remaining after the previous collection
+reaches this percentage. SetGCPercent returns the previous setting. The initial setting is the
+value of the GOGC environment variable at startup, or 100 if the variable is not set.
-### Using POST with a form
+This setting may be effectively reduced in order to maintain a memory limit.
+A negative percentage effectively disables garbage collection, unless the memory limit is reached.
-```
-curl --data "potato=1" --data "sausage=2" http://localhost:5572/rc/noop
-```
+See https://pkg.go.dev/runtime/debug#SetMemoryLimit for more details.
-Response
+Parameters:
-```
-{
- "potato": "1",
- "sausage": "2"
-}
-```
+- gc-percent - int
-Note that you can combine these with URL parameters too with the POST
-parameters taking precedence.
+### debug/set-mutex-profile-fraction: Set runtime.SetMutexProfileFraction for mutex profiling. {#debug-set-mutex-profile-fraction}
-```
-curl --data "potato=1" --data "sausage=2" "http://localhost:5572/rc/noop?rutabaga=3&sausage=4"
-```
+SetMutexProfileFraction controls the fraction of mutex contention
+events that are reported in the mutex profile. On average 1/rate
+events are reported. The previous rate is returned.
-Response
+To turn off profiling entirely, pass rate 0. To just read the current
+rate, pass rate < 0. (For n>1 the details of sampling may change.)
-```
-{
- "potato": "1",
- "rutabaga": "3",
- "sausage": "4"
-}
+Once this is set you can look use this to profile the mutex contention:
-```
+ go tool pprof http://localhost:5572/debug/pprof/mutex
-### Using POST with a JSON blob
+Parameters:
-```
-curl -H "Content-Type: application/json" -X POST -d '{"potato":2,"sausage":1}' http://localhost:5572/rc/noop
-```
+- rate - int
-response
+Results:
-```
-{
- "password": "xyz",
- "username": "xyz"
-}
-```
+- previousRate - int
-This can be combined with URL parameters too if required. The JSON
-blob takes precedence.
+### debug/set-soft-memory-limit: Call runtime/debug.SetMemoryLimit for setting a soft memory limit for the runtime. {#debug-set-soft-memory-limit}
-```
-curl -H "Content-Type: application/json" -X POST -d '{"potato":2,"sausage":1}' 'http://localhost:5572/rc/noop?rutabaga=3&potato=4'
-```
+SetMemoryLimit provides the runtime with a soft memory limit.
-```
-{
- "potato": 2,
- "rutabaga": "3",
- "sausage": 1
-}
-```
+The runtime undertakes several processes to try to respect this memory limit, including
+adjustments to the frequency of garbage collections and returning memory to the underlying
+system more aggressively. This limit will be respected even if GOGC=off (or, if SetGCPercent(-1) is executed).
-## Debugging rclone with pprof ##
+The input limit is provided as bytes, and includes all memory mapped, managed, and not
+released by the Go runtime. Notably, it does not account for space used by the Go binary
+and memory external to Go, such as memory managed by the underlying system on behalf of
+the process, or memory managed by non-Go code inside the same process.
+Examples of excluded memory sources include: OS kernel memory held on behalf of the process,
+memory allocated by C code, and memory mapped by syscall.Mmap (because it is not managed by the Go runtime).
-If you use the `--rc` flag this will also enable the use of the go
-profiling tools on the same port.
+A zero limit or a limit that's lower than the amount of memory used by the Go runtime may cause
+the garbage collector to run nearly continuously. However, the application may still make progress.
-To use these, first [install go](https://golang.org/doc/install).
+The memory limit is always respected by the Go runtime, so to effectively disable this behavior,
+set the limit very high. math.MaxInt64 is the canonical value for disabling the limit, but values
+much greater than the available memory on the underlying system work just as well.
-### Debugging memory use
+See https://go.dev/doc/gc-guide for a detailed guide explaining the soft memory limit in more detail,
+as well as a variety of common use-cases and scenarios.
-To profile rclone's memory use you can run:
+SetMemoryLimit returns the previously set memory limit. A negative input does not adjust the limit,
+and allows for retrieval of the currently set memory limit.
- go tool pprof -web http://localhost:5572/debug/pprof/heap
+Parameters:
-This should open a page in your browser showing what is using what
-memory.
+- mem-limit - int
-You can also use the `-text` flag to produce a textual summary
+### fscache/clear: Clear the Fs cache. {#fscache-clear}
-```
-$ go tool pprof -text http://localhost:5572/debug/pprof/heap
-Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
- flat flat% sum% cum cum%
- 1024.03kB 66.62% 66.62% 1024.03kB 66.62% github.com/rclone/rclone/vendor/golang.org/x/net/http2/hpack.addDecoderNode
- 513kB 33.38% 100% 513kB 33.38% net/http.newBufioWriterSize
- 0 0% 100% 1024.03kB 66.62% github.com/rclone/rclone/cmd/all.init
- 0 0% 100% 1024.03kB 66.62% github.com/rclone/rclone/cmd/serve.init
- 0 0% 100% 1024.03kB 66.62% github.com/rclone/rclone/cmd/serve/restic.init
- 0 0% 100% 1024.03kB 66.62% github.com/rclone/rclone/vendor/golang.org/x/net/http2.init
- 0 0% 100% 1024.03kB 66.62% github.com/rclone/rclone/vendor/golang.org/x/net/http2/hpack.init
- 0 0% 100% 1024.03kB 66.62% github.com/rclone/rclone/vendor/golang.org/x/net/http2/hpack.init.0
- 0 0% 100% 1024.03kB 66.62% main.init
- 0 0% 100% 513kB 33.38% net/http.(*conn).readRequest
- 0 0% 100% 513kB 33.38% net/http.(*conn).serve
- 0 0% 100% 1024.03kB 66.62% runtime.main
-```
+This clears the fs cache. This is where remotes created from backends
+are cached for a short while to make repeated rc calls more efficient.
-### Debugging go routine leaks
+If you change the parameters of a backend then you may want to call
+this to clear an existing remote out of the cache before re-creating
+it.
-Memory leaks are most often caused by go routine leaks keeping memory
-alive which should have been garbage collected.
+**Authentication is required for this call.**
-See all active go routines using
+### fscache/entries: Returns the number of entries in the fs cache. {#fscache-entries}
- curl http://localhost:5572/debug/pprof/goroutine?debug=1
+This returns the number of entries in the fs cache.
-Or go to http://localhost:5572/debug/pprof/goroutine?debug=1 in your browser.
+Returns
+- entries - number of items in the cache
-### Other profiles to look at
+**Authentication is required for this call.**
-You can see a summary of profiles available at http://localhost:5572/debug/pprof/
+### job/list: Lists the IDs of the running jobs {#job-list}
-Here is how to use some of them:
+Parameters: None.
-- Memory: `go tool pprof http://localhost:5572/debug/pprof/heap`
-- Go routines: `curl http://localhost:5572/debug/pprof/goroutine?debug=1`
-- 30-second CPU profile: `go tool pprof http://localhost:5572/debug/pprof/profile`
-- 5-second execution trace: `wget http://localhost:5572/debug/pprof/trace?seconds=5`
-- Goroutine blocking profile
- - Enable first with: `rclone rc debug/set-block-profile-rate rate=1` ([docs](#debug-set-block-profile-rate))
- - `go tool pprof http://localhost:5572/debug/pprof/block`
-- Contended mutexes:
- - Enable first with: `rclone rc debug/set-mutex-profile-fraction rate=1` ([docs](#debug-set-mutex-profile-fraction))
- - `go tool pprof http://localhost:5572/debug/pprof/mutex`
+Results:
-See the [net/http/pprof docs](https://golang.org/pkg/net/http/pprof/)
-for more info on how to use the profiling and for a general overview
-see [the Go team's blog post on profiling go programs](https://blog.golang.org/profiling-go-programs).
+- executeId - string id of rclone executing (change after restart)
+- jobids - array of integer job ids (starting at 1 on each restart)
-The profiling hook is [zero overhead unless it is used](https://stackoverflow.com/q/26545159/164234).
+### job/status: Reads the status of the job ID {#job-status}
-# Overview of cloud storage systems #
+Parameters:
-Each cloud storage system is slightly different. Rclone attempts to
-provide a unified interface to them, but some underlying differences
-show through.
+- jobid - id of the job (integer).
-## Features ##
+Results:
-Here is an overview of the major features of each cloud storage system.
+- finished - boolean
+- duration - time in seconds that the job ran for
+- endTime - time the job finished (e.g. "2018-10-26T18:50:20.528746884+01:00")
+- error - error from the job or empty string for no error
+- finished - boolean whether the job has finished or not
+- id - as passed in above
+- startTime - time the job started (e.g. "2018-10-26T18:50:20.528336039+01:00")
+- success - boolean - true for success false otherwise
+- output - output of the job as would have been returned if called synchronously
+- progress - output of the progress related to the underlying job
-| Name | Hash | ModTime | Case Insensitive | Duplicate Files | MIME Type | Metadata |
-| ---------------------------- |:----------------:|:-------:|:----------------:|:---------------:|:---------:|:--------:|
-| 1Fichier | Whirlpool | - | No | Yes | R | - |
-| Akamai Netstorage | MD5, SHA256 | R/W | No | No | R | - |
-| Amazon Drive | MD5 | - | Yes | No | R | - |
-| Amazon S3 (or S3 compatible) | MD5 | R/W | No | No | R/W | RWU |
-| Backblaze B2 | SHA1 | R/W | No | No | R/W | - |
-| Box | SHA1 | R/W | Yes | No | - | - |
-| Citrix ShareFile | MD5 | R/W | Yes | No | - | - |
-| Dropbox | DBHASH ¹ | R | Yes | No | - | - |
-| Enterprise File Fabric | - | R/W | Yes | No | R/W | - |
-| FTP | - | R/W ¹Ⱐ| No | No | - | - |
-| Google Cloud Storage | MD5 | R/W | No | No | R/W | - |
-| Google Drive | MD5 | R/W | No | Yes | R/W | - |
-| Google Photos | - | - | No | Yes | R | - |
-| HDFS | - | R/W | No | No | - | - |
-| HiDrive | HiDrive ¹² | R/W | No | No | - | - |
-| HTTP | - | R | No | No | R | - |
-| Internet Archive | MD5, SHA1, CRC32 | R/W ¹¹ | No | No | - | RWU |
-| Jottacloud | MD5 | R/W | Yes | No | R | - |
-| Koofr | MD5 | - | Yes | No | - | - |
-| Mail.ru Cloud | Mailru ⶠ| R/W | Yes | No | - | - |
-| Mega | - | - | No | Yes | - | - |
-| Memory | MD5 | R/W | No | No | - | - |
-| Microsoft Azure Blob Storage | MD5 | R/W | No | No | R/W | - |
-| Microsoft OneDrive | QuickXorHash âµ | R/W | Yes | No | R | - |
-| OpenDrive | MD5 | R/W | Yes | Partial ⸠| - | - |
-| OpenStack Swift | MD5 | R/W | No | No | R/W | - |
-| Oracle Object Storage | MD5 | R/W | No | No | R/W | - |
-| pCloud | MD5, SHA1 â· | R | No | No | W | - |
-| PikPak | MD5 | R | No | No | R | - |
-| premiumize.me | - | - | Yes | No | R | - |
-| put.io | CRC-32 | R/W | No | Yes | R | - |
-| Proton Drive | SHA1 | R/W | No | No | R | - |
-| QingStor | MD5 | - â¹ | No | No | R/W | - |
-| Quatrix by Maytech | - | R/W | No | No | - | - |
-| Seafile | - | - | No | No | - | - |
-| SFTP | MD5, SHA1 ² | R/W | Depends | No | - | - |
-| Sia | - | - | No | No | - | - |
-| SMB | - | - | Yes | No | - | - |
-| SugarSync | - | - | No | No | - | - |
-| Storj | - | R | No | No | - | - |
-| Uptobox | - | - | No | Yes | - | - |
-| WebDAV | MD5, SHA1 ³ | R ⴠ| Depends | No | - | - |
-| Yandex Disk | MD5 | R/W | No | No | R | - |
-| Zoho WorkDrive | - | - | No | No | - | - |
-| The local filesystem | All | R/W | Depends | No | - | RWU |
-
-### Notes
+### job/stop: Stop the running job {#job-stop}
-¹ Dropbox supports [its own custom
-hash](https://www.dropbox.com/developers/reference/content-hash).
-This is an SHA256 sum of all the 4 MiB block SHA256s.
+Parameters:
-² SFTP supports checksums if the same login has shell access and
-`md5sum` or `sha1sum` as well as `echo` are in the remote's PATH.
+- jobid - id of the job (integer).
-³ WebDAV supports hashes when used with Fastmail Files, Owncloud and Nextcloud only.
+### job/stopgroup: Stop all running jobs in a group {#job-stopgroup}
-â´ WebDAV supports modtimes when used with Fastmail Files, Owncloud and Nextcloud only.
+Parameters:
-âµ [QuickXorHash](https://docs.microsoft.com/en-us/onedrive/developer/code-snippets/quickxorhash) is Microsoft's own hash.
+- group - name of the group (string).
-ⶠMail.ru uses its own modified SHA1 hash
+### mount/listmounts: Show current mount points {#mount-listmounts}
-â· pCloud only supports SHA1 (not MD5) in its EU region
+This shows currently mounted points, which can be used for performing an unmount.
-⸠Opendrive does not support creation of duplicate files using
-their web client interface or other stock clients, but the underlying
-storage platform has been determined to allow duplicate files, and it
-is possible to create them with `rclone`. It may be that this is a
-mistake or an unsupported feature.
+This takes no parameters and returns
-â¹ QingStor does not support SetModTime for objects bigger than 5 GiB.
+- mountPoints: list of current mount points
-¹ⰠFTP supports modtimes for the major FTP servers, and also others
-if they advertised required protocol extensions. See [this](https://rclone.org/ftp/#modified-time)
-for more details.
+Eg
-¹¹ Internet Archive requires option `wait_archive` to be set to a non-zero value
-for full modtime support.
+ rclone rc mount/listmounts
-¹² HiDrive supports [its own custom
-hash](https://static.hidrive.com/dev/0001).
-It combines SHA1 sums for each 4 KiB block hierarchically to a single
-top-level sum.
+**Authentication is required for this call.**
-### Hash ###
+### mount/mount: Create a new mount point {#mount-mount}
-The cloud storage system supports various hash types of the objects.
-The hashes are used when transferring data as an integrity check and
-can be specifically used with the `--checksum` flag in syncs and in
-the `check` command.
+rclone allows Linux, FreeBSD, macOS and Windows to mount any of
+Rclone's cloud storage systems as a file system with FUSE.
-To use the verify checksums when transferring between cloud storage
-systems they must support a common hash type.
+If no mountType is provided, the priority is given as follows: 1. mount 2.cmount 3.mount2
-### ModTime ###
+This takes the following parameters:
-Almost all cloud storage systems store some sort of timestamp
-on objects, but several of them not something that is appropriate
-to use for syncing. E.g. some backends will only write a timestamp
-that represent the time of the upload. To be relevant for syncing
-it should be able to store the modification time of the source
-object. If this is not the case, rclone will only check the file
-size by default, though can be configured to check the file hash
-(with the `--checksum` flag). Ideally it should also be possible to
-change the timestamp of an existing file without having to re-upload it.
+- fs - a remote path to be mounted (required)
+- mountPoint: valid path on the local machine (required)
+- mountType: one of the values (mount, cmount, mount2) specifies the mount implementation to use
+- mountOpt: a JSON object with Mount options in.
+- vfsOpt: a JSON object with VFS options in.
-Storage systems with a `-` in the ModTime column, means the
-modification read on objects is not the modification time of the
-file when uploaded. It is most likely the time the file was uploaded,
-or possibly something else (like the time the picture was taken in
-Google Photos).
+Example:
-Storage systems with a `R` (for read-only) in the ModTime column,
-means the it keeps modification times on objects, and updates them
-when uploading objects, but it does not support changing only the
-modification time (`SetModTime` operation) without re-uploading,
-possibly not even without deleting existing first. Some operations
-in rclone, such as `copy` and `sync` commands, will automatically
-check for `SetModTime` support and re-upload if necessary to keep
-the modification times in sync. Other commands will not work
-without `SetModTime` support, e.g. `touch` command on an existing
-file will fail, and changes to modification time only on a files
-in a `mount` will be silently ignored.
+ rclone rc mount/mount fs=mydrive: mountPoint=/home//mountPoint
+ rclone rc mount/mount fs=mydrive: mountPoint=/home//mountPoint mountType=mount
+ rclone rc mount/mount fs=TestDrive: mountPoint=/mnt/tmp vfsOpt='{"CacheMode": 2}' mountOpt='{"AllowOther": true}'
-Storage systems with `R/W` (for read/write) in the ModTime column,
-means they do also support modtime-only operations.
+The vfsOpt are as described in options/get and can be seen in the the
+"vfs" section when running and the mountOpt can be seen in the "mount" section:
-### Case Insensitive ###
+ rclone rc options/get
-If a cloud storage systems is case sensitive then it is possible to
-have two files which differ only in case, e.g. `file.txt` and
-`FILE.txt`. If a cloud storage system is case insensitive then that
-isn't possible.
+**Authentication is required for this call.**
-This can cause problems when syncing between a case insensitive
-system and a case sensitive system. The symptom of this is that no
-matter how many times you run the sync it never completes fully.
+### mount/types: Show all possible mount types {#mount-types}
-The local filesystem and SFTP may or may not be case sensitive
-depending on OS.
+This shows all possible mount types and returns them as a list.
- * Windows - usually case insensitive, though case is preserved
- * OSX - usually case insensitive, though it is possible to format case sensitive
- * Linux - usually case sensitive, but there are case insensitive file systems (e.g. FAT formatted USB keys)
+This takes no parameters and returns
-Most of the time this doesn't cause any problems as people tend to
-avoid files whose name differs only by case even on case sensitive
-systems.
+- mountTypes: list of mount types
-### Duplicate files ###
+The mount types are strings like "mount", "mount2", "cmount" and can
+be passed to mount/mount as the mountType parameter.
-If a cloud storage system allows duplicate files then it can have two
-objects with the same name.
+Eg
-This confuses rclone greatly when syncing - use the `rclone dedupe`
-command to rename or remove duplicates.
+ rclone rc mount/types
-### Restricted filenames ###
+**Authentication is required for this call.**
-Some cloud storage systems might have restrictions on the characters
-that are usable in file or directory names.
-When `rclone` detects such a name during a file upload, it will
-transparently replace the restricted characters with similar looking
-Unicode characters. To handle the different sets of restricted characters
-for different backends, rclone uses something it calls [encoding](#encoding).
+### mount/unmount: Unmount selected active mount {#mount-unmount}
-This process is designed to avoid ambiguous file names as much as
-possible and allow to move files between many cloud storage systems
-transparently.
+rclone allows Linux, FreeBSD, macOS and Windows to
+mount any of Rclone's cloud storage systems as a file system with
+FUSE.
-The name shown by `rclone` to the user or during log output will only
-contain a minimal set of [replaced characters](#restricted-characters)
-to ensure correct formatting and not necessarily the actual name used
-on the cloud storage.
+This takes the following parameters:
-This transformation is reversed when downloading a file or parsing
-`rclone` arguments. For example, when uploading a file named `my file?.txt`
-to Onedrive, it will be displayed as `my file?.txt` on the console, but
-stored as `my file？.txt` to Onedrive (the `?` gets replaced by the similar
-looking `？` character, the so-called "fullwidth question mark").
-The reverse transformation allows to read a file `unusual/name.txt`
-from Google Drive, by passing the name `unusualï¼name.txt` on the command line
-(the `/` needs to be replaced by the similar looking `ï¼` character).
+- mountPoint: valid path on the local machine where the mount was created (required)
-#### Caveats {#restricted-filenames-caveats}
+Example:
-The filename encoding system works well in most cases, at least
-where file names are written in English or similar languages.
-You might not even notice it: It just works. In some cases it may
-lead to issues, though. E.g. when file names are written in Chinese,
-or Japanese, where it is always the Unicode fullwidth variants of the
-punctuation marks that are used.
+ rclone rc mount/unmount mountPoint=/home//mountPoint
-On Windows, the characters `:`, `*` and `?` are examples of restricted
-characters. If these are used in filenames on a remote that supports it,
-Rclone will transparently convert them to their fullwidth Unicode
-variants `＊`, `？` and `：` when downloading to Windows, and back again
-when uploading. This way files with names that are not allowed on Windows
-can still be stored.
+**Authentication is required for this call.**
-However, if you have files on your Windows system originally with these same
-Unicode characters in their names, they will be included in the same conversion
-process. E.g. if you create a file in your Windows filesystem with name
-`Test：1.jpg`, where `：` is the Unicode fullwidth colon symbol, and use
-rclone to upload it to Google Drive, which supports regular `:` (halfwidth
-question mark), rclone will replace the fullwidth `:` with the
-halfwidth `:` and store the file as `Test:1.jpg` in Google Drive. Since
-both Windows and Google Drive allows the name `Test：1.jpg`, it would
-probably be better if rclone just kept the name as is in this case.
+### mount/unmountall: Unmount all active mounts {#mount-unmountall}
-With the opposite situation; if you have a file named `Test:1.jpg`,
-in your Google Drive, e.g. uploaded from a Linux system where `:` is valid
-in file names. Then later use rclone to copy this file to your Windows
-computer you will notice that on your local disk it gets renamed
-to `Test：1.jpg`. The original filename is not legal on Windows, due to
-the `:`, and rclone therefore renames it to make the copy possible.
-That is all good. However, this can also lead to an issue: If you already
-had a *different* file named `Test：1.jpg` on Windows, and then use rclone
-to copy either way. Rclone will then treat the file originally named
-`Test:1.jpg` on Google Drive and the file originally named `Test：1.jpg`
-on Windows as the same file, and replace the contents from one with the other.
+rclone allows Linux, FreeBSD, macOS and Windows to
+mount any of Rclone's cloud storage systems as a file system with
+FUSE.
-Its virtually impossible to handle all cases like these correctly in all
-situations, but by customizing the [encoding option](#encoding), changing the
-set of characters that rclone should convert, you should be able to
-create a configuration that works well for your specific situation.
-See also the [example](https://rclone.org/overview/#encoding-example-windows) below.
+This takes no parameters and returns error if unmount does not succeed.
-(Windows was used as an example of a file system with many restricted
-characters, and Google drive a storage system with few.)
+Eg
-#### Default restricted characters {#restricted-characters}
+ rclone rc mount/unmountall
-The table below shows the characters that are replaced by default.
+**Authentication is required for this call.**
-When a replacement character is found in a filename, this character
-will be escaped with the `‛` character to avoid ambiguous file names.
-(e.g. a file named `â€.txt` would shown as `‛â€.txt`)
+### operations/about: Return the space used on the remote {#operations-about}
-Each cloud storage backend can use a different set of characters,
-which will be specified in the documentation for each backend.
+This takes the following parameters:
-| Character | Value | Replacement |
-| --------- |:-----:|:-----------:|
-| NUL | 0x00 | †|
-| SOH | 0x01 | â |
-| STX | 0x02 | â‚ |
-| ETX | 0x03 | ⃠|
-| EOT | 0x04 | â„ |
-| ENQ | 0x05 | â… |
-| ACK | 0x06 | ↠|
-| BEL | 0x07 | ⇠|
-| BS | 0x08 | ∠|
-| HT | 0x09 | ≠|
-| LF | 0x0A | ⊠|
-| VT | 0x0B | â‹ |
-| FF | 0x0C | ⌠|
-| CR | 0x0D | â |
-| SO | 0x0E | ⎠|
-| SI | 0x0F | â |
-| DLE | 0x10 | â |
-| DC1 | 0x11 | â‘ |
-| DC2 | 0x12 | â’ |
-| DC3 | 0x13 | â“ |
-| DC4 | 0x14 | â” |
-| NAK | 0x15 | â• |
-| SYN | 0x16 | â– |
-| ETB | 0x17 | â— |
-| CAN | 0x18 | ☠|
-| EM | 0x19 | â™ |
-| SUB | 0x1A | âš |
-| ESC | 0x1B | â› |
-| FS | 0x1C | ✠|
-| GS | 0x1D | â |
-| RS | 0x1E | âž |
-| US | 0x1F | ⟠|
-| / | 0x2F | ï¼ |
-| DEL | 0x7F | â¡ |
+- fs - a remote name string e.g. "drive:"
-The default encoding will also encode these file names as they are
-problematic with many cloud storage systems.
+The result is as returned from rclone about --json
-| File name | Replacement |
-| --------- |:-----------:|
-| . | ． |
-| .. | ．． |
+See the [about](https://rclone.org/commands/rclone_about/) command for more information on the above.
-#### Invalid UTF-8 bytes {#invalid-utf8}
+**Authentication is required for this call.**
-Some backends only support a sequence of well formed UTF-8 bytes
-as file or directory names.
+### operations/check: check the source and destination are the same {#operations-check}
-In this case all invalid UTF-8 bytes will be replaced with a quoted
-representation of the byte value to allow uploading a file to such a
-backend. For example, the invalid byte `0xFE` will be encoded as `‛FE`.
+Checks the files in the source and destination match. It compares
+sizes and hashes and logs a report of files that don't
+match. It doesn't alter the source or destination.
-A common source of invalid UTF-8 bytes are local filesystems, that store
-names in a different encoding than UTF-8 or UTF-16, like latin1. See the
-[local filenames](https://rclone.org/local/#filenames) section for details.
+This takes the following parameters:
-#### Encoding option {#encoding}
+- srcFs - a remote name string e.g. "drive:" for the source, "/" for local filesystem
+- dstFs - a remote name string e.g. "drive2:" for the destination, "/" for local filesystem
+- download - check by downloading rather than with hash
+- checkFileHash - treat checkFileFs:checkFileRemote as a SUM file with hashes of given type
+- checkFileFs - treat checkFileFs:checkFileRemote as a SUM file with hashes of given type
+- checkFileRemote - treat checkFileFs:checkFileRemote as a SUM file with hashes of given type
+- oneWay - check one way only, source files must exist on remote
+- combined - make a combined report of changes (default false)
+- missingOnSrc - report all files missing from the source (default true)
+- missingOnDst - report all files missing from the destination (default true)
+- match - report all matching files (default false)
+- differ - report all non-matching files (default true)
+- error - report all files with errors (hashing or reading) (default true)
+
+If you supply the download flag, it will download the data from
+both remotes and check them against each other on the fly. This can
+be useful for remotes that don't support hashes or if you really want
+to check all the data.
-Most backends have an encoding option, specified as a flag
-`--backend-encoding` where `backend` is the name of the backend, or as
-a config parameter `encoding` (you'll need to select the Advanced
-config in `rclone config` to see it).
+If you supply the size-only global flag, it will only compare the sizes not
+the hashes as well. Use this for a quick check.
-This will have default value which encodes and decodes characters in
-such a way as to preserve the maximum number of characters (see
-above).
+If you supply the checkFileHash option with a valid hash name, the
+checkFileFs:checkFileRemote must point to a text file in the SUM
+format. This treats the checksum file as the source and dstFs as the
+destination. Note that srcFs is not used and should not be supplied in
+this case.
-However this can be incorrect in some scenarios, for example if you
-have a Windows file system with Unicode fullwidth characters
-`＊`, `？` or `：`, that you want to remain as those characters on the
-remote rather than being translated to regular (halfwidth) `*`, `?` and `:`.
+Returns:
-The `--backend-encoding` flags allow you to change that. You can
-disable the encoding completely with `--backend-encoding None` or set
-`encoding = None` in the config file.
+- success - true if no error, false otherwise
+- status - textual summary of check, OK or text string
+- hashType - hash used in check, may be missing
+- combined - array of strings of combined report of changes
+- missingOnSrc - array of strings of all files missing from the source
+- missingOnDst - array of strings of all files missing from the destination
+- match - array of strings of all matching files
+- differ - array of strings of all non-matching files
+- error - array of strings of all files with errors (hashing or reading)
-Encoding takes a comma separated list of encodings. You can see the
-list of all possible values by passing an invalid value to this
-flag, e.g. `--local-encoding "help"`. The command `rclone help flags encoding`
-will show you the defaults for the backends.
+**Authentication is required for this call.**
-| Encoding | Characters | Encoded as |
-| --------- | ---------- | ---------- |
-| Asterisk | `*` | `＊` |
-| BackQuote | `` ` `` | `ï½€` |
-| BackSlash | `\` | `ï¼¼` |
-| Colon | `:` | `：` |
-| CrLf | CR 0x0D, LF 0x0A | `â`, `âŠ` |
-| Ctl | All control characters 0x00-0x1F | `â€ââ‚âƒâ„â…â†â‡âˆâ‰âŠâ‹âŒââŽâââ‘â’â“â”â•â–â—â˜â™âšâ›âœââžâŸ` |
-| Del | DEL 0x7F | `â¡` |
-| Dollar | `$` | `＄` |
-| Dot | `.` or `..` as entire string | `．`, `．．` |
-| DoubleQuote | `"` | `＂` |
-| Hash | `#` | `＃` |
-| InvalidUtf8 | An invalid UTF-8 character (e.g. latin1) | `�` |
-| LeftCrLfHtVt | CR 0x0D, LF 0x0A, HT 0x09, VT 0x0B on the left of a string | `â`, `âŠ`, `â‰`, `â‹` |
-| LeftPeriod | `.` on the left of a string | `.` |
-| LeftSpace | SPACE on the left of a string | `â ` |
-| LeftTilde | `~` on the left of a string | `～` |
-| LtGt | `<`, `>` | `＜`, `＞` |
-| None | No characters are encoded | |
-| Percent | `%` | `ï¼…` |
-| Pipe | \| | `｜` |
-| Question | `?` | `？` |
-| RightCrLfHtVt | CR 0x0D, LF 0x0A, HT 0x09, VT 0x0B on the right of a string | `â`, `âŠ`, `â‰`, `â‹` |
-| RightPeriod | `.` on the right of a string | `.` |
-| RightSpace | SPACE on the right of a string | `â ` |
-| Semicolon | `;` | `ï¼›` |
-| SingleQuote | `'` | `＇` |
-| Slash | `/` | `ï¼` |
-| SquareBracket | `[`, `]` | `ï¼»`, `ï¼½` |
+### operations/cleanup: Remove trashed files in the remote or path {#operations-cleanup}
-##### Encoding example: FTP
+This takes the following parameters:
-To take a specific example, the FTP backend's default encoding is
+- fs - a remote name string e.g. "drive:"
- --ftp-encoding "Slash,Del,Ctl,RightSpace,Dot"
+See the [cleanup](https://rclone.org/commands/rclone_cleanup/) command for more information on the above.
-However, let's say the FTP server is running on Windows and can't have
-any of the invalid Windows characters in file names. You are backing
-up Linux servers to this FTP server which do have those characters in
-file names. So you would add the Windows set which are
+**Authentication is required for this call.**
- Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot
+### operations/copyfile: Copy a file from source remote to destination remote {#operations-copyfile}
-to the existing ones, giving:
+This takes the following parameters:
- Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot,Del,RightSpace
+- srcFs - a remote name string e.g. "drive:" for the source, "/" for local filesystem
+- srcRemote - a path within that remote e.g. "file.txt" for the source
+- dstFs - a remote name string e.g. "drive2:" for the destination, "/" for local filesystem
+- dstRemote - a path within that remote e.g. "file2.txt" for the destination
-This can be specified using the `--ftp-encoding` flag or using an `encoding` parameter in the config file.
+**Authentication is required for this call.**
-##### Encoding example: Windows
+### operations/copyurl: Copy the URL to the object {#operations-copyurl}
-As a nother example, take a Windows system where there is a file with
-name `Test：1.jpg`, where `：` is the Unicode fullwidth colon symbol.
-When using rclone to copy this to a remote which supports `:`,
-the regular (halfwidth) colon (such as Google Drive), you will notice
-that the file gets renamed to `Test:1.jpg`.
+This takes the following parameters:
-To avoid this you can change the set of characters rclone should convert
-for the local filesystem, using command-line argument `--local-encoding`.
-Rclone's default behavior on Windows corresponds to
+- fs - a remote name string e.g. "drive:"
+- remote - a path within that remote e.g. "dir"
+- url - string, URL to read from
+ - autoFilename - boolean, set to true to retrieve destination file name from url
-```
---local-encoding "Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot"
-```
+See the [copyurl](https://rclone.org/commands/rclone_copyurl/) command for more information on the above.
-If you want to use fullwidth characters `：`, `＊` and `？` in your filenames
-without rclone changing them when uploading to a remote, then set the same as
-the default value but without `Colon,Question,Asterisk`:
+**Authentication is required for this call.**
-```
---local-encoding "Slash,LtGt,DoubleQuote,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot"
-```
+### operations/delete: Remove files in the path {#operations-delete}
-Alternatively, you can disable the conversion of any characters with `--local-encoding None`.
+This takes the following parameters:
-Instead of using command-line argument `--local-encoding`, you may also set it
-as [environment variable](https://rclone.org/docs/#environment-variables) `RCLONE_LOCAL_ENCODING`,
-or [configure](https://rclone.org/docs/#configure) a remote of type `local` in your config,
-and set the `encoding` option there.
+- fs - a remote name string e.g. "drive:"
-The risk by doing this is that if you have a filename with the regular (halfwidth)
-`:`, `*` and `?` in your cloud storage, and you try to download
-it to your Windows filesystem, this will fail. These characters are not
-valid in filenames on Windows, and you have told rclone not to work around
-this by converting them to valid fullwidth variants.
+See the [delete](https://rclone.org/commands/rclone_delete/) command for more information on the above.
-### MIME Type ###
+**Authentication is required for this call.**
-MIME types (also known as media types) classify types of documents
-using a simple text classification, e.g. `text/html` or
-`application/pdf`.
+### operations/deletefile: Remove the single file pointed to {#operations-deletefile}
-Some cloud storage systems support reading (`R`) the MIME type of
-objects and some support writing (`W`) the MIME type of objects.
+This takes the following parameters:
-The MIME type can be important if you are serving files directly to
-HTTP from the storage system.
+- fs - a remote name string e.g. "drive:"
+- remote - a path within that remote e.g. "dir"
-If you are copying from a remote which supports reading (`R`) to a
-remote which supports writing (`W`) then rclone will preserve the MIME
-types. Otherwise they will be guessed from the extension, or the
-remote itself may assign the MIME type.
+See the [deletefile](https://rclone.org/commands/rclone_deletefile/) command for more information on the above.
-### Metadata
+**Authentication is required for this call.**
-Backends may or may support reading or writing metadata. They may
-support reading and writing system metadata (metadata intrinsic to
-that backend) and/or user metadata (general purpose metadata).
+### operations/fsinfo: Return information about the remote {#operations-fsinfo}
-The levels of metadata support are
+This takes the following parameters:
-| Key | Explanation |
-|-----|-------------|
-| `R` | Read only System Metadata |
-| `RW` | Read and write System Metadata |
-| `RWU` | Read and write System Metadata and read and write User Metadata |
+- fs - a remote name string e.g. "drive:"
-See [the metadata docs](https://rclone.org/docs/#metadata) for more info.
+This returns info about the remote passed in;
-## Optional Features ##
+```
+{
+ // optional features and whether they are available or not
+ "Features": {
+ "About": true,
+ "BucketBased": false,
+ "BucketBasedRootOK": false,
+ "CanHaveEmptyDirectories": true,
+ "CaseInsensitive": false,
+ "ChangeNotify": false,
+ "CleanUp": false,
+ "Command": true,
+ "Copy": false,
+ "DirCacheFlush": false,
+ "DirMove": true,
+ "Disconnect": false,
+ "DuplicateFiles": false,
+ "GetTier": false,
+ "IsLocal": true,
+ "ListR": false,
+ "MergeDirs": false,
+ "MetadataInfo": true,
+ "Move": true,
+ "OpenWriterAt": true,
+ "PublicLink": false,
+ "Purge": true,
+ "PutStream": true,
+ "PutUnchecked": false,
+ "ReadMetadata": true,
+ "ReadMimeType": false,
+ "ServerSideAcrossConfigs": false,
+ "SetTier": false,
+ "SetWrapper": false,
+ "Shutdown": false,
+ "SlowHash": true,
+ "SlowModTime": false,
+ "UnWrap": false,
+ "UserInfo": false,
+ "UserMetadata": true,
+ "WrapFs": false,
+ "WriteMetadata": true,
+ "WriteMimeType": false
+ },
+ // Names of hashes available
+ "Hashes": [
+ "md5",
+ "sha1",
+ "whirlpool",
+ "crc32",
+ "sha256",
+ "dropbox",
+ "mailru",
+ "quickxor"
+ ],
+ "Name": "local", // Name as created
+ "Precision": 1, // Precision of timestamps in ns
+ "Root": "/", // Path as created
+ "String": "Local file system at /", // how the remote will appear in logs
+ // Information about the system metadata for this backend
+ "MetadataInfo": {
+ "System": {
+ "atime": {
+ "Help": "Time of last access",
+ "Type": "RFC 3339",
+ "Example": "2006-01-02T15:04:05.999999999Z07:00"
+ },
+ "btime": {
+ "Help": "Time of file birth (creation)",
+ "Type": "RFC 3339",
+ "Example": "2006-01-02T15:04:05.999999999Z07:00"
+ },
+ "gid": {
+ "Help": "Group ID of owner",
+ "Type": "decimal number",
+ "Example": "500"
+ },
+ "mode": {
+ "Help": "File type and mode",
+ "Type": "octal, unix style",
+ "Example": "0100664"
+ },
+ "mtime": {
+ "Help": "Time of last modification",
+ "Type": "RFC 3339",
+ "Example": "2006-01-02T15:04:05.999999999Z07:00"
+ },
+ "rdev": {
+ "Help": "Device ID (if special file)",
+ "Type": "hexadecimal",
+ "Example": "1abc"
+ },
+ "uid": {
+ "Help": "User ID of owner",
+ "Type": "decimal number",
+ "Example": "500"
+ }
+ },
+ "Help": "Textual help string\n"
+ }
+}
+```
-All rclone remotes support a base command set. Other features depend
-upon backend-specific capabilities.
+This command does not have a command line equivalent so use this instead:
-| Name | Purge | Copy | Move | DirMove | CleanUp | ListR | StreamUpload | MultithreadUpload | LinkSharing | About | EmptyDir |
-| ---------------------------- |:-----:|:----:|:----:|:-------:|:-------:|:-----:|:------------:|:------------------|:------------:|:-----:|:--------:|
-| 1Fichier | No | Yes | Yes | No | No | No | No | No | Yes | No | Yes |
-| Akamai Netstorage | Yes | No | No | No | No | Yes | Yes | No | No | No | Yes |
-| Amazon Drive | Yes | No | Yes | Yes | No | No | No | No | No | No | Yes |
-| Amazon S3 (or S3 compatible) | No | Yes | No | No | Yes | Yes | Yes | Yes | Yes | No | No |
-| Backblaze B2 | No | Yes | No | No | Yes | Yes | Yes | Yes | Yes | No | No |
-| Box | Yes | Yes | Yes | Yes | Yes ‡‡ | No | Yes | No | Yes | Yes | Yes |
-| Citrix ShareFile | Yes | Yes | Yes | Yes | No | No | No | No | No | No | Yes |
-| Dropbox | Yes | Yes | Yes | Yes | No | No | Yes | No | Yes | Yes | Yes |
-| Enterprise File Fabric | Yes | Yes | Yes | Yes | Yes | No | No | No | No | No | Yes |
-| FTP | No | No | Yes | Yes | No | No | Yes | No | No | No | Yes |
-| Google Cloud Storage | Yes | Yes | No | No | No | Yes | Yes | No | No | No | No |
-| Google Drive | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No | Yes | Yes | Yes |
-| Google Photos | No | No | No | No | No | No | No | No | No | No | No |
-| HDFS | Yes | No | Yes | Yes | No | No | Yes | No | No | Yes | Yes |
-| HiDrive | Yes | Yes | Yes | Yes | No | No | Yes | No | No | No | Yes |
-| HTTP | No | No | No | No | No | No | No | No | No | No | Yes |
-| Internet Archive | No | Yes | No | No | Yes | Yes | No | No | Yes | Yes | No |
-| Jottacloud | Yes | Yes | Yes | Yes | Yes | Yes | No | No | Yes | Yes | Yes |
-| Koofr | Yes | Yes | Yes | Yes | No | No | Yes | No | Yes | Yes | Yes |
-| Mail.ru Cloud | Yes | Yes | Yes | Yes | Yes | No | No | No | Yes | Yes | Yes |
-| Mega | Yes | No | Yes | Yes | Yes | No | No | No | Yes | Yes | Yes |
-| Memory | No | Yes | No | No | No | Yes | Yes | No | No | No | No |
-| Microsoft Azure Blob Storage | Yes | Yes | No | No | No | Yes | Yes | Yes | No | No | No |
-| Microsoft OneDrive | Yes | Yes | Yes | Yes | Yes | No | No | No | Yes | Yes | Yes |
-| OpenDrive | Yes | Yes | Yes | Yes | No | No | No | No | No | No | Yes |
-| OpenStack Swift | Yes †| Yes | No | No | No | Yes | Yes | No | No | Yes | No |
-| Oracle Object Storage | No | Yes | No | No | Yes | Yes | Yes | No | No | No | No |
-| pCloud | Yes | Yes | Yes | Yes | Yes | No | No | No | Yes | Yes | Yes |
-| PikPak | Yes | Yes | Yes | Yes | Yes | No | No | No | Yes | Yes | Yes |
-| premiumize.me | Yes | No | Yes | Yes | No | No | No | No | Yes | Yes | Yes |
-| put.io | Yes | No | Yes | Yes | Yes | No | Yes | No | No | Yes | Yes |
-| Proton Drive | Yes | No | Yes | Yes | Yes | No | No | No | No | Yes | Yes |
-| QingStor | No | Yes | No | No | Yes | Yes | No | No | No | No | No |
-| Quatrix by Maytech | Yes | Yes | Yes | Yes | No | No | No | No | No | Yes | Yes |
-| Seafile | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No | Yes | Yes | Yes |
-| SFTP | No | No | Yes | Yes | No | No | Yes | No | No | Yes | Yes |
-| Sia | No | No | No | No | No | No | Yes | No | No | No | Yes |
-| SMB | No | No | Yes | Yes | No | No | Yes | Yes | No | No | Yes |
-| SugarSync | Yes | Yes | Yes | Yes | No | No | Yes | No | Yes | No | Yes |
-| Storj | Yes ☨ | Yes | Yes | No | No | Yes | Yes | No | Yes | No | No |
-| Uptobox | No | Yes | Yes | Yes | No | No | No | No | No | No | No |
-| WebDAV | Yes | Yes | Yes | Yes | No | No | Yes ‡ | No | No | Yes | Yes |
-| Yandex Disk | Yes | Yes | Yes | Yes | Yes | No | Yes | No | Yes | Yes | Yes |
-| Zoho WorkDrive | Yes | Yes | Yes | Yes | No | No | No | No | No | Yes | Yes |
-| The local filesystem | Yes | No | Yes | Yes | No | No | Yes | Yes | No | Yes | Yes |
+ rclone rc --loopback operations/fsinfo fs=remote:
-### Purge ###
+### operations/list: List the given remote and path in JSON format {#operations-list}
-This deletes a directory quicker than just deleting all the files in
-the directory.
+This takes the following parameters:
-†Note Swift implements this in order to delete directory markers but
-they don't actually have a quicker way of deleting files other than
-deleting them individually.
+- fs - a remote name string e.g. "drive:"
+- remote - a path within that remote e.g. "dir"
+- opt - a dictionary of options to control the listing (optional)
+ - recurse - If set recurse directories
+ - noModTime - If set return modification time
+ - showEncrypted - If set show decrypted names
+ - showOrigIDs - If set show the IDs for each item if known
+ - showHash - If set return a dictionary of hashes
+ - noMimeType - If set don't show mime types
+ - dirsOnly - If set only show directories
+ - filesOnly - If set only show files
+ - metadata - If set return metadata of objects also
+ - hashTypes - array of strings of hash types to show if showHash set
-☨ Storj implements this efficiently only for entire buckets. If
-purging a directory inside a bucket, files are deleted individually.
+Returns:
-‡ StreamUpload is not supported with Nextcloud
+- list
+ - This is an array of objects as described in the lsjson command
-### Copy ###
+See the [lsjson](https://rclone.org/commands/rclone_lsjson/) command for more information on the above and examples.
-Used when copying an object to and from the same remote. This known
-as a server-side copy so you can copy a file without downloading it
-and uploading it again. It is used if you use `rclone copy` or
-`rclone move` if the remote doesn't support `Move` directly.
+**Authentication is required for this call.**
-If the server doesn't support `Copy` directly then for copy operations
-the file is downloaded then re-uploaded.
+### operations/mkdir: Make a destination directory or container {#operations-mkdir}
-### Move ###
+This takes the following parameters:
-Used when moving/renaming an object on the same remote. This is known
-as a server-side move of a file. This is used in `rclone move` if the
-server doesn't support `DirMove`.
+- fs - a remote name string e.g. "drive:"
+- remote - a path within that remote e.g. "dir"
-If the server isn't capable of `Move` then rclone simulates it with
-`Copy` then delete. If the server doesn't support `Copy` then rclone
-will download the file and re-upload it.
+See the [mkdir](https://rclone.org/commands/rclone_mkdir/) command for more information on the above.
-### DirMove ###
+**Authentication is required for this call.**
-This is used to implement `rclone move` to move a directory if
-possible. If it isn't then it will use `Move` on each file (which
-falls back to `Copy` then download and upload - see `Move` section).
+### operations/movefile: Move a file from source remote to destination remote {#operations-movefile}
-### CleanUp ###
+This takes the following parameters:
-This is used for emptying the trash for a remote by `rclone cleanup`.
+- srcFs - a remote name string e.g. "drive:" for the source, "/" for local filesystem
+- srcRemote - a path within that remote e.g. "file.txt" for the source
+- dstFs - a remote name string e.g. "drive2:" for the destination, "/" for local filesystem
+- dstRemote - a path within that remote e.g. "file2.txt" for the destination
-If the server can't do `CleanUp` then `rclone cleanup` will return an
-error.
+**Authentication is required for this call.**
-‡‡ Note that while Box implements this it has to delete every file
-individually so it will be slower than emptying the trash via the WebUI
+### operations/publiclink: Create or retrieve a public link to the given file or folder. {#operations-publiclink}
-### ListR ###
+This takes the following parameters:
-The remote supports a recursive list to list all the contents beneath
-a directory quickly. This enables the `--fast-list` flag to work.
-See the [rclone docs](https://rclone.org/docs/#fast-list) for more details.
+- fs - a remote name string e.g. "drive:"
+- remote - a path within that remote e.g. "dir"
+- unlink - boolean - if set removes the link rather than adding it (optional)
+- expire - string - the expiry time of the link e.g. "1d" (optional)
-### StreamUpload ###
+Returns:
-Some remotes allow files to be uploaded without knowing the file size
-in advance. This allows certain operations to work without spooling the
-file to local disk first, e.g. `rclone rcat`.
+- url - URL of the resource
-### MultithreadUpload ###
+See the [link](https://rclone.org/commands/rclone_link/) command for more information on the above.
-Some remotes allow transfers to the remote to be sent as chunks in
-parallel. If this is supported then rclone will use multi-thread
-copying to transfer files much faster.
+**Authentication is required for this call.**
-### LinkSharing ###
+### operations/purge: Remove a directory or container and all of its contents {#operations-purge}
-Sets the necessary permissions on a file or folder and prints a link
-that allows others to access them, even if they don't have an account
-on the particular cloud provider.
+This takes the following parameters:
-### About ###
+- fs - a remote name string e.g. "drive:"
+- remote - a path within that remote e.g. "dir"
-Rclone `about` prints quota information for a remote. Typical output
-includes bytes used, free, quota and in trash.
+See the [purge](https://rclone.org/commands/rclone_purge/) command for more information on the above.
-If a remote lacks about capability `rclone about remote:`returns
-an error.
+**Authentication is required for this call.**
-Backends without about capability cannot determine free space for an
-rclone mount, or use policy `mfs` (most free space) as a member of an
-rclone union remote.
+### operations/rmdir: Remove an empty directory or container {#operations-rmdir}
-See [rclone about command](https://rclone.org/commands/rclone_about/)
+This takes the following parameters:
-### EmptyDir ###
+- fs - a remote name string e.g. "drive:"
+- remote - a path within that remote e.g. "dir"
-The remote supports empty directories. See [Limitations](https://rclone.org/bugs/#limitations)
- for details. Most Object/Bucket-based remotes do not support this.
+See the [rmdir](https://rclone.org/commands/rclone_rmdir/) command for more information on the above.
-# Global Flags
+**Authentication is required for this call.**
-This describes the global flags available to every rclone command
-split into groups.
+### operations/rmdirs: Remove all the empty directories in the path {#operations-rmdirs}
+This takes the following parameters:
-## Copy
+- fs - a remote name string e.g. "drive:"
+- remote - a path within that remote e.g. "dir"
+- leaveRoot - boolean, set to true not to delete the root
-Flags for anything which can Copy a file.
+See the [rmdirs](https://rclone.org/commands/rclone_rmdirs/) command for more information on the above.
-```
- --check-first Do all the checks before starting transfers
- -c, --checksum Check for changes with size & checksum (if available, or fallback to size only).
- --compare-dest stringArray Include additional comma separated server-side paths during comparison
- --copy-dest stringArray Implies --compare-dest but also copies files from paths into destination
- --cutoff-mode string Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default "HARD")
- --ignore-case-sync Ignore case when synchronizing
- --ignore-checksum Skip post copy check of checksums
- --ignore-existing Skip all files that exist on destination
- --ignore-size Ignore size when skipping use mod-time or checksum
- -I, --ignore-times Don't skip files that match size and time - transfer all files
- --immutable Do not modify files, fail if existing files have been modified
- --inplace Download directly to destination file instead of atomic download to temp/rename
- --max-backlog int Maximum number of objects in sync or check backlog (default 10000)
- --max-duration Duration Maximum duration rclone will transfer data for (default 0s)
- --max-transfer SizeSuffix Maximum size of data to transfer (default off)
- -M, --metadata If set, preserve metadata when copying objects
- --modify-window Duration Max time diff to be considered the same (default 1ns)
- --multi-thread-chunk-size SizeSuffix Chunk size for multi-thread downloads / uploads, if not set by filesystem (default 64Mi)
- --multi-thread-cutoff SizeSuffix Use multi-thread downloads for files above this size (default 256Mi)
- --multi-thread-streams int Number of streams to use for multi-thread downloads (default 4)
- --multi-thread-write-buffer-size SizeSuffix In memory buffer size for writing when in multi-thread mode (default 128Ki)
- --no-check-dest Don't check the destination, copy regardless
- --no-traverse Don't traverse destination file system on copy
- --no-update-modtime Don't update destination mod-time if files identical
- --order-by string Instructions on how to order the transfers, e.g. 'size,descending'
- --refresh-times Refresh the modtime of remote files
- --server-side-across-configs Allow server-side operations (e.g. copy) to work across different configs
- --size-only Skip based on size only, not mod-time or checksum
- --streaming-upload-cutoff SizeSuffix Cutoff for switching to chunked upload if file size is unknown, upload starts after reaching cutoff or when file ends (default 100Ki)
- -u, --update Skip files that are newer on the destination
-```
+**Authentication is required for this call.**
+### operations/settier: Changes storage tier or class on all files in the path {#operations-settier}
-## Sync
+This takes the following parameters:
-Flags just used for `rclone sync`.
+- fs - a remote name string e.g. "drive:"
-```
- --backup-dir string Make backups into hierarchy based in DIR
- --delete-after When synchronizing, delete files on destination after transferring (default)
- --delete-before When synchronizing, delete files on destination before transferring
- --delete-during When synchronizing, delete files during transfer
- --ignore-errors Delete even if there are I/O errors
- --max-delete int When synchronizing, limit the number of deletes (default -1)
- --max-delete-size SizeSuffix When synchronizing, limit the total size of deletes (default off)
- --suffix string Suffix to add to changed files
- --suffix-keep-extension Preserve the extension when using --suffix
- --track-renames When synchronizing, track file renames and do a server-side move if possible
- --track-renames-strategy string Strategies to use when synchronizing using track-renames hash|modtime|leaf (default "hash")
-```
+See the [settier](https://rclone.org/commands/rclone_settier/) command for more information on the above.
+**Authentication is required for this call.**
-## Important
+### operations/settierfile: Changes storage tier or class on the single file pointed to {#operations-settierfile}
-Important flags useful for most commands.
+This takes the following parameters:
-```
- -n, --dry-run Do a trial run with no permanent changes
- -i, --interactive Enable interactive mode
- -v, --verbose count Print lots more stuff (repeat for more)
-```
+- fs - a remote name string e.g. "drive:"
+- remote - a path within that remote e.g. "dir"
+See the [settierfile](https://rclone.org/commands/rclone_settierfile/) command for more information on the above.
-## Check
+**Authentication is required for this call.**
-Flags used for `rclone check`.
+### operations/size: Count the number of bytes and files in remote {#operations-size}
-```
- --max-backlog int Maximum number of objects in sync or check backlog (default 10000)
-```
+This takes the following parameters:
+- fs - a remote name string e.g. "drive:path/to/dir"
-## Networking
+Returns:
-General networking and HTTP stuff.
+- count - number of files
+- bytes - number of bytes in those files
-```
- --bind string Local address to bind to for outgoing connections, IPv4, IPv6 or name
- --bwlimit BwTimetable Bandwidth limit in KiB/s, or use suffix B|K|M|G|T|P or a full timetable
- --bwlimit-file BwTimetable Bandwidth limit per file in KiB/s, or use suffix B|K|M|G|T|P or a full timetable
- --ca-cert stringArray CA certificate used to verify servers
- --client-cert string Client SSL certificate (PEM) for mutual TLS auth
- --client-key string Client SSL private key (PEM) for mutual TLS auth
- --contimeout Duration Connect timeout (default 1m0s)
- --disable-http-keep-alives Disable HTTP keep-alives and use each connection once.
- --disable-http2 Disable HTTP/2 in the global transport
- --dscp string Set DSCP value to connections, value or name, e.g. CS1, LE, DF, AF21
- --expect-continue-timeout Duration Timeout when using expect / 100-continue in HTTP (default 1s)
- --header stringArray Set HTTP header for all transactions
- --header-download stringArray Set HTTP header for download transactions
- --header-upload stringArray Set HTTP header for upload transactions
- --no-check-certificate Do not verify the server SSL certificate (insecure)
- --no-gzip-encoding Don't set Accept-Encoding: gzip
- --timeout Duration IO idle timeout (default 5m0s)
- --tpslimit float Limit HTTP transactions per second to this
- --tpslimit-burst int Max burst of transactions for --tpslimit (default 1)
- --use-cookies Enable session cookiejar
- --user-agent string Set the user-agent to a specified string (default "rclone/v1.64.0")
-```
+See the [size](https://rclone.org/commands/rclone_size/) command for more information on the above.
+**Authentication is required for this call.**
-## Performance
+### operations/stat: Give information about the supplied file or directory {#operations-stat}
-Flags helpful for increasing performance.
+This takes the following parameters
-```
- --buffer-size SizeSuffix In memory buffer size when reading files for each --transfer (default 16Mi)
- --checkers int Number of checkers to run in parallel (default 8)
- --transfers int Number of file transfers to run in parallel (default 4)
-```
+- fs - a remote name string eg "drive:"
+- remote - a path within that remote eg "dir"
+- opt - a dictionary of options to control the listing (optional)
+ - see operations/list for the options
+The result is
-## Config
+- item - an object as described in the lsjson command. Will be null if not found.
-General configuration of rclone.
+Note that if you are only interested in files then it is much more
+efficient to set the filesOnly flag in the options.
-```
- --ask-password Allow prompt for password for encrypted configuration (default true)
- --auto-confirm If enabled, do not request console confirmation
- --cache-dir string Directory rclone will use for caching (default "$HOME/.cache/rclone")
- --color string When to show colors (and other ANSI codes) AUTO|NEVER|ALWAYS (default "AUTO")
- --config string Config file (default "$HOME/.config/rclone/rclone.conf")
- --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z)
- --disable string Disable a comma separated list of features (use --disable help to see a list)
- -n, --dry-run Do a trial run with no permanent changes
- --error-on-no-transfer Sets exit code 9 if no files are transferred, useful in scripts
- --fs-cache-expire-duration Duration Cache remotes for this long (0 to disable caching) (default 5m0s)
- --fs-cache-expire-interval Duration Interval to check for expired remotes (default 1m0s)
- --human-readable Print numbers in a human-readable format, sizes with suffix Ki|Mi|Gi|Ti|Pi
- -i, --interactive Enable interactive mode
- --kv-lock-time Duration Maximum time to keep key-value database locked by process (default 1s)
- --low-level-retries int Number of low level retries to do (default 10)
- --no-console Hide console window (supported on Windows only)
- --no-unicode-normalization Don't normalize unicode characters in filenames
- --password-command SpaceSepList Command for supplying password for encrypted configuration
- --retries int Retry operations this many times if they fail (default 3)
- --retries-sleep Duration Interval between retrying operations if they fail, e.g. 500ms, 60s, 5m (0 to disable) (default 0s)
- --temp-dir string Directory rclone will use for temporary files (default "/tmp")
- --use-mmap Use mmap allocator (see docs)
- --use-server-modtime Use server modified time instead of object metadata
-```
+See the [lsjson](https://rclone.org/commands/rclone_lsjson/) command for more information on the above and examples.
+**Authentication is required for this call.**
-## Debugging
+### operations/uploadfile: Upload file using multiform/form-data {#operations-uploadfile}
-Flags for developers.
+This takes the following parameters:
-```
- --cpuprofile string Write cpu profile to file
- --dump DumpFlags List of items to dump from: headers,bodies,requests,responses,auth,filters,goroutines,openfiles
- --dump-bodies Dump HTTP headers and bodies - may contain sensitive info
- --dump-headers Dump HTTP headers - may contain sensitive info
- --memprofile string Write memory profile to file
-```
+- fs - a remote name string e.g. "drive:"
+- remote - a path within that remote e.g. "dir"
+- each part in body represents a file to be uploaded
+See the [uploadfile](https://rclone.org/commands/rclone_uploadfile/) command for more information on the above.
-## Filter
+**Authentication is required for this call.**
-Flags for filtering directory listings.
+### options/blocks: List all the option blocks {#options-blocks}
-```
- --delete-excluded Delete files on dest excluded from sync
- --exclude stringArray Exclude files matching pattern
- --exclude-from stringArray Read file exclude patterns from file (use - to read from stdin)
- --exclude-if-present stringArray Exclude directories if filename is present
- --files-from stringArray Read list of source-file names from file (use - to read from stdin)
- --files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin)
- -f, --filter stringArray Add a file filtering rule
- --filter-from stringArray Read file filtering patterns from a file (use - to read from stdin)
- --ignore-case Ignore case in filters (case insensitive)
- --include stringArray Include files matching pattern
- --include-from stringArray Read file include patterns from file (use - to read from stdin)
- --max-age Duration Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off)
- --max-depth int If set limits the recursion depth to this (default -1)
- --max-size SizeSuffix Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off)
- --metadata-exclude stringArray Exclude metadatas matching pattern
- --metadata-exclude-from stringArray Read metadata exclude patterns from file (use - to read from stdin)
- --metadata-filter stringArray Add a metadata filtering rule
- --metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin)
- --metadata-include stringArray Include metadatas matching pattern
- --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin)
- --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off)
- --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off)
-```
+Returns:
+- options - a list of the options block names
+### options/get: Get all the global options {#options-get}
-## Listing
+Returns an object where keys are option block names and values are an
+object with the current option values in.
-Flags for listing directories.
+Note that these are the global options which are unaffected by use of
+the _config and _filter parameters. If you wish to read the parameters
+set in _config then use options/config and for _filter use options/filter.
-```
- --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z)
- --fast-list Use recursive list if available; uses more memory but fewer transactions
-```
+This shows the internal names of the option within rclone which should
+map to the external options very easily with a few exceptions.
+### options/local: Get the currently active config for this call {#options-local}
-## Logging
+Returns an object with the keys "config" and "filter".
+The "config" key contains the local config and the "filter" key contains
+the local filters.
-Logging and statistics.
+Note that these are the local options specific to this rc call. If
+_config was not supplied then they will be the global options.
+Likewise with "_filter".
-```
- --log-file string Log everything to this file
- --log-format string Comma separated list of log format options (default "date,time")
- --log-level string Log level DEBUG|INFO|NOTICE|ERROR (default "NOTICE")
- --log-systemd Activate systemd integration for the logger
- --max-stats-groups int Maximum number of stats groups to keep in memory, on max oldest is discarded (default 1000)
- -P, --progress Show progress during transfer
- --progress-terminal-title Show progress on the terminal title (requires -P/--progress)
- -q, --quiet Print as little stuff as possible
- --stats Duration Interval between printing stats, e.g. 500ms, 60s, 5m (0 to disable) (default 1m0s)
- --stats-file-name-length int Max file name length in stats (0 for no limit) (default 45)
- --stats-log-level string Log level to show --stats output DEBUG|INFO|NOTICE|ERROR (default "INFO")
- --stats-one-line Make the stats fit on one line
- --stats-one-line-date Enable --stats-one-line and add current date/time prefix
- --stats-one-line-date-format string Enable --stats-one-line-date and use custom formatted date: Enclose date string in double quotes ("), see https://golang.org/pkg/time/#Time.Format
- --stats-unit string Show data rate in stats as either 'bits' or 'bytes' per second (default "bytes")
- --syslog Use Syslog for logging
- --syslog-facility string Facility for syslog, e.g. KERN,USER,... (default "DAEMON")
- --use-json-log Use json log format
- -v, --verbose count Print lots more stuff (repeat for more)
-```
+This call is mostly useful for seeing if _config and _filter passing
+is working.
+This shows the internal names of the option within rclone which should
+map to the external options very easily with a few exceptions.
-## Metadata
+### options/set: Set an option {#options-set}
-Flags to control metadata.
+Parameters:
-```
- -M, --metadata If set, preserve metadata when copying objects
- --metadata-exclude stringArray Exclude metadatas matching pattern
- --metadata-exclude-from stringArray Read metadata exclude patterns from file (use - to read from stdin)
- --metadata-filter stringArray Add a metadata filtering rule
- --metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin)
- --metadata-include stringArray Include metadatas matching pattern
- --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin)
- --metadata-set stringArray Add metadata key=value when uploading
-```
+- option block name containing an object with
+ - key: value
+Repeated as often as required.
-## RC
+Only supply the options you wish to change. If an option is unknown
+it will be silently ignored. Not all options will have an effect when
+changed like this.
-Flags to control the Remote Control API.
+For example:
-```
- --rc Enable the remote control server
- --rc-addr stringArray IPaddress:Port or :Port to bind server to (default [localhost:5572])
- --rc-allow-origin string Origin which cross-domain request (CORS) can be executed from
- --rc-baseurl string Prefix for URLs - leave blank for root
- --rc-cert string TLS PEM key (concatenation of certificate and CA certificate)
- --rc-client-ca string Client certificate authority to verify clients with
- --rc-enable-metrics Enable prometheus metrics on /metrics
- --rc-files string Path to local files to serve on the HTTP server
- --rc-htpasswd string A htpasswd file - if not provided no authentication is done
- --rc-job-expire-duration Duration Expire finished async jobs older than this value (default 1m0s)
- --rc-job-expire-interval Duration Interval to check for expired async jobs (default 10s)
- --rc-key string TLS PEM Private key
- --rc-max-header-bytes int Maximum size of request header (default 4096)
- --rc-min-tls-version string Minimum TLS version that is acceptable (default "tls1.0")
- --rc-no-auth Don't require auth for certain methods
- --rc-pass string Password for authentication
- --rc-realm string Realm for authentication
- --rc-salt string Password hashing salt (default "dlPL2MqE")
- --rc-serve Enable the serving of remote objects
- --rc-server-read-timeout Duration Timeout for server reading data (default 1h0m0s)
- --rc-server-write-timeout Duration Timeout for server writing data (default 1h0m0s)
- --rc-template string User-specified template
- --rc-user string User name for authentication
- --rc-web-fetch-url string URL to fetch the releases for webgui (default "https://api.github.com/repos/rclone/rclone-webui-react/releases/latest")
- --rc-web-gui Launch WebGUI on localhost
- --rc-web-gui-force-update Force update to latest version of web gui
- --rc-web-gui-no-open-browser Don't open the browser automatically
- --rc-web-gui-update Check and update to latest version of web gui
-```
+This sets DEBUG level logs (-vv) (these can be set by number or string)
+ rclone rc options/set --json '{"main": {"LogLevel": "DEBUG"}}'
+ rclone rc options/set --json '{"main": {"LogLevel": 8}}'
-## Backend
+And this sets INFO level logs (-v)
-Backend only flags. These can be set in the config file also.
+ rclone rc options/set --json '{"main": {"LogLevel": "INFO"}}'
-```
- --acd-auth-url string Auth server URL
- --acd-client-id string OAuth Client Id
- --acd-client-secret string OAuth Client Secret
- --acd-encoding MultiEncoder The encoding for the backend (default Slash,InvalidUtf8,Dot)
- --acd-templink-threshold SizeSuffix Files >= this size will be downloaded via their tempLink (default 9Gi)
- --acd-token string OAuth Access Token as a JSON blob
- --acd-token-url string Token server url
- --acd-upload-wait-per-gb Duration Additional time per GiB to wait after a failed complete upload to see if it appears (default 3m0s)
- --alias-remote string Remote or path to alias
- --azureblob-access-tier string Access tier of blob: hot, cool or archive
- --azureblob-account string Azure Storage Account Name
- --azureblob-archive-tier-delete Delete archive tier blobs before overwriting
- --azureblob-chunk-size SizeSuffix Upload chunk size (default 4Mi)
- --azureblob-client-certificate-password string Password for the certificate file (optional) (obscured)
- --azureblob-client-certificate-path string Path to a PEM or PKCS12 certificate file including the private key
- --azureblob-client-id string The ID of the client in use
- --azureblob-client-secret string One of the service principal's client secrets
- --azureblob-client-send-certificate-chain Send the certificate chain when using certificate auth
- --azureblob-directory-markers Upload an empty object with a trailing slash when a new directory is created
- --azureblob-disable-checksum Don't store MD5 checksum with object metadata
- --azureblob-encoding MultiEncoder The encoding for the backend (default Slash,BackSlash,Del,Ctl,RightPeriod,InvalidUtf8)
- --azureblob-endpoint string Endpoint for the service
- --azureblob-env-auth Read credentials from runtime (environment variables, CLI or MSI)
- --azureblob-key string Storage Account Shared Key
- --azureblob-list-chunk int Size of blob list (default 5000)
- --azureblob-msi-client-id string Object ID of the user-assigned MSI to use, if any
- --azureblob-msi-mi-res-id string Azure resource ID of the user-assigned MSI to use, if any
- --azureblob-msi-object-id string Object ID of the user-assigned MSI to use, if any
- --azureblob-no-check-container If set, don't attempt to check the container exists or create it
- --azureblob-no-head-object If set, do not do HEAD before GET when getting objects
- --azureblob-password string The user's password (obscured)
- --azureblob-public-access string Public access level of a container: blob or container
- --azureblob-sas-url string SAS URL for container level access only
- --azureblob-service-principal-file string Path to file containing credentials for use with a service principal
- --azureblob-tenant string ID of the service principal's tenant. Also called its directory ID
- --azureblob-upload-concurrency int Concurrency for multipart uploads (default 16)
- --azureblob-upload-cutoff string Cutoff for switching to chunked upload (<= 256 MiB) (deprecated)
- --azureblob-use-emulator Uses local storage emulator if provided as 'true'
- --azureblob-use-msi Use a managed service identity to authenticate (only works in Azure)
- --azureblob-username string User name (usually an email address)
- --b2-account string Account ID or Application Key ID
- --b2-chunk-size SizeSuffix Upload chunk size (default 96Mi)
- --b2-copy-cutoff SizeSuffix Cutoff for switching to multipart copy (default 4Gi)
- --b2-disable-checksum Disable checksums for large (> upload cutoff) files
- --b2-download-auth-duration Duration Time before the authorization token will expire in s or suffix ms|s|m|h|d (default 1w)
- --b2-download-url string Custom endpoint for downloads
- --b2-encoding MultiEncoder The encoding for the backend (default Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot)
- --b2-endpoint string Endpoint for the service
- --b2-hard-delete Permanently delete files on remote removal, otherwise hide files
- --b2-key string Application Key
- --b2-test-mode string A flag string for X-Bz-Test-Mode header for debugging
- --b2-upload-concurrency int Concurrency for multipart uploads (default 16)
- --b2-upload-cutoff SizeSuffix Cutoff for switching to chunked upload (default 200Mi)
- --b2-version-at Time Show file versions as they were at the specified time (default off)
- --b2-versions Include old versions in directory listings
- --box-access-token string Box App Primary Access Token
- --box-auth-url string Auth server URL
- --box-box-config-file string Box App config.json location
- --box-box-sub-type string (default "user")
- --box-client-id string OAuth Client Id
- --box-client-secret string OAuth Client Secret
- --box-commit-retries int Max number of times to try committing a multipart file (default 100)
- --box-encoding MultiEncoder The encoding for the backend (default Slash,BackSlash,Del,Ctl,RightSpace,InvalidUtf8,Dot)
- --box-impersonate string Impersonate this user ID when using a service account
- --box-list-chunk int Size of listing chunk 1-1000 (default 1000)
- --box-owned-by string Only show items owned by the login (email address) passed in
- --box-root-folder-id string Fill in for rclone to use a non root folder as its starting point
- --box-token string OAuth Access Token as a JSON blob
- --box-token-url string Token server url
- --box-upload-cutoff SizeSuffix Cutoff for switching to multipart upload (>= 50 MiB) (default 50Mi)
- --cache-chunk-clean-interval Duration How often should the cache perform cleanups of the chunk storage (default 1m0s)
- --cache-chunk-no-memory Disable the in-memory cache for storing chunks during streaming
- --cache-chunk-path string Directory to cache chunk files (default "$HOME/.cache/rclone/cache-backend")
- --cache-chunk-size SizeSuffix The size of a chunk (partial file data) (default 5Mi)
- --cache-chunk-total-size SizeSuffix The total size that the chunks can take up on the local disk (default 10Gi)
- --cache-db-path string Directory to store file structure metadata DB (default "$HOME/.cache/rclone/cache-backend")
- --cache-db-purge Clear all the cached data for this remote on start
- --cache-db-wait-time Duration How long to wait for the DB to be available - 0 is unlimited (default 1s)
- --cache-info-age Duration How long to cache file structure information (directory listings, file size, times, etc.) (default 6h0m0s)
- --cache-plex-insecure string Skip all certificate verification when connecting to the Plex server
- --cache-plex-password string The password of the Plex user (obscured)
- --cache-plex-url string The URL of the Plex server
- --cache-plex-username string The username of the Plex user
- --cache-read-retries int How many times to retry a read from a cache storage (default 10)
- --cache-remote string Remote to cache
- --cache-rps int Limits the number of requests per second to the source FS (-1 to disable) (default -1)
- --cache-tmp-upload-path string Directory to keep temporary files until they are uploaded
- --cache-tmp-wait-time Duration How long should files be stored in local cache before being uploaded (default 15s)
- --cache-workers int How many workers should run in parallel to download chunks (default 4)
- --cache-writes Cache file data on writes through the FS
- --chunker-chunk-size SizeSuffix Files larger than chunk size will be split in chunks (default 2Gi)
- --chunker-fail-hard Choose how chunker should handle files with missing or invalid chunks
- --chunker-hash-type string Choose how chunker handles hash sums (default "md5")
- --chunker-remote string Remote to chunk/unchunk
- --combine-upstreams SpaceSepList Upstreams for combining
- --compress-level int GZIP compression level (-2 to 9) (default -1)
- --compress-mode string Compression mode (default "gzip")
- --compress-ram-cache-limit SizeSuffix Some remotes don't allow the upload of files with unknown size (default 20Mi)
- --compress-remote string Remote to compress
- -L, --copy-links Follow symlinks and copy the pointed to item
- --crypt-directory-name-encryption Option to either encrypt directory names or leave them intact (default true)
- --crypt-filename-encoding string How to encode the encrypted filename to text string (default "base32")
- --crypt-filename-encryption string How to encrypt the filenames (default "standard")
- --crypt-no-data-encryption Option to either encrypt file data or leave it unencrypted
- --crypt-pass-bad-blocks If set this will pass bad blocks through as all 0
- --crypt-password string Password or pass phrase for encryption (obscured)
- --crypt-password2 string Password or pass phrase for salt (obscured)
- --crypt-remote string Remote to encrypt/decrypt
- --crypt-server-side-across-configs Deprecated: use --server-side-across-configs instead
- --crypt-show-mapping For all files listed show how the names encrypt
- --crypt-suffix string If this is set it will override the default suffix of ".bin" (default ".bin")
- --drive-acknowledge-abuse Set to allow files which return cannotDownloadAbusiveFile to be downloaded
- --drive-allow-import-name-change Allow the filetype to change when uploading Google docs
- --drive-auth-owner-only Only consider files owned by the authenticated user
- --drive-auth-url string Auth server URL
- --drive-chunk-size SizeSuffix Upload chunk size (default 8Mi)
- --drive-client-id string Google Application Client Id
- --drive-client-secret string OAuth Client Secret
- --drive-copy-shortcut-content Server side copy contents of shortcuts instead of the shortcut
- --drive-disable-http2 Disable drive using http2 (default true)
- --drive-encoding MultiEncoder The encoding for the backend (default InvalidUtf8)
- --drive-env-auth Get IAM credentials from runtime (environment variables or instance meta data if no env vars)
- --drive-export-formats string Comma separated list of preferred formats for downloading Google docs (default "docx,xlsx,pptx,svg")
- --drive-fast-list-bug-fix Work around a bug in Google Drive listing (default true)
- --drive-formats string Deprecated: See export_formats
- --drive-impersonate string Impersonate this user when using a service account
- --drive-import-formats string Comma separated list of preferred formats for uploading Google docs
- --drive-keep-revision-forever Keep new head revision of each file forever
- --drive-list-chunk int Size of listing chunk 100-1000, 0 to disable (default 1000)
- --drive-pacer-burst int Number of API calls to allow without sleeping (default 100)
- --drive-pacer-min-sleep Duration Minimum time to sleep between API calls (default 100ms)
- --drive-resource-key string Resource key for accessing a link-shared file
- --drive-root-folder-id string ID of the root folder
- --drive-scope string Scope that rclone should use when requesting access from drive
- --drive-server-side-across-configs Deprecated: use --server-side-across-configs instead
- --drive-service-account-credentials string Service Account Credentials JSON blob
- --drive-service-account-file string Service Account Credentials JSON file path
- --drive-shared-with-me Only show files that are shared with me
- --drive-size-as-quota Show sizes as storage quota usage, not actual size
- --drive-skip-checksum-gphotos Skip MD5 checksum on Google photos and videos only
- --drive-skip-dangling-shortcuts If set skip dangling shortcut files
- --drive-skip-gdocs Skip google documents in all listings
- --drive-skip-shortcuts If set skip shortcut files
- --drive-starred-only Only show files that are starred
- --drive-stop-on-download-limit Make download limit errors be fatal
- --drive-stop-on-upload-limit Make upload limit errors be fatal
- --drive-team-drive string ID of the Shared Drive (Team Drive)
- --drive-token string OAuth Access Token as a JSON blob
- --drive-token-url string Token server url
- --drive-trashed-only Only show files that are in the trash
- --drive-upload-cutoff SizeSuffix Cutoff for switching to chunked upload (default 8Mi)
- --drive-use-created-date Use file created date instead of modified date
- --drive-use-shared-date Use date file was shared instead of modified date
- --drive-use-trash Send files to the trash instead of deleting permanently (default true)
- --drive-v2-download-min-size SizeSuffix If Object's are greater, use drive v2 API to download (default off)
- --dropbox-auth-url string Auth server URL
- --dropbox-batch-commit-timeout Duration Max time to wait for a batch to finish committing (default 10m0s)
- --dropbox-batch-mode string Upload file batching sync|async|off (default "sync")
- --dropbox-batch-size int Max number of files in upload batch
- --dropbox-batch-timeout Duration Max time to allow an idle upload batch before uploading (default 0s)
- --dropbox-chunk-size SizeSuffix Upload chunk size (< 150Mi) (default 48Mi)
- --dropbox-client-id string OAuth Client Id
- --dropbox-client-secret string OAuth Client Secret
- --dropbox-encoding MultiEncoder The encoding for the backend (default Slash,BackSlash,Del,RightSpace,InvalidUtf8,Dot)
- --dropbox-impersonate string Impersonate this user when using a business account
- --dropbox-pacer-min-sleep Duration Minimum time to sleep between API calls (default 10ms)
- --dropbox-shared-files Instructs rclone to work on individual shared files
- --dropbox-shared-folders Instructs rclone to work on shared folders
- --dropbox-token string OAuth Access Token as a JSON blob
- --dropbox-token-url string Token server url
- --fichier-api-key string Your API Key, get it from https://1fichier.com/console/params.pl
- --fichier-cdn Set if you wish to use CDN download links
- --fichier-encoding MultiEncoder The encoding for the backend (default Slash,LtGt,DoubleQuote,SingleQuote,BackQuote,Dollar,BackSlash,Del,Ctl,LeftSpace,RightSpace,InvalidUtf8,Dot)
- --fichier-file-password string If you want to download a shared file that is password protected, add this parameter (obscured)
- --fichier-folder-password string If you want to list the files in a shared folder that is password protected, add this parameter (obscured)
- --fichier-shared-folder string If you want to download a shared folder, add this parameter
- --filefabric-encoding MultiEncoder The encoding for the backend (default Slash,Del,Ctl,InvalidUtf8,Dot)
- --filefabric-permanent-token string Permanent Authentication Token
- --filefabric-root-folder-id string ID of the root folder
- --filefabric-token string Session Token
- --filefabric-token-expiry string Token expiry time
- --filefabric-url string URL of the Enterprise File Fabric to connect to
- --filefabric-version string Version read from the file fabric
- --ftp-ask-password Allow asking for FTP password when needed
- --ftp-close-timeout Duration Maximum time to wait for a response to close (default 1m0s)
- --ftp-concurrency int Maximum number of FTP simultaneous connections, 0 for unlimited
- --ftp-disable-epsv Disable using EPSV even if server advertises support
- --ftp-disable-mlsd Disable using MLSD even if server advertises support
- --ftp-disable-tls13 Disable TLS 1.3 (workaround for FTP servers with buggy TLS)
- --ftp-disable-utf8 Disable using UTF-8 even if server advertises support
- --ftp-encoding MultiEncoder The encoding for the backend (default Slash,Del,Ctl,RightSpace,Dot)
- --ftp-explicit-tls Use Explicit FTPS (FTP over TLS)
- --ftp-force-list-hidden Use LIST -a to force listing of hidden files and folders. This will disable the use of MLSD
- --ftp-host string FTP host to connect to
- --ftp-idle-timeout Duration Max time before closing idle connections (default 1m0s)
- --ftp-no-check-certificate Do not verify the TLS certificate of the server
- --ftp-pass string FTP password (obscured)
- --ftp-port int FTP port number (default 21)
- --ftp-shut-timeout Duration Maximum time to wait for data connection closing status (default 1m0s)
- --ftp-socks-proxy string Socks 5 proxy host
- --ftp-tls Use Implicit FTPS (FTP over TLS)
- --ftp-tls-cache-size int Size of TLS session cache for all control and data connections (default 32)
- --ftp-user string FTP username (default "$USER")
- --ftp-writing-mdtm Use MDTM to set modification time (VsFtpd quirk)
- --gcs-anonymous Access public buckets and objects without credentials
- --gcs-auth-url string Auth server URL
- --gcs-bucket-acl string Access Control List for new buckets
- --gcs-bucket-policy-only Access checks should use bucket-level IAM policies
- --gcs-client-id string OAuth Client Id
- --gcs-client-secret string OAuth Client Secret
- --gcs-decompress If set this will decompress gzip encoded objects
- --gcs-directory-markers Upload an empty object with a trailing slash when a new directory is created
- --gcs-encoding MultiEncoder The encoding for the backend (default Slash,CrLf,InvalidUtf8,Dot)
- --gcs-endpoint string Endpoint for the service
- --gcs-env-auth Get GCP IAM credentials from runtime (environment variables or instance meta data if no env vars)
- --gcs-location string Location for the newly created buckets
- --gcs-no-check-bucket If set, don't attempt to check the bucket exists or create it
- --gcs-object-acl string Access Control List for new objects
- --gcs-project-number string Project number
- --gcs-service-account-file string Service Account Credentials JSON file path
- --gcs-storage-class string The storage class to use when storing objects in Google Cloud Storage
- --gcs-token string OAuth Access Token as a JSON blob
- --gcs-token-url string Token server url
- --gcs-user-project string User project
- --gphotos-auth-url string Auth server URL
- --gphotos-client-id string OAuth Client Id
- --gphotos-client-secret string OAuth Client Secret
- --gphotos-encoding MultiEncoder The encoding for the backend (default Slash,CrLf,InvalidUtf8,Dot)
- --gphotos-include-archived Also view and download archived media
- --gphotos-read-only Set to make the Google Photos backend read only
- --gphotos-read-size Set to read the size of media items
- --gphotos-start-year int Year limits the photos to be downloaded to those which are uploaded after the given year (default 2000)
- --gphotos-token string OAuth Access Token as a JSON blob
- --gphotos-token-url string Token server url
- --hasher-auto-size SizeSuffix Auto-update checksum for files smaller than this size (disabled by default)
- --hasher-hashes CommaSepList Comma separated list of supported checksum types (default md5,sha1)
- --hasher-max-age Duration Maximum time to keep checksums in cache (0 = no cache, off = cache forever) (default off)
- --hasher-remote string Remote to cache checksums for (e.g. myRemote:path)
+And this sets NOTICE level logs (normal without -v)
+
+ rclone rc options/set --json '{"main": {"LogLevel": "NOTICE"}}'
+
+### pluginsctl/addPlugin: Add a plugin using url {#pluginsctl-addPlugin}
+
+Used for adding a plugin to the webgui.
+
+This takes the following parameters:
+
+- url - http url of the github repo where the plugin is hosted (http://github.com/rclone/rclone-webui-react).
+
+Example:
+
+ rclone rc pluginsctl/addPlugin
+
+**Authentication is required for this call.**
+
+### pluginsctl/getPluginsForType: Get plugins with type criteria {#pluginsctl-getPluginsForType}
+
+This shows all possible plugins by a mime type.
+
+This takes the following parameters:
+
+- type - supported mime type by a loaded plugin e.g. (video/mp4, audio/mp3).
+- pluginType - filter plugins based on their type e.g. (DASHBOARD, FILE_HANDLER, TERMINAL).
+
+Returns:
+
+- loadedPlugins - list of current production plugins.
+- testPlugins - list of temporarily loaded development plugins, usually running on a different server.
+
+Example:
+
+ rclone rc pluginsctl/getPluginsForType type=video/mp4
+
+**Authentication is required for this call.**
+
+### pluginsctl/listPlugins: Get the list of currently loaded plugins {#pluginsctl-listPlugins}
+
+This allows you to get the currently enabled plugins and their details.
+
+This takes no parameters and returns:
+
+- loadedPlugins - list of current production plugins.
+- testPlugins - list of temporarily loaded development plugins, usually running on a different server.
+
+E.g.
+
+ rclone rc pluginsctl/listPlugins
+
+**Authentication is required for this call.**
+
+### pluginsctl/listTestPlugins: Show currently loaded test plugins {#pluginsctl-listTestPlugins}
+
+Allows listing of test plugins with the rclone.test set to true in package.json of the plugin.
+
+This takes no parameters and returns:
+
+- loadedTestPlugins - list of currently available test plugins.
+
+E.g.
+
+ rclone rc pluginsctl/listTestPlugins
+
+**Authentication is required for this call.**
+
+### pluginsctl/removePlugin: Remove a loaded plugin {#pluginsctl-removePlugin}
+
+This allows you to remove a plugin using it's name.
+
+This takes parameters:
+
+- name - name of the plugin in the format `author`/`plugin_name`.
+
+E.g.
+
+ rclone rc pluginsctl/removePlugin name=rclone/video-plugin
+
+**Authentication is required for this call.**
+
+### pluginsctl/removeTestPlugin: Remove a test plugin {#pluginsctl-removeTestPlugin}
+
+This allows you to remove a plugin using it's name.
+
+This takes the following parameters:
+
+- name - name of the plugin in the format `author`/`plugin_name`.
+
+Example:
+
+ rclone rc pluginsctl/removeTestPlugin name=rclone/rclone-webui-react
+
+**Authentication is required for this call.**
+
+### rc/error: This returns an error {#rc-error}
+
+This returns an error with the input as part of its error string.
+Useful for testing error handling.
+
+### rc/list: List all the registered remote control commands {#rc-list}
+
+This lists all the registered remote control commands as a JSON map in
+the commands response.
+
+### rc/noop: Echo the input to the output parameters {#rc-noop}
+
+This echoes the input parameters to the output parameters for testing
+purposes. It can be used to check that rclone is still alive and to
+check that parameter passing is working properly.
+
+### rc/noopauth: Echo the input to the output parameters requiring auth {#rc-noopauth}
+
+This echoes the input parameters to the output parameters for testing
+purposes. It can be used to check that rclone is still alive and to
+check that parameter passing is working properly.
+
+**Authentication is required for this call.**
+
+### sync/bisync: Perform bidirectional synchronization between two paths. {#sync-bisync}
+
+This takes the following parameters
+
+- path1 - a remote directory string e.g. `drive:path1`
+- path2 - a remote directory string e.g. `drive:path2`
+- dryRun - dry-run mode
+- resync - performs the resync run
+- checkAccess - abort if RCLONE_TEST files are not found on both filesystems
+- checkFilename - file name for checkAccess (default: RCLONE_TEST)
+- maxDelete - abort sync if percentage of deleted files is above
+ this threshold (default: 50)
+- force - Bypass maxDelete safety check and run the sync
+- checkSync - `true` by default, `false` disables comparison of final listings,
+ `only` will skip sync, only compare listings from the last run
+- createEmptySrcDirs - Sync creation and deletion of empty directories.
+ (Not compatible with --remove-empty-dirs)
+- removeEmptyDirs - remove empty directories at the final cleanup step
+- filtersFile - read filtering patterns from a file
+- ignoreListingChecksum - Do not use checksums for listings
+- resilient - Allow future runs to retry after certain less-serious errors, instead of requiring resync.
+ Use at your own risk!
+- workdir - server directory for history files (default: /home/ncw/.cache/rclone/bisync)
+- noCleanup - retain working files
+
+See [bisync command help](https://rclone.org/commands/rclone_bisync/)
+and [full bisync description](https://rclone.org/bisync/)
+for more information.
+
+**Authentication is required for this call.**
+
+### sync/copy: copy a directory from source remote to destination remote {#sync-copy}
+
+This takes the following parameters:
+
+- srcFs - a remote name string e.g. "drive:src" for the source
+- dstFs - a remote name string e.g. "drive:dst" for the destination
+- createEmptySrcDirs - create empty src directories on destination if set
+
+
+See the [copy](https://rclone.org/commands/rclone_copy/) command for more information on the above.
+
+**Authentication is required for this call.**
+
+### sync/move: move a directory from source remote to destination remote {#sync-move}
+
+This takes the following parameters:
+
+- srcFs - a remote name string e.g. "drive:src" for the source
+- dstFs - a remote name string e.g. "drive:dst" for the destination
+- createEmptySrcDirs - create empty src directories on destination if set
+- deleteEmptySrcDirs - delete empty src directories if set
+
+
+See the [move](https://rclone.org/commands/rclone_move/) command for more information on the above.
+
+**Authentication is required for this call.**
+
+### sync/sync: sync a directory from source remote to destination remote {#sync-sync}
+
+This takes the following parameters:
+
+- srcFs - a remote name string e.g. "drive:src" for the source
+- dstFs - a remote name string e.g. "drive:dst" for the destination
+- createEmptySrcDirs - create empty src directories on destination if set
+
+
+See the [sync](https://rclone.org/commands/rclone_sync/) command for more information on the above.
+
+**Authentication is required for this call.**
+
+### vfs/forget: Forget files or directories in the directory cache. {#vfs-forget}
+
+This forgets the paths in the directory cache causing them to be
+re-read from the remote when needed.
+
+If no paths are passed in then it will forget all the paths in the
+directory cache.
+
+ rclone rc vfs/forget
+
+Otherwise pass files or dirs in as file=path or dir=path. Any
+parameter key starting with file will forget that file and any
+starting with dir will forget that dir, e.g.
+
+ rclone rc vfs/forget file=hello file2=goodbye dir=home/junk
+
+This command takes an "fs" parameter. If this parameter is not
+supplied and if there is only one VFS in use then that VFS will be
+used. If there is more than one VFS in use then the "fs" parameter
+must be supplied.
+
+### vfs/list: List active VFSes. {#vfs-list}
+
+This lists the active VFSes.
+
+It returns a list under the key "vfses" where the values are the VFS
+names that could be passed to the other VFS commands in the "fs"
+parameter.
+
+### vfs/poll-interval: Get the status or update the value of the poll-interval option. {#vfs-poll-interval}
+
+Without any parameter given this returns the current status of the
+poll-interval setting.
+
+When the interval=duration parameter is set, the poll-interval value
+is updated and the polling function is notified.
+Setting interval=0 disables poll-interval.
+
+ rclone rc vfs/poll-interval interval=5m
+
+The timeout=duration parameter can be used to specify a time to wait
+for the current poll function to apply the new value.
+If timeout is less or equal 0, which is the default, wait indefinitely.
+
+The new poll-interval value will only be active when the timeout is
+not reached.
+
+If poll-interval is updated or disabled temporarily, some changes
+might not get picked up by the polling function, depending on the
+used remote.
+
+This command takes an "fs" parameter. If this parameter is not
+supplied and if there is only one VFS in use then that VFS will be
+used. If there is more than one VFS in use then the "fs" parameter
+must be supplied.
+
+### vfs/refresh: Refresh the directory cache. {#vfs-refresh}
+
+This reads the directories for the specified paths and freshens the
+directory cache.
+
+If no paths are passed in then it will refresh the root directory.
+
+ rclone rc vfs/refresh
+
+Otherwise pass directories in as dir=path. Any parameter key
+starting with dir will refresh that directory, e.g.
+
+ rclone rc vfs/refresh dir=home/junk dir2=data/misc
+
+If the parameter recursive=true is given the whole directory tree
+will get refreshed. This refresh will use --fast-list if enabled.
+
+This command takes an "fs" parameter. If this parameter is not
+supplied and if there is only one VFS in use then that VFS will be
+used. If there is more than one VFS in use then the "fs" parameter
+must be supplied.
+
+### vfs/stats: Stats for a VFS. {#vfs-stats}
+
+This returns stats for the selected VFS.
+
+ {
+ // Status of the disk cache - only present if --vfs-cache-mode > off
+ "diskCache": {
+ "bytesUsed": 0,
+ "erroredFiles": 0,
+ "files": 0,
+ "hashType": 1,
+ "outOfSpace": false,
+ "path": "/home/user/.cache/rclone/vfs/local/mnt/a",
+ "pathMeta": "/home/user/.cache/rclone/vfsMeta/local/mnt/a",
+ "uploadsInProgress": 0,
+ "uploadsQueued": 0
+ },
+ "fs": "/mnt/a",
+ "inUse": 1,
+ // Status of the in memory metadata cache
+ "metadataCache": {
+ "dirs": 1,
+ "files": 0
+ },
+ // Options as returned by options/get
+ "opt": {
+ "CacheMaxAge": 3600000000000,
+ // ...
+ "WriteWait": 1000000000
+ }
+ }
+
+
+This command takes an "fs" parameter. If this parameter is not
+supplied and if there is only one VFS in use then that VFS will be
+used. If there is more than one VFS in use then the "fs" parameter
+must be supplied.
+
+
+
+## Accessing the remote control via HTTP {#api-http}
+
+Rclone implements a simple HTTP based protocol.
+
+Each endpoint takes an JSON object and returns a JSON object or an
+error. The JSON objects are essentially a map of string names to
+values.
+
+All calls must made using POST.
+
+The input objects can be supplied using URL parameters, POST
+parameters or by supplying "Content-Type: application/json" and a JSON
+blob in the body. There are examples of these below using `curl`.
+
+The response will be a JSON blob in the body of the response. This is
+formatted to be reasonably human-readable.
+
+### Error returns
+
+If an error occurs then there will be an HTTP error status (e.g. 500)
+and the body of the response will contain a JSON encoded error object,
+e.g.
+
+```
+{
+ "error": "Expecting string value for key \"remote\" (was float64)",
+ "input": {
+ "fs": "/tmp",
+ "remote": 3
+ },
+ "status": 400
+ "path": "operations/rmdir",
+}
+```
+
+The keys in the error response are
+- error - error string
+- input - the input parameters to the call
+- status - the HTTP status code
+- path - the path of the call
+
+### CORS
+
+The sever implements basic CORS support and allows all origins for that.
+The response to a preflight OPTIONS request will echo the requested "Access-Control-Request-Headers" back.
+
+### Using POST with URL parameters only
+
+```
+curl -X POST 'http://localhost:5572/rc/noop?potato=1&sausage=2'
+```
+
+Response
+
+```
+{
+ "potato": "1",
+ "sausage": "2"
+}
+```
+
+Here is what an error response looks like:
+
+```
+curl -X POST 'http://localhost:5572/rc/error?potato=1&sausage=2'
+```
+
+```
+{
+ "error": "arbitrary error on input map[potato:1 sausage:2]",
+ "input": {
+ "potato": "1",
+ "sausage": "2"
+ }
+}
+```
+
+Note that curl doesn't return errors to the shell unless you use the `-f` option
+
+```
+$ curl -f -X POST 'http://localhost:5572/rc/error?potato=1&sausage=2'
+curl: (22) The requested URL returned error: 400 Bad Request
+$ echo $?
+22
+```
+
+### Using POST with a form
+
+```
+curl --data "potato=1" --data "sausage=2" http://localhost:5572/rc/noop
+```
+
+Response
+
+```
+{
+ "potato": "1",
+ "sausage": "2"
+}
+```
+
+Note that you can combine these with URL parameters too with the POST
+parameters taking precedence.
+
+```
+curl --data "potato=1" --data "sausage=2" "http://localhost:5572/rc/noop?rutabaga=3&sausage=4"
+```
+
+Response
+
+```
+{
+ "potato": "1",
+ "rutabaga": "3",
+ "sausage": "4"
+}
+
+```
+
+### Using POST with a JSON blob
+
+```
+curl -H "Content-Type: application/json" -X POST -d '{"potato":2,"sausage":1}' http://localhost:5572/rc/noop
+```
+
+response
+
+```
+{
+ "password": "xyz",
+ "username": "xyz"
+}
+```
+
+This can be combined with URL parameters too if required. The JSON
+blob takes precedence.
+
+```
+curl -H "Content-Type: application/json" -X POST -d '{"potato":2,"sausage":1}' 'http://localhost:5572/rc/noop?rutabaga=3&potato=4'
+```
+
+```
+{
+ "potato": 2,
+ "rutabaga": "3",
+ "sausage": 1
+}
+```
+
+## Debugging rclone with pprof ##
+
+If you use the `--rc` flag this will also enable the use of the go
+profiling tools on the same port.
+
+To use these, first [install go](https://golang.org/doc/install).
+
+### Debugging memory use
+
+To profile rclone's memory use you can run:
+
+ go tool pprof -web http://localhost:5572/debug/pprof/heap
+
+This should open a page in your browser showing what is using what
+memory.
+
+You can also use the `-text` flag to produce a textual summary
+
+```
+$ go tool pprof -text http://localhost:5572/debug/pprof/heap
+Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
+ flat flat% sum% cum cum%
+ 1024.03kB 66.62% 66.62% 1024.03kB 66.62% github.com/rclone/rclone/vendor/golang.org/x/net/http2/hpack.addDecoderNode
+ 513kB 33.38% 100% 513kB 33.38% net/http.newBufioWriterSize
+ 0 0% 100% 1024.03kB 66.62% github.com/rclone/rclone/cmd/all.init
+ 0 0% 100% 1024.03kB 66.62% github.com/rclone/rclone/cmd/serve.init
+ 0 0% 100% 1024.03kB 66.62% github.com/rclone/rclone/cmd/serve/restic.init
+ 0 0% 100% 1024.03kB 66.62% github.com/rclone/rclone/vendor/golang.org/x/net/http2.init
+ 0 0% 100% 1024.03kB 66.62% github.com/rclone/rclone/vendor/golang.org/x/net/http2/hpack.init
+ 0 0% 100% 1024.03kB 66.62% github.com/rclone/rclone/vendor/golang.org/x/net/http2/hpack.init.0
+ 0 0% 100% 1024.03kB 66.62% main.init
+ 0 0% 100% 513kB 33.38% net/http.(*conn).readRequest
+ 0 0% 100% 513kB 33.38% net/http.(*conn).serve
+ 0 0% 100% 1024.03kB 66.62% runtime.main
+```
+
+### Debugging go routine leaks
+
+Memory leaks are most often caused by go routine leaks keeping memory
+alive which should have been garbage collected.
+
+See all active go routines using
+
+ curl http://localhost:5572/debug/pprof/goroutine?debug=1
+
+Or go to http://localhost:5572/debug/pprof/goroutine?debug=1 in your browser.
+
+### Other profiles to look at
+
+You can see a summary of profiles available at http://localhost:5572/debug/pprof/
+
+Here is how to use some of them:
+
+- Memory: `go tool pprof http://localhost:5572/debug/pprof/heap`
+- Go routines: `curl http://localhost:5572/debug/pprof/goroutine?debug=1`
+- 30-second CPU profile: `go tool pprof http://localhost:5572/debug/pprof/profile`
+- 5-second execution trace: `wget http://localhost:5572/debug/pprof/trace?seconds=5`
+- Goroutine blocking profile
+ - Enable first with: `rclone rc debug/set-block-profile-rate rate=1` ([docs](#debug-set-block-profile-rate))
+ - `go tool pprof http://localhost:5572/debug/pprof/block`
+- Contended mutexes:
+ - Enable first with: `rclone rc debug/set-mutex-profile-fraction rate=1` ([docs](#debug-set-mutex-profile-fraction))
+ - `go tool pprof http://localhost:5572/debug/pprof/mutex`
+
+See the [net/http/pprof docs](https://golang.org/pkg/net/http/pprof/)
+for more info on how to use the profiling and for a general overview
+see [the Go team's blog post on profiling go programs](https://blog.golang.org/profiling-go-programs).
+
+The profiling hook is [zero overhead unless it is used](https://stackoverflow.com/q/26545159/164234).
+
+# Overview of cloud storage systems #
+
+Each cloud storage system is slightly different. Rclone attempts to
+provide a unified interface to them, but some underlying differences
+show through.
+
+## Features ##
+
+Here is an overview of the major features of each cloud storage system.
+
+| Name | Hash | ModTime | Case Insensitive | Duplicate Files | MIME Type | Metadata |
+| ---------------------------- |:-----------------:|:-------:|:----------------:|:---------------:|:---------:|:--------:|
+| 1Fichier | Whirlpool | - | No | Yes | R | - |
+| Akamai Netstorage | MD5, SHA256 | R/W | No | No | R | - |
+| Amazon Drive | MD5 | - | Yes | No | R | - |
+| Amazon S3 (or S3 compatible) | MD5 | R/W | No | No | R/W | RWU |
+| Backblaze B2 | SHA1 | R/W | No | No | R/W | - |
+| Box | SHA1 | R/W | Yes | No | - | - |
+| Citrix ShareFile | MD5 | R/W | Yes | No | - | - |
+| Dropbox | DBHASH ¹ | R | Yes | No | - | - |
+| Enterprise File Fabric | - | R/W | Yes | No | R/W | - |
+| FTP | - | R/W ¹Ⱐ| No | No | - | - |
+| Google Cloud Storage | MD5 | R/W | No | No | R/W | - |
+| Google Drive | MD5, SHA1, SHA256 | R/W | No | Yes | R/W | - |
+| Google Photos | - | - | No | Yes | R | - |
+| HDFS | - | R/W | No | No | - | - |
+| HiDrive | HiDrive ¹² | R/W | No | No | - | - |
+| HTTP | - | R | No | No | R | - |
+| Internet Archive | MD5, SHA1, CRC32 | R/W ¹¹ | No | No | - | RWU |
+| Jottacloud | MD5 | R/W | Yes | No | R | RW |
+| Koofr | MD5 | - | Yes | No | - | - |
+| Linkbox | - | R | No | No | - | - |
+| Mail.ru Cloud | Mailru ⶠ| R/W | Yes | No | - | - |
+| Mega | - | - | No | Yes | - | - |
+| Memory | MD5 | R/W | No | No | - | - |
+| Microsoft Azure Blob Storage | MD5 | R/W | No | No | R/W | - |
+| Microsoft Azure Files Storage | MD5 | R/W | Yes | No | R/W | - |
+| Microsoft OneDrive | QuickXorHash âµ | R/W | Yes | No | R | - |
+| OpenDrive | MD5 | R/W | Yes | Partial ⸠| - | - |
+| OpenStack Swift | MD5 | R/W | No | No | R/W | - |
+| Oracle Object Storage | MD5 | R/W | No | No | R/W | - |
+| pCloud | MD5, SHA1 â· | R | No | No | W | - |
+| PikPak | MD5 | R | No | No | R | - |
+| premiumize.me | - | - | Yes | No | R | - |
+| put.io | CRC-32 | R/W | No | Yes | R | - |
+| Proton Drive | SHA1 | R/W | No | No | R | - |
+| QingStor | MD5 | - â¹ | No | No | R/W | - |
+| Quatrix by Maytech | - | R/W | No | No | - | - |
+| Seafile | - | - | No | No | - | - |
+| SFTP | MD5, SHA1 ² | R/W | Depends | No | - | - |
+| Sia | - | - | No | No | - | - |
+| SMB | - | R/W | Yes | No | - | - |
+| SugarSync | - | - | No | No | - | - |
+| Storj | - | R | No | No | - | - |
+| Uptobox | - | - | No | Yes | - | - |
+| WebDAV | MD5, SHA1 ³ | R ⴠ| Depends | No | - | - |
+| Yandex Disk | MD5 | R/W | No | No | R | - |
+| Zoho WorkDrive | - | - | No | No | - | - |
+| The local filesystem | All | R/W | Depends | No | - | RWU |
+
+¹ Dropbox supports [its own custom
+hash](https://www.dropbox.com/developers/reference/content-hash).
+This is an SHA256 sum of all the 4 MiB block SHA256s.
+
+² SFTP supports checksums if the same login has shell access and
+`md5sum` or `sha1sum` as well as `echo` are in the remote's PATH.
+
+³ WebDAV supports hashes when used with Fastmail Files, Owncloud and Nextcloud only.
+
+â´ WebDAV supports modtimes when used with Fastmail Files, Owncloud and Nextcloud only.
+
+âµ [QuickXorHash](https://docs.microsoft.com/en-us/onedrive/developer/code-snippets/quickxorhash) is Microsoft's own hash.
+
+ⶠMail.ru uses its own modified SHA1 hash
+
+â· pCloud only supports SHA1 (not MD5) in its EU region
+
+⸠Opendrive does not support creation of duplicate files using
+their web client interface or other stock clients, but the underlying
+storage platform has been determined to allow duplicate files, and it
+is possible to create them with `rclone`. It may be that this is a
+mistake or an unsupported feature.
+
+â¹ QingStor does not support SetModTime for objects bigger than 5 GiB.
+
+¹ⰠFTP supports modtimes for the major FTP servers, and also others
+if they advertised required protocol extensions. See [this](https://rclone.org/ftp/#modification-times)
+for more details.
+
+¹¹ Internet Archive requires option `wait_archive` to be set to a non-zero value
+for full modtime support.
+
+¹² HiDrive supports [its own custom
+hash](https://static.hidrive.com/dev/0001).
+It combines SHA1 sums for each 4 KiB block hierarchically to a single
+top-level sum.
+
+### Hash ###
+
+The cloud storage system supports various hash types of the objects.
+The hashes are used when transferring data as an integrity check and
+can be specifically used with the `--checksum` flag in syncs and in
+the `check` command.
+
+To use the verify checksums when transferring between cloud storage
+systems they must support a common hash type.
+
+### ModTime ###
+
+Almost all cloud storage systems store some sort of timestamp
+on objects, but several of them not something that is appropriate
+to use for syncing. E.g. some backends will only write a timestamp
+that represent the time of the upload. To be relevant for syncing
+it should be able to store the modification time of the source
+object. If this is not the case, rclone will only check the file
+size by default, though can be configured to check the file hash
+(with the `--checksum` flag). Ideally it should also be possible to
+change the timestamp of an existing file without having to re-upload it.
+
+Storage systems with a `-` in the ModTime column, means the
+modification read on objects is not the modification time of the
+file when uploaded. It is most likely the time the file was uploaded,
+or possibly something else (like the time the picture was taken in
+Google Photos).
+
+Storage systems with a `R` (for read-only) in the ModTime column,
+means the it keeps modification times on objects, and updates them
+when uploading objects, but it does not support changing only the
+modification time (`SetModTime` operation) without re-uploading,
+possibly not even without deleting existing first. Some operations
+in rclone, such as `copy` and `sync` commands, will automatically
+check for `SetModTime` support and re-upload if necessary to keep
+the modification times in sync. Other commands will not work
+without `SetModTime` support, e.g. `touch` command on an existing
+file will fail, and changes to modification time only on a files
+in a `mount` will be silently ignored.
+
+Storage systems with `R/W` (for read/write) in the ModTime column,
+means they do also support modtime-only operations.
+
+### Case Insensitive ###
+
+If a cloud storage systems is case sensitive then it is possible to
+have two files which differ only in case, e.g. `file.txt` and
+`FILE.txt`. If a cloud storage system is case insensitive then that
+isn't possible.
+
+This can cause problems when syncing between a case insensitive
+system and a case sensitive system. The symptom of this is that no
+matter how many times you run the sync it never completes fully.
+
+The local filesystem and SFTP may or may not be case sensitive
+depending on OS.
+
+ * Windows - usually case insensitive, though case is preserved
+ * OSX - usually case insensitive, though it is possible to format case sensitive
+ * Linux - usually case sensitive, but there are case insensitive file systems (e.g. FAT formatted USB keys)
+
+Most of the time this doesn't cause any problems as people tend to
+avoid files whose name differs only by case even on case sensitive
+systems.
+
+### Duplicate files ###
+
+If a cloud storage system allows duplicate files then it can have two
+objects with the same name.
+
+This confuses rclone greatly when syncing - use the `rclone dedupe`
+command to rename or remove duplicates.
+
+### Restricted filenames ###
+
+Some cloud storage systems might have restrictions on the characters
+that are usable in file or directory names.
+When `rclone` detects such a name during a file upload, it will
+transparently replace the restricted characters with similar looking
+Unicode characters. To handle the different sets of restricted characters
+for different backends, rclone uses something it calls [encoding](#encoding).
+
+This process is designed to avoid ambiguous file names as much as
+possible and allow to move files between many cloud storage systems
+transparently.
+
+The name shown by `rclone` to the user or during log output will only
+contain a minimal set of [replaced characters](#restricted-characters)
+to ensure correct formatting and not necessarily the actual name used
+on the cloud storage.
+
+This transformation is reversed when downloading a file or parsing
+`rclone` arguments. For example, when uploading a file named `my file?.txt`
+to Onedrive, it will be displayed as `my file?.txt` on the console, but
+stored as `my file？.txt` to Onedrive (the `?` gets replaced by the similar
+looking `？` character, the so-called "fullwidth question mark").
+The reverse transformation allows to read a file `unusual/name.txt`
+from Google Drive, by passing the name `unusualï¼name.txt` on the command line
+(the `/` needs to be replaced by the similar looking `ï¼` character).
+
+#### Caveats {#restricted-filenames-caveats}
+
+The filename encoding system works well in most cases, at least
+where file names are written in English or similar languages.
+You might not even notice it: It just works. In some cases it may
+lead to issues, though. E.g. when file names are written in Chinese,
+or Japanese, where it is always the Unicode fullwidth variants of the
+punctuation marks that are used.
+
+On Windows, the characters `:`, `*` and `?` are examples of restricted
+characters. If these are used in filenames on a remote that supports it,
+Rclone will transparently convert them to their fullwidth Unicode
+variants `＊`, `？` and `：` when downloading to Windows, and back again
+when uploading. This way files with names that are not allowed on Windows
+can still be stored.
+
+However, if you have files on your Windows system originally with these same
+Unicode characters in their names, they will be included in the same conversion
+process. E.g. if you create a file in your Windows filesystem with name
+`Test：1.jpg`, where `：` is the Unicode fullwidth colon symbol, and use
+rclone to upload it to Google Drive, which supports regular `:` (halfwidth
+question mark), rclone will replace the fullwidth `:` with the
+halfwidth `:` and store the file as `Test:1.jpg` in Google Drive. Since
+both Windows and Google Drive allows the name `Test：1.jpg`, it would
+probably be better if rclone just kept the name as is in this case.
+
+With the opposite situation; if you have a file named `Test:1.jpg`,
+in your Google Drive, e.g. uploaded from a Linux system where `:` is valid
+in file names. Then later use rclone to copy this file to your Windows
+computer you will notice that on your local disk it gets renamed
+to `Test：1.jpg`. The original filename is not legal on Windows, due to
+the `:`, and rclone therefore renames it to make the copy possible.
+That is all good. However, this can also lead to an issue: If you already
+had a *different* file named `Test：1.jpg` on Windows, and then use rclone
+to copy either way. Rclone will then treat the file originally named
+`Test:1.jpg` on Google Drive and the file originally named `Test：1.jpg`
+on Windows as the same file, and replace the contents from one with the other.
+
+Its virtually impossible to handle all cases like these correctly in all
+situations, but by customizing the [encoding option](#encoding), changing the
+set of characters that rclone should convert, you should be able to
+create a configuration that works well for your specific situation.
+See also the [example](https://rclone.org/overview/#encoding-example-windows) below.
+
+(Windows was used as an example of a file system with many restricted
+characters, and Google drive a storage system with few.)
+
+#### Default restricted characters {#restricted-characters}
+
+The table below shows the characters that are replaced by default.
+
+When a replacement character is found in a filename, this character
+will be escaped with the `‛` character to avoid ambiguous file names.
+(e.g. a file named `â€.txt` would shown as `‛â€.txt`)
+
+Each cloud storage backend can use a different set of characters,
+which will be specified in the documentation for each backend.
+
+| Character | Value | Replacement |
+| --------- |:-----:|:-----------:|
+| NUL | 0x00 | †|
+| SOH | 0x01 | â |
+| STX | 0x02 | â‚ |
+| ETX | 0x03 | ⃠|
+| EOT | 0x04 | â„ |
+| ENQ | 0x05 | â… |
+| ACK | 0x06 | ↠|
+| BEL | 0x07 | ⇠|
+| BS | 0x08 | ∠|
+| HT | 0x09 | ≠|
+| LF | 0x0A | ⊠|
+| VT | 0x0B | â‹ |
+| FF | 0x0C | ⌠|
+| CR | 0x0D | â |
+| SO | 0x0E | ⎠|
+| SI | 0x0F | â |
+| DLE | 0x10 | â |
+| DC1 | 0x11 | â‘ |
+| DC2 | 0x12 | â’ |
+| DC3 | 0x13 | â“ |
+| DC4 | 0x14 | â” |
+| NAK | 0x15 | â• |
+| SYN | 0x16 | â– |
+| ETB | 0x17 | â— |
+| CAN | 0x18 | ☠|
+| EM | 0x19 | â™ |
+| SUB | 0x1A | âš |
+| ESC | 0x1B | â› |
+| FS | 0x1C | ✠|
+| GS | 0x1D | â |
+| RS | 0x1E | âž |
+| US | 0x1F | ⟠|
+| / | 0x2F | ï¼ |
+| DEL | 0x7F | â¡ |
+
+The default encoding will also encode these file names as they are
+problematic with many cloud storage systems.
+
+| File name | Replacement |
+| --------- |:-----------:|
+| . | ． |
+| .. | ．． |
+
+#### Invalid UTF-8 bytes {#invalid-utf8}
+
+Some backends only support a sequence of well formed UTF-8 bytes
+as file or directory names.
+
+In this case all invalid UTF-8 bytes will be replaced with a quoted
+representation of the byte value to allow uploading a file to such a
+backend. For example, the invalid byte `0xFE` will be encoded as `‛FE`.
+
+A common source of invalid UTF-8 bytes are local filesystems, that store
+names in a different encoding than UTF-8 or UTF-16, like latin1. See the
+[local filenames](https://rclone.org/local/#filenames) section for details.
+
+#### Encoding option {#encoding}
+
+Most backends have an encoding option, specified as a flag
+`--backend-encoding` where `backend` is the name of the backend, or as
+a config parameter `encoding` (you'll need to select the Advanced
+config in `rclone config` to see it).
+
+This will have default value which encodes and decodes characters in
+such a way as to preserve the maximum number of characters (see
+above).
+
+However this can be incorrect in some scenarios, for example if you
+have a Windows file system with Unicode fullwidth characters
+`＊`, `？` or `：`, that you want to remain as those characters on the
+remote rather than being translated to regular (halfwidth) `*`, `?` and `:`.
+
+The `--backend-encoding` flags allow you to change that. You can
+disable the encoding completely with `--backend-encoding None` or set
+`encoding = None` in the config file.
+
+Encoding takes a comma separated list of encodings. You can see the
+list of all possible values by passing an invalid value to this
+flag, e.g. `--local-encoding "help"`. The command `rclone help flags encoding`
+will show you the defaults for the backends.
+
+| Encoding | Characters | Encoded as |
+| --------- | ---------- | ---------- |
+| Asterisk | `*` | `＊` |
+| BackQuote | `` ` `` | `ï½€` |
+| BackSlash | `\` | `ï¼¼` |
+| Colon | `:` | `：` |
+| CrLf | CR 0x0D, LF 0x0A | `â`, `âŠ` |
+| Ctl | All control characters 0x00-0x1F | `â€ââ‚âƒâ„â…â†â‡âˆâ‰âŠâ‹âŒââŽâââ‘â’â“â”â•â–â—â˜â™âšâ›âœââžâŸ` |
+| Del | DEL 0x7F | `â¡` |
+| Dollar | `$` | `＄` |
+| Dot | `.` or `..` as entire string | `．`, `．．` |
+| DoubleQuote | `"` | `＂` |
+| Hash | `#` | `＃` |
+| InvalidUtf8 | An invalid UTF-8 character (e.g. latin1) | `�` |
+| LeftCrLfHtVt | CR 0x0D, LF 0x0A, HT 0x09, VT 0x0B on the left of a string | `â`, `âŠ`, `â‰`, `â‹` |
+| LeftPeriod | `.` on the left of a string | `.` |
+| LeftSpace | SPACE on the left of a string | `â ` |
+| LeftTilde | `~` on the left of a string | `～` |
+| LtGt | `<`, `>` | `＜`, `＞` |
+| None | No characters are encoded | |
+| Percent | `%` | `ï¼…` |
+| Pipe | \| | `｜` |
+| Question | `?` | `？` |
+| RightCrLfHtVt | CR 0x0D, LF 0x0A, HT 0x09, VT 0x0B on the right of a string | `â`, `âŠ`, `â‰`, `â‹` |
+| RightPeriod | `.` on the right of a string | `.` |
+| RightSpace | SPACE on the right of a string | `â ` |
+| Semicolon | `;` | `ï¼›` |
+| SingleQuote | `'` | `＇` |
+| Slash | `/` | `ï¼` |
+| SquareBracket | `[`, `]` | `ï¼»`, `ï¼½` |
+
+##### Encoding example: FTP
+
+To take a specific example, the FTP backend's default encoding is
+
+ --ftp-encoding "Slash,Del,Ctl,RightSpace,Dot"
+
+However, let's say the FTP server is running on Windows and can't have
+any of the invalid Windows characters in file names. You are backing
+up Linux servers to this FTP server which do have those characters in
+file names. So you would add the Windows set which are
+
+ Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot
+
+to the existing ones, giving:
+
+ Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot,Del,RightSpace
+
+This can be specified using the `--ftp-encoding` flag or using an `encoding` parameter in the config file.
+
+##### Encoding example: Windows
+
+As a nother example, take a Windows system where there is a file with
+name `Test：1.jpg`, where `：` is the Unicode fullwidth colon symbol.
+When using rclone to copy this to a remote which supports `:`,
+the regular (halfwidth) colon (such as Google Drive), you will notice
+that the file gets renamed to `Test:1.jpg`.
+
+To avoid this you can change the set of characters rclone should convert
+for the local filesystem, using command-line argument `--local-encoding`.
+Rclone's default behavior on Windows corresponds to
+
+```
+--local-encoding "Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot"
+```
+
+If you want to use fullwidth characters `：`, `＊` and `？` in your filenames
+without rclone changing them when uploading to a remote, then set the same as
+the default value but without `Colon,Question,Asterisk`:
+
+```
+--local-encoding "Slash,LtGt,DoubleQuote,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot"
+```
+
+Alternatively, you can disable the conversion of any characters with `--local-encoding None`.
+
+Instead of using command-line argument `--local-encoding`, you may also set it
+as [environment variable](https://rclone.org/docs/#environment-variables) `RCLONE_LOCAL_ENCODING`,
+or [configure](https://rclone.org/docs/#configure) a remote of type `local` in your config,
+and set the `encoding` option there.
+
+The risk by doing this is that if you have a filename with the regular (halfwidth)
+`:`, `*` and `?` in your cloud storage, and you try to download
+it to your Windows filesystem, this will fail. These characters are not
+valid in filenames on Windows, and you have told rclone not to work around
+this by converting them to valid fullwidth variants.
+
+### MIME Type ###
+
+MIME types (also known as media types) classify types of documents
+using a simple text classification, e.g. `text/html` or
+`application/pdf`.
+
+Some cloud storage systems support reading (`R`) the MIME type of
+objects and some support writing (`W`) the MIME type of objects.
+
+The MIME type can be important if you are serving files directly to
+HTTP from the storage system.
+
+If you are copying from a remote which supports reading (`R`) to a
+remote which supports writing (`W`) then rclone will preserve the MIME
+types. Otherwise they will be guessed from the extension, or the
+remote itself may assign the MIME type.
+
+### Metadata
+
+Backends may or may support reading or writing metadata. They may
+support reading and writing system metadata (metadata intrinsic to
+that backend) and/or user metadata (general purpose metadata).
+
+The levels of metadata support are
+
+| Key | Explanation |
+|-----|-------------|
+| `R` | Read only System Metadata |
+| `RW` | Read and write System Metadata |
+| `RWU` | Read and write System Metadata and read and write User Metadata |
+
+See [the metadata docs](https://rclone.org/docs/#metadata) for more info.
+
+## Optional Features ##
+
+All rclone remotes support a base command set. Other features depend
+upon backend-specific capabilities.
+
+| Name | Purge | Copy | Move | DirMove | CleanUp | ListR | StreamUpload | MultithreadUpload | LinkSharing | About | EmptyDir |
+| ---------------------------- |:-----:|:----:|:----:|:-------:|:-------:|:-----:|:------------:|:------------------|:------------:|:-----:|:--------:|
+| 1Fichier | No | Yes | Yes | No | No | No | No | No | Yes | No | Yes |
+| Akamai Netstorage | Yes | No | No | No | No | Yes | Yes | No | No | No | Yes |
+| Amazon Drive | Yes | No | Yes | Yes | No | No | No | No | No | No | Yes |
+| Amazon S3 (or S3 compatible) | No | Yes | No | No | Yes | Yes | Yes | Yes | Yes | No | No |
+| Backblaze B2 | No | Yes | No | No | Yes | Yes | Yes | Yes | Yes | No | No |
+| Box | Yes | Yes | Yes | Yes | Yes | No | Yes | No | Yes | Yes | Yes |
+| Citrix ShareFile | Yes | Yes | Yes | Yes | No | No | No | No | No | No | Yes |
+| Dropbox | Yes | Yes | Yes | Yes | No | No | Yes | No | Yes | Yes | Yes |
+| Enterprise File Fabric | Yes | Yes | Yes | Yes | Yes | No | No | No | No | No | Yes |
+| FTP | No | No | Yes | Yes | No | No | Yes | No | No | No | Yes |
+| Google Cloud Storage | Yes | Yes | No | No | No | Yes | Yes | No | No | No | No |
+| Google Drive | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No | Yes | Yes | Yes |
+| Google Photos | No | No | No | No | No | No | No | No | No | No | No |
+| HDFS | Yes | No | Yes | Yes | No | No | Yes | No | No | Yes | Yes |
+| HiDrive | Yes | Yes | Yes | Yes | No | No | Yes | No | No | No | Yes |
+| HTTP | No | No | No | No | No | No | No | No | No | No | Yes |
+| Internet Archive | No | Yes | No | No | Yes | Yes | No | No | Yes | Yes | No |
+| Jottacloud | Yes | Yes | Yes | Yes | Yes | Yes | No | No | Yes | Yes | Yes |
+| Koofr | Yes | Yes | Yes | Yes | No | No | Yes | No | Yes | Yes | Yes |
+| Mail.ru Cloud | Yes | Yes | Yes | Yes | Yes | No | No | No | Yes | Yes | Yes |
+| Mega | Yes | No | Yes | Yes | Yes | No | No | No | Yes | Yes | Yes |
+| Memory | No | Yes | No | No | No | Yes | Yes | No | No | No | No |
+| Microsoft Azure Blob Storage | Yes | Yes | No | No | No | Yes | Yes | Yes | No | No | No |
+| Microsoft Azure Files Storage | No | Yes | Yes | Yes | No | No | Yes | Yes | No | Yes | Yes |
+| Microsoft OneDrive | Yes | Yes | Yes | Yes | Yes | Yes âµ | No | No | Yes | Yes | Yes |
+| OpenDrive | Yes | Yes | Yes | Yes | No | No | No | No | No | No | Yes |
+| OpenStack Swift | Yes ¹ | Yes | No | No | No | Yes | Yes | No | No | Yes | No |
+| Oracle Object Storage | No | Yes | No | No | Yes | Yes | Yes | Yes | No | No | No |
+| pCloud | Yes | Yes | Yes | Yes | Yes | No | No | No | Yes | Yes | Yes |
+| PikPak | Yes | Yes | Yes | Yes | Yes | No | No | No | Yes | Yes | Yes |
+| premiumize.me | Yes | No | Yes | Yes | No | No | No | No | Yes | Yes | Yes |
+| put.io | Yes | No | Yes | Yes | Yes | No | Yes | No | No | Yes | Yes |
+| Proton Drive | Yes | No | Yes | Yes | Yes | No | No | No | No | Yes | Yes |
+| QingStor | No | Yes | No | No | Yes | Yes | No | No | No | No | No |
+| Quatrix by Maytech | Yes | Yes | Yes | Yes | No | No | No | No | No | Yes | Yes |
+| Seafile | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No | Yes | Yes | Yes |
+| SFTP | No | Yes â´| Yes | Yes | No | No | Yes | No | No | Yes | Yes |
+| Sia | No | No | No | No | No | No | Yes | No | No | No | Yes |
+| SMB | No | No | Yes | Yes | No | No | Yes | Yes | No | No | Yes |
+| SugarSync | Yes | Yes | Yes | Yes | No | No | Yes | No | Yes | No | Yes |
+| Storj | Yes ² | Yes | Yes | No | No | Yes | Yes | No | Yes | No | No |
+| Uptobox | No | Yes | Yes | Yes | No | No | No | No | No | No | No |
+| WebDAV | Yes | Yes | Yes | Yes | No | No | Yes ³ | No | No | Yes | Yes |
+| Yandex Disk | Yes | Yes | Yes | Yes | Yes | No | Yes | No | Yes | Yes | Yes |
+| Zoho WorkDrive | Yes | Yes | Yes | Yes | No | No | No | No | No | Yes | Yes |
+| The local filesystem | Yes | No | Yes | Yes | No | No | Yes | Yes | No | Yes | Yes |
+
+¹ Note Swift implements this in order to delete directory markers but
+it doesn't actually have a quicker way of deleting files other than
+deleting them individually.
+
+² Storj implements this efficiently only for entire buckets. If
+purging a directory inside a bucket, files are deleted individually.
+
+³ StreamUpload is not supported with Nextcloud
+
+â´ Use the `--sftp-copy-is-hardlink` flag to enable.
+
+âµ Use the `--onedrive-delta` flag to enable.
+
+### Purge ###
+
+This deletes a directory quicker than just deleting all the files in
+the directory.
+
+### Copy ###
+
+Used when copying an object to and from the same remote. This known
+as a server-side copy so you can copy a file without downloading it
+and uploading it again. It is used if you use `rclone copy` or
+`rclone move` if the remote doesn't support `Move` directly.
+
+If the server doesn't support `Copy` directly then for copy operations
+the file is downloaded then re-uploaded.
+
+### Move ###
+
+Used when moving/renaming an object on the same remote. This is known
+as a server-side move of a file. This is used in `rclone move` if the
+server doesn't support `DirMove`.
+
+If the server isn't capable of `Move` then rclone simulates it with
+`Copy` then delete. If the server doesn't support `Copy` then rclone
+will download the file and re-upload it.
+
+### DirMove ###
+
+This is used to implement `rclone move` to move a directory if
+possible. If it isn't then it will use `Move` on each file (which
+falls back to `Copy` then download and upload - see `Move` section).
+
+### CleanUp ###
+
+This is used for emptying the trash for a remote by `rclone cleanup`.
+
+If the server can't do `CleanUp` then `rclone cleanup` will return an
+error.
+
+‡‡ Note that while Box implements this it has to delete every file
+individually so it will be slower than emptying the trash via the WebUI
+
+### ListR ###
+
+The remote supports a recursive list to list all the contents beneath
+a directory quickly. This enables the `--fast-list` flag to work.
+See the [rclone docs](https://rclone.org/docs/#fast-list) for more details.
+
+### StreamUpload ###
+
+Some remotes allow files to be uploaded without knowing the file size
+in advance. This allows certain operations to work without spooling the
+file to local disk first, e.g. `rclone rcat`.
+
+### MultithreadUpload ###
+
+Some remotes allow transfers to the remote to be sent as chunks in
+parallel. If this is supported then rclone will use multi-thread
+copying to transfer files much faster.
+
+### LinkSharing ###
+
+Sets the necessary permissions on a file or folder and prints a link
+that allows others to access them, even if they don't have an account
+on the particular cloud provider.
+
+### About ###
+
+Rclone `about` prints quota information for a remote. Typical output
+includes bytes used, free, quota and in trash.
+
+If a remote lacks about capability `rclone about remote:`returns
+an error.
+
+Backends without about capability cannot determine free space for an
+rclone mount, or use policy `mfs` (most free space) as a member of an
+rclone union remote.
+
+See [rclone about command](https://rclone.org/commands/rclone_about/)
+
+### EmptyDir ###
+
+The remote supports empty directories. See [Limitations](https://rclone.org/bugs/#limitations)
+ for details. Most Object/Bucket-based remotes do not support this.
+
+# Global Flags
+
+This describes the global flags available to every rclone command
+split into groups.
+
+
+## Copy
+
+Flags for anything which can Copy a file.
+
+```
+ --check-first Do all the checks before starting transfers
+ -c, --checksum Check for changes with size & checksum (if available, or fallback to size only).
+ --compare-dest stringArray Include additional comma separated server-side paths during comparison
+ --copy-dest stringArray Implies --compare-dest but also copies files from paths into destination
+ --cutoff-mode HARD|SOFT|CAUTIOUS Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default HARD)
+ --ignore-case-sync Ignore case when synchronizing
+ --ignore-checksum Skip post copy check of checksums
+ --ignore-existing Skip all files that exist on destination
+ --ignore-size Ignore size when skipping use modtime or checksum
+ -I, --ignore-times Don't skip files that match size and time - transfer all files
+ --immutable Do not modify files, fail if existing files have been modified
+ --inplace Download directly to destination file instead of atomic download to temp/rename
+ --max-backlog int Maximum number of objects in sync or check backlog (default 10000)
+ --max-duration Duration Maximum duration rclone will transfer data for (default 0s)
+ --max-transfer SizeSuffix Maximum size of data to transfer (default off)
+ -M, --metadata If set, preserve metadata when copying objects
+ --modify-window Duration Max time diff to be considered the same (default 1ns)
+ --multi-thread-chunk-size SizeSuffix Chunk size for multi-thread downloads / uploads, if not set by filesystem (default 64Mi)
+ --multi-thread-cutoff SizeSuffix Use multi-thread downloads for files above this size (default 256Mi)
+ --multi-thread-streams int Number of streams to use for multi-thread downloads (default 4)
+ --multi-thread-write-buffer-size SizeSuffix In memory buffer size for writing when in multi-thread mode (default 128Ki)
+ --no-check-dest Don't check the destination, copy regardless
+ --no-traverse Don't traverse destination file system on copy
+ --no-update-modtime Don't update destination modtime if files identical
+ --order-by string Instructions on how to order the transfers, e.g. 'size,descending'
+ --partial-suffix string Add partial-suffix to temporary file name when --inplace is not used (default ".partial")
+ --refresh-times Refresh the modtime of remote files
+ --server-side-across-configs Allow server-side operations (e.g. copy) to work across different configs
+ --size-only Skip based on size only, not modtime or checksum
+ --streaming-upload-cutoff SizeSuffix Cutoff for switching to chunked upload if file size is unknown, upload starts after reaching cutoff or when file ends (default 100Ki)
+ -u, --update Skip files that are newer on the destination
+```
+
+
+## Sync
+
+Flags just used for `rclone sync`.
+
+```
+ --backup-dir string Make backups into hierarchy based in DIR
+ --delete-after When synchronizing, delete files on destination after transferring (default)
+ --delete-before When synchronizing, delete files on destination before transferring
+ --delete-during When synchronizing, delete files during transfer
+ --ignore-errors Delete even if there are I/O errors
+ --max-delete int When synchronizing, limit the number of deletes (default -1)
+ --max-delete-size SizeSuffix When synchronizing, limit the total size of deletes (default off)
+ --suffix string Suffix to add to changed files
+ --suffix-keep-extension Preserve the extension when using --suffix
+ --track-renames When synchronizing, track file renames and do a server-side move if possible
+ --track-renames-strategy string Strategies to use when synchronizing using track-renames hash|modtime|leaf (default "hash")
+```
+
+
+## Important
+
+Important flags useful for most commands.
+
+```
+ -n, --dry-run Do a trial run with no permanent changes
+ -i, --interactive Enable interactive mode
+ -v, --verbose count Print lots more stuff (repeat for more)
+```
+
+
+## Check
+
+Flags used for `rclone check`.
+
+```
+ --max-backlog int Maximum number of objects in sync or check backlog (default 10000)
+```
+
+
+## Networking
+
+General networking and HTTP stuff.
+
+```
+ --bind string Local address to bind to for outgoing connections, IPv4, IPv6 or name
+ --bwlimit BwTimetable Bandwidth limit in KiB/s, or use suffix B|K|M|G|T|P or a full timetable
+ --bwlimit-file BwTimetable Bandwidth limit per file in KiB/s, or use suffix B|K|M|G|T|P or a full timetable
+ --ca-cert stringArray CA certificate used to verify servers
+ --client-cert string Client SSL certificate (PEM) for mutual TLS auth
+ --client-key string Client SSL private key (PEM) for mutual TLS auth
+ --contimeout Duration Connect timeout (default 1m0s)
+ --disable-http-keep-alives Disable HTTP keep-alives and use each connection once.
+ --disable-http2 Disable HTTP/2 in the global transport
+ --dscp string Set DSCP value to connections, value or name, e.g. CS1, LE, DF, AF21
+ --expect-continue-timeout Duration Timeout when using expect / 100-continue in HTTP (default 1s)
+ --header stringArray Set HTTP header for all transactions
+ --header-download stringArray Set HTTP header for download transactions
+ --header-upload stringArray Set HTTP header for upload transactions
+ --no-check-certificate Do not verify the server SSL certificate (insecure)
+ --no-gzip-encoding Don't set Accept-Encoding: gzip
+ --timeout Duration IO idle timeout (default 5m0s)
+ --tpslimit float Limit HTTP transactions per second to this
+ --tpslimit-burst int Max burst of transactions for --tpslimit (default 1)
+ --use-cookies Enable session cookiejar
+ --user-agent string Set the user-agent to a specified string (default "rclone/v1.65.0")
+```
+
+
+## Performance
+
+Flags helpful for increasing performance.
+
+```
+ --buffer-size SizeSuffix In memory buffer size when reading files for each --transfer (default 16Mi)
+ --checkers int Number of checkers to run in parallel (default 8)
+ --transfers int Number of file transfers to run in parallel (default 4)
+```
+
+
+## Config
+
+General configuration of rclone.
+
+```
+ --ask-password Allow prompt for password for encrypted configuration (default true)
+ --auto-confirm If enabled, do not request console confirmation
+ --cache-dir string Directory rclone will use for caching (default "$HOME/.cache/rclone")
+ --color AUTO|NEVER|ALWAYS When to show colors (and other ANSI codes) AUTO|NEVER|ALWAYS (default AUTO)
+ --config string Config file (default "$HOME/.config/rclone/rclone.conf")
+ --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z)
+ --disable string Disable a comma separated list of features (use --disable help to see a list)
+ -n, --dry-run Do a trial run with no permanent changes
+ --error-on-no-transfer Sets exit code 9 if no files are transferred, useful in scripts
+ --fs-cache-expire-duration Duration Cache remotes for this long (0 to disable caching) (default 5m0s)
+ --fs-cache-expire-interval Duration Interval to check for expired remotes (default 1m0s)
+ --human-readable Print numbers in a human-readable format, sizes with suffix Ki|Mi|Gi|Ti|Pi
+ -i, --interactive Enable interactive mode
+ --kv-lock-time Duration Maximum time to keep key-value database locked by process (default 1s)
+ --low-level-retries int Number of low level retries to do (default 10)
+ --no-console Hide console window (supported on Windows only)
+ --no-unicode-normalization Don't normalize unicode characters in filenames
+ --password-command SpaceSepList Command for supplying password for encrypted configuration
+ --retries int Retry operations this many times if they fail (default 3)
+ --retries-sleep Duration Interval between retrying operations if they fail, e.g. 500ms, 60s, 5m (0 to disable) (default 0s)
+ --temp-dir string Directory rclone will use for temporary files (default "/tmp")
+ --use-mmap Use mmap allocator (see docs)
+ --use-server-modtime Use server modified time instead of object metadata
+```
+
+
+## Debugging
+
+Flags for developers.
+
+```
+ --cpuprofile string Write cpu profile to file
+ --dump DumpFlags List of items to dump from: headers, bodies, requests, responses, auth, filters, goroutines, openfiles, mapper
+ --dump-bodies Dump HTTP headers and bodies - may contain sensitive info
+ --dump-headers Dump HTTP headers - may contain sensitive info
+ --memprofile string Write memory profile to file
+```
+
+
+## Filter
+
+Flags for filtering directory listings.
+
+```
+ --delete-excluded Delete files on dest excluded from sync
+ --exclude stringArray Exclude files matching pattern
+ --exclude-from stringArray Read file exclude patterns from file (use - to read from stdin)
+ --exclude-if-present stringArray Exclude directories if filename is present
+ --files-from stringArray Read list of source-file names from file (use - to read from stdin)
+ --files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin)
+ -f, --filter stringArray Add a file filtering rule
+ --filter-from stringArray Read file filtering patterns from a file (use - to read from stdin)
+ --ignore-case Ignore case in filters (case insensitive)
+ --include stringArray Include files matching pattern
+ --include-from stringArray Read file include patterns from file (use - to read from stdin)
+ --max-age Duration Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off)
+ --max-depth int If set limits the recursion depth to this (default -1)
+ --max-size SizeSuffix Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off)
+ --metadata-exclude stringArray Exclude metadatas matching pattern
+ --metadata-exclude-from stringArray Read metadata exclude patterns from file (use - to read from stdin)
+ --metadata-filter stringArray Add a metadata filtering rule
+ --metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin)
+ --metadata-include stringArray Include metadatas matching pattern
+ --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin)
+ --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off)
+ --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off)
+```
+
+
+## Listing
+
+Flags for listing directories.
+
+```
+ --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z)
+ --fast-list Use recursive list if available; uses more memory but fewer transactions
+```
+
+
+## Logging
+
+Logging and statistics.
+
+```
+ --log-file string Log everything to this file
+ --log-format string Comma separated list of log format options (default "date,time")
+ --log-level LogLevel Log level DEBUG|INFO|NOTICE|ERROR (default NOTICE)
+ --log-systemd Activate systemd integration for the logger
+ --max-stats-groups int Maximum number of stats groups to keep in memory, on max oldest is discarded (default 1000)
+ -P, --progress Show progress during transfer
+ --progress-terminal-title Show progress on the terminal title (requires -P/--progress)
+ -q, --quiet Print as little stuff as possible
+ --stats Duration Interval between printing stats, e.g. 500ms, 60s, 5m (0 to disable) (default 1m0s)
+ --stats-file-name-length int Max file name length in stats (0 for no limit) (default 45)
+ --stats-log-level LogLevel Log level to show --stats output DEBUG|INFO|NOTICE|ERROR (default INFO)
+ --stats-one-line Make the stats fit on one line
+ --stats-one-line-date Enable --stats-one-line and add current date/time prefix
+ --stats-one-line-date-format string Enable --stats-one-line-date and use custom formatted date: Enclose date string in double quotes ("), see https://golang.org/pkg/time/#Time.Format
+ --stats-unit string Show data rate in stats as either 'bits' or 'bytes' per second (default "bytes")
+ --syslog Use Syslog for logging
+ --syslog-facility string Facility for syslog, e.g. KERN,USER,... (default "DAEMON")
+ --use-json-log Use json log format
+ -v, --verbose count Print lots more stuff (repeat for more)
+```
+
+
+## Metadata
+
+Flags to control metadata.
+
+```
+ -M, --metadata If set, preserve metadata when copying objects
+ --metadata-exclude stringArray Exclude metadatas matching pattern
+ --metadata-exclude-from stringArray Read metadata exclude patterns from file (use - to read from stdin)
+ --metadata-filter stringArray Add a metadata filtering rule
+ --metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin)
+ --metadata-include stringArray Include metadatas matching pattern
+ --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin)
+ --metadata-mapper SpaceSepList Program to run to transforming metadata before upload
+ --metadata-set stringArray Add metadata key=value when uploading
+```
+
+
+## RC
+
+Flags to control the Remote Control API.
+
+```
+ --rc Enable the remote control server
+ --rc-addr stringArray IPaddress:Port or :Port to bind server to (default [localhost:5572])
+ --rc-allow-origin string Origin which cross-domain request (CORS) can be executed from
+ --rc-baseurl string Prefix for URLs - leave blank for root
+ --rc-cert string TLS PEM key (concatenation of certificate and CA certificate)
+ --rc-client-ca string Client certificate authority to verify clients with
+ --rc-enable-metrics Enable prometheus metrics on /metrics
+ --rc-files string Path to local files to serve on the HTTP server
+ --rc-htpasswd string A htpasswd file - if not provided no authentication is done
+ --rc-job-expire-duration Duration Expire finished async jobs older than this value (default 1m0s)
+ --rc-job-expire-interval Duration Interval to check for expired async jobs (default 10s)
+ --rc-key string TLS PEM Private key
+ --rc-max-header-bytes int Maximum size of request header (default 4096)
+ --rc-min-tls-version string Minimum TLS version that is acceptable (default "tls1.0")
+ --rc-no-auth Don't require auth for certain methods
+ --rc-pass string Password for authentication
+ --rc-realm string Realm for authentication
+ --rc-salt string Password hashing salt (default "dlPL2MqE")
+ --rc-serve Enable the serving of remote objects
+ --rc-server-read-timeout Duration Timeout for server reading data (default 1h0m0s)
+ --rc-server-write-timeout Duration Timeout for server writing data (default 1h0m0s)
+ --rc-template string User-specified template
+ --rc-user string User name for authentication
+ --rc-web-fetch-url string URL to fetch the releases for webgui (default "https://api.github.com/repos/rclone/rclone-webui-react/releases/latest")
+ --rc-web-gui Launch WebGUI on localhost
+ --rc-web-gui-force-update Force update to latest version of web gui
+ --rc-web-gui-no-open-browser Don't open the browser automatically
+ --rc-web-gui-update Check and update to latest version of web gui
+```
+
+
+## Backend
+
+Backend only flags. These can be set in the config file also.
+
+```
+ --acd-auth-url string Auth server URL
+ --acd-client-id string OAuth Client Id
+ --acd-client-secret string OAuth Client Secret
+ --acd-encoding Encoding The encoding for the backend (default Slash,InvalidUtf8,Dot)
+ --acd-templink-threshold SizeSuffix Files >= this size will be downloaded via their tempLink (default 9Gi)
+ --acd-token string OAuth Access Token as a JSON blob
+ --acd-token-url string Token server url
+ --acd-upload-wait-per-gb Duration Additional time per GiB to wait after a failed complete upload to see if it appears (default 3m0s)
+ --alias-remote string Remote or path to alias
+ --azureblob-access-tier string Access tier of blob: hot, cool, cold or archive
+ --azureblob-account string Azure Storage Account Name
+ --azureblob-archive-tier-delete Delete archive tier blobs before overwriting
+ --azureblob-chunk-size SizeSuffix Upload chunk size (default 4Mi)
+ --azureblob-client-certificate-password string Password for the certificate file (optional) (obscured)
+ --azureblob-client-certificate-path string Path to a PEM or PKCS12 certificate file including the private key
+ --azureblob-client-id string The ID of the client in use
+ --azureblob-client-secret string One of the service principal's client secrets
+ --azureblob-client-send-certificate-chain Send the certificate chain when using certificate auth
+ --azureblob-directory-markers Upload an empty object with a trailing slash when a new directory is created
+ --azureblob-disable-checksum Don't store MD5 checksum with object metadata
+ --azureblob-encoding Encoding The encoding for the backend (default Slash,BackSlash,Del,Ctl,RightPeriod,InvalidUtf8)
+ --azureblob-endpoint string Endpoint for the service
+ --azureblob-env-auth Read credentials from runtime (environment variables, CLI or MSI)
+ --azureblob-key string Storage Account Shared Key
+ --azureblob-list-chunk int Size of blob list (default 5000)
+ --azureblob-msi-client-id string Object ID of the user-assigned MSI to use, if any
+ --azureblob-msi-mi-res-id string Azure resource ID of the user-assigned MSI to use, if any
+ --azureblob-msi-object-id string Object ID of the user-assigned MSI to use, if any
+ --azureblob-no-check-container If set, don't attempt to check the container exists or create it
+ --azureblob-no-head-object If set, do not do HEAD before GET when getting objects
+ --azureblob-password string The user's password (obscured)
+ --azureblob-public-access string Public access level of a container: blob or container
+ --azureblob-sas-url string SAS URL for container level access only
+ --azureblob-service-principal-file string Path to file containing credentials for use with a service principal
+ --azureblob-tenant string ID of the service principal's tenant. Also called its directory ID
+ --azureblob-upload-concurrency int Concurrency for multipart uploads (default 16)
+ --azureblob-upload-cutoff string Cutoff for switching to chunked upload (<= 256 MiB) (deprecated)
+ --azureblob-use-emulator Uses local storage emulator if provided as 'true'
+ --azureblob-use-msi Use a managed service identity to authenticate (only works in Azure)
+ --azureblob-username string User name (usually an email address)
+ --azurefiles-account string Azure Storage Account Name
+ --azurefiles-chunk-size SizeSuffix Upload chunk size (default 4Mi)
+ --azurefiles-client-certificate-password string Password for the certificate file (optional) (obscured)
+ --azurefiles-client-certificate-path string Path to a PEM or PKCS12 certificate file including the private key
+ --azurefiles-client-id string The ID of the client in use
+ --azurefiles-client-secret string One of the service principal's client secrets
+ --azurefiles-client-send-certificate-chain Send the certificate chain when using certificate auth
+ --azurefiles-connection-string string Azure Files Connection String
+ --azurefiles-encoding Encoding The encoding for the backend (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Del,Ctl,RightPeriod,InvalidUtf8,Dot)
+ --azurefiles-endpoint string Endpoint for the service
+ --azurefiles-env-auth Read credentials from runtime (environment variables, CLI or MSI)
+ --azurefiles-key string Storage Account Shared Key
+ --azurefiles-max-stream-size SizeSuffix Max size for streamed files (default 10Gi)
+ --azurefiles-msi-client-id string Object ID of the user-assigned MSI to use, if any
+ --azurefiles-msi-mi-res-id string Azure resource ID of the user-assigned MSI to use, if any
+ --azurefiles-msi-object-id string Object ID of the user-assigned MSI to use, if any
+ --azurefiles-password string The user's password (obscured)
+ --azurefiles-sas-url string SAS URL
+ --azurefiles-service-principal-file string Path to file containing credentials for use with a service principal
+ --azurefiles-share-name string Azure Files Share Name
+ --azurefiles-tenant string ID of the service principal's tenant. Also called its directory ID
+ --azurefiles-upload-concurrency int Concurrency for multipart uploads (default 16)
+ --azurefiles-use-msi Use a managed service identity to authenticate (only works in Azure)
+ --azurefiles-username string User name (usually an email address)
+ --b2-account string Account ID or Application Key ID
+ --b2-chunk-size SizeSuffix Upload chunk size (default 96Mi)
+ --b2-copy-cutoff SizeSuffix Cutoff for switching to multipart copy (default 4Gi)
+ --b2-disable-checksum Disable checksums for large (> upload cutoff) files
+ --b2-download-auth-duration Duration Time before the authorization token will expire in s or suffix ms|s|m|h|d (default 1w)
+ --b2-download-url string Custom endpoint for downloads
+ --b2-encoding Encoding The encoding for the backend (default Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot)
+ --b2-endpoint string Endpoint for the service
+ --b2-hard-delete Permanently delete files on remote removal, otherwise hide files
+ --b2-key string Application Key
+ --b2-lifecycle int Set the number of days deleted files should be kept when creating a bucket
+ --b2-test-mode string A flag string for X-Bz-Test-Mode header for debugging
+ --b2-upload-concurrency int Concurrency for multipart uploads (default 4)
+ --b2-upload-cutoff SizeSuffix Cutoff for switching to chunked upload (default 200Mi)
+ --b2-version-at Time Show file versions as they were at the specified time (default off)
+ --b2-versions Include old versions in directory listings
+ --box-access-token string Box App Primary Access Token
+ --box-auth-url string Auth server URL
+ --box-box-config-file string Box App config.json location
+ --box-box-sub-type string (default "user")
+ --box-client-id string OAuth Client Id
+ --box-client-secret string OAuth Client Secret
+ --box-commit-retries int Max number of times to try committing a multipart file (default 100)
+ --box-encoding Encoding The encoding for the backend (default Slash,BackSlash,Del,Ctl,RightSpace,InvalidUtf8,Dot)
+ --box-impersonate string Impersonate this user ID when using a service account
+ --box-list-chunk int Size of listing chunk 1-1000 (default 1000)
+ --box-owned-by string Only show items owned by the login (email address) passed in
+ --box-root-folder-id string Fill in for rclone to use a non root folder as its starting point
+ --box-token string OAuth Access Token as a JSON blob
+ --box-token-url string Token server url
+ --box-upload-cutoff SizeSuffix Cutoff for switching to multipart upload (>= 50 MiB) (default 50Mi)
+ --cache-chunk-clean-interval Duration How often should the cache perform cleanups of the chunk storage (default 1m0s)
+ --cache-chunk-no-memory Disable the in-memory cache for storing chunks during streaming
+ --cache-chunk-path string Directory to cache chunk files (default "$HOME/.cache/rclone/cache-backend")
+ --cache-chunk-size SizeSuffix The size of a chunk (partial file data) (default 5Mi)
+ --cache-chunk-total-size SizeSuffix The total size that the chunks can take up on the local disk (default 10Gi)
+ --cache-db-path string Directory to store file structure metadata DB (default "$HOME/.cache/rclone/cache-backend")
+ --cache-db-purge Clear all the cached data for this remote on start
+ --cache-db-wait-time Duration How long to wait for the DB to be available - 0 is unlimited (default 1s)
+ --cache-info-age Duration How long to cache file structure information (directory listings, file size, times, etc.) (default 6h0m0s)
+ --cache-plex-insecure string Skip all certificate verification when connecting to the Plex server
+ --cache-plex-password string The password of the Plex user (obscured)
+ --cache-plex-url string The URL of the Plex server
+ --cache-plex-username string The username of the Plex user
+ --cache-read-retries int How many times to retry a read from a cache storage (default 10)
+ --cache-remote string Remote to cache
+ --cache-rps int Limits the number of requests per second to the source FS (-1 to disable) (default -1)
+ --cache-tmp-upload-path string Directory to keep temporary files until they are uploaded
+ --cache-tmp-wait-time Duration How long should files be stored in local cache before being uploaded (default 15s)
+ --cache-workers int How many workers should run in parallel to download chunks (default 4)
+ --cache-writes Cache file data on writes through the FS
+ --chunker-chunk-size SizeSuffix Files larger than chunk size will be split in chunks (default 2Gi)
+ --chunker-fail-hard Choose how chunker should handle files with missing or invalid chunks
+ --chunker-hash-type string Choose how chunker handles hash sums (default "md5")
+ --chunker-remote string Remote to chunk/unchunk
+ --combine-upstreams SpaceSepList Upstreams for combining
+ --compress-level int GZIP compression level (-2 to 9) (default -1)
+ --compress-mode string Compression mode (default "gzip")
+ --compress-ram-cache-limit SizeSuffix Some remotes don't allow the upload of files with unknown size (default 20Mi)
+ --compress-remote string Remote to compress
+ -L, --copy-links Follow symlinks and copy the pointed to item
+ --crypt-directory-name-encryption Option to either encrypt directory names or leave them intact (default true)
+ --crypt-filename-encoding string How to encode the encrypted filename to text string (default "base32")
+ --crypt-filename-encryption string How to encrypt the filenames (default "standard")
+ --crypt-no-data-encryption Option to either encrypt file data or leave it unencrypted
+ --crypt-pass-bad-blocks If set this will pass bad blocks through as all 0
+ --crypt-password string Password or pass phrase for encryption (obscured)
+ --crypt-password2 string Password or pass phrase for salt (obscured)
+ --crypt-remote string Remote to encrypt/decrypt
+ --crypt-server-side-across-configs Deprecated: use --server-side-across-configs instead
+ --crypt-show-mapping For all files listed show how the names encrypt
+ --crypt-suffix string If this is set it will override the default suffix of ".bin" (default ".bin")
+ --drive-acknowledge-abuse Set to allow files which return cannotDownloadAbusiveFile to be downloaded
+ --drive-allow-import-name-change Allow the filetype to change when uploading Google docs
+ --drive-auth-owner-only Only consider files owned by the authenticated user
+ --drive-auth-url string Auth server URL
+ --drive-chunk-size SizeSuffix Upload chunk size (default 8Mi)
+ --drive-client-id string Google Application Client Id
+ --drive-client-secret string OAuth Client Secret
+ --drive-copy-shortcut-content Server side copy contents of shortcuts instead of the shortcut
+ --drive-disable-http2 Disable drive using http2 (default true)
+ --drive-encoding Encoding The encoding for the backend (default InvalidUtf8)
+ --drive-env-auth Get IAM credentials from runtime (environment variables or instance meta data if no env vars)
+ --drive-export-formats string Comma separated list of preferred formats for downloading Google docs (default "docx,xlsx,pptx,svg")
+ --drive-fast-list-bug-fix Work around a bug in Google Drive listing (default true)
+ --drive-formats string Deprecated: See export_formats
+ --drive-impersonate string Impersonate this user when using a service account
+ --drive-import-formats string Comma separated list of preferred formats for uploading Google docs
+ --drive-keep-revision-forever Keep new head revision of each file forever
+ --drive-list-chunk int Size of listing chunk 100-1000, 0 to disable (default 1000)
+ --drive-metadata-labels Bits Control whether labels should be read or written in metadata (default off)
+ --drive-metadata-owner Bits Control whether owner should be read or written in metadata (default read)
+ --drive-metadata-permissions Bits Control whether permissions should be read or written in metadata (default off)
+ --drive-pacer-burst int Number of API calls to allow without sleeping (default 100)
+ --drive-pacer-min-sleep Duration Minimum time to sleep between API calls (default 100ms)
+ --drive-resource-key string Resource key for accessing a link-shared file
+ --drive-root-folder-id string ID of the root folder
+ --drive-scope string Comma separated list of scopes that rclone should use when requesting access from drive
+ --drive-server-side-across-configs Deprecated: use --server-side-across-configs instead
+ --drive-service-account-credentials string Service Account Credentials JSON blob
+ --drive-service-account-file string Service Account Credentials JSON file path
+ --drive-shared-with-me Only show files that are shared with me
+ --drive-show-all-gdocs Show all Google Docs including non-exportable ones in listings
+ --drive-size-as-quota Show sizes as storage quota usage, not actual size
+ --drive-skip-checksum-gphotos Skip checksums on Google photos and videos only
+ --drive-skip-dangling-shortcuts If set skip dangling shortcut files
+ --drive-skip-gdocs Skip google documents in all listings
+ --drive-skip-shortcuts If set skip shortcut files
+ --drive-starred-only Only show files that are starred
+ --drive-stop-on-download-limit Make download limit errors be fatal
+ --drive-stop-on-upload-limit Make upload limit errors be fatal
+ --drive-team-drive string ID of the Shared Drive (Team Drive)
+ --drive-token string OAuth Access Token as a JSON blob
+ --drive-token-url string Token server url
+ --drive-trashed-only Only show files that are in the trash
+ --drive-upload-cutoff SizeSuffix Cutoff for switching to chunked upload (default 8Mi)
+ --drive-use-created-date Use file created date instead of modified date
+ --drive-use-shared-date Use date file was shared instead of modified date
+ --drive-use-trash Send files to the trash instead of deleting permanently (default true)
+ --drive-v2-download-min-size SizeSuffix If Object's are greater, use drive v2 API to download (default off)
+ --dropbox-auth-url string Auth server URL
+ --dropbox-batch-commit-timeout Duration Max time to wait for a batch to finish committing (default 10m0s)
+ --dropbox-batch-mode string Upload file batching sync|async|off (default "sync")
+ --dropbox-batch-size int Max number of files in upload batch
+ --dropbox-batch-timeout Duration Max time to allow an idle upload batch before uploading (default 0s)
+ --dropbox-chunk-size SizeSuffix Upload chunk size (< 150Mi) (default 48Mi)
+ --dropbox-client-id string OAuth Client Id
+ --dropbox-client-secret string OAuth Client Secret
+ --dropbox-encoding Encoding The encoding for the backend (default Slash,BackSlash,Del,RightSpace,InvalidUtf8,Dot)
+ --dropbox-impersonate string Impersonate this user when using a business account
+ --dropbox-pacer-min-sleep Duration Minimum time to sleep between API calls (default 10ms)
+ --dropbox-shared-files Instructs rclone to work on individual shared files
+ --dropbox-shared-folders Instructs rclone to work on shared folders
+ --dropbox-token string OAuth Access Token as a JSON blob
+ --dropbox-token-url string Token server url
+ --fichier-api-key string Your API Key, get it from https://1fichier.com/console/params.pl
+ --fichier-cdn Set if you wish to use CDN download links
+ --fichier-encoding Encoding The encoding for the backend (default Slash,LtGt,DoubleQuote,SingleQuote,BackQuote,Dollar,BackSlash,Del,Ctl,LeftSpace,RightSpace,InvalidUtf8,Dot)
+ --fichier-file-password string If you want to download a shared file that is password protected, add this parameter (obscured)
+ --fichier-folder-password string If you want to list the files in a shared folder that is password protected, add this parameter (obscured)
+ --fichier-shared-folder string If you want to download a shared folder, add this parameter
+ --filefabric-encoding Encoding The encoding for the backend (default Slash,Del,Ctl,InvalidUtf8,Dot)
+ --filefabric-permanent-token string Permanent Authentication Token
+ --filefabric-root-folder-id string ID of the root folder
+ --filefabric-token string Session Token
+ --filefabric-token-expiry string Token expiry time
+ --filefabric-url string URL of the Enterprise File Fabric to connect to
+ --filefabric-version string Version read from the file fabric
+ --ftp-ask-password Allow asking for FTP password when needed
+ --ftp-close-timeout Duration Maximum time to wait for a response to close (default 1m0s)
+ --ftp-concurrency int Maximum number of FTP simultaneous connections, 0 for unlimited
+ --ftp-disable-epsv Disable using EPSV even if server advertises support
+ --ftp-disable-mlsd Disable using MLSD even if server advertises support
+ --ftp-disable-tls13 Disable TLS 1.3 (workaround for FTP servers with buggy TLS)
+ --ftp-disable-utf8 Disable using UTF-8 even if server advertises support
+ --ftp-encoding Encoding The encoding for the backend (default Slash,Del,Ctl,RightSpace,Dot)
+ --ftp-explicit-tls Use Explicit FTPS (FTP over TLS)
+ --ftp-force-list-hidden Use LIST -a to force listing of hidden files and folders. This will disable the use of MLSD
+ --ftp-host string FTP host to connect to
+ --ftp-idle-timeout Duration Max time before closing idle connections (default 1m0s)
+ --ftp-no-check-certificate Do not verify the TLS certificate of the server
+ --ftp-pass string FTP password (obscured)
+ --ftp-port int FTP port number (default 21)
+ --ftp-shut-timeout Duration Maximum time to wait for data connection closing status (default 1m0s)
+ --ftp-socks-proxy string Socks 5 proxy host
+ --ftp-tls Use Implicit FTPS (FTP over TLS)
+ --ftp-tls-cache-size int Size of TLS session cache for all control and data connections (default 32)
+ --ftp-user string FTP username (default "$USER")
+ --ftp-writing-mdtm Use MDTM to set modification time (VsFtpd quirk)
+ --gcs-anonymous Access public buckets and objects without credentials
+ --gcs-auth-url string Auth server URL
+ --gcs-bucket-acl string Access Control List for new buckets
+ --gcs-bucket-policy-only Access checks should use bucket-level IAM policies
+ --gcs-client-id string OAuth Client Id
+ --gcs-client-secret string OAuth Client Secret
+ --gcs-decompress If set this will decompress gzip encoded objects
+ --gcs-directory-markers Upload an empty object with a trailing slash when a new directory is created
+ --gcs-encoding Encoding The encoding for the backend (default Slash,CrLf,InvalidUtf8,Dot)
+ --gcs-endpoint string Endpoint for the service
+ --gcs-env-auth Get GCP IAM credentials from runtime (environment variables or instance meta data if no env vars)
+ --gcs-location string Location for the newly created buckets
+ --gcs-no-check-bucket If set, don't attempt to check the bucket exists or create it
+ --gcs-object-acl string Access Control List for new objects
+ --gcs-project-number string Project number
+ --gcs-service-account-file string Service Account Credentials JSON file path
+ --gcs-storage-class string The storage class to use when storing objects in Google Cloud Storage
+ --gcs-token string OAuth Access Token as a JSON blob
+ --gcs-token-url string Token server url
+ --gcs-user-project string User project
+ --gphotos-auth-url string Auth server URL
+ --gphotos-batch-commit-timeout Duration Max time to wait for a batch to finish committing (default 10m0s)
+ --gphotos-batch-mode string Upload file batching sync|async|off (default "sync")
+ --gphotos-batch-size int Max number of files in upload batch
+ --gphotos-batch-timeout Duration Max time to allow an idle upload batch before uploading (default 0s)
+ --gphotos-client-id string OAuth Client Id
+ --gphotos-client-secret string OAuth Client Secret
+ --gphotos-encoding Encoding The encoding for the backend (default Slash,CrLf,InvalidUtf8,Dot)
+ --gphotos-include-archived Also view and download archived media
+ --gphotos-read-only Set to make the Google Photos backend read only
+ --gphotos-read-size Set to read the size of media items
+ --gphotos-start-year int Year limits the photos to be downloaded to those which are uploaded after the given year (default 2000)
+ --gphotos-token string OAuth Access Token as a JSON blob
+ --gphotos-token-url string Token server url
+ --hasher-auto-size SizeSuffix Auto-update checksum for files smaller than this size (disabled by default)
+ --hasher-hashes CommaSepList Comma separated list of supported checksum types (default md5,sha1)
+ --hasher-max-age Duration Maximum time to keep checksums in cache (0 = no cache, off = cache forever) (default off)
+ --hasher-remote string Remote to cache checksums for (e.g. myRemote:path)
--hdfs-data-transfer-protection string Kerberos data transfer protection: authentication|integrity|privacy
- --hdfs-encoding MultiEncoder The encoding for the backend (default Slash,Colon,Del,Ctl,InvalidUtf8,Dot)
- --hdfs-namenode string Hadoop name node and port
+ --hdfs-encoding Encoding The encoding for the backend (default Slash,Colon,Del,Ctl,InvalidUtf8,Dot)
+ --hdfs-namenode CommaSepList Hadoop name nodes and ports
--hdfs-service-principal-name string Kerberos service principal name for the namenode
--hdfs-username string Hadoop user name
--hidrive-auth-url string Auth server URL
@@ -17369,7 +18721,7 @@ Backend only flags. These can be set in the config file also.
--hidrive-client-id string OAuth Client Id
--hidrive-client-secret string OAuth Client Secret
--hidrive-disable-fetching-member-count Do not fetch number of objects in directories unless it is absolutely necessary
- --hidrive-encoding MultiEncoder The encoding for the backend (default Slash,Dot)
+ --hidrive-encoding Encoding The encoding for the backend (default Slash,Dot)
--hidrive-endpoint string Endpoint for the service (default "https://api.hidrive.strato.com/2.1")
--hidrive-root-prefix string The root/parent folder for all paths (default "/")
--hidrive-scope-access string Access permissions that rclone should use when requesting access from HiDrive (default "rw")
@@ -17382,9 +18734,16 @@ Backend only flags. These can be set in the config file also.
--http-no-head Don't use HEAD requests
--http-no-slash Set this if the site doesn't end directories with /
--http-url string URL of HTTP host to connect to
+ --imagekit-encoding Encoding The encoding for the backend (default Slash,LtGt,DoubleQuote,Dollar,Question,Hash,Percent,BackSlash,Del,Ctl,InvalidUtf8,Dot,SquareBracket)
+ --imagekit-endpoint string You can find your ImageKit.io URL endpoint in your [dashboard](https://imagekit.io/dashboard/developer/api-keys)
+ --imagekit-only-signed Restrict unsigned image URLs If you have configured Restrict unsigned image URLs in your dashboard settings, set this to true
+ --imagekit-private-key string You can find your ImageKit.io private key in your [dashboard](https://imagekit.io/dashboard/developer/api-keys)
+ --imagekit-public-key string You can find your ImageKit.io public key in your [dashboard](https://imagekit.io/dashboard/developer/api-keys)
+ --imagekit-upload-tags string Tags to add to the uploaded files, e.g. "tag1,tag2"
+ --imagekit-versions Include old versions in directory listings
--internetarchive-access-key-id string IAS3 Access Key
--internetarchive-disable-checksum Don't ask the server to test against MD5 checksum calculated by rclone (default true)
- --internetarchive-encoding MultiEncoder The encoding for the backend (default Slash,LtGt,CrLf,Del,Ctl,InvalidUtf8,Dot)
+ --internetarchive-encoding Encoding The encoding for the backend (default Slash,LtGt,CrLf,Del,Ctl,InvalidUtf8,Dot)
--internetarchive-endpoint string IAS3 Endpoint (default "https://s3.us.archive.org")
--internetarchive-front-endpoint string Host of InternetArchive Frontend (default "https://archive.org")
--internetarchive-secret-access-key string IAS3 Secret Key (password)
@@ -17392,7 +18751,7 @@ Backend only flags. These can be set in the config file also.
--jottacloud-auth-url string Auth server URL
--jottacloud-client-id string OAuth Client Id
--jottacloud-client-secret string OAuth Client Secret
- --jottacloud-encoding MultiEncoder The encoding for the backend (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,Del,Ctl,InvalidUtf8,Dot)
+ --jottacloud-encoding Encoding The encoding for the backend (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,Del,Ctl,InvalidUtf8,Dot)
--jottacloud-hard-delete Delete files permanently rather than putting them into the trash
--jottacloud-md5-memory-limit SizeSuffix Files bigger than this will be cached on disk to calculate the MD5 if required (default 10Mi)
--jottacloud-no-versions Avoid server side versioning by deleting files and recreating files instead of overwriting them
@@ -17400,17 +18759,18 @@ Backend only flags. These can be set in the config file also.
--jottacloud-token-url string Token server url
--jottacloud-trashed-only Only show files that are in the trash
--jottacloud-upload-resume-limit SizeSuffix Files bigger than this can be resumed if the upload fail's (default 10Mi)
- --koofr-encoding MultiEncoder The encoding for the backend (default Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot)
+ --koofr-encoding Encoding The encoding for the backend (default Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot)
--koofr-endpoint string The Koofr API endpoint to use
--koofr-mountid string Mount ID of the mount to use
--koofr-password string Your password for rclone (generate one at https://app.koofr.net/app/admin/preferences/password) (obscured)
--koofr-provider string Choose your storage provider
--koofr-setmtime Does the backend support setting modification time (default true)
--koofr-user string Your user name
+ --linkbox-token string Token from https://www.linkbox.to/admin/account
-l, --links Translate symlinks to/from regular files with a '.rclonelink' extension
--local-case-insensitive Force the filesystem to report itself as case insensitive
--local-case-sensitive Force the filesystem to report itself as case sensitive
- --local-encoding MultiEncoder The encoding for the backend (default Slash,Dot)
+ --local-encoding Encoding The encoding for the backend (default Slash,Dot)
--local-no-check-updated Don't check to see if the files change during upload
--local-no-preallocate Disable preallocation of disk space for transferred files
--local-no-set-modtime Disable setting modtime
@@ -17422,7 +18782,7 @@ Backend only flags. These can be set in the config file also.
--mailru-check-hash What should copy do if file checksum is mismatched or invalid (default true)
--mailru-client-id string OAuth Client Id
--mailru-client-secret string OAuth Client Secret
- --mailru-encoding MultiEncoder The encoding for the backend (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Del,Ctl,InvalidUtf8,Dot)
+ --mailru-encoding Encoding The encoding for the backend (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Del,Ctl,InvalidUtf8,Dot)
--mailru-pass string Password (obscured)
--mailru-speedup-enable Skip full upload if there is another file with same data hash (default true)
--mailru-speedup-file-patterns string Comma separated list of file name patterns eligible for speedup (put by hash) (default "*.mkv,*.avi,*.mp4,*.mp3,*.zip,*.gz,*.rar,*.pdf")
@@ -17432,7 +18792,7 @@ Backend only flags. These can be set in the config file also.
--mailru-token-url string Token server url
--mailru-user string User name (usually email)
--mega-debug Output more debug from Mega
- --mega-encoding MultiEncoder The encoding for the backend (default Slash,InvalidUtf8,Dot)
+ --mega-encoding Encoding The encoding for the backend (default Slash,InvalidUtf8,Dot)
--mega-hard-delete Delete files permanently rather than putting them into the trash
--mega-pass string Password (obscured)
--mega-use-https Use HTTPS for transfers
@@ -17448,9 +18808,10 @@ Backend only flags. These can be set in the config file also.
--onedrive-chunk-size SizeSuffix Chunk size to upload files with - must be multiple of 320k (327,680 bytes) (default 10Mi)
--onedrive-client-id string OAuth Client Id
--onedrive-client-secret string OAuth Client Secret
+ --onedrive-delta If set rclone will use delta listing to implement recursive listings
--onedrive-drive-id string The ID of the drive to use
--onedrive-drive-type string The type of the drive (personal | business | documentLibrary)
- --onedrive-encoding MultiEncoder The encoding for the backend (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Del,Ctl,LeftSpace,LeftTilde,RightSpace,RightPeriod,InvalidUtf8,Dot)
+ --onedrive-encoding Encoding The encoding for the backend (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Del,Ctl,LeftSpace,LeftTilde,RightSpace,RightPeriod,InvalidUtf8,Dot)
--onedrive-expose-onenote-files Set to make OneNote files show up in directory listings
--onedrive-hash-type string Specify the hash in use for the backend (default "auto")
--onedrive-link-password string Set the password for links created by the link command
@@ -17471,7 +18832,7 @@ Backend only flags. These can be set in the config file also.
--oos-copy-cutoff SizeSuffix Cutoff for switching to multipart copy (default 4.656Gi)
--oos-copy-timeout Duration Timeout for copy (default 1m0s)
--oos-disable-checksum Don't store MD5 checksum with object metadata
- --oos-encoding MultiEncoder The encoding for the backend (default Slash,InvalidUtf8,Dot)
+ --oos-encoding Encoding The encoding for the backend (default Slash,InvalidUtf8,Dot)
--oos-endpoint string Endpoint for Object storage API
--oos-leave-parts-on-error If true avoid calling abort upload on a failure, leaving all successfully uploaded parts for manual recovery
--oos-max-upload-parts int Maximum number of parts in a multipart upload (default 10000)
@@ -17488,13 +18849,13 @@ Backend only flags. These can be set in the config file also.
--oos-upload-concurrency int Concurrency for multipart uploads (default 10)
--oos-upload-cutoff SizeSuffix Cutoff for switching to chunked upload (default 200Mi)
--opendrive-chunk-size SizeSuffix Files will be uploaded in chunks this size (default 10Mi)
- --opendrive-encoding MultiEncoder The encoding for the backend (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,LeftSpace,LeftCrLfHtVt,RightSpace,RightCrLfHtVt,InvalidUtf8,Dot)
+ --opendrive-encoding Encoding The encoding for the backend (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,LeftSpace,LeftCrLfHtVt,RightSpace,RightCrLfHtVt,InvalidUtf8,Dot)
--opendrive-password string Password (obscured)
--opendrive-username string Username
--pcloud-auth-url string Auth server URL
--pcloud-client-id string OAuth Client Id
--pcloud-client-secret string OAuth Client Secret
- --pcloud-encoding MultiEncoder The encoding for the backend (default Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot)
+ --pcloud-encoding Encoding The encoding for the backend (default Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot)
--pcloud-hostname string Hostname to connect to (default "api.pcloud.com")
--pcloud-password string Your pcloud password (obscured)
--pcloud-root-folder-id string Fill in for rclone to use a non root folder as its starting point (default "d0")
@@ -17504,7 +18865,7 @@ Backend only flags. These can be set in the config file also.
--pikpak-auth-url string Auth server URL
--pikpak-client-id string OAuth Client Id
--pikpak-client-secret string OAuth Client Secret
- --pikpak-encoding MultiEncoder The encoding for the backend (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,LeftSpace,RightSpace,RightPeriod,InvalidUtf8,Dot)
+ --pikpak-encoding Encoding The encoding for the backend (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,LeftSpace,RightSpace,RightPeriod,InvalidUtf8,Dot)
--pikpak-hash-memory-limit SizeSuffix Files bigger than this will be cached on disk to calculate hash if required (default 10Mi)
--pikpak-pass string Pikpak password (obscured)
--pikpak-root-folder-id string ID of the root folder
@@ -17516,13 +18877,13 @@ Backend only flags. These can be set in the config file also.
--premiumizeme-auth-url string Auth server URL
--premiumizeme-client-id string OAuth Client Id
--premiumizeme-client-secret string OAuth Client Secret
- --premiumizeme-encoding MultiEncoder The encoding for the backend (default Slash,DoubleQuote,BackSlash,Del,Ctl,InvalidUtf8,Dot)
+ --premiumizeme-encoding Encoding The encoding for the backend (default Slash,DoubleQuote,BackSlash,Del,Ctl,InvalidUtf8,Dot)
--premiumizeme-token string OAuth Access Token as a JSON blob
--premiumizeme-token-url string Token server url
--protondrive-2fa string The 2FA code
--protondrive-app-version string The app version string (default "macos-drive@1.0.0-alpha.1+rclone")
--protondrive-enable-caching Caches the files and folders metadata to reduce API calls (default true)
- --protondrive-encoding MultiEncoder The encoding for the backend (default Slash,LeftSpace,RightSpace,InvalidUtf8,Dot)
+ --protondrive-encoding Encoding The encoding for the backend (default Slash,LeftSpace,RightSpace,InvalidUtf8,Dot)
--protondrive-mailbox-password string The mailbox password of your two-password proton account (obscured)
--protondrive-original-file-size Return the file size before encryption (default true)
--protondrive-password string The password of your proton account (obscured)
@@ -17531,13 +18892,13 @@ Backend only flags. These can be set in the config file also.
--putio-auth-url string Auth server URL
--putio-client-id string OAuth Client Id
--putio-client-secret string OAuth Client Secret
- --putio-encoding MultiEncoder The encoding for the backend (default Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot)
+ --putio-encoding Encoding The encoding for the backend (default Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot)
--putio-token string OAuth Access Token as a JSON blob
--putio-token-url string Token server url
--qingstor-access-key-id string QingStor Access Key ID
--qingstor-chunk-size SizeSuffix Chunk size to use for uploading (default 4Mi)
--qingstor-connection-retries int Number of connection retries (default 3)
- --qingstor-encoding MultiEncoder The encoding for the backend (default Slash,Ctl,InvalidUtf8)
+ --qingstor-encoding Encoding The encoding for the backend (default Slash,Ctl,InvalidUtf8)
--qingstor-endpoint string Enter an endpoint URL to connection QingStor API
--qingstor-env-auth Get QingStor credentials from runtime
--qingstor-secret-access-key string QingStor Secret Access Key (password)
@@ -17546,7 +18907,7 @@ Backend only flags. These can be set in the config file also.
--qingstor-zone string Zone to connect to
--quatrix-api-key string API key for accessing Quatrix account
--quatrix-effective-upload-time string Wanted upload time for one chunk (default "4s")
- --quatrix-encoding MultiEncoder The encoding for the backend (default Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot)
+ --quatrix-encoding Encoding The encoding for the backend (default Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot)
--quatrix-hard-delete Delete files permanently rather than putting them into the trash
--quatrix-host string Host name of Quatrix account
--quatrix-maximal-summary-chunk-size SizeSuffix The maximal summary for all chunks. It should not be less than 'transfers'*'minimal_chunk_size' (default 95.367Mi)
@@ -17561,7 +18922,7 @@ Backend only flags. These can be set in the config file also.
--s3-disable-checksum Don't store MD5 checksum with object metadata
--s3-disable-http2 Disable usage of http2 for S3 backends
--s3-download-url string Custom endpoint for downloads
- --s3-encoding MultiEncoder The encoding for the backend (default Slash,InvalidUtf8,Dot)
+ --s3-encoding Encoding The encoding for the backend (default Slash,InvalidUtf8,Dot)
--s3-endpoint string Endpoint for S3 API
--s3-env-auth Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars)
--s3-force-path-style If true use path style access if false use virtual hosted style (default true)
@@ -17595,14 +18956,16 @@ Backend only flags. These can be set in the config file also.
--s3-upload-cutoff SizeSuffix Cutoff for switching to chunked upload (default 200Mi)
--s3-use-accelerate-endpoint If true use the AWS S3 accelerated endpoint
--s3-use-accept-encoding-gzip Accept-Encoding: gzip Whether to send Accept-Encoding: gzip header (default unset)
+ --s3-use-already-exists Tristate Set if rclone should report BucketAlreadyExists errors on bucket creation (default unset)
--s3-use-multipart-etag Tristate Whether to use ETag in multipart uploads for verification (default unset)
+ --s3-use-multipart-uploads Tristate Set if rclone should use multipart uploads (default unset)
--s3-use-presigned-request Whether to use a presigned request or PutObject for single part uploads
--s3-v2-auth If true use v2 authentication
--s3-version-at Time Show file versions as they were at the specified time (default off)
--s3-versions Include old versions in directory listings
--seafile-2fa Two-factor authentication ('true' if the account has 2FA enabled)
--seafile-create-library Should rclone create a library if it doesn't exist
- --seafile-encoding MultiEncoder The encoding for the backend (default Slash,DoubleQuote,BackSlash,Ctl,InvalidUtf8)
+ --seafile-encoding Encoding The encoding for the backend (default Slash,DoubleQuote,BackSlash,Ctl,InvalidUtf8)
--seafile-library string Name of the library
--seafile-library-key string Library password (for encrypted libraries only) (obscured)
--seafile-pass string Password (obscured)
@@ -17612,6 +18975,7 @@ Backend only flags. These can be set in the config file also.
--sftp-chunk-size SizeSuffix Upload and download chunk size (default 32Ki)
--sftp-ciphers SpaceSepList Space separated list of ciphers to be used for session encryption, ordered by preference
--sftp-concurrency int The maximum number of outstanding requests for one file (default 64)
+ --sftp-copy-is-hardlink Set to enable server side copies using hardlinks
--sftp-disable-concurrent-reads If set don't use concurrent reads
--sftp-disable-concurrent-writes If set don't use concurrent writes
--sftp-disable-hashcheck Disable the execution of SSH commands to determine if remote file hashing is available
@@ -17646,7 +19010,7 @@ Backend only flags. These can be set in the config file also.
--sharefile-chunk-size SizeSuffix Upload chunk size (default 64Mi)
--sharefile-client-id string OAuth Client Id
--sharefile-client-secret string OAuth Client Secret
- --sharefile-encoding MultiEncoder The encoding for the backend (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,LeftSpace,LeftPeriod,RightSpace,RightPeriod,InvalidUtf8,Dot)
+ --sharefile-encoding Encoding The encoding for the backend (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,LeftSpace,LeftPeriod,RightSpace,RightPeriod,InvalidUtf8,Dot)
--sharefile-endpoint string Endpoint for API calls
--sharefile-root-folder-id string ID of the root folder
--sharefile-token string OAuth Access Token as a JSON blob
@@ -17654,12 +19018,12 @@ Backend only flags. These can be set in the config file also.
--sharefile-upload-cutoff SizeSuffix Cutoff for switching to multipart upload (default 128Mi)
--sia-api-password string Sia Daemon API Password (obscured)
--sia-api-url string Sia daemon API URL, like http://sia.daemon.host:9980 (default "http://127.0.0.1:9980")
- --sia-encoding MultiEncoder The encoding for the backend (default Slash,Question,Hash,Percent,Del,Ctl,InvalidUtf8,Dot)
+ --sia-encoding Encoding The encoding for the backend (default Slash,Question,Hash,Percent,Del,Ctl,InvalidUtf8,Dot)
--sia-user-agent string Siad User Agent (default "Sia-Agent")
--skip-links Don't warn about skipped symlinks
--smb-case-insensitive Whether the server is configured to be case-insensitive (default true)
--smb-domain string Domain name for NTLM authentication (default "WORKGROUP")
- --smb-encoding MultiEncoder The encoding for the backend (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot)
+ --smb-encoding Encoding The encoding for the backend (default Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot)
--smb-hide-special-share Hide special shares (e.g. print$) which users aren't supposed to access (default true)
--smb-host string SMB server hostname to connect to
--smb-idle-timeout Duration Max time before closing idle connections (default 1m0s)
@@ -17677,7 +19041,7 @@ Backend only flags. These can be set in the config file also.
--sugarsync-authorization string Sugarsync authorization
--sugarsync-authorization-expiry string Sugarsync authorization expiry
--sugarsync-deleted-id string Sugarsync deleted folder id
- --sugarsync-encoding MultiEncoder The encoding for the backend (default Slash,Ctl,InvalidUtf8,Dot)
+ --sugarsync-encoding Encoding The encoding for the backend (default Slash,Ctl,InvalidUtf8,Dot)
--sugarsync-hard-delete Permanently delete files if true
--sugarsync-private-access-key string Sugarsync Private Access Key
--sugarsync-refresh-token string Sugarsync refresh token
@@ -17691,7 +19055,7 @@ Backend only flags. These can be set in the config file also.
--swift-auth-version int AuthVersion - optional - set to (1,2,3) if your auth URL has no version (ST_AUTH_VERSION)
--swift-chunk-size SizeSuffix Above this size files will be chunked into a _segments container (default 5Gi)
--swift-domain string User domain - optional (v3 auth) (OS_USER_DOMAIN_NAME)
- --swift-encoding MultiEncoder The encoding for the backend (default Slash,InvalidUtf8)
+ --swift-encoding Encoding The encoding for the backend (default Slash,InvalidUtf8)
--swift-endpoint-type string Endpoint type to choose from the service catalogue (OS_ENDPOINT_TYPE) (default "public")
--swift-env-auth Get swift credentials from environment variables in standard OpenStack form
--swift-key string API key or password (OS_PASSWORD)
@@ -17713,7 +19077,7 @@ Backend only flags. These can be set in the config file also.
--union-search-policy string Policy to choose upstream on SEARCH category (default "ff")
--union-upstreams string List of space separated upstreams
--uptobox-access-token string Your access token
- --uptobox-encoding MultiEncoder The encoding for the backend (default Slash,LtGt,DoubleQuote,BackQuote,Del,Ctl,LeftSpace,InvalidUtf8,Dot)
+ --uptobox-encoding Encoding The encoding for the backend (default Slash,LtGt,DoubleQuote,BackQuote,Del,Ctl,LeftSpace,InvalidUtf8,Dot)
--uptobox-private Set to make uploaded files private
--webdav-bearer-token string Bearer token instead of user/pass (e.g. a Macaroon)
--webdav-bearer-token-command string Command to run to get a bearer token
@@ -17728,9421 +19092,10046 @@ Backend only flags. These can be set in the config file also.
--yandex-auth-url string Auth server URL
--yandex-client-id string OAuth Client Id
--yandex-client-secret string OAuth Client Secret
- --yandex-encoding MultiEncoder The encoding for the backend (default Slash,Del,Ctl,InvalidUtf8,Dot)
+ --yandex-encoding Encoding The encoding for the backend (default Slash,Del,Ctl,InvalidUtf8,Dot)
--yandex-hard-delete Delete files permanently rather than putting them into the trash
--yandex-token string OAuth Access Token as a JSON blob
--yandex-token-url string Token server url
--zoho-auth-url string Auth server URL
--zoho-client-id string OAuth Client Id
--zoho-client-secret string OAuth Client Secret
- --zoho-encoding MultiEncoder The encoding for the backend (default Del,Ctl,InvalidUtf8)
+ --zoho-encoding Encoding The encoding for the backend (default Del,Ctl,InvalidUtf8)
--zoho-region string Zoho region to connect to
--zoho-token string OAuth Access Token as a JSON blob
--zoho-token-url string Token server url
```
-# Docker Volume Plugin
+# Docker Volume Plugin
+
+## Introduction
+
+Docker 1.9 has added support for creating
+[named volumes](https://docs.docker.com/storage/volumes/) via
+[command-line interface](https://docs.docker.com/engine/reference/commandline/volume_create/)
+and mounting them in containers as a way to share data between them.
+Since Docker 1.10 you can create named volumes with
+[Docker Compose](https://docs.docker.com/compose/) by descriptions in
+[docker-compose.yml](https://docs.docker.com/compose/compose-file/compose-file-v2/#volume-configuration-reference)
+files for use by container groups on a single host.
+As of Docker 1.12 volumes are supported by
+[Docker Swarm](https://docs.docker.com/engine/swarm/key-concepts/)
+included with Docker Engine and created from descriptions in
+[swarm compose v3](https://docs.docker.com/compose/compose-file/compose-file-v3/#volume-configuration-reference)
+files for use with _swarm stacks_ across multiple cluster nodes.
+
+[Docker Volume Plugins](https://docs.docker.com/engine/extend/plugins_volume/)
+augment the default `local` volume driver included in Docker with stateful
+volumes shared across containers and hosts. Unlike local volumes, your
+data will _not_ be deleted when such volume is removed. Plugins can run
+managed by the docker daemon, as a native system service
+(under systemd, _sysv_ or _upstart_) or as a standalone executable.
+Rclone can run as docker volume plugin in all these modes.
+It interacts with the local docker daemon
+via [plugin API](https://docs.docker.com/engine/extend/plugin_api/) and
+handles mounting of remote file systems into docker containers so it must
+run on the same host as the docker daemon or on every Swarm node.
+
+## Getting started
+
+In the first example we will use the [SFTP](https://rclone.org/sftp/)
+rclone volume with Docker engine on a standalone Ubuntu machine.
+
+Start from [installing Docker](https://docs.docker.com/engine/install/)
+on the host.
+
+The _FUSE_ driver is a prerequisite for rclone mounting and should be
+installed on host:
+```
+sudo apt-get -y install fuse
+```
+
+Create two directories required by rclone docker plugin:
+```
+sudo mkdir -p /var/lib/docker-plugins/rclone/config
+sudo mkdir -p /var/lib/docker-plugins/rclone/cache
+```
+
+Install the managed rclone docker plugin for your architecture (here `amd64`):
+```
+docker plugin install rclone/docker-volume-rclone:amd64 args="-v" --alias rclone --grant-all-permissions
+docker plugin list
+```
+
+Create your [SFTP volume](https://rclone.org/sftp/#standard-options):
+```
+docker volume create firstvolume -d rclone -o type=sftp -o sftp-host=_hostname_ -o sftp-user=_username_ -o sftp-pass=_password_ -o allow-other=true
+```
+
+Note that since all options are static, you don't even have to run
+`rclone config` or create the `rclone.conf` file (but the `config` directory
+should still be present). In the simplest case you can use `localhost`
+as _hostname_ and your SSH credentials as _username_ and _password_.
+You can also change the remote path to your home directory on the host,
+for example `-o path=/home/username`.
+
+
+Time to create a test container and mount the volume into it:
+```
+docker run --rm -it -v firstvolume:/mnt --workdir /mnt ubuntu:latest bash
+```
+
+If all goes well, you will enter the new container and change right to
+the mounted SFTP remote. You can type `ls` to list the mounted directory
+or otherwise play with it. Type `exit` when you are done.
+The container will stop but the volume will stay, ready to be reused.
+When it's not needed anymore, remove it:
+```
+docker volume list
+docker volume remove firstvolume
+```
+
+Now let us try **something more elaborate**:
+[Google Drive](https://rclone.org/drive/) volume on multi-node Docker Swarm.
+
+You should start from installing Docker and FUSE, creating plugin
+directories and installing rclone plugin on _every_ swarm node.
+Then [setup the Swarm](https://docs.docker.com/engine/swarm/swarm-mode/).
+
+Google Drive volumes need an access token which can be setup via web
+browser and will be periodically renewed by rclone. The managed
+plugin cannot run a browser so we will use a technique similar to the
+[rclone setup on a headless box](https://rclone.org/remote_setup/).
+
+Run [rclone config](https://rclone.org/commands/rclone_config_create/)
+on _another_ machine equipped with _web browser_ and graphical user interface.
+Create the [Google Drive remote](https://rclone.org/drive/#standard-options).
+When done, transfer the resulting `rclone.conf` to the Swarm cluster
+and save as `/var/lib/docker-plugins/rclone/config/rclone.conf`
+on _every_ node. By default this location is accessible only to the
+root user so you will need appropriate privileges. The resulting config
+will look like this:
+```
+[gdrive]
+type = drive
+scope = drive
+drive_id = 1234567...
+root_folder_id = 0Abcd...
+token = {"access_token":...}
+```
+
+Now create the file named `example.yml` with a swarm stack description
+like this:
+```
+version: '3'
+services:
+ heimdall:
+ image: linuxserver/heimdall:latest
+ ports: [8080:80]
+ volumes: [configdata:/config]
+volumes:
+ configdata:
+ driver: rclone
+ driver_opts:
+ remote: 'gdrive:heimdall'
+ allow_other: 'true'
+ vfs_cache_mode: full
+ poll_interval: 0
+```
+
+and run the stack:
+```
+docker stack deploy example -c ./example.yml
+```
+
+After a few seconds docker will spread the parsed stack description
+over cluster, create the `example_heimdall` service on port _8080_,
+run service containers on one or more cluster nodes and request
+the `example_configdata` volume from rclone plugins on the node hosts.
+You can use the following commands to confirm results:
+```
+docker service ls
+docker service ps example_heimdall
+docker volume ls
+```
+
+Point your browser to `http://cluster.host.address:8080` and play with
+the service. Stop it with `docker stack remove example` when you are done.
+Note that the `example_configdata` volume(s) created on demand at the
+cluster nodes will not be automatically removed together with the stack
+but stay for future reuse. You can remove them manually by invoking
+the `docker volume remove example_configdata` command on every node.
+
+## Creating Volumes via CLI
+
+Volumes can be created with [docker volume create](https://docs.docker.com/engine/reference/commandline/volume_create/).
+Here are a few examples:
+```
+docker volume create vol1 -d rclone -o remote=storj: -o vfs-cache-mode=full
+docker volume create vol2 -d rclone -o remote=:storj,access_grant=xxx:heimdall
+docker volume create vol3 -d rclone -o type=storj -o path=heimdall -o storj-access-grant=xxx -o poll-interval=0
+```
+
+Note the `-d rclone` flag that tells docker to request volume from the
+rclone driver. This works even if you installed managed driver by its full
+name `rclone/docker-volume-rclone` because you provided the `--alias rclone`
+option.
+
+Volumes can be inspected as follows:
+```
+docker volume list
+docker volume inspect vol1
+```
+
+## Volume Configuration
+
+Rclone flags and volume options are set via the `-o` flag to the
+`docker volume create` command. They include backend-specific parameters
+as well as mount and _VFS_ options. Also there are a few
+special `-o` options:
+`remote`, `fs`, `type`, `path`, `mount-type` and `persist`.
+
+`remote` determines an existing remote name from the config file, with
+trailing colon and optionally with a remote path. See the full syntax in
+the [rclone documentation](https://rclone.org/docs/#syntax-of-remote-paths).
+This option can be aliased as `fs` to prevent confusion with the
+_remote_ parameter of such backends as _crypt_ or _alias_.
+
+The `remote=:backend:dir/subdir` syntax can be used to create
+[on-the-fly (config-less) remotes](https://rclone.org/docs/#backend-path-to-dir),
+while the `type` and `path` options provide a simpler alternative for this.
+Using two split options
+```
+-o type=backend -o path=dir/subdir
+```
+is equivalent to the combined syntax
+```
+-o remote=:backend:dir/subdir
+```
+but is arguably easier to parameterize in scripts.
+The `path` part is optional.
+
+[Mount and VFS options](https://rclone.org/commands/rclone_serve_docker/#options)
+as well as [backend parameters](https://rclone.org/flags/#backend-flags) are named
+like their twin command-line flags without the `--` CLI prefix.
+Optionally you can use underscores instead of dashes in option names.
+For example, `--vfs-cache-mode full` becomes
+`-o vfs-cache-mode=full` or `-o vfs_cache_mode=full`.
+Boolean CLI flags without value will gain the `true` value, e.g.
+`--allow-other` becomes `-o allow-other=true` or `-o allow_other=true`.
+
+Please note that you can provide parameters only for the backend immediately
+referenced by the backend type of mounted `remote`.
+If this is a wrapping backend like _alias, chunker or crypt_, you cannot
+provide options for the referred to remote or backend. This limitation is
+imposed by the rclone connection string parser. The only workaround is to
+feed plugin with `rclone.conf` or configure plugin arguments (see below).
+
+## Special Volume Options
+
+`mount-type` determines the mount method and in general can be one of:
+`mount`, `cmount`, or `mount2`. This can be aliased as `mount_type`.
+It should be noted that the managed rclone docker plugin currently does
+not support the `cmount` method and `mount2` is rarely needed.
+This option defaults to the first found method, which is usually `mount`
+so you generally won't need it.
+
+`persist` is a reserved boolean (true/false) option.
+In future it will allow to persist on-the-fly remotes in the plugin
+`rclone.conf` file.
+
+## Connection Strings
+
+The `remote` value can be extended
+with [connection strings](https://rclone.org/docs/#connection-strings)
+as an alternative way to supply backend parameters. This is equivalent
+to the `-o` backend options with one _syntactic difference_.
+Inside connection string the backend prefix must be dropped from parameter
+names but in the `-o param=value` array it must be present.
+For instance, compare the following option array
+```
+-o remote=:sftp:/home -o sftp-host=localhost
+```
+with equivalent connection string:
+```
+-o remote=:sftp,host=localhost:/home
+```
+This difference exists because flag options `-o key=val` include not only
+backend parameters but also mount/VFS flags and possibly other settings.
+Also it allows to discriminate the `remote` option from the `crypt-remote`
+(or similarly named backend parameters) and arguably simplifies scripting
+due to clearer value substitution.
+
+## Using with Swarm or Compose
+
+Both _Docker Swarm_ and _Docker Compose_ use
+[YAML](http://yaml.org/spec/1.2/spec.html)-formatted text files to describe
+groups (stacks) of containers, their properties, networks and volumes.
+_Compose_ uses the [compose v2](https://docs.docker.com/compose/compose-file/compose-file-v2/#volume-configuration-reference) format,
+_Swarm_ uses the [compose v3](https://docs.docker.com/compose/compose-file/compose-file-v3/#volume-configuration-reference) format.
+They are mostly similar, differences are explained in the
+[docker documentation](https://docs.docker.com/compose/compose-file/compose-versioning/#upgrading).
+
+Volumes are described by the children of the top-level `volumes:` node.
+Each of them should be named after its volume and have at least two
+elements, the self-explanatory `driver: rclone` value and the
+`driver_opts:` structure playing the same role as `-o key=val` CLI flags:
+
+```
+volumes:
+ volume_name_1:
+ driver: rclone
+ driver_opts:
+ remote: 'gdrive:'
+ allow_other: 'true'
+ vfs_cache_mode: full
+ token: '{"type": "borrower", "expires": "2021-12-31"}'
+ poll_interval: 0
+```
+
+Notice a few important details:
+- YAML prefers `_` in option names instead of `-`.
+- YAML treats single and double quotes interchangeably.
+ Simple strings and integers can be left unquoted.
+- Boolean values must be quoted like `'true'` or `"false"` because
+ these two words are reserved by YAML.
+- The filesystem string is keyed with `remote` (or with `fs`).
+ Normally you can omit quotes here, but if the string ends with colon,
+ you **must** quote it like `remote: "storage_box:"`.
+- YAML is picky about surrounding braces in values as this is in fact
+ another [syntax for key/value mappings](http://yaml.org/spec/1.2/spec.html#id2790832).
+ For example, JSON access tokens usually contain double quotes and
+ surrounding braces, so you must put them in single quotes.
+
+## Installing as Managed Plugin
+
+Docker daemon can install plugins from an image registry and run them managed.
+We maintain the
+[docker-volume-rclone](https://hub.docker.com/p/rclone/docker-volume-rclone/)
+plugin image on [Docker Hub](https://hub.docker.com).
+
+Rclone volume plugin requires **Docker Engine >= 19.03.15**
+
+The plugin requires presence of two directories on the host before it can
+be installed. Note that plugin will **not** create them automatically.
+By default they must exist on host at the following locations
+(though you can tweak the paths):
+- `/var/lib/docker-plugins/rclone/config`
+ is reserved for the `rclone.conf` config file and **must** exist
+ even if it's empty and the config file is not present.
+- `/var/lib/docker-plugins/rclone/cache`
+ holds the plugin state file as well as optional VFS caches.
+
+You can [install managed plugin](https://docs.docker.com/engine/reference/commandline/plugin_install/)
+with default settings as follows:
+```
+docker plugin install rclone/docker-volume-rclone:amd64 --grant-all-permissions --alias rclone
+```
+
+The `:amd64` part of the image specification after colon is called a _tag_.
+Usually you will want to install the latest plugin for your architecture. In
+this case the tag will just name it, like `amd64` above. The following plugin
+architectures are currently available:
+- `amd64`
+- `arm64`
+- `arm-v7`
+
+Sometimes you might want a concrete plugin version, not the latest one.
+Then you should use image tag in the form `:ARCHITECTURE-VERSION`.
+For example, to install plugin version `v1.56.2` on architecture `arm64`
+you will use tag `arm64-1.56.2` (note the removed `v`) so the full image
+specification becomes `rclone/docker-volume-rclone:arm64-1.56.2`.
+
+We also provide the `latest` plugin tag, but since docker does not support
+multi-architecture plugins as of the time of this writing, this tag is
+currently an **alias for `amd64`**.
+By convention the `latest` tag is the default one and can be omitted, thus
+both `rclone/docker-volume-rclone:latest` and just `rclone/docker-volume-rclone`
+will refer to the latest plugin release for the `amd64` platform.
+
+Also the `amd64` part can be omitted from the versioned rclone plugin tags.
+For example, rclone image reference `rclone/docker-volume-rclone:amd64-1.56.2`
+can be abbreviated as `rclone/docker-volume-rclone:1.56.2` for convenience.
+However, for non-intel architectures you still have to use the full tag as
+`amd64` or `latest` will fail to start.
+
+Managed plugin is in fact a special container running in a namespace separate
+from normal docker containers. Inside it runs the `rclone serve docker`
+command. The config and cache directories are bind-mounted into the
+container at start. The docker daemon connects to a unix socket created
+by the command inside the container. The command creates on-demand remote
+mounts right inside, then docker machinery propagates them through kernel
+mount namespaces and bind-mounts into requesting user containers.
+
+You can tweak a few plugin settings after installation when it's disabled
+(not in use), for instance:
+```
+docker plugin disable rclone
+docker plugin set rclone RCLONE_VERBOSE=2 config=/etc/rclone args="--vfs-cache-mode=writes --allow-other"
+docker plugin enable rclone
+docker plugin inspect rclone
+```
+
+Note that if docker refuses to disable the plugin, you should find and
+remove all active volumes connected with it as well as containers and
+swarm services that use them. This is rather tedious so please carefully
+plan in advance.
+
+You can tweak the following settings:
+`args`, `config`, `cache`, `HTTP_PROXY`, `HTTPS_PROXY`, `NO_PROXY`
+and `RCLONE_VERBOSE`.
+It's _your_ task to keep plugin settings in sync across swarm cluster nodes.
+
+`args` sets command-line arguments for the `rclone serve docker` command
+(_none_ by default). Arguments should be separated by space so you will
+normally want to put them in quotes on the
+[docker plugin set](https://docs.docker.com/engine/reference/commandline/plugin_set/)
+command line. Both [serve docker flags](https://rclone.org/commands/rclone_serve_docker/#options)
+and [generic rclone flags](https://rclone.org/flags/) are supported, including backend
+parameters that will be used as defaults for volume creation.
+Note that plugin will fail (due to [this docker bug](https://github.com/moby/moby/blob/v20.10.7/plugin/v2/plugin.go#L195))
+if the `args` value is empty. Use e.g. `args="-v"` as a workaround.
+
+`config=/host/dir` sets alternative host location for the config directory.
+Plugin will look for `rclone.conf` here. It's not an error if the config
+file is not present but the directory must exist. Please note that plugin
+can periodically rewrite the config file, for example when it renews
+storage access tokens. Keep this in mind and try to avoid races between
+the plugin and other instances of rclone on the host that might try to
+change the config simultaneously resulting in corrupted `rclone.conf`.
+You can also put stuff like private key files for SFTP remotes in this
+directory. Just note that it's bind-mounted inside the plugin container
+at the predefined path `/data/config`. For example, if your key file is
+named `sftp-box1.key` on the host, the corresponding volume config option
+should read `-o sftp-key-file=/data/config/sftp-box1.key`.
+
+`cache=/host/dir` sets alternative host location for the _cache_ directory.
+The plugin will keep VFS caches here. Also it will create and maintain
+the `docker-plugin.state` file in this directory. When the plugin is
+restarted or reinstalled, it will look in this file to recreate any volumes
+that existed previously. However, they will not be re-mounted into
+consuming containers after restart. Usually this is not a problem as
+the docker daemon normally will restart affected user containers after
+failures, daemon restarts or host reboots.
+
+`RCLONE_VERBOSE` sets plugin verbosity from `0` (errors only, by default)
+to `2` (debugging). Verbosity can be also tweaked via `args="-v [-v] ..."`.
+Since arguments are more generic, you will rarely need this setting.
+The plugin output by default feeds the docker daemon log on local host.
+Log entries are reflected as _errors_ in the docker log but retain their
+actual level assigned by rclone in the encapsulated message string.
+
+`HTTP_PROXY`, `HTTPS_PROXY`, `NO_PROXY` customize the plugin proxy settings.
+
+You can set custom plugin options right when you install it, _in one go_:
+```
+docker plugin remove rclone
+docker plugin install rclone/docker-volume-rclone:amd64 \
+ --alias rclone --grant-all-permissions \
+ args="-v --allow-other" config=/etc/rclone
+docker plugin inspect rclone
+```
+
+## Healthchecks
+
+The docker plugin volume protocol doesn't provide a way for plugins
+to inform the docker daemon that a volume is (un-)available.
+As a workaround you can setup a healthcheck to verify that the mount
+is responding, for example:
+```
+services:
+ my_service:
+ image: my_image
+ healthcheck:
+ test: ls /path/to/rclone/mount || exit 1
+ interval: 1m
+ timeout: 15s
+ retries: 3
+ start_period: 15s
+```
+
+## Running Plugin under Systemd
+
+In most cases you should prefer managed mode. Moreover, MacOS and Windows
+do not support native Docker plugins. Please use managed mode on these
+systems. Proceed further only if you are on Linux.
+
+First, [install rclone](https://rclone.org/install/).
+You can just run it (type `rclone serve docker` and hit enter) for the test.
+
+Install _FUSE_:
+```
+sudo apt-get -y install fuse
+```
+
+Download two systemd configuration files:
+[docker-volume-rclone.service](https://raw.githubusercontent.com/rclone/rclone/master/contrib/docker-plugin/systemd/docker-volume-rclone.service)
+and [docker-volume-rclone.socket](https://raw.githubusercontent.com/rclone/rclone/master/contrib/docker-plugin/systemd/docker-volume-rclone.socket).
+
+Put them to the `/etc/systemd/system/` directory:
+```
+cp docker-volume-plugin.service /etc/systemd/system/
+cp docker-volume-plugin.socket /etc/systemd/system/
+```
+
+Please note that all commands in this section must be run as _root_ but
+we omit `sudo` prefix for brevity.
+Now create directories required by the service:
+```
+mkdir -p /var/lib/docker-volumes/rclone
+mkdir -p /var/lib/docker-plugins/rclone/config
+mkdir -p /var/lib/docker-plugins/rclone/cache
+```
+
+Run the docker plugin service in the socket activated mode:
+```
+systemctl daemon-reload
+systemctl start docker-volume-rclone.service
+systemctl enable docker-volume-rclone.socket
+systemctl start docker-volume-rclone.socket
+systemctl restart docker
+```
+
+Or run the service directly:
+- run `systemctl daemon-reload` to let systemd pick up new config
+- run `systemctl enable docker-volume-rclone.service` to make the new
+ service start automatically when you power on your machine.
+- run `systemctl start docker-volume-rclone.service`
+ to start the service now.
+- run `systemctl restart docker` to restart docker daemon and let it
+ detect the new plugin socket. Note that this step is not needed in
+ managed mode where docker knows about plugin state changes.
+
+The two methods are equivalent from the user perspective, but I personally
+prefer socket activation.
+
+## Troubleshooting
+
+You can [see managed plugin settings](https://docs.docker.com/engine/extend/#debugging-plugins)
+with
+```
+docker plugin list
+docker plugin inspect rclone
+```
+Note that docker (including latest 20.10.7) will not show actual values
+of `args`, just the defaults.
+
+Use `journalctl --unit docker` to see managed plugin output as part of
+the docker daemon log. Note that docker reflects plugin lines as _errors_
+but their actual level can be seen from encapsulated message string.
+
+You will usually install the latest version of managed plugin for your platform.
+Use the following commands to print the actual installed version:
+```
+PLUGID=$(docker plugin list --no-trunc | awk '/rclone/{print$1}')
+sudo runc --root /run/docker/runtime-runc/plugins.moby exec $PLUGID rclone version
+```
+
+You can even use `runc` to run shell inside the plugin container:
+```
+sudo runc --root /run/docker/runtime-runc/plugins.moby exec --tty $PLUGID bash
+```
+
+Also you can use curl to check the plugin socket connectivity:
+```
+docker plugin list --no-trunc
+PLUGID=123abc...
+sudo curl -H Content-Type:application/json -XPOST -d {} --unix-socket /run/docker/plugins/$PLUGID/rclone.sock http://localhost/Plugin.Activate
+```
+though this is rarely needed.
+
+## Caveats
+
+Finally I'd like to mention a _caveat with updating volume settings_.
+Docker CLI does not have a dedicated command like `docker volume update`.
+It may be tempting to invoke `docker volume create` with updated options
+on existing volume, but there is a gotcha. The command will do nothing,
+it won't even return an error. I hope that docker maintainers will fix
+this some day. In the meantime be aware that you must remove your volume
+before recreating it with new settings:
+```
+docker volume remove my_vol
+docker volume create my_vol -d rclone -o opt1=new_val1 ...
+```
+
+and verify that settings did update:
+```
+docker volume list
+docker volume inspect my_vol
+```
+
+If docker refuses to remove the volume, you should find containers
+or swarm services that use it and stop them first.
+
+## Getting started {#getting-started}
+
+- [Install rclone](https://rclone.org/install/) and setup your remotes.
+- Bisync will create its working directory
+ at `~/.cache/rclone/bisync` on Linux
+ or `C:\Users\MyLogin\AppData\Local\rclone\bisync` on Windows.
+ Make sure that this location is writable.
+- Run bisync with the `--resync` flag, specifying the paths
+ to the local and remote sync directory roots.
+- For successive sync runs, leave off the `--resync` flag.
+- Consider using a [filters file](#filtering) for excluding
+ unnecessary files and directories from the sync.
+- Consider setting up the [--check-access](#check-access) feature
+ for safety.
+- On Linux, consider setting up a [crontab entry](#cron). bisync can
+ safely run in concurrent cron jobs thanks to lock files it maintains.
+
+Here is a typical run log (with timestamps removed for clarity):
+
+```
+rclone bisync /testdir/path1/ /testdir/path2/ --verbose
+INFO : Synching Path1 "/testdir/path1/" with Path2 "/testdir/path2/"
+INFO : Path1 checking for diffs
+INFO : - Path1 File is new - file11.txt
+INFO : - Path1 File is newer - file2.txt
+INFO : - Path1 File is newer - file5.txt
+INFO : - Path1 File is newer - file7.txt
+INFO : - Path1 File was deleted - file4.txt
+INFO : - Path1 File was deleted - file6.txt
+INFO : - Path1 File was deleted - file8.txt
+INFO : Path1: 7 changes: 1 new, 3 newer, 0 older, 3 deleted
+INFO : Path2 checking for diffs
+INFO : - Path2 File is new - file10.txt
+INFO : - Path2 File is newer - file1.txt
+INFO : - Path2 File is newer - file5.txt
+INFO : - Path2 File is newer - file6.txt
+INFO : - Path2 File was deleted - file3.txt
+INFO : - Path2 File was deleted - file7.txt
+INFO : - Path2 File was deleted - file8.txt
+INFO : Path2: 7 changes: 1 new, 3 newer, 0 older, 3 deleted
+INFO : Applying changes
+INFO : - Path1 Queue copy to Path2 - /testdir/path2/file11.txt
+INFO : - Path1 Queue copy to Path2 - /testdir/path2/file2.txt
+INFO : - Path2 Queue delete - /testdir/path2/file4.txt
+NOTICE: - WARNING New or changed in both paths - file5.txt
+NOTICE: - Path1 Renaming Path1 copy - /testdir/path1/file5.txt..path1
+NOTICE: - Path1 Queue copy to Path2 - /testdir/path2/file5.txt..path1
+NOTICE: - Path2 Renaming Path2 copy - /testdir/path2/file5.txt..path2
+NOTICE: - Path2 Queue copy to Path1 - /testdir/path1/file5.txt..path2
+INFO : - Path2 Queue copy to Path1 - /testdir/path1/file6.txt
+INFO : - Path1 Queue copy to Path2 - /testdir/path2/file7.txt
+INFO : - Path2 Queue copy to Path1 - /testdir/path1/file1.txt
+INFO : - Path2 Queue copy to Path1 - /testdir/path1/file10.txt
+INFO : - Path1 Queue delete - /testdir/path1/file3.txt
+INFO : - Path2 Do queued copies to - Path1
+INFO : - Path1 Do queued copies to - Path2
+INFO : - Do queued deletes on - Path1
+INFO : - Do queued deletes on - Path2
+INFO : Updating listings
+INFO : Validating listings for Path1 "/testdir/path1/" vs Path2 "/testdir/path2/"
+INFO : Bisync successful
+```
+
+## Command line syntax
+
+```
+$ rclone bisync --help
+Usage:
+ rclone bisync remote1:path1 remote2:path2 [flags]
+
+Positional arguments:
+ Path1, Path2 Local path, or remote storage with ':' plus optional path.
+ Type 'rclone listremotes' for list of configured remotes.
+
+Optional Flags:
+ --check-access Ensure expected `RCLONE_TEST` files are found on
+ both Path1 and Path2 filesystems, else abort.
+ --check-filename FILENAME Filename for `--check-access` (default: `RCLONE_TEST`)
+ --check-sync CHOICE Controls comparison of final listings:
+ `true | false | only` (default: true)
+ If set to `only`, bisync will only compare listings
+ from the last run but skip actual sync.
+ --filters-file PATH Read filtering patterns from a file
+ --max-delete PERCENT Safety check on maximum percentage of deleted files allowed.
+ If exceeded, the bisync run will abort. (default: 50%)
+ --force Bypass `--max-delete` safety check and run the sync.
+ Consider using with `--verbose`
+ --create-empty-src-dirs Sync creation and deletion of empty directories.
+ (Not compatible with --remove-empty-dirs)
+ --remove-empty-dirs Remove empty directories at the final cleanup step.
+ -1, --resync Performs the resync run.
+ Warning: Path1 files may overwrite Path2 versions.
+ Consider using `--verbose` or `--dry-run` first.
+ --ignore-listing-checksum Do not use checksums for listings
+ (add --ignore-checksum to additionally skip post-copy checksum checks)
+ --resilient Allow future runs to retry after certain less-serious errors,
+ instead of requiring --resync. Use at your own risk!
+ --localtime Use local time in listings (default: UTC)
+ --no-cleanup Retain working files (useful for troubleshooting and testing).
+ --workdir PATH Use custom working directory (useful for testing).
+ (default: `~/.cache/rclone/bisync`)
+ -n, --dry-run Go through the motions - No files are copied/deleted.
+ -v, --verbose Increases logging verbosity.
+ May be specified more than once for more details.
+ -h, --help help for bisync
+```
+
+Arbitrary rclone flags may be specified on the
+[bisync command line](https://rclone.org/commands/rclone_bisync/), for example
+`rclone bisync ./testdir/path1/ gdrive:testdir/path2/ --drive-skip-gdocs -v -v --timeout 10s`
+Note that interactions of various rclone flags with bisync process flow
+has not been fully tested yet.
+
+### Paths
+
+Path1 and Path2 arguments may be references to any mix of local directory
+paths (absolute or relative), UNC paths (`//server/share/path`),
+Windows drive paths (with a drive letter and `:`) or configured
+[remotes](https://rclone.org/docs/#syntax-of-remote-paths) with optional subdirectory paths.
+Cloud references are distinguished by having a `:` in the argument
+(see [Windows support](#windows) below).
+
+Path1 and Path2 are treated equally, in that neither has priority for
+file changes (except during [`--resync`](#resync)), and access efficiency does not change whether a remote
+is on Path1 or Path2.
+
+The listings in bisync working directory (default: `~/.cache/rclone/bisync`)
+are named based on the Path1 and Path2 arguments so that separate syncs
+to individual directories within the tree may be set up, e.g.:
+`path_to_local_tree..dropbox_subdir.lst`.
+
+Any empty directories after the sync on both the Path1 and Path2
+filesystems are not deleted by default, unless `--create-empty-src-dirs` is specified.
+If the `--remove-empty-dirs` flag is specified, then both paths will have ALL empty directories purged
+as the last step in the process.
+
+## Command-line flags
+
+#### --resync
+
+This will effectively make both Path1 and Path2 filesystems contain a
+matching superset of all files. Path2 files that do not exist in Path1 will
+be copied to Path1, and the process will then copy the Path1 tree to Path2.
+
+The `--resync` sequence is roughly equivalent to:
+```
+rclone copy Path2 Path1 --ignore-existing
+rclone copy Path1 Path2
+```
+Or, if using `--create-empty-src-dirs`:
+```
+rclone copy Path2 Path1 --ignore-existing
+rclone copy Path1 Path2 --create-empty-src-dirs
+rclone copy Path2 Path1 --create-empty-src-dirs
+```
+
+The base directories on both Path1 and Path2 filesystems must exist
+or bisync will fail. This is required for safety - that bisync can verify
+that both paths are valid.
+
+When using `--resync`, a newer version of a file on the Path2 filesystem
+will be overwritten by the Path1 filesystem version.
+(Note that this is [NOT entirely symmetrical](https://github.com/rclone/rclone/issues/5681#issuecomment-938761815).)
+Carefully evaluate deltas using [--dry-run](https://rclone.org/flags/#non-backend-flags).
+
+[//]: # (I reverted a recent change in the above paragraph, as it was incorrect.
+https://github.com/rclone/rclone/commit/dd72aff98a46c6e20848ac7ae5f7b19d45802493 )
+
+For a resync run, one of the paths may be empty (no files in the path tree).
+The resync run should result in files on both paths, else a normal non-resync
+run will fail.
+
+For a non-resync run, either path being empty (no files in the tree) fails with
+`Empty current PathN listing. Cannot sync to an empty directory: X.pathN.lst`
+This is a safety check that an unexpected empty path does not result in
+deleting **everything** in the other path.
+
+#### --check-access
+
+Access check files are an additional safety measure against data loss.
+bisync will ensure it can find matching `RCLONE_TEST` files in the same places
+in the Path1 and Path2 filesystems.
+`RCLONE_TEST` files are not generated automatically.
+For `--check-access` to succeed, you must first either:
+**A)** Place one or more `RCLONE_TEST` files in both systems, or
+**B)** Set `--check-filename` to a filename already in use in various locations
+throughout your sync'd fileset. Recommended methods for **A)** include:
+* `rclone touch Path1/RCLONE_TEST` (create a new file)
+* `rclone copyto Path1/RCLONE_TEST Path2/RCLONE_TEST` (copy an existing file)
+* `rclone copy Path1/RCLONE_TEST Path2/RCLONE_TEST --include "RCLONE_TEST"` (copy multiple files at once, recursively)
+* create the files manually (outside of rclone)
+* run `bisync` once *without* `--check-access` to set matching files on both filesystems
+will also work, but is not preferred, due to potential for user error
+(you are temporarily disabling the safety feature).
+
+Note that `--check-access` is still enforced on `--resync`, so `bisync --resync --check-access`
+will not work as a method of initially setting the files (this is to ensure that bisync can't
+[inadvertently circumvent its own safety switch](https://forum.rclone.org/t/bisync-bugs-and-feature-requests/37636#:~:text=3.%20%2D%2Dcheck%2Daccess%20doesn%27t%20always%20fail%20when%20it%20should).)
+
+Time stamps and file contents for `RCLONE_TEST` files are not important, just the names and locations.
+If you have symbolic links in your sync tree it is recommended to place
+`RCLONE_TEST` files in the linked-to directory tree to protect against
+bisync assuming a bunch of deleted files if the linked-to tree should not be
+accessible.
+See also the [--check-filename](--check-filename) flag.
+
+#### --check-filename
+
+Name of the file(s) used in access health validation.
+The default `--check-filename` is `RCLONE_TEST`.
+One or more files having this filename must exist, synchronized between your
+source and destination filesets, in order for `--check-access` to succeed.
+See [--check-access](#check-access) for additional details.
+
+#### --max-delete
+
+As a safety check, if greater than the `--max-delete` percent of files were
+deleted on either the Path1 or Path2 filesystem, then bisync will abort with
+a warning message, without making any changes.
+The default `--max-delete` is `50%`.
+One way to trigger this limit is to rename a directory that contains more
+than half of your files. This will appear to bisync as a bunch of deleted
+files and a bunch of new files.
+This safety check is intended to block bisync from deleting all of the
+files on both filesystems due to a temporary network access issue, or if
+the user had inadvertently deleted the files on one side or the other.
+To force the sync, either set a different delete percentage limit,
+e.g. `--max-delete 75` (allows up to 75% deletion), or use `--force`
+to bypass the check.
+
+Also see the [all files changed](#all-files-changed) check.
+
+#### --filters-file {#filters-file}
+
+By using rclone filter features you can exclude file types or directory
+sub-trees from the sync.
+See the [bisync filters](#filtering) section and generic
+[--filter-from](https://rclone.org/filtering/#filter-from-read-filtering-patterns-from-a-file)
+documentation.
+An [example filters file](#example-filters-file) contains filters for
+non-allowed files for synching with Dropbox.
+
+If you make changes to your filters file then bisync requires a run
+with `--resync`. This is a safety feature, which prevents existing files
+on the Path1 and/or Path2 side from seeming to disappear from view
+(since they are excluded in the new listings), which would fool bisync
+into seeing them as deleted (as compared to the prior run listings),
+and then bisync would proceed to delete them for real.
+
+To block this from happening, bisync calculates an MD5 hash of the filters file
+and stores the hash in a `.md5` file in the same place as your filters file.
+On the next run with `--filters-file` set, bisync re-calculates the MD5 hash
+of the current filters file and compares it to the hash stored in the `.md5` file.
+If they don't match, the run aborts with a critical error and thus forces you
+to do a `--resync`, likely avoiding a disaster.
+
+#### --check-sync
+
+Enabled by default, the check-sync function checks that all of the same
+files exist in both the Path1 and Path2 history listings. This _check-sync_
+integrity check is performed at the end of the sync run by default.
+Any untrapped failing copy/deletes between the two paths might result
+in differences between the two listings and in the untracked file content
+differences between the two paths. A resync run would correct the error.
+
+Note that the default-enabled integrity check locally executes a load of both
+the final Path1 and Path2 listings, and thus adds to the run time of a sync.
+Using `--check-sync=false` will disable it and may significantly reduce the
+sync run times for very large numbers of files.
+
+The check may be run manually with `--check-sync=only`. It runs only the
+integrity check and terminates without actually synching.
+
+See also: [Concurrent modifications](#concurrent-modifications)
+
+
+#### --ignore-listing-checksum
+
+By default, bisync will retrieve (or generate) checksums (for backends that support them)
+when creating the listings for both paths, and store the checksums in the listing files.
+`--ignore-listing-checksum` will disable this behavior, which may speed things up considerably,
+especially on backends (such as [local](https://rclone.org/local/)) where hashes must be computed on the fly instead of retrieved.
+Please note the following:
+
+* While checksums are (by default) generated and stored in the listing files,
+they are NOT currently used for determining diffs (deltas).
+It is anticipated that full checksum support will be added in a future version.
+* `--ignore-listing-checksum` is NOT the same as [`--ignore-checksum`](https://rclone.org/docs/#ignore-checksum),
+and you may wish to use one or the other, or both. In a nutshell:
+`--ignore-listing-checksum` controls whether checksums are considered when scanning for diffs,
+while `--ignore-checksum` controls whether checksums are considered during the copy/sync operations that follow,
+if there ARE diffs.
+* Unless `--ignore-listing-checksum` is passed, bisync currently computes hashes for one path
+*even when there's no common hash with the other path*
+(for example, a [crypt](https://rclone.org/crypt/#modification-times-and-hashes) remote.)
+* If both paths support checksums and have a common hash,
+AND `--ignore-listing-checksum` was not specified when creating the listings,
+`--check-sync=only` can be used to compare Path1 vs. Path2 checksums (as of the time the previous listings were created.)
+However, `--check-sync=only` will NOT include checksums if the previous listings
+were generated on a run using `--ignore-listing-checksum`. For a more robust integrity check of the current state,
+consider using [`check`](commands/rclone_check/)
+(or [`cryptcheck`](https://rclone.org/commands/rclone_cryptcheck/), if at least one path is a `crypt` remote.)
+
+#### --resilient
+
+***Caution: this is an experimental feature. Use at your own risk!***
+
+By default, most errors or interruptions will cause bisync to abort and
+require [`--resync`](#resync) to recover. This is a safety feature,
+to prevent bisync from running again until a user checks things out.
+However, in some cases, bisync can go too far and enforce a lockout when one isn't actually necessary,
+like for certain less-serious errors that might resolve themselves on the next run.
+When `--resilient` is specified, bisync tries its best to recover and self-correct,
+and only requires `--resync` as a last resort when a human's involvement is absolutely necessary.
+The intended use case is for running bisync as a background process (such as via scheduled [cron](#cron)).
+
+When using `--resilient` mode, bisync will still report the error and abort,
+however it will not lock out future runs -- allowing the possibility of retrying at the next normally scheduled time,
+without requiring a `--resync` first. Examples of such retryable errors include
+access test failures, missing listing files, and filter change detections.
+These safety features will still prevent the *current* run from proceeding --
+the difference is that if conditions have improved by the time of the *next* run,
+that next run will be allowed to proceed.
+Certain more serious errors will still enforce a `--resync` lockout, even in `--resilient` mode, to prevent data loss.
+
+Behavior of `--resilient` may change in a future version.
+
+## Operation
+
+### Runtime flow details
+
+bisync retains the listings of the `Path1` and `Path2` filesystems
+from the prior run.
+On each successive run it will:
+
+- list files on `path1` and `path2`, and check for changes on each side.
+ Changes include `New`, `Newer`, `Older`, and `Deleted` files.
+- Propagate changes on `path1` to `path2`, and vice-versa.
+
+### Safety measures
+
+- Lock file prevents multiple simultaneous runs when taking a while.
+ This can be particularly useful if bisync is run by cron scheduler.
+- Handle change conflicts non-destructively by creating
+ `..path1` and `..path2` file versions.
+- File system access health check using `RCLONE_TEST` files
+ (see the `--check-access` flag).
+- Abort on excessive deletes - protects against a failed listing
+ being interpreted as all the files were deleted.
+ See the `--max-delete` and `--force` flags.
+- If something evil happens, bisync goes into a safe state to block
+ damage by later runs. (See [Error Handling](#error-handling))
+
+### Normal sync checks
+
+ Type | Description | Result | Implementation
+--------------|-----------------------------------------------|--------------------------|-----------------------------
+Path2 new | File is new on Path2, does not exist on Path1 | Path2 version survives | `rclone copy` Path2 to Path1
+Path2 newer | File is newer on Path2, unchanged on Path1 | Path2 version survives | `rclone copy` Path2 to Path1
+Path2 deleted | File is deleted on Path2, unchanged on Path1 | File is deleted | `rclone delete` Path1
+Path1 new | File is new on Path1, does not exist on Path2 | Path1 version survives | `rclone copy` Path1 to Path2
+Path1 newer | File is newer on Path1, unchanged on Path2 | Path1 version survives | `rclone copy` Path1 to Path2
+Path1 older | File is older on Path1, unchanged on Path2 | _Path1 version survives_ | `rclone copy` Path1 to Path2
+Path2 older | File is older on Path2, unchanged on Path1 | _Path2 version survives_ | `rclone copy` Path2 to Path1
+Path1 deleted | File no longer exists on Path1 | File is deleted | `rclone delete` Path2
+
+### Unusual sync checks
+
+ Type | Description | Result | Implementation
+--------------------------------|---------------------------------------|------------------------------------|-----------------------
+Path1 new/changed AND Path2 new/changed AND Path1 == Path2 | File is new/changed on Path1 AND new/changed on Path2 AND Path1 version is currently identical to Path2 | No change | None
+Path1 new AND Path2 new | File is new on Path1 AND new on Path2 (and Path1 version is NOT identical to Path2) | Files renamed to _Path1 and _Path2 | `rclone copy` _Path2 file to Path1, `rclone copy` _Path1 file to Path2
+Path2 newer AND Path1 changed | File is newer on Path2 AND also changed (newer/older/size) on Path1 (and Path1 version is NOT identical to Path2) | Files renamed to _Path1 and _Path2 | `rclone copy` _Path2 file to Path1, `rclone copy` _Path1 file to Path2
+Path2 newer AND Path1 deleted | File is newer on Path2 AND also deleted on Path1 | Path2 version survives | `rclone copy` Path2 to Path1
+Path2 deleted AND Path1 changed | File is deleted on Path2 AND changed (newer/older/size) on Path1 | Path1 version survives |`rclone copy` Path1 to Path2
+Path1 deleted AND Path2 changed | File is deleted on Path1 AND changed (newer/older/size) on Path2 | Path2 version survives | `rclone copy` Path2 to Path1
+
+As of `rclone v1.64`, bisync is now better at detecting *false positive* sync conflicts,
+which would previously have resulted in unnecessary renames and duplicates.
+Now, when bisync comes to a file that it wants to rename (because it is new/changed on both sides),
+it first checks whether the Path1 and Path2 versions are currently *identical*
+(using the same underlying function as [`check`](commands/rclone_check/).)
+If bisync concludes that the files are identical, it will skip them and move on.
+Otherwise, it will create renamed `..Path1` and `..Path2` duplicates, as before.
+This behavior also [improves the experience of renaming directories](https://forum.rclone.org/t/bisync-bugs-and-feature-requests/37636#:~:text=Renamed%20directories),
+as a `--resync` is no longer required, so long as the same change has been made on both sides.
+
+### All files changed check {#all-files-changed}
+
+If _all_ prior existing files on either of the filesystems have changed
+(e.g. timestamps have changed due to changing the system's timezone)
+then bisync will abort without making any changes.
+Any new files are not considered for this check. You could use `--force`
+to force the sync (whichever side has the changed timestamp files wins).
+Alternately, a `--resync` may be used (Path1 versions will be pushed
+to Path2). Consider the situation carefully and perhaps use `--dry-run`
+before you commit to the changes.
+
+### Modification times
+
+Bisync relies on file timestamps to identify changed files and will
+_refuse_ to operate if backend lacks the modification time support.
+
+If you or your application should change the content of a file
+without changing the modification time then bisync will _not_
+notice the change, and thus will not copy it to the other side.
+
+Note that on some cloud storage systems it is not possible to have file
+timestamps that match _precisely_ between the local and other filesystems.
+
+Bisync's approach to this problem is by tracking the changes on each side
+_separately_ over time with a local database of files in that side then
+applying the resulting changes on the other side.
+
+### Error handling {#error-handling}
+
+Certain bisync critical errors, such as file copy/move failing, will result in
+a bisync lockout of following runs. The lockout is asserted because the sync
+status and history of the Path1 and Path2 filesystems cannot be trusted,
+so it is safer to block any further changes until someone checks things out.
+The recovery is to do a `--resync` again.
+
+It is recommended to use `--resync --dry-run --verbose` initially and
+_carefully_ review what changes will be made before running the `--resync`
+without `--dry-run`.
+
+Most of these events come up due to an error status from an internal call.
+On such a critical error the `{...}.path1.lst` and `{...}.path2.lst`
+listing files are renamed to extension `.lst-err`, which blocks any future
+bisync runs (since the normal `.lst` files are not found).
+Bisync keeps them under `bisync` subdirectory of the rclone cache directory,
+typically at `${HOME}/.cache/rclone/bisync/` on Linux.
+
+Some errors are considered temporary and re-running the bisync is not blocked.
+The _critical return_ blocks further bisync runs.
+
+See also: [`--resilient`](#resilient)
+
+### Lock file
+
+When bisync is running, a lock file is created in the bisync working directory,
+typically at `~/.cache/rclone/bisync/PATH1..PATH2.lck` on Linux.
+If bisync should crash or hang, the lock file will remain in place and block
+any further runs of bisync _for the same paths_.
+Delete the lock file as part of debugging the situation.
+The lock file effectively blocks follow-on (e.g., scheduled by _cron_) runs
+when the prior invocation is taking a long time.
+The lock file contains _PID_ of the blocking process, which may help in debug.
+
+**Note**
+that while concurrent bisync runs are allowed, _be very cautious_
+that there is no overlap in the trees being synched between concurrent runs,
+lest there be replicated files, deleted files and general mayhem.
+
+### Return codes
+
+`rclone bisync` returns the following codes to calling program:
+- `0` on a successful run,
+- `1` for a non-critical failing run (a rerun may be successful),
+- `2` for a critically aborted run (requires a `--resync` to recover).
+
+## Limitations
+
+### Supported backends
+
+Bisync is considered _BETA_ and has been tested with the following backends:
+- Local filesystem
+- Google Drive
+- Dropbox
+- OneDrive
+- S3
+- SFTP
+- Yandex Disk
+
+It has not been fully tested with other services yet.
+If it works, or sorta works, please let us know and we'll update the list.
+Run the test suite to check for proper operation as described below.
+
+First release of `rclone bisync` requires that underlying backend supports
+the modification time feature and will refuse to run otherwise.
+This limitation will be lifted in a future `rclone bisync` release.
+
+### Concurrent modifications
+
+When using **Local, FTP or SFTP** remotes rclone does not create _temporary_
+files at the destination when copying, and thus if the connection is lost
+the created file may be corrupt, which will likely propagate back to the
+original path on the next sync, resulting in data loss.
+This will be solved in a future release, there is no workaround at the moment.
+
+Files that **change during** a bisync run may result in data loss.
+This has been seen in a highly dynamic environment, where the filesystem
+is getting hammered by running processes during the sync.
+The currently recommended solution is to sync at quiet times or [filter out](#filtering)
+unnecessary directories and files.
+
+As an [alternative approach](https://forum.rclone.org/t/bisync-bugs-and-feature-requests/37636#:~:text=scans%2C%20to%20avoid-,errors%20if%20files%20changed%20during%20sync,-Given%20the%20number),
+consider using `--check-sync=false` (and possibly `--resilient`) to make bisync more forgiving
+of filesystems that change during the sync.
+Be advised that this may cause bisync to miss events that occur during a bisync run,
+so it is a good idea to supplement this with a periodic independent integrity check,
+and corrective sync if diffs are found. For example, a possible sequence could look like this:
+
+1. Normally scheduled bisync run:
+
+```
+rclone bisync Path1 Path2 -MPc --check-access --max-delete 10 --filters-file /path/to/filters.txt -v --check-sync=false --no-cleanup --ignore-listing-checksum --disable ListR --checkers=16 --drive-pacer-min-sleep=10ms --create-empty-src-dirs --resilient
+```
+
+2. Periodic independent integrity check (perhaps scheduled nightly or weekly):
+
+```
+rclone check -MvPc Path1 Path2 --filter-from /path/to/filters.txt
+```
+
+3. If diffs are found, you have some choices to correct them.
+If one side is more up-to-date and you want to make the other side match it, you could run:
+
+```
+rclone sync Path1 Path2 --filter-from /path/to/filters.txt --create-empty-src-dirs -MPc -v
+```
+(or switch Path1 and Path2 to make Path2 the source-of-truth)
+
+Or, if neither side is totally up-to-date, you could run a `--resync` to bring them back into agreement
+(but remember that this could cause deleted files to re-appear.)
+
+*Note also that `rclone check` does not currently include empty directories,
+so if you want to know if any empty directories are out of sync,
+consider alternatively running the above `rclone sync` command with `--dry-run` added.
+
+### Empty directories
+
+By default, new/deleted empty directories on one path are _not_ propagated to the other side.
+This is because bisync (and rclone) natively works on files, not directories.
+However, this can be changed with the `--create-empty-src-dirs` flag, which works in
+much the same way as in [`sync`](https://rclone.org/commands/rclone_sync/) and [`copy`](https://rclone.org/commands/rclone_copy/).
+When used, empty directories created or deleted on one side will also be created or deleted on the other side.
+The following should be noted:
+* `--create-empty-src-dirs` is not compatible with `--remove-empty-dirs`. Use only one or the other (or neither).
+* It is not recommended to switch back and forth between `--create-empty-src-dirs`
+and the default (no `--create-empty-src-dirs`) without running `--resync`.
+This is because it may appear as though all directories (not just the empty ones) were created/deleted,
+when actually you've just toggled between making them visible/invisible to bisync.
+It looks scarier than it is, but it's still probably best to stick to one or the other,
+and use `--resync` when you need to switch.
+
+### Renamed directories
+
+Renaming a folder on the Path1 side results in deleting all files on
+the Path2 side and then copying all files again from Path1 to Path2.
+Bisync sees this as all files in the old directory name as deleted and all
+files in the new directory name as new.
+Currently, the most effective and efficient method of renaming a directory
+is to rename it to the same name on both sides. (As of `rclone v1.64`,
+a `--resync` is no longer required after doing so, as bisync will automatically
+detect that Path1 and Path2 are in agreement.)
+
+### `--fast-list` used by default
+
+Unlike most other rclone commands, bisync uses [`--fast-list`](https://rclone.org/docs/#fast-list) by default,
+for backends that support it. In many cases this is desirable, however,
+there are some scenarios in which bisync could be faster *without* `--fast-list`,
+and there is also a [known issue concerning Google Drive users with many empty directories](https://github.com/rclone/rclone/commit/cbf3d4356135814921382dd3285d859d15d0aa77).
+For now, the recommended way to avoid using `--fast-list` is to add `--disable ListR`
+to all bisync commands. The default behavior may change in a future version.
+
+### Overridden Configs
+
+When rclone detects an overridden config, it adds a suffix like `{ABCDE}` on the fly
+to the internal name of the remote. Bisync follows suit by including this suffix in its listing filenames.
+However, this suffix does not necessarily persist from run to run, especially if different flags are provided.
+So if next time the suffix assigned is `{FGHIJ}`, bisync will get confused,
+because it's looking for a listing file with `{FGHIJ}`, when the file it wants has `{ABCDE}`.
+As a result, it throws
+`Bisync critical error: cannot find prior Path1 or Path2 listings, likely due to critical error on prior run`
+and refuses to run again until the user runs a `--resync` (unless using `--resilient`).
+The best workaround at the moment is to set any backend-specific flags in the [config file](https://rclone.org/commands/rclone_config/)
+instead of specifying them with command flags. (You can still override them as needed for other rclone commands.)
+
+### Case sensitivity
+
+Synching with **case-insensitive** filesystems, such as Windows or `Box`,
+can result in file name conflicts. This will be fixed in a future release.
+The near-term workaround is to make sure that files on both sides
+don't have spelling case differences (`Smile.jpg` vs. `smile.jpg`).
+
+## Windows support {#windows}
+
+Bisync has been tested on Windows 8.1, Windows 10 Pro 64-bit and on Windows
+GitHub runners.
+
+Drive letters are allowed, including drive letters mapped to network drives
+(`rclone bisync J:\localsync GDrive:`).
+If a drive letter is omitted, the shell current drive is the default.
+Drive letters are a single character follows by `:`, so cloud names
+must be more than one character long.
+
+Absolute paths (with or without a drive letter), and relative paths
+(with or without a drive letter) are supported.
+
+Working directory is created at `C:\Users\MyLogin\AppData\Local\rclone\bisync`.
+
+Note that bisync output may show a mix of forward `/` and back `\` slashes.
+
+Be careful of case independent directory and file naming on Windows
+vs. case dependent Linux
+
+## Filtering {#filtering}
+
+See [filtering documentation](https://rclone.org/filtering/)
+for how filter rules are written and interpreted.
+
+Bisync's [`--filters-file`](#filters-file) flag slightly extends the rclone's
+[--filter-from](https://rclone.org/filtering/#filter-from-read-filtering-patterns-from-a-file)
+filtering mechanism.
+For a given bisync run you may provide _only one_ `--filters-file`.
+The `--include*`, `--exclude*`, and `--filter` flags are also supported.
+
+### How to filter directories
+
+Filtering portions of the directory tree is a critical feature for synching.
+
+Examples of directory trees (always beneath the Path1/Path2 root level)
+you may want to exclude from your sync:
+- Directory trees containing only software build intermediate files.
+- Directory trees containing application temporary files and data
+ such as the Windows `C:\Users\MyLogin\AppData\` tree.
+- Directory trees containing files that are large, less important,
+ or are getting thrashed continuously by ongoing processes.
+
+On the other hand, there may be only select directories that you
+actually want to sync, and exclude all others. See the
+[Example include-style filters for Windows user directories](#include-filters)
+below.
+
+### Filters file writing guidelines
+
+1. Begin with excluding directory trees:
+ - e.g. `- /AppData/`
+ - `**` on the end is not necessary. Once a given directory level
+ is excluded then everything beneath it won't be looked at by rclone.
+ - Exclude such directories that are unneeded, are big, dynamically thrashed,
+ or where there may be access permission issues.
+ - Excluding such dirs first will make rclone operations (much) faster.
+ - Specific files may also be excluded, as with the Dropbox exclusions
+ example below.
+2. Decide if it's easier (or cleaner) to:
+ - Include select directories and therefore _exclude everything else_ -- or --
+ - Exclude select directories and therefore _include everything else_
+3. Include select directories:
+ - Add lines like: `+ /Documents/PersonalFiles/**` to select which
+ directories to include in the sync.
+ - `**` on the end specifies to include the full depth of the specified tree.
+ - With Include-style filters, files at the Path1/Path2 root are not included.
+ They may be included with `+ /*`.
+ - Place RCLONE_TEST files within these included directory trees.
+ They will only be looked for in these directory trees.
+ - Finish by excluding everything else by adding `- **` at the end
+ of the filters file.
+ - Disregard step 4.
+4. Exclude select directories:
+ - Add more lines like in step 1.
+ For example: `-/Desktop/tempfiles/`, or `- /testdir/`.
+ Again, a `**` on the end is not necessary.
+ - Do _not_ add a `- **` in the file. Without this line, everything
+ will be included that has not been explicitly excluded.
+ - Disregard step 3.
+
+A few rules for the syntax of a filter file expanding on
+[filtering documentation](https://rclone.org/filtering/):
+
+- Lines may start with spaces and tabs - rclone strips leading whitespace.
+- If the first non-whitespace character is a `#` then the line is a comment
+ and will be ignored.
+- Blank lines are ignored.
+- The first non-whitespace character on a filter line must be a `+` or `-`.
+- Exactly 1 space is allowed between the `+/-` and the path term.
+- Only forward slashes (`/`) are used in path terms, even on Windows.
+- The rest of the line is taken as the path term.
+ Trailing whitespace is taken literally, and probably is an error.
+
+### Example include-style filters for Windows user directories {#include-filters}
+
+This Windows _include-style_ example is based on the sync root (Path1)
+set to `C:\Users\MyLogin`. The strategy is to select specific directories
+to be synched with a network drive (Path2).
+
+- `- /AppData/` excludes an entire tree of Windows stored stuff
+ that need not be synched.
+ In my case, AppData has >11 GB of stuff I don't care about, and there are
+ some subdirectories beneath AppData that are not accessible to my
+ user login, resulting in bisync critical aborts.
+- Windows creates cache files starting with both upper and
+ lowercase `NTUSER` at `C:\Users\MyLogin`. These files may be dynamic,
+ locked, and are generally _don't care_.
+- There are just a few directories with _my_ data that I do want synched,
+ in the form of `+ /`. By selecting only the directory trees I
+ want to avoid the dozen plus directories that various apps make
+ at `C:\Users\MyLogin\Documents`.
+- Include files in the root of the sync point, `C:\Users\MyLogin`,
+ by adding the `+ /*` line.
+- This is an Include-style filters file, therefore it ends with `- **`
+ which excludes everything not explicitly included.
+
+```
+- /AppData/
+- NTUSER*
+- ntuser*
++ /Documents/Family/**
++ /Documents/Sketchup/**
++ /Documents/Microcapture_Photo/**
++ /Documents/Microcapture_Video/**
++ /Desktop/**
++ /Pictures/**
++ /*
+- **
+```
+
+Note also that Windows implements several "library" links such as
+`C:\Users\MyLogin\My Documents\My Music` pointing to `C:\Users\MyLogin\Music`.
+rclone sees these as links, so you must add `--links` to the
+bisync command line if you which to follow these links. I find that I get
+permission errors in trying to follow the links, so I don't include the
+rclone `--links` flag, but then you get lots of `Can't follow symlink…`
+noise from rclone about not following the links. This noise can be
+quashed by adding `--quiet` to the bisync command line.
+
+## Example exclude-style filters files for use with Dropbox {#exclude-filters}
+
+- Dropbox disallows synching the listed temporary and configuration/data files.
+ The `- ` filters exclude these files where ever they may occur
+ in the sync tree. Consider adding similar exclusions for file types
+ you don't need to sync, such as core dump and software build files.
+- bisync testing creates `/testdir/` at the top level of the sync tree,
+ and usually deletes the tree after the test. If a normal sync should run
+ while the `/testdir/` tree exists the `--check-access` phase may fail
+ due to unbalanced RCLONE_TEST files.
+ The `- /testdir/` filter blocks this tree from being synched.
+ You don't need this exclusion if you are not doing bisync development testing.
+- Everything else beneath the Path1/Path2 root will be synched.
+- RCLONE_TEST files may be placed anywhere within the tree, including the root.
+
+### Example filters file for Dropbox {#example-filters-file}
+
+```
+# Filter file for use with bisync
+# See https://rclone.org/filtering/ for filtering rules
+# NOTICE: If you make changes to this file you MUST do a --resync run.
+# Run with --dry-run to see what changes will be made.
+
+# Dropbox won't sync some files so filter them away here.
+# See https://help.dropbox.com/installs-integrations/sync-uploads/files-not-syncing
+- .dropbox.attr
+- ~*.tmp
+- ~$*
+- .~*
+- desktop.ini
+- .dropbox
+
+# Used for bisync testing, so excluded from normal runs
+- /testdir/
+
+# Other example filters
+#- /TiBU/
+#- /Photos/
+```
+
+### How --check-access handles filters
+
+At the start of a bisync run, listings are gathered for Path1 and Path2
+while using the user's `--filters-file`. During the check access phase,
+bisync scans these listings for `RCLONE_TEST` files.
+Any `RCLONE_TEST` files hidden by the `--filters-file` are _not_ in the
+listings and thus not checked during the check access phase.
+
+## Troubleshooting {#troubleshooting}
+
+### Reading bisync logs
+
+Here are two normal runs. The first one has a newer file on the remote.
+The second has no deltas between local and remote.
+
+```
+2021/05/16 00:24:38 INFO : Synching Path1 "/path/to/local/tree/" with Path2 "dropbox:/"
+2021/05/16 00:24:38 INFO : Path1 checking for diffs
+2021/05/16 00:24:38 INFO : - Path1 File is new - file.txt
+2021/05/16 00:24:38 INFO : Path1: 1 changes: 1 new, 0 newer, 0 older, 0 deleted
+2021/05/16 00:24:38 INFO : Path2 checking for diffs
+2021/05/16 00:24:38 INFO : Applying changes
+2021/05/16 00:24:38 INFO : - Path1 Queue copy to Path2 - dropbox:/file.txt
+2021/05/16 00:24:38 INFO : - Path1 Do queued copies to - Path2
+2021/05/16 00:24:38 INFO : Updating listings
+2021/05/16 00:24:38 INFO : Validating listings for Path1 "/path/to/local/tree/" vs Path2 "dropbox:/"
+2021/05/16 00:24:38 INFO : Bisync successful
+
+2021/05/16 00:36:52 INFO : Synching Path1 "/path/to/local/tree/" with Path2 "dropbox:/"
+2021/05/16 00:36:52 INFO : Path1 checking for diffs
+2021/05/16 00:36:52 INFO : Path2 checking for diffs
+2021/05/16 00:36:52 INFO : No changes found
+2021/05/16 00:36:52 INFO : Updating listings
+2021/05/16 00:36:52 INFO : Validating listings for Path1 "/path/to/local/tree/" vs Path2 "dropbox:/"
+2021/05/16 00:36:52 INFO : Bisync successful
+```
+
+### Dry run oddity
+
+The `--dry-run` messages may indicate that it would try to delete some files.
+For example, if a file is new on Path2 and does not exist on Path1 then
+it would normally be copied to Path1, but with `--dry-run` enabled those
+copies don't happen, which leads to the attempted delete on Path2,
+blocked again by --dry-run: `... Not deleting as --dry-run`.
+
+This whole confusing situation is an artifact of the `--dry-run` flag.
+Scrutinize the proposed deletes carefully, and if the files would have been
+copied to Path1 then the threatened deletes on Path2 may be disregarded.
+
+### Retries
+
+Rclone has built-in retries. If you run with `--verbose` you'll see
+error and retry messages such as shown below. This is usually not a bug.
+If at the end of the run, you see `Bisync successful` and not
+`Bisync critical error` or `Bisync aborted` then the run was successful,
+and you can ignore the error messages.
+
+The following run shows an intermittent fail. Lines _5_ and _6- are
+low-level messages. Line _6_ is a bubbled-up _warning_ message, conveying
+the error. Rclone normally retries failing commands, so there may be
+numerous such messages in the log.
+
+Since there are no final error/warning messages on line _7_, rclone has
+recovered from failure after a retry, and the overall sync was successful.
+
+```
+1: 2021/05/14 00:44:12 INFO : Synching Path1 "/path/to/local/tree" with Path2 "dropbox:"
+2: 2021/05/14 00:44:12 INFO : Path1 checking for diffs
+3: 2021/05/14 00:44:12 INFO : Path2 checking for diffs
+4: 2021/05/14 00:44:12 INFO : Path2: 113 changes: 22 new, 0 newer, 0 older, 91 deleted
+5: 2021/05/14 00:44:12 ERROR : /path/to/local/tree/objects/af: error listing: unexpected end of JSON input
+6: 2021/05/14 00:44:12 NOTICE: WARNING listing try 1 failed. - dropbox:
+7: 2021/05/14 00:44:12 INFO : Bisync successful
+```
+
+This log shows a _Critical failure_ which requires a `--resync` to recover from.
+See the [Runtime Error Handling](#error-handling) section.
+
+```
+2021/05/12 00:49:40 INFO : Google drive root '': Waiting for checks to finish
+2021/05/12 00:49:40 INFO : Google drive root '': Waiting for transfers to finish
+2021/05/12 00:49:40 INFO : Google drive root '': not deleting files as there were IO errors
+2021/05/12 00:49:40 ERROR : Attempt 3/3 failed with 3 errors and: not deleting files as there were IO errors
+2021/05/12 00:49:40 ERROR : Failed to sync: not deleting files as there were IO errors
+2021/05/12 00:49:40 NOTICE: WARNING rclone sync try 3 failed. - /path/to/local/tree/
+2021/05/12 00:49:40 ERROR : Bisync aborted. Must run --resync to recover.
+```
+
+### Denied downloads of "infected" or "abusive" files
+
+Google Drive has a filter for certain file types (`.exe`, `.apk`, et cetera)
+that by default cannot be copied from Google Drive to the local filesystem.
+If you are having problems, run with `--verbose` to see specifically which
+files are generating complaints. If the error is
+`This file has been identified as malware or spam and cannot be downloaded`,
+consider using the flag
+[--drive-acknowledge-abuse](https://rclone.org/drive/#drive-acknowledge-abuse).
+
+### Google Doc files
+
+Google docs exist as virtual files on Google Drive and cannot be transferred
+to other filesystems natively. While it is possible to export a Google doc to
+a normal file (with `.xlsx` extension, for example), it is not possible
+to import a normal file back into a Google document.
+
+Bisync's handling of Google Doc files is to flag them in the run log output
+for user's attention and ignore them for any file transfers, deletes, or syncs.
+They will show up with a length of `-1` in the listings.
+This bisync run is otherwise successful:
+
+```
+2021/05/11 08:23:15 INFO : Synching Path1 "/path/to/local/tree/base/" with Path2 "GDrive:"
+2021/05/11 08:23:15 INFO : ...path2.lst-new: Ignoring incorrect line: "- -1 - - 2018-07-29T08:49:30.136000000+0000 GoogleDoc.docx"
+2021/05/11 08:23:15 INFO : Bisync successful
+```
+
+## Usage examples
+
+### Cron {#cron}
+
+Rclone does not yet have a built-in capability to monitor the local file
+system for changes and must be blindly run periodically.
+On Windows this can be done using a _Task Scheduler_,
+on Linux you can use _Cron_ which is described below.
+
+The 1st example runs a sync every 5 minutes between a local directory
+and an OwnCloud server, with output logged to a runlog file:
+
+```
+# Minute (0-59)
+# Hour (0-23)
+# Day of Month (1-31)
+# Month (1-12 or Jan-Dec)
+# Day of Week (0-6 or Sun-Sat)
+# Command
+ */5 * * * * /path/to/rclone bisync /local/files MyCloud: --check-access --filters-file /path/to/bysync-filters.txt --log-file /path/to//bisync.log
+```
+
+See [crontab syntax](https://www.man7.org/linux/man-pages/man1/crontab.1p.html#INPUT_FILES)
+for the details of crontab time interval expressions.
+
+If you run `rclone bisync` as a cron job, redirect stdout/stderr to a file.
+The 2nd example runs a sync to Dropbox every hour and logs all stdout (via the `>>`)
+and stderr (via `2>&1`) to a log file.
+
+```
+0 * * * * /path/to/rclone bisync /path/to/local/dropbox Dropbox: --check-access --filters-file /home/user/filters.txt >> /path/to/logs/dropbox-run.log 2>&1
+```
+
+### Sharing an encrypted folder tree between hosts
+
+bisync can keep a local folder in sync with a cloud service,
+but what if you have some highly sensitive files to be synched?
+
+Usage of a cloud service is for exchanging both routine and sensitive
+personal files between one's home network, one's personal notebook when on the
+road, and with one's work computer. The routine data is not sensitive.
+For the sensitive data, configure an rclone [crypt remote](https://rclone.org/crypt/) to point to
+a subdirectory within the local disk tree that is bisync'd to Dropbox,
+and then set up an bisync for this local crypt directory to a directory
+outside of the main sync tree.
+
+### Linux server setup
+
+- `/path/to/DBoxroot` is the root of my local sync tree.
+ There are numerous subdirectories.
+- `/path/to/DBoxroot/crypt` is the root subdirectory for files
+ that are encrypted. This local directory target is setup as an
+ rclone crypt remote named `Dropcrypt:`.
+ See [rclone.conf](#rclone-conf-snippet) snippet below.
+- `/path/to/my/unencrypted/files` is the root of my sensitive
+ files - not encrypted, not within the tree synched to Dropbox.
+- To sync my local unencrypted files with the encrypted Dropbox versions
+ I manually run `bisync /path/to/my/unencrypted/files DropCrypt:`.
+ This step could be bundled into a script to run before and after
+ the full Dropbox tree sync in the last step,
+ thus actively keeping the sensitive files in sync.
+- `bisync /path/to/DBoxroot Dropbox:` runs periodically via cron,
+ keeping my full local sync tree in sync with Dropbox.
+
+### Windows notebook setup
+
+- The Dropbox client runs keeping the local tree `C:\Users\MyLogin\Dropbox`
+ always in sync with Dropbox. I could have used `rclone bisync` instead.
+- A separate directory tree at `C:\Users\MyLogin\Documents\DropLocal`
+ hosts the tree of unencrypted files/folders.
+- To sync my local unencrypted files with the encrypted
+ Dropbox versions I manually run the following command:
+ `rclone bisync C:\Users\MyLogin\Documents\DropLocal Dropcrypt:`.
+- The Dropbox client then syncs the changes with Dropbox.
+
+### rclone.conf snippet {#rclone-conf-snippet}
+
+```
+[Dropbox]
+type = dropbox
+...
+
+[Dropcrypt]
+type = crypt
+remote = /path/to/DBoxroot/crypt # on the Linux server
+remote = C:\Users\MyLogin\Dropbox\crypt # on the Windows notebook
+filename_encryption = standard
+directory_name_encryption = true
+password = ...
+...
+```
+
+## Testing {#testing}
+
+You should read this section only if you are developing for rclone.
+You need to have rclone source code locally to work with bisync tests.
+
+Bisync has a dedicated test framework implemented in the `bisync_test.go`
+file located in the rclone source tree. The test suite is based on the
+`go test` command. Series of tests are stored in subdirectories below the
+`cmd/bisync/testdata` directory. Individual tests can be invoked by their
+directory name, e.g.
+`go test . -case basic -remote local -remote2 gdrive: -v`
+
+Tests will make a temporary folder on remote and purge it afterwards.
+If during test run there are intermittent errors and rclone retries,
+these errors will be captured and flagged as invalid MISCOMPAREs.
+Rerunning the test will let it pass. Consider such failures as noise.
+
+### Test command syntax
+
+```
+usage: go test ./cmd/bisync [options...]
+
+Options:
+ -case NAME Name(s) of the test case(s) to run. Multiple names should
+ be separated by commas. You can remove the `test_` prefix
+ and replace `_` by `-` in test name for convenience.
+ If not `all`, the name(s) should map to a directory under
+ `./cmd/bisync/testdata`.
+ Use `all` to run all tests (default: all)
+ -remote PATH1 `local` or name of cloud service with `:` (default: local)
+ -remote2 PATH2 `local` or name of cloud service with `:` (default: local)
+ -no-compare Disable comparing test results with the golden directory
+ (default: compare)
+ -no-cleanup Disable cleanup of Path1 and Path2 testdirs.
+ Useful for troubleshooting. (default: cleanup)
+ -golden Store results in the golden directory (default: false)
+ This flag can be used with multiple tests.
+ -debug Print debug messages
+ -stop-at NUM Stop test after given step number. (default: run to the end)
+ Implies `-no-compare` and `-no-cleanup`, if the test really
+ ends prematurely. Only meaningful for a single test case.
+ -refresh-times Force refreshing the target modtime, useful for Dropbox
+ (default: false)
+ -verbose Run tests verbosely
+```
+
+Note: unlike rclone flags which must be prefixed by double dash (`--`), the
+test command flags can be equally prefixed by a single `-` or double dash.
+
+### Running tests
+
+- `go test . -case basic -remote local -remote2 local`
+ runs the `test_basic` test case using only the local filesystem,
+ synching one local directory with another local directory.
+ Test script output is to the console, while commands within scenario.txt
+ have their output sent to the `.../workdir/test.log` file,
+ which is finally compared to the golden copy.
+- The first argument after `go test` should be a relative name of the
+ directory containing bisync source code. If you run tests right from there,
+ the argument will be `.` (current directory) as in most examples below.
+ If you run bisync tests from the rclone source directory, the command
+ should be `go test ./cmd/bisync ...`.
+- The test engine will mangle rclone output to ensure comparability
+ with golden listings and logs.
+- Test scenarios are located in `./cmd/bisync/testdata`. The test `-case`
+ argument should match the full name of a subdirectory under that
+ directory. Every test subdirectory name on disk must start with `test_`,
+ this prefix can be omitted on command line for brevity. Also, underscores
+ in the name can be replaced by dashes for convenience.
+- `go test . -remote local -remote2 local -case all` runs all tests.
+- Path1 and Path2 may either be the keyword `local`
+ or may be names of configured cloud services.
+ `go test . -remote gdrive: -remote2 dropbox: -case basic`
+ will run the test between these two services, without transferring
+ any files to the local filesystem.
+- Test run stdout and stderr console output may be directed to a file, e.g.
+ `go test . -remote gdrive: -remote2 local -case all > runlog.txt 2>&1`
+
+### Test execution flow
+
+1. The base setup in the `initial` directory of the testcase is applied
+ on the Path1 and Path2 filesystems (via rclone copy the initial directory
+ to Path1, then rclone sync Path1 to Path2).
+2. The commands in the scenario.txt file are applied, with output directed
+ to the `test.log` file in the test working directory.
+ Typically, the first actual command in the `scenario.txt` file is
+ to do a `--resync`, which establishes the baseline
+ `{...}.path1.lst` and `{...}.path2.lst` files in the test working
+ directory (`.../workdir/` relative to the temporary test directory).
+ Various commands and listing snapshots are done within the test.
+3. Finally, the contents of the test working directory are compared
+ to the contents of the testcase's golden directory.
+
+### Notes about testing
+
+- Test cases are in individual directories beneath `./cmd/bisync/testdata`.
+ A command line reference to a test is understood to reference a directory
+ beneath `testdata`. For example,
+ `go test ./cmd/bisync -case dry-run -remote gdrive: -remote2 local`
+ refers to the test case in `./cmd/bisync/testdata/test_dry_run`.
+- The test working directory is located at `.../workdir` relative to a
+ temporary test directory, usually under `/tmp` on Linux.
+- The local test sync tree is created at a temporary directory named
+ like `bisync.XXX` under system temporary directory.
+- The remote test sync tree is located at a temporary directory
+ under `/bisync.XXX/`.
+- `path1` and/or `path2` subdirectories are created in a temporary
+ directory under the respective local or cloud test remote.
+- By default, the Path1 and Path2 test dirs and workdir will be deleted
+ after each test run. The `-no-cleanup` flag disables purging these
+ directories when validating and debugging a given test.
+ These directories will be flushed before running another test,
+ independent of the `-no-cleanup` usage.
+- You will likely want to add `- /testdir/` to your normal
+ bisync `--filters-file` so that normal syncs do not attempt to sync
+ the test temporary directories, which may have `RCLONE_TEST` miscompares
+ in some testcases which would otherwise trip the `--check-access` system.
+ The `--check-access` mechanism is hard-coded to ignore `RCLONE_TEST`
+ files beneath `bisync/testdata`, so the test cases may reside on the
+ synched tree even if there are check file mismatches in the test tree.
+- Some Dropbox tests can fail, notably printing the following message:
+ `src and dst identical but can't set mod time without deleting and re-uploading`
+ This is expected and happens due to the way Dropbox handles modification times.
+ You should use the `-refresh-times` test flag to make up for this.
+- If Dropbox tests hit request limit for you and print error message
+ `too_many_requests/...: Too many requests or write operations.`
+ then follow the
+ [Dropbox App ID instructions](https://rclone.org/dropbox/#get-your-own-dropbox-app-id).
+
+### Updating golden results
+
+Sometimes even a slight change in the bisync source can cause little changes
+spread around many log files. Updating them manually would be a nightmare.
+
+The `-golden` flag will store the `test.log` and `*.lst` listings from each
+test case into respective golden directories. Golden results will
+automatically contain generic strings instead of local or cloud paths which
+means that they should match when run with a different cloud service.
+
+Your normal workflow might be as follows:
+1. Git-clone the rclone sources locally
+2. Modify bisync source and check that it builds
+3. Run the whole test suite `go test ./cmd/bisync -remote local`
+4. If some tests show log difference, recheck them individually, e.g.:
+ `go test ./cmd/bisync -remote local -case basic`
+5. If you are convinced with the difference, goldenize all tests at once:
+ `go test ./cmd/bisync -remote local -golden`
+6. Use word diff: `git diff --word-diff ./cmd/bisync/testdata/`.
+ Please note that normal line-level diff is generally useless here.
+7. Check the difference _carefully_!
+8. Commit the change (`git commit`) _only_ if you are sure.
+ If unsure, save your code changes then wipe the log diffs from git:
+ `git reset [--hard]`.
+
+### Structure of test scenarios
+
+- `/initial/` contains a tree of files that will be set
+ as the initial condition on both Path1 and Path2 testdirs.
+- `/modfiles/` contains files that will be used to
+ modify the Path1 and/or Path2 filesystems.
+- `/golden/` contains the expected content of the test
+ working directory (`workdir`) at the completion of the testcase.
+- `/scenario.txt` contains the body of the test, in the form of
+ various commands to modify files, run bisync, and snapshot listings.
+ Output from these commands is captured to `.../workdir/test.log`
+ for comparison to the golden files.
+
+### Supported test commands
+
+- `test `
+ Print the line to the console and to the `test.log`:
+ `test sync is working correctly with options x, y, z`
+- `copy-listings `
+ Save a copy of all `.lst` listings in the test working directory
+ with the specified prefix:
+ `save-listings exclude-pass-run`
+- `move-listings `
+ Similar to `copy-listings` but removes the source
+- `purge-children `
+ This will delete all child files and purge all child subdirs under given
+ directory but keep the parent intact. This behavior is important for tests
+ with Google Drive because removing and re-creating the parent would change
+ its ID.
+- `delete-file `
+ Delete a single file.
+- `delete-glob `
+ Delete a group of files located one level deep in the given directory
+ with names matching a given glob pattern.
+- `touch-glob YYYY-MM-DD `
+ Change modification time on a group of files.
+- `touch-copy YYYY-MM-DD `
+ Change file modification time then copy it to destination.
+- `copy-file `
+ Copy a single file to given directory.
+- `copy-as `
+ Similar to above but destination must include both directory
+ and the new file name at destination.
+- `copy-dir ` and `sync-dir `
+ Copy/sync a directory. Equivalent of `rclone copy` and `rclone sync`.
+- `list-dirs `
+ Equivalent to `rclone lsf -R --dirs-only `
+- `bisync [options]`
+ Runs bisync against `-remote` and `-remote2`.
+
+### Supported substitution terms
-## Introduction
+- `{testdir/}` - the root dir of the testcase
+- `{datadir/}` - the `modfiles` dir under the testcase root
+- `{workdir/}` - the temporary test working directory
+- `{path1/}` - the root of the Path1 test directory tree
+- `{path2/}` - the root of the Path2 test directory tree
+- `{session}` - base name of the test listings
+- `{/}` - OS-specific path separator
+- `{spc}`, `{tab}`, `{eol}` - whitespace
+- `{chr:HH}` - raw byte with given hexadecimal code
-Docker 1.9 has added support for creating
-[named volumes](https://docs.docker.com/storage/volumes/) via
-[command-line interface](https://docs.docker.com/engine/reference/commandline/volume_create/)
-and mounting them in containers as a way to share data between them.
-Since Docker 1.10 you can create named volumes with
-[Docker Compose](https://docs.docker.com/compose/) by descriptions in
-[docker-compose.yml](https://docs.docker.com/compose/compose-file/compose-file-v2/#volume-configuration-reference)
-files for use by container groups on a single host.
-As of Docker 1.12 volumes are supported by
-[Docker Swarm](https://docs.docker.com/engine/swarm/key-concepts/)
-included with Docker Engine and created from descriptions in
-[swarm compose v3](https://docs.docker.com/compose/compose-file/compose-file-v3/#volume-configuration-reference)
-files for use with _swarm stacks_ across multiple cluster nodes.
+Substitution results of the terms named like `{dir/}` will end with
+`/` (or backslash on Windows), so it is not necessary to include
+slash in the usage, for example `delete-file {path1/}file1.txt`.
-[Docker Volume Plugins](https://docs.docker.com/engine/extend/plugins_volume/)
-augment the default `local` volume driver included in Docker with stateful
-volumes shared across containers and hosts. Unlike local volumes, your
-data will _not_ be deleted when such volume is removed. Plugins can run
-managed by the docker daemon, as a native system service
-(under systemd, _sysv_ or _upstart_) or as a standalone executable.
-Rclone can run as docker volume plugin in all these modes.
-It interacts with the local docker daemon
-via [plugin API](https://docs.docker.com/engine/extend/plugin_api/) and
-handles mounting of remote file systems into docker containers so it must
-run on the same host as the docker daemon or on every Swarm node.
+## Benchmarks
-## Getting started
+_This section is work in progress._
-In the first example we will use the [SFTP](https://rclone.org/sftp/)
-rclone volume with Docker engine on a standalone Ubuntu machine.
+Here are a few data points for scale, execution times, and memory usage.
-Start from [installing Docker](https://docs.docker.com/engine/install/)
-on the host.
+The first set of data was taken between a local disk to Dropbox.
+The [speedtest.net](https://speedtest.net) download speed was ~170 Mbps,
+and upload speed was ~10 Mbps. 500 files (~9.5 MB each) had been already
+synched. 50 files were added in a new directory, each ~9.5 MB, ~475 MB total.
-The _FUSE_ driver is a prerequisite for rclone mounting and should be
-installed on host:
-```
-sudo apt-get -y install fuse
-```
+Change | Operations and times | Overall run time
+--------------------------------------|--------------------------------------------------------|------------------
+500 files synched (nothing to move) | 1x listings for Path1 & Path2 | 1.5 sec
+500 files synched with --check-access | 1x listings for Path1 & Path2 | 1.5 sec
+50 new files on remote | Queued 50 copies down: 27 sec | 29 sec
+Moved local dir | Queued 50 copies up: 410 sec, 50 deletes up: 9 sec | 421 sec
+Moved remote dir | Queued 50 copies down: 31 sec, 50 deletes down: <1 sec | 33 sec
+Delete local dir | Queued 50 deletes up: 9 sec | 13 sec
-Create two directories required by rclone docker plugin:
-```
-sudo mkdir -p /var/lib/docker-plugins/rclone/config
-sudo mkdir -p /var/lib/docker-plugins/rclone/cache
-```
+This next data is from a user's application. They had ~400GB of data
+over 1.96 million files being sync'ed between a Windows local disk and some
+remote cloud. The file full path length was on average 35 characters
+(which factors into load time and RAM required).
-Install the managed rclone docker plugin for your architecture (here `amd64`):
-```
-docker plugin install rclone/docker-volume-rclone:amd64 args="-v" --alias rclone --grant-all-permissions
-docker plugin list
-```
+- Loading the prior listing into memory (1.96 million files, listing file
+ size 140 MB) took ~30 sec and occupied about 1 GB of RAM.
+- Getting a fresh listing of the local file system (producing the
+ 140 MB output file) took about XXX sec.
+- Getting a fresh listing of the remote file system (producing the 140 MB
+ output file) took about XXX sec. The network download speed was measured
+ at XXX Mb/s.
+- Once the prior and current Path1 and Path2 listings were loaded (a total
+ of four to be loaded, two at a time), determining the deltas was pretty
+ quick (a few seconds for this test case), and the transfer time for any
+ files to be copied was dominated by the network bandwidth.
-Create your [SFTP volume](https://rclone.org/sftp/#standard-options):
-```
-docker volume create firstvolume -d rclone -o type=sftp -o sftp-host=_hostname_ -o sftp-user=_username_ -o sftp-pass=_password_ -o allow-other=true
-```
+## References
-Note that since all options are static, you don't even have to run
-`rclone config` or create the `rclone.conf` file (but the `config` directory
-should still be present). In the simplest case you can use `localhost`
-as _hostname_ and your SSH credentials as _username_ and _password_.
-You can also change the remote path to your home directory on the host,
-for example `-o path=/home/username`.
+rclone's bisync implementation was derived from
+the [rclonesync-V2](https://github.com/cjnaz/rclonesync-V2) project,
+including documentation and test mechanisms,
+with [@cjnaz](https://github.com/cjnaz)'s full support and encouragement.
+`rclone bisync` is similar in nature to a range of other projects:
-Time to create a test container and mount the volume into it:
-```
-docker run --rm -it -v firstvolume:/mnt --workdir /mnt ubuntu:latest bash
-```
+- [unison](https://github.com/bcpierce00/unison)
+- [syncthing](https://github.com/syncthing/syncthing)
+- [cjnaz/rclonesync](https://github.com/cjnaz/rclonesync-V2)
+- [ConorWilliams/rsinc](https://github.com/ConorWilliams/rsinc)
+- [jwink3101/syncrclone](https://github.com/Jwink3101/syncrclone)
+- [DavideRossi/upback](https://github.com/DavideRossi/upback)
-If all goes well, you will enter the new container and change right to
-the mounted SFTP remote. You can type `ls` to list the mounted directory
-or otherwise play with it. Type `exit` when you are done.
-The container will stop but the volume will stay, ready to be reused.
-When it's not needed anymore, remove it:
-```
-docker volume list
-docker volume remove firstvolume
-```
+Bisync adopts the differential synchronization technique, which is
+based on keeping history of changes performed by both synchronizing sides.
+See the _Dual Shadow Method_ section in
+[Neil Fraser's article](https://neil.fraser.name/writing/sync/).
-Now let us try **something more elaborate**:
-[Google Drive](https://rclone.org/drive/) volume on multi-node Docker Swarm.
+Also note a number of academic publications by
+[Benjamin Pierce](http://www.cis.upenn.edu/%7Ebcpierce/papers/index.shtml#File%20Synchronization)
+about _Unison_ and synchronization in general.
-You should start from installing Docker and FUSE, creating plugin
-directories and installing rclone plugin on _every_ swarm node.
-Then [setup the Swarm](https://docs.docker.com/engine/swarm/swarm-mode/).
+## Changelog
-Google Drive volumes need an access token which can be setup via web
-browser and will be periodically renewed by rclone. The managed
-plugin cannot run a browser so we will use a technique similar to the
-[rclone setup on a headless box](https://rclone.org/remote_setup/).
+### `v1.64`
+* Fixed an [issue](https://forum.rclone.org/t/bisync-bugs-and-feature-requests/37636#:~:text=1.%20Dry%20runs%20are%20not%20completely%20dry)
+causing dry runs to inadvertently commit filter changes
+* Fixed an [issue](https://forum.rclone.org/t/bisync-bugs-and-feature-requests/37636#:~:text=2.%20%2D%2Dresync%20deletes%20data%2C%20contrary%20to%20docs)
+causing `--resync` to erroneously delete empty folders and duplicate files unique to Path2
+* `--check-access` is now enforced during `--resync`, preventing data loss in [certain user error scenarios](https://forum.rclone.org/t/bisync-bugs-and-feature-requests/37636#:~:text=%2D%2Dcheck%2Daccess%20doesn%27t%20always%20fail%20when%20it%20should)
+* Fixed an [issue](https://forum.rclone.org/t/bisync-bugs-and-feature-requests/37636#:~:text=5.%20Bisync%20reads%20files%20in%20excluded%20directories%20during%20delete%20operations)
+causing bisync to consider more files than necessary due to overbroad filters during delete operations
+* [Improved detection of false positive change conflicts](https://forum.rclone.org/t/bisync-bugs-and-feature-requests/37636#:~:text=1.%20Identical%20files%20should%20be%20left%20alone%2C%20even%20if%20new/newer/changed%20on%20both%20sides)
+(identical files are now left alone instead of renamed)
+* Added [support for `--create-empty-src-dirs`](https://forum.rclone.org/t/bisync-bugs-and-feature-requests/37636#:~:text=3.%20Bisync%20should%20create/delete%20empty%20directories%20as%20sync%20does%2C%20when%20%2D%2Dcreate%2Dempty%2Dsrc%2Ddirs%20is%20passed)
+* Added experimental `--resilient` mode to allow [recovery from self-correctable errors](https://forum.rclone.org/t/bisync-bugs-and-feature-requests/37636#:~:text=2.%20Bisync%20should%20be%20more%20resilient%20to%20self%2Dcorrectable%20errors)
+* Added [new `--ignore-listing-checksum` flag](https://forum.rclone.org/t/bisync-bugs-and-feature-requests/37636#:~:text=6.%20%2D%2Dignore%2Dchecksum%20should%20be%20split%20into%20two%20flags%20for%20separate%20purposes)
+to distinguish from `--ignore-checksum`
+* [Performance improvements](https://forum.rclone.org/t/bisync-bugs-and-feature-requests/37636#:~:text=6.%20Deletes%20take%20several%20times%20longer%20than%20copies) for large remotes
+* Documentation and testing improvements
-Run [rclone config](https://rclone.org/commands/rclone_config_create/)
-on _another_ machine equipped with _web browser_ and graphical user interface.
-Create the [Google Drive remote](https://rclone.org/drive/#standard-options).
-When done, transfer the resulting `rclone.conf` to the Swarm cluster
-and save as `/var/lib/docker-plugins/rclone/config/rclone.conf`
-on _every_ node. By default this location is accessible only to the
-root user so you will need appropriate privileges. The resulting config
-will look like this:
-```
-[gdrive]
-type = drive
-scope = drive
-drive_id = 1234567...
-root_folder_id = 0Abcd...
-token = {"access_token":...}
-```
+# Release signing
-Now create the file named `example.yml` with a swarm stack description
-like this:
-```
-version: '3'
-services:
- heimdall:
- image: linuxserver/heimdall:latest
- ports: [8080:80]
- volumes: [configdata:/config]
-volumes:
- configdata:
- driver: rclone
- driver_opts:
- remote: 'gdrive:heimdall'
- allow_other: 'true'
- vfs_cache_mode: full
- poll_interval: 0
-```
+The hashes of the binary artefacts of the rclone release are signed
+with a public PGP/GPG key. This can be verified manually as described
+below.
-and run the stack:
-```
-docker stack deploy example -c ./example.yml
-```
+The same mechanism is also used by [rclone selfupdate](https://rclone.org/commands/rclone_selfupdate/)
+to verify that the release has not been tampered with before the new
+update is installed. This checks the SHA256 hash and the signature
+with a public key compiled into the rclone binary.
+
+## Release signing key
+
+You may obtain the release signing key from:
+
+- From [KEYS](/KEYS) on this website - this file contains all past signing keys also.
+- The git repository hosted on GitHub - https://github.com/rclone/rclone/blob/master/docs/content/KEYS
+- `gpg --keyserver hkps://keys.openpgp.org --search nick@craig-wood.com`
+- `gpg --keyserver hkps://keyserver.ubuntu.com --search nick@craig-wood.com`
+- https://www.craig-wood.com/nick/pub/pgp-key.txt
+
+After importing the key, verify that the fingerprint of one of the
+keys matches: `FBF737ECE9F8AB18604BD2AC93935E02FF3B54FA` as this key is used for signing.
+
+We recommend that you cross-check the fingerprint shown above through
+the domains listed below. By cross-checking the integrity of the
+fingerprint across multiple domains you can be confident that you
+obtained the correct key.
+
+- The [source for this page on GitHub](https://github.com/rclone/rclone/blob/master/docs/content/release_signing.md).
+- Through DNS `dig key.rclone.org txt`
+
+If you find anything that doesn't not match, please contact the
+developers at once.
+
+## How to verify the release
+
+In the release directory you will see the release files and some files called `MD5SUMS`, `SHA1SUMS` and `SHA256SUMS`.
-After a few seconds docker will spread the parsed stack description
-over cluster, create the `example_heimdall` service on port _8080_,
-run service containers on one or more cluster nodes and request
-the `example_configdata` volume from rclone plugins on the node hosts.
-You can use the following commands to confirm results:
```
-docker service ls
-docker service ps example_heimdall
-docker volume ls
+$ rclone lsf --http-url https://downloads.rclone.org/v1.63.1 :http:
+MD5SUMS
+SHA1SUMS
+SHA256SUMS
+rclone-v1.63.1-freebsd-386.zip
+rclone-v1.63.1-freebsd-amd64.zip
+...
+rclone-v1.63.1-windows-arm64.zip
+rclone-v1.63.1.tar.gz
+version.txt
```
-Point your browser to `http://cluster.host.address:8080` and play with
-the service. Stop it with `docker stack remove example` when you are done.
-Note that the `example_configdata` volume(s) created on demand at the
-cluster nodes will not be automatically removed together with the stack
-but stay for future reuse. You can remove them manually by invoking
-the `docker volume remove example_configdata` command on every node.
+The `MD5SUMS`, `SHA1SUMS` and `SHA256SUMS` contain hashes of the
+binary files in the release directory along with a signature.
-## Creating Volumes via CLI
+For example:
-Volumes can be created with [docker volume create](https://docs.docker.com/engine/reference/commandline/volume_create/).
-Here are a few examples:
```
-docker volume create vol1 -d rclone -o remote=storj: -o vfs-cache-mode=full
-docker volume create vol2 -d rclone -o remote=:storj,access_grant=xxx:heimdall
-docker volume create vol3 -d rclone -o type=storj -o path=heimdall -o storj-access-grant=xxx -o poll-interval=0
+$ rclone cat --http-url https://downloads.rclone.org/v1.63.1 :http:SHA256SUMS
+-----BEGIN PGP SIGNED MESSAGE-----
+Hash: SHA1
+
+f6d1b2d7477475ce681bdce8cb56f7870f174cb6b2a9ac5d7b3764296ea4a113 rclone-v1.63.1-freebsd-386.zip
+7266febec1f01a25d6575de51c44ddf749071a4950a6384e4164954dff7ac37e rclone-v1.63.1-freebsd-amd64.zip
+...
+66ca083757fb22198309b73879831ed2b42309892394bf193ff95c75dff69c73 rclone-v1.63.1-windows-amd64.zip
+bbb47c16882b6c5f2e8c1b04229378e28f68734c613321ef0ea2263760f74cd0 rclone-v1.63.1-windows-arm64.zip
+-----BEGIN PGP SIGNATURE-----
+
+iF0EARECAB0WIQT79zfs6firGGBL0qyTk14C/ztU+gUCZLVKJQAKCRCTk14C/ztU
++pZuAJ0XJ+QWLP/3jCtkmgcgc4KAwd/rrwCcCRZQ7E+oye1FPY46HOVzCFU3L7g=
+=8qrL
+-----END PGP SIGNATURE-----
```
-Note the `-d rclone` flag that tells docker to request volume from the
-rclone driver. This works even if you installed managed driver by its full
-name `rclone/docker-volume-rclone` because you provided the `--alias rclone`
-option.
+### Download the files
+
+The first step is to download the binary and SUMs file and verify that
+the SUMs you have downloaded match. Here we download
+`rclone-v1.63.1-windows-amd64.zip` - choose the binary (or binaries)
+appropriate to your architecture. We've also chosen the `SHA256SUMS`
+as these are the most secure. You could verify the other types of hash
+also for extra security. `rclone selfupdate` verifies just the
+`SHA256SUMS`.
-Volumes can be inspected as follows:
```
-docker volume list
-docker volume inspect vol1
+$ mkdir /tmp/check
+$ cd /tmp/check
+$ rclone copy --http-url https://downloads.rclone.org/v1.63.1 :http:SHA256SUMS .
+$ rclone copy --http-url https://downloads.rclone.org/v1.63.1 :http:rclone-v1.63.1-windows-amd64.zip .
```
-## Volume Configuration
+### Verify the signatures
-Rclone flags and volume options are set via the `-o` flag to the
-`docker volume create` command. They include backend-specific parameters
-as well as mount and _VFS_ options. Also there are a few
-special `-o` options:
-`remote`, `fs`, `type`, `path`, `mount-type` and `persist`.
+First verify the signatures on the SHA256 file.
-`remote` determines an existing remote name from the config file, with
-trailing colon and optionally with a remote path. See the full syntax in
-the [rclone documentation](https://rclone.org/docs/#syntax-of-remote-paths).
-This option can be aliased as `fs` to prevent confusion with the
-_remote_ parameter of such backends as _crypt_ or _alias_.
+Import the key. See above for ways to verify this key is correct.
-The `remote=:backend:dir/subdir` syntax can be used to create
-[on-the-fly (config-less) remotes](https://rclone.org/docs/#backend-path-to-dir),
-while the `type` and `path` options provide a simpler alternative for this.
-Using two split options
```
--o type=backend -o path=dir/subdir
+$ gpg --keyserver keyserver.ubuntu.com --receive-keys FBF737ECE9F8AB18604BD2AC93935E02FF3B54FA
+gpg: key 93935E02FF3B54FA: public key "Nick Craig-Wood " imported
+gpg: Total number processed: 1
+gpg: imported: 1
```
-is equivalent to the combined syntax
+
+Then check the signature:
+
```
--o remote=:backend:dir/subdir
+$ gpg --verify SHA256SUMS
+gpg: Signature made Mon 17 Jul 2023 15:03:17 BST
+gpg: using DSA key FBF737ECE9F8AB18604BD2AC93935E02FF3B54FA
+gpg: Good signature from "Nick Craig-Wood " [ultimate]
```
-but is arguably easier to parameterize in scripts.
-The `path` part is optional.
-[Mount and VFS options](https://rclone.org/commands/rclone_serve_docker/#options)
-as well as [backend parameters](https://rclone.org/flags/#backend-flags) are named
-like their twin command-line flags without the `--` CLI prefix.
-Optionally you can use underscores instead of dashes in option names.
-For example, `--vfs-cache-mode full` becomes
-`-o vfs-cache-mode=full` or `-o vfs_cache_mode=full`.
-Boolean CLI flags without value will gain the `true` value, e.g.
-`--allow-other` becomes `-o allow-other=true` or `-o allow_other=true`.
+Verify the signature was good and is using the fingerprint shown above.
-Please note that you can provide parameters only for the backend immediately
-referenced by the backend type of mounted `remote`.
-If this is a wrapping backend like _alias, chunker or crypt_, you cannot
-provide options for the referred to remote or backend. This limitation is
-imposed by the rclone connection string parser. The only workaround is to
-feed plugin with `rclone.conf` or configure plugin arguments (see below).
+Repeat for `MD5SUMS` and `SHA1SUMS` if desired.
-## Special Volume Options
+### Verify the hashes
-`mount-type` determines the mount method and in general can be one of:
-`mount`, `cmount`, or `mount2`. This can be aliased as `mount_type`.
-It should be noted that the managed rclone docker plugin currently does
-not support the `cmount` method and `mount2` is rarely needed.
-This option defaults to the first found method, which is usually `mount`
-so you generally won't need it.
+Now that we know the signatures on the hashes are OK we can verify the
+binaries match the hashes, completing the verification.
-`persist` is a reserved boolean (true/false) option.
-In future it will allow to persist on-the-fly remotes in the plugin
-`rclone.conf` file.
+```
+$ sha256sum -c SHA256SUMS 2>&1 | grep OK
+rclone-v1.63.1-windows-amd64.zip: OK
+```
-## Connection Strings
+Or do the check with rclone
-The `remote` value can be extended
-with [connection strings](https://rclone.org/docs/#connection-strings)
-as an alternative way to supply backend parameters. This is equivalent
-to the `-o` backend options with one _syntactic difference_.
-Inside connection string the backend prefix must be dropped from parameter
-names but in the `-o param=value` array it must be present.
-For instance, compare the following option array
```
--o remote=:sftp:/home -o sftp-host=localhost
+$ rclone hashsum sha256 -C SHA256SUMS rclone-v1.63.1-windows-amd64.zip
+2023/09/11 10:53:58 NOTICE: SHA256SUMS: improperly formatted checksum line 0
+2023/09/11 10:53:58 NOTICE: SHA256SUMS: improperly formatted checksum line 1
+2023/09/11 10:53:58 NOTICE: SHA256SUMS: improperly formatted checksum line 49
+2023/09/11 10:53:58 NOTICE: SHA256SUMS: 4 warning(s) suppressed...
+= rclone-v1.63.1-windows-amd64.zip
+2023/09/11 10:53:58 NOTICE: Local file system at /tmp/check: 0 differences found
+2023/09/11 10:53:58 NOTICE: Local file system at /tmp/check: 1 matching files
```
-with equivalent connection string:
+
+### Verify signatures and hashes together
+
+You can verify the signatures and hashes in one command line like this:
+
```
--o remote=:sftp,host=localhost:/home
+$ gpg --decrypt SHA256SUMS | sha256sum -c --ignore-missing
+gpg: Signature made Mon 17 Jul 2023 15:03:17 BST
+gpg: using DSA key FBF737ECE9F8AB18604BD2AC93935E02FF3B54FA
+gpg: Good signature from "Nick Craig-Wood " [ultimate]
+gpg: aka "Nick Craig-Wood " [unknown]
+rclone-v1.63.1-windows-amd64.zip: OK
```
-This difference exists because flag options `-o key=val` include not only
-backend parameters but also mount/VFS flags and possibly other settings.
-Also it allows to discriminate the `remote` option from the `crypt-remote`
-(or similarly named backend parameters) and arguably simplifies scripting
-due to clearer value substitution.
-## Using with Swarm or Compose
+# 1Fichier
-Both _Docker Swarm_ and _Docker Compose_ use
-[YAML](http://yaml.org/spec/1.2/spec.html)-formatted text files to describe
-groups (stacks) of containers, their properties, networks and volumes.
-_Compose_ uses the [compose v2](https://docs.docker.com/compose/compose-file/compose-file-v2/#volume-configuration-reference) format,
-_Swarm_ uses the [compose v3](https://docs.docker.com/compose/compose-file/compose-file-v3/#volume-configuration-reference) format.
-They are mostly similar, differences are explained in the
-[docker documentation](https://docs.docker.com/compose/compose-file/compose-versioning/#upgrading).
+This is a backend for the [1fichier](https://1fichier.com) cloud
+storage service. Note that a Premium subscription is required to use
+the API.
-Volumes are described by the children of the top-level `volumes:` node.
-Each of them should be named after its volume and have at least two
-elements, the self-explanatory `driver: rclone` value and the
-`driver_opts:` structure playing the same role as `-o key=val` CLI flags:
+Paths are specified as `remote:path`
-```
-volumes:
- volume_name_1:
- driver: rclone
- driver_opts:
- remote: 'gdrive:'
- allow_other: 'true'
- vfs_cache_mode: full
- token: '{"type": "borrower", "expires": "2021-12-31"}'
- poll_interval: 0
-```
+Paths may be as deep as required, e.g. `remote:directory/subdirectory`.
-Notice a few important details:
-- YAML prefers `_` in option names instead of `-`.
-- YAML treats single and double quotes interchangeably.
- Simple strings and integers can be left unquoted.
-- Boolean values must be quoted like `'true'` or `"false"` because
- these two words are reserved by YAML.
-- The filesystem string is keyed with `remote` (or with `fs`).
- Normally you can omit quotes here, but if the string ends with colon,
- you **must** quote it like `remote: "storage_box:"`.
-- YAML is picky about surrounding braces in values as this is in fact
- another [syntax for key/value mappings](http://yaml.org/spec/1.2/spec.html#id2790832).
- For example, JSON access tokens usually contain double quotes and
- surrounding braces, so you must put them in single quotes.
+## Configuration
-## Installing as Managed Plugin
+The initial setup for 1Fichier involves getting the API key from the website which you
+need to do in your browser.
-Docker daemon can install plugins from an image registry and run them managed.
-We maintain the
-[docker-volume-rclone](https://hub.docker.com/p/rclone/docker-volume-rclone/)
-plugin image on [Docker Hub](https://hub.docker.com).
+Here is an example of how to make a remote called `remote`. First run:
-Rclone volume plugin requires **Docker Engine >= 19.03.15**
+ rclone config
-The plugin requires presence of two directories on the host before it can
-be installed. Note that plugin will **not** create them automatically.
-By default they must exist on host at the following locations
-(though you can tweak the paths):
-- `/var/lib/docker-plugins/rclone/config`
- is reserved for the `rclone.conf` config file and **must** exist
- even if it's empty and the config file is not present.
-- `/var/lib/docker-plugins/rclone/cache`
- holds the plugin state file as well as optional VFS caches.
+This will guide you through an interactive setup process:
-You can [install managed plugin](https://docs.docker.com/engine/reference/commandline/plugin_install/)
-with default settings as follows:
-```
-docker plugin install rclone/docker-volume-rclone:amd64 --grant-all-permissions --alias rclone
```
+No remotes found, make a new one?
+n) New remote
+s) Set configuration password
+q) Quit config
+n/s/q> n
+name> remote
+Type of storage to configure.
+Enter a string value. Press Enter for the default ("").
+Choose a number from below, or type in your own value
+[snip]
+XX / 1Fichier
+ \ "fichier"
+[snip]
+Storage> fichier
+** See help for fichier backend at: https://rclone.org/fichier/ **
-The `:amd64` part of the image specification after colon is called a _tag_.
-Usually you will want to install the latest plugin for your architecture. In
-this case the tag will just name it, like `amd64` above. The following plugin
-architectures are currently available:
-- `amd64`
-- `arm64`
-- `arm-v7`
+Your API Key, get it from https://1fichier.com/console/params.pl
+Enter a string value. Press Enter for the default ("").
+api_key> example_key
+
+Edit advanced config? (y/n)
+y) Yes
+n) No
+y/n>
+Remote config
+--------------------
+[remote]
+type = fichier
+api_key = example_key
+--------------------
+y) Yes this is OK
+e) Edit this remote
+d) Delete this remote
+y/e/d> y
+```
-Sometimes you might want a concrete plugin version, not the latest one.
-Then you should use image tag in the form `:ARCHITECTURE-VERSION`.
-For example, to install plugin version `v1.56.2` on architecture `arm64`
-you will use tag `arm64-1.56.2` (note the removed `v`) so the full image
-specification becomes `rclone/docker-volume-rclone:arm64-1.56.2`.
+Once configured you can then use `rclone` like this,
-We also provide the `latest` plugin tag, but since docker does not support
-multi-architecture plugins as of the time of this writing, this tag is
-currently an **alias for `amd64`**.
-By convention the `latest` tag is the default one and can be omitted, thus
-both `rclone/docker-volume-rclone:latest` and just `rclone/docker-volume-rclone`
-will refer to the latest plugin release for the `amd64` platform.
+List directories in top level of your 1Fichier account
-Also the `amd64` part can be omitted from the versioned rclone plugin tags.
-For example, rclone image reference `rclone/docker-volume-rclone:amd64-1.56.2`
-can be abbreviated as `rclone/docker-volume-rclone:1.56.2` for convenience.
-However, for non-intel architectures you still have to use the full tag as
-`amd64` or `latest` will fail to start.
+ rclone lsd remote:
-Managed plugin is in fact a special container running in a namespace separate
-from normal docker containers. Inside it runs the `rclone serve docker`
-command. The config and cache directories are bind-mounted into the
-container at start. The docker daemon connects to a unix socket created
-by the command inside the container. The command creates on-demand remote
-mounts right inside, then docker machinery propagates them through kernel
-mount namespaces and bind-mounts into requesting user containers.
+List all the files in your 1Fichier account
-You can tweak a few plugin settings after installation when it's disabled
-(not in use), for instance:
-```
-docker plugin disable rclone
-docker plugin set rclone RCLONE_VERBOSE=2 config=/etc/rclone args="--vfs-cache-mode=writes --allow-other"
-docker plugin enable rclone
-docker plugin inspect rclone
-```
+ rclone ls remote:
-Note that if docker refuses to disable the plugin, you should find and
-remove all active volumes connected with it as well as containers and
-swarm services that use them. This is rather tedious so please carefully
-plan in advance.
+To copy a local directory to a 1Fichier directory called backup
-You can tweak the following settings:
-`args`, `config`, `cache`, `HTTP_PROXY`, `HTTPS_PROXY`, `NO_PROXY`
-and `RCLONE_VERBOSE`.
-It's _your_ task to keep plugin settings in sync across swarm cluster nodes.
+ rclone copy /home/source remote:backup
-`args` sets command-line arguments for the `rclone serve docker` command
-(_none_ by default). Arguments should be separated by space so you will
-normally want to put them in quotes on the
-[docker plugin set](https://docs.docker.com/engine/reference/commandline/plugin_set/)
-command line. Both [serve docker flags](https://rclone.org/commands/rclone_serve_docker/#options)
-and [generic rclone flags](https://rclone.org/flags/) are supported, including backend
-parameters that will be used as defaults for volume creation.
-Note that plugin will fail (due to [this docker bug](https://github.com/moby/moby/blob/v20.10.7/plugin/v2/plugin.go#L195))
-if the `args` value is empty. Use e.g. `args="-v"` as a workaround.
+### Modification times and hashes
-`config=/host/dir` sets alternative host location for the config directory.
-Plugin will look for `rclone.conf` here. It's not an error if the config
-file is not present but the directory must exist. Please note that plugin
-can periodically rewrite the config file, for example when it renews
-storage access tokens. Keep this in mind and try to avoid races between
-the plugin and other instances of rclone on the host that might try to
-change the config simultaneously resulting in corrupted `rclone.conf`.
-You can also put stuff like private key files for SFTP remotes in this
-directory. Just note that it's bind-mounted inside the plugin container
-at the predefined path `/data/config`. For example, if your key file is
-named `sftp-box1.key` on the host, the corresponding volume config option
-should read `-o sftp-key-file=/data/config/sftp-box1.key`.
+1Fichier does not support modification times. It supports the Whirlpool hash algorithm.
-`cache=/host/dir` sets alternative host location for the _cache_ directory.
-The plugin will keep VFS caches here. Also it will create and maintain
-the `docker-plugin.state` file in this directory. When the plugin is
-restarted or reinstalled, it will look in this file to recreate any volumes
-that existed previously. However, they will not be re-mounted into
-consuming containers after restart. Usually this is not a problem as
-the docker daemon normally will restart affected user containers after
-failures, daemon restarts or host reboots.
+### Duplicated files
-`RCLONE_VERBOSE` sets plugin verbosity from `0` (errors only, by default)
-to `2` (debugging). Verbosity can be also tweaked via `args="-v [-v] ..."`.
-Since arguments are more generic, you will rarely need this setting.
-The plugin output by default feeds the docker daemon log on local host.
-Log entries are reflected as _errors_ in the docker log but retain their
-actual level assigned by rclone in the encapsulated message string.
+1Fichier can have two files with exactly the same name and path (unlike a
+normal file system).
-`HTTP_PROXY`, `HTTPS_PROXY`, `NO_PROXY` customize the plugin proxy settings.
+Duplicated files cause problems with the syncing and you will see
+messages in the log about duplicates.
-You can set custom plugin options right when you install it, _in one go_:
-```
-docker plugin remove rclone
-docker plugin install rclone/docker-volume-rclone:amd64 \
- --alias rclone --grant-all-permissions \
- args="-v --allow-other" config=/etc/rclone
-docker plugin inspect rclone
-```
+### Restricted filename characters
-## Healthchecks
+In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters)
+the following characters are also replaced:
-The docker plugin volume protocol doesn't provide a way for plugins
-to inform the docker daemon that a volume is (un-)available.
-As a workaround you can setup a healthcheck to verify that the mount
-is responding, for example:
-```
-services:
- my_service:
- image: my_image
- healthcheck:
- test: ls /path/to/rclone/mount || exit 1
- interval: 1m
- timeout: 15s
- retries: 3
- start_period: 15s
-```
+| Character | Value | Replacement |
+| --------- |:-----:|:-----------:|
+| \ | 0x5C | ï¼¼ |
+| < | 0x3C | ＜ |
+| > | 0x3E | ＞ |
+| " | 0x22 | ＂ |
+| $ | 0x24 | ＄ |
+| ` | 0x60 | ï½€ |
+| ' | 0x27 | ＇ |
-## Running Plugin under Systemd
+File names can also not start or end with the following characters.
+These only get replaced if they are the first or last character in the
+name:
-In most cases you should prefer managed mode. Moreover, MacOS and Windows
-do not support native Docker plugins. Please use managed mode on these
-systems. Proceed further only if you are on Linux.
+| Character | Value | Replacement |
+| --------- |:-----:|:-----------:|
+| SP | 0x20 | â |
-First, [install rclone](https://rclone.org/install/).
-You can just run it (type `rclone serve docker` and hit enter) for the test.
+Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8),
+as they can't be used in JSON strings.
-Install _FUSE_:
-```
-sudo apt-get -y install fuse
-```
-Download two systemd configuration files:
-[docker-volume-rclone.service](https://raw.githubusercontent.com/rclone/rclone/master/contrib/docker-plugin/systemd/docker-volume-rclone.service)
-and [docker-volume-rclone.socket](https://raw.githubusercontent.com/rclone/rclone/master/contrib/docker-plugin/systemd/docker-volume-rclone.socket).
+### Standard options
-Put them to the `/etc/systemd/system/` directory:
-```
-cp docker-volume-plugin.service /etc/systemd/system/
-cp docker-volume-plugin.socket /etc/systemd/system/
-```
+Here are the Standard options specific to fichier (1Fichier).
-Please note that all commands in this section must be run as _root_ but
-we omit `sudo` prefix for brevity.
-Now create directories required by the service:
-```
-mkdir -p /var/lib/docker-volumes/rclone
-mkdir -p /var/lib/docker-plugins/rclone/config
-mkdir -p /var/lib/docker-plugins/rclone/cache
-```
+#### --fichier-api-key
-Run the docker plugin service in the socket activated mode:
-```
-systemctl daemon-reload
-systemctl start docker-volume-rclone.service
-systemctl enable docker-volume-rclone.socket
-systemctl start docker-volume-rclone.socket
-systemctl restart docker
-```
+Your API Key, get it from https://1fichier.com/console/params.pl.
-Or run the service directly:
-- run `systemctl daemon-reload` to let systemd pick up new config
-- run `systemctl enable docker-volume-rclone.service` to make the new
- service start automatically when you power on your machine.
-- run `systemctl start docker-volume-rclone.service`
- to start the service now.
-- run `systemctl restart docker` to restart docker daemon and let it
- detect the new plugin socket. Note that this step is not needed in
- managed mode where docker knows about plugin state changes.
+Properties:
-The two methods are equivalent from the user perspective, but I personally
-prefer socket activation.
+- Config: api_key
+- Env Var: RCLONE_FICHIER_API_KEY
+- Type: string
+- Required: false
-## Troubleshooting
+### Advanced options
-You can [see managed plugin settings](https://docs.docker.com/engine/extend/#debugging-plugins)
-with
-```
-docker plugin list
-docker plugin inspect rclone
-```
-Note that docker (including latest 20.10.7) will not show actual values
-of `args`, just the defaults.
+Here are the Advanced options specific to fichier (1Fichier).
-Use `journalctl --unit docker` to see managed plugin output as part of
-the docker daemon log. Note that docker reflects plugin lines as _errors_
-but their actual level can be seen from encapsulated message string.
+#### --fichier-shared-folder
-You will usually install the latest version of managed plugin for your platform.
-Use the following commands to print the actual installed version:
-```
-PLUGID=$(docker plugin list --no-trunc | awk '/rclone/{print$1}')
-sudo runc --root /run/docker/runtime-runc/plugins.moby exec $PLUGID rclone version
-```
+If you want to download a shared folder, add this parameter.
-You can even use `runc` to run shell inside the plugin container:
-```
-sudo runc --root /run/docker/runtime-runc/plugins.moby exec --tty $PLUGID bash
-```
+Properties:
-Also you can use curl to check the plugin socket connectivity:
-```
-docker plugin list --no-trunc
-PLUGID=123abc...
-sudo curl -H Content-Type:application/json -XPOST -d {} --unix-socket /run/docker/plugins/$PLUGID/rclone.sock http://localhost/Plugin.Activate
-```
-though this is rarely needed.
+- Config: shared_folder
+- Env Var: RCLONE_FICHIER_SHARED_FOLDER
+- Type: string
+- Required: false
-## Caveats
+#### --fichier-file-password
-Finally I'd like to mention a _caveat with updating volume settings_.
-Docker CLI does not have a dedicated command like `docker volume update`.
-It may be tempting to invoke `docker volume create` with updated options
-on existing volume, but there is a gotcha. The command will do nothing,
-it won't even return an error. I hope that docker maintainers will fix
-this some day. In the meantime be aware that you must remove your volume
-before recreating it with new settings:
-```
-docker volume remove my_vol
-docker volume create my_vol -d rclone -o opt1=new_val1 ...
-```
+If you want to download a shared file that is password protected, add this parameter.
-and verify that settings did update:
-```
-docker volume list
-docker volume inspect my_vol
-```
+**NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/).
-If docker refuses to remove the volume, you should find containers
-or swarm services that use it and stop them first.
+Properties:
-## Getting started {#getting-started}
+- Config: file_password
+- Env Var: RCLONE_FICHIER_FILE_PASSWORD
+- Type: string
+- Required: false
-- [Install rclone](https://rclone.org/install/) and setup your remotes.
-- Bisync will create its working directory
- at `~/.cache/rclone/bisync` on Linux
- or `C:\Users\MyLogin\AppData\Local\rclone\bisync` on Windows.
- Make sure that this location is writable.
-- Run bisync with the `--resync` flag, specifying the paths
- to the local and remote sync directory roots.
-- For successive sync runs, leave off the `--resync` flag.
-- Consider using a [filters file](#filtering) for excluding
- unnecessary files and directories from the sync.
-- Consider setting up the [--check-access](#check-access) feature
- for safety.
-- On Linux, consider setting up a [crontab entry](#cron). bisync can
- safely run in concurrent cron jobs thanks to lock files it maintains.
+#### --fichier-folder-password
-Here is a typical run log (with timestamps removed for clarity):
+If you want to list the files in a shared folder that is password protected, add this parameter.
-```
-rclone bisync /testdir/path1/ /testdir/path2/ --verbose
-INFO : Synching Path1 "/testdir/path1/" with Path2 "/testdir/path2/"
-INFO : Path1 checking for diffs
-INFO : - Path1 File is new - file11.txt
-INFO : - Path1 File is newer - file2.txt
-INFO : - Path1 File is newer - file5.txt
-INFO : - Path1 File is newer - file7.txt
-INFO : - Path1 File was deleted - file4.txt
-INFO : - Path1 File was deleted - file6.txt
-INFO : - Path1 File was deleted - file8.txt
-INFO : Path1: 7 changes: 1 new, 3 newer, 0 older, 3 deleted
-INFO : Path2 checking for diffs
-INFO : - Path2 File is new - file10.txt
-INFO : - Path2 File is newer - file1.txt
-INFO : - Path2 File is newer - file5.txt
-INFO : - Path2 File is newer - file6.txt
-INFO : - Path2 File was deleted - file3.txt
-INFO : - Path2 File was deleted - file7.txt
-INFO : - Path2 File was deleted - file8.txt
-INFO : Path2: 7 changes: 1 new, 3 newer, 0 older, 3 deleted
-INFO : Applying changes
-INFO : - Path1 Queue copy to Path2 - /testdir/path2/file11.txt
-INFO : - Path1 Queue copy to Path2 - /testdir/path2/file2.txt
-INFO : - Path2 Queue delete - /testdir/path2/file4.txt
-NOTICE: - WARNING New or changed in both paths - file5.txt
-NOTICE: - Path1 Renaming Path1 copy - /testdir/path1/file5.txt..path1
-NOTICE: - Path1 Queue copy to Path2 - /testdir/path2/file5.txt..path1
-NOTICE: - Path2 Renaming Path2 copy - /testdir/path2/file5.txt..path2
-NOTICE: - Path2 Queue copy to Path1 - /testdir/path1/file5.txt..path2
-INFO : - Path2 Queue copy to Path1 - /testdir/path1/file6.txt
-INFO : - Path1 Queue copy to Path2 - /testdir/path2/file7.txt
-INFO : - Path2 Queue copy to Path1 - /testdir/path1/file1.txt
-INFO : - Path2 Queue copy to Path1 - /testdir/path1/file10.txt
-INFO : - Path1 Queue delete - /testdir/path1/file3.txt
-INFO : - Path2 Do queued copies to - Path1
-INFO : - Path1 Do queued copies to - Path2
-INFO : - Do queued deletes on - Path1
-INFO : - Do queued deletes on - Path2
-INFO : Updating listings
-INFO : Validating listings for Path1 "/testdir/path1/" vs Path2 "/testdir/path2/"
-INFO : Bisync successful
-```
+**NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/).
-## Command line syntax
+Properties:
-```
-$ rclone bisync --help
-Usage:
- rclone bisync remote1:path1 remote2:path2 [flags]
+- Config: folder_password
+- Env Var: RCLONE_FICHIER_FOLDER_PASSWORD
+- Type: string
+- Required: false
-Positional arguments:
- Path1, Path2 Local path, or remote storage with ':' plus optional path.
- Type 'rclone listremotes' for list of configured remotes.
+#### --fichier-cdn
-Optional Flags:
- --check-access Ensure expected `RCLONE_TEST` files are found on
- both Path1 and Path2 filesystems, else abort.
- --check-filename FILENAME Filename for `--check-access` (default: `RCLONE_TEST`)
- --check-sync CHOICE Controls comparison of final listings:
- `true | false | only` (default: true)
- If set to `only`, bisync will only compare listings
- from the last run but skip actual sync.
- --filters-file PATH Read filtering patterns from a file
- --max-delete PERCENT Safety check on maximum percentage of deleted files allowed.
- If exceeded, the bisync run will abort. (default: 50%)
- --force Bypass `--max-delete` safety check and run the sync.
- Consider using with `--verbose`
- --create-empty-src-dirs Sync creation and deletion of empty directories.
- (Not compatible with --remove-empty-dirs)
- --remove-empty-dirs Remove empty directories at the final cleanup step.
- -1, --resync Performs the resync run.
- Warning: Path1 files may overwrite Path2 versions.
- Consider using `--verbose` or `--dry-run` first.
- --ignore-listing-checksum Do not use checksums for listings
- (add --ignore-checksum to additionally skip post-copy checksum checks)
- --resilient Allow future runs to retry after certain less-serious errors,
- instead of requiring --resync. Use at your own risk!
- --localtime Use local time in listings (default: UTC)
- --no-cleanup Retain working files (useful for troubleshooting and testing).
- --workdir PATH Use custom working directory (useful for testing).
- (default: `~/.cache/rclone/bisync`)
- -n, --dry-run Go through the motions - No files are copied/deleted.
- -v, --verbose Increases logging verbosity.
- May be specified more than once for more details.
- -h, --help help for bisync
-```
+Set if you wish to use CDN download links.
-Arbitrary rclone flags may be specified on the
-[bisync command line](https://rclone.org/commands/rclone_bisync/), for example
-`rclone bisync ./testdir/path1/ gdrive:testdir/path2/ --drive-skip-gdocs -v -v --timeout 10s`
-Note that interactions of various rclone flags with bisync process flow
-has not been fully tested yet.
+Properties:
-### Paths
+- Config: cdn
+- Env Var: RCLONE_FICHIER_CDN
+- Type: bool
+- Default: false
-Path1 and Path2 arguments may be references to any mix of local directory
-paths (absolute or relative), UNC paths (`//server/share/path`),
-Windows drive paths (with a drive letter and `:`) or configured
-[remotes](https://rclone.org/docs/#syntax-of-remote-paths) with optional subdirectory paths.
-Cloud references are distinguished by having a `:` in the argument
-(see [Windows support](#windows) below).
+#### --fichier-encoding
-Path1 and Path2 are treated equally, in that neither has priority for
-file changes (except during [`--resync`](#resync)), and access efficiency does not change whether a remote
-is on Path1 or Path2.
+The encoding for the backend.
-The listings in bisync working directory (default: `~/.cache/rclone/bisync`)
-are named based on the Path1 and Path2 arguments so that separate syncs
-to individual directories within the tree may be set up, e.g.:
-`path_to_local_tree..dropbox_subdir.lst`.
+See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info.
-Any empty directories after the sync on both the Path1 and Path2
-filesystems are not deleted by default, unless `--create-empty-src-dirs` is specified.
-If the `--remove-empty-dirs` flag is specified, then both paths will have ALL empty directories purged
-as the last step in the process.
+Properties:
-## Command-line flags
+- Config: encoding
+- Env Var: RCLONE_FICHIER_ENCODING
+- Type: Encoding
+- Default: Slash,LtGt,DoubleQuote,SingleQuote,BackQuote,Dollar,BackSlash,Del,Ctl,LeftSpace,RightSpace,InvalidUtf8,Dot
-#### --resync
-This will effectively make both Path1 and Path2 filesystems contain a
-matching superset of all files. Path2 files that do not exist in Path1 will
-be copied to Path1, and the process will then copy the Path1 tree to Path2.
-The `--resync` sequence is roughly equivalent to:
-```
-rclone copy Path2 Path1 --ignore-existing
-rclone copy Path1 Path2
-```
-Or, if using `--create-empty-src-dirs`:
-```
-rclone copy Path2 Path1 --ignore-existing
-rclone copy Path1 Path2 --create-empty-src-dirs
-rclone copy Path2 Path1 --create-empty-src-dirs
-```
+## Limitations
-The base directories on both Path1 and Path2 filesystems must exist
-or bisync will fail. This is required for safety - that bisync can verify
-that both paths are valid.
+`rclone about` is not supported by the 1Fichier backend. Backends without
+this capability cannot determine free space for an rclone mount or
+use policy `mfs` (most free space) as a member of an rclone union
+remote.
-When using `--resync`, a newer version of a file on the Path2 filesystem
-will be overwritten by the Path1 filesystem version.
-(Note that this is [NOT entirely symmetrical](https://github.com/rclone/rclone/issues/5681#issuecomment-938761815).)
-Carefully evaluate deltas using [--dry-run](https://rclone.org/flags/#non-backend-flags).
+See [List of backends that do not support rclone about](https://rclone.org/overview/#optional-features) and [rclone about](https://rclone.org/commands/rclone_about/)
-[//]: # (I reverted a recent change in the above paragraph, as it was incorrect.
-https://github.com/rclone/rclone/commit/dd72aff98a46c6e20848ac7ae5f7b19d45802493 )
+# Alias
-For a resync run, one of the paths may be empty (no files in the path tree).
-The resync run should result in files on both paths, else a normal non-resync
-run will fail.
+The `alias` remote provides a new name for another remote.
-For a non-resync run, either path being empty (no files in the tree) fails with
-`Empty current PathN listing. Cannot sync to an empty directory: X.pathN.lst`
-This is a safety check that an unexpected empty path does not result in
-deleting **everything** in the other path.
+Paths may be as deep as required or a local path,
+e.g. `remote:directory/subdirectory` or `/directory/subdirectory`.
-#### --check-access
+During the initial setup with `rclone config` you will specify the target
+remote. The target remote can either be a local path or another remote.
-Access check files are an additional safety measure against data loss.
-bisync will ensure it can find matching `RCLONE_TEST` files in the same places
-in the Path1 and Path2 filesystems.
-`RCLONE_TEST` files are not generated automatically.
-For `--check-access` to succeed, you must first either:
-**A)** Place one or more `RCLONE_TEST` files in both systems, or
-**B)** Set `--check-filename` to a filename already in use in various locations
-throughout your sync'd fileset. Recommended methods for **A)** include:
-* `rclone touch Path1/RCLONE_TEST` (create a new file)
-* `rclone copyto Path1/RCLONE_TEST Path2/RCLONE_TEST` (copy an existing file)
-* `rclone copy Path1/RCLONE_TEST Path2/RCLONE_TEST --include "RCLONE_TEST"` (copy multiple files at once, recursively)
-* create the files manually (outside of rclone)
-* run `bisync` once *without* `--check-access` to set matching files on both filesystems
-will also work, but is not preferred, due to potential for user error
-(you are temporarily disabling the safety feature).
+Subfolders can be used in target remote. Assume an alias remote named `backup`
+with the target `mydrive:private/backup`. Invoking `rclone mkdir backup:desktop`
+is exactly the same as invoking `rclone mkdir mydrive:private/backup/desktop`.
-Note that `--check-access` is still enforced on `--resync`, so `bisync --resync --check-access`
-will not work as a method of initially setting the files (this is to ensure that bisync can't
-[inadvertently circumvent its own safety switch](https://forum.rclone.org/t/bisync-bugs-and-feature-requests/37636#:~:text=3.%20%2D%2Dcheck%2Daccess%20doesn%27t%20always%20fail%20when%20it%20should).)
+There will be no special handling of paths containing `..` segments.
+Invoking `rclone mkdir backup:../desktop` is exactly the same as invoking
+`rclone mkdir mydrive:private/backup/../desktop`.
+The empty path is not allowed as a remote. To alias the current directory
+use `.` instead.
-Time stamps and file contents for `RCLONE_TEST` files are not important, just the names and locations.
-If you have symbolic links in your sync tree it is recommended to place
-`RCLONE_TEST` files in the linked-to directory tree to protect against
-bisync assuming a bunch of deleted files if the linked-to tree should not be
-accessible.
-See also the [--check-filename](--check-filename) flag.
+The target remote can also be a [connection string](https://rclone.org/docs/#connection-strings).
+This can be used to modify the config of a remote for different uses, e.g.
+the alias `myDriveTrash` with the target remote `myDrive,trashed_only:`
+can be used to only show the trashed files in `myDrive`.
-#### --check-filename
+## Configuration
-Name of the file(s) used in access health validation.
-The default `--check-filename` is `RCLONE_TEST`.
-One or more files having this filename must exist, synchronized between your
-source and destination filesets, in order for `--check-access` to succeed.
-See [--check-access](#check-access) for additional details.
+Here is an example of how to make an alias called `remote` for local folder.
+First run:
-#### --max-delete
+ rclone config
+
+This will guide you through an interactive setup process:
+
+```
+No remotes found, make a new one?
+n) New remote
+s) Set configuration password
+q) Quit config
+n/s/q> n
+name> remote
+Type of storage to configure.
+Choose a number from below, or type in your own value
+[snip]
+XX / Alias for an existing remote
+ \ "alias"
+[snip]
+Storage> alias
+Remote or path to alias.
+Can be "myremote:path/to/dir", "myremote:bucket", "myremote:" or "/local/path".
+remote> /mnt/storage/backup
+Remote config
+--------------------
+[remote]
+remote = /mnt/storage/backup
+--------------------
+y) Yes this is OK
+e) Edit this remote
+d) Delete this remote
+y/e/d> y
+Current remotes:
-As a safety check, if greater than the `--max-delete` percent of files were
-deleted on either the Path1 or Path2 filesystem, then bisync will abort with
-a warning message, without making any changes.
-The default `--max-delete` is `50%`.
-One way to trigger this limit is to rename a directory that contains more
-than half of your files. This will appear to bisync as a bunch of deleted
-files and a bunch of new files.
-This safety check is intended to block bisync from deleting all of the
-files on both filesystems due to a temporary network access issue, or if
-the user had inadvertently deleted the files on one side or the other.
-To force the sync, either set a different delete percentage limit,
-e.g. `--max-delete 75` (allows up to 75% deletion), or use `--force`
-to bypass the check.
+Name Type
+==== ====
+remote alias
-Also see the [all files changed](#all-files-changed) check.
+e) Edit existing remote
+n) New remote
+d) Delete remote
+r) Rename remote
+c) Copy remote
+s) Set configuration password
+q) Quit config
+e/n/d/r/c/s/q> q
+```
-#### --filters-file {#filters-file}
+Once configured you can then use `rclone` like this,
-By using rclone filter features you can exclude file types or directory
-sub-trees from the sync.
-See the [bisync filters](#filtering) section and generic
-[--filter-from](https://rclone.org/filtering/#filter-from-read-filtering-patterns-from-a-file)
-documentation.
-An [example filters file](#example-filters-file) contains filters for
-non-allowed files for synching with Dropbox.
+List directories in top level in `/mnt/storage/backup`
-If you make changes to your filters file then bisync requires a run
-with `--resync`. This is a safety feature, which prevents existing files
-on the Path1 and/or Path2 side from seeming to disappear from view
-(since they are excluded in the new listings), which would fool bisync
-into seeing them as deleted (as compared to the prior run listings),
-and then bisync would proceed to delete them for real.
+ rclone lsd remote:
-To block this from happening, bisync calculates an MD5 hash of the filters file
-and stores the hash in a `.md5` file in the same place as your filters file.
-On the next run with `--filters-file` set, bisync re-calculates the MD5 hash
-of the current filters file and compares it to the hash stored in the `.md5` file.
-If they don't match, the run aborts with a critical error and thus forces you
-to do a `--resync`, likely avoiding a disaster.
+List all the files in `/mnt/storage/backup`
-#### --check-sync
+ rclone ls remote:
-Enabled by default, the check-sync function checks that all of the same
-files exist in both the Path1 and Path2 history listings. This _check-sync_
-integrity check is performed at the end of the sync run by default.
-Any untrapped failing copy/deletes between the two paths might result
-in differences between the two listings and in the untracked file content
-differences between the two paths. A resync run would correct the error.
+Copy another local directory to the alias directory called source
-Note that the default-enabled integrity check locally executes a load of both
-the final Path1 and Path2 listings, and thus adds to the run time of a sync.
-Using `--check-sync=false` will disable it and may significantly reduce the
-sync run times for very large numbers of files.
+ rclone copy /home/source remote:source
-The check may be run manually with `--check-sync=only`. It runs only the
-integrity check and terminates without actually synching.
-See also: [Concurrent modifications](#concurrent-modifications)
+### Standard options
+Here are the Standard options specific to alias (Alias for an existing remote).
-#### --ignore-listing-checksum
+#### --alias-remote
-By default, bisync will retrieve (or generate) checksums (for backends that support them)
-when creating the listings for both paths, and store the checksums in the listing files.
-`--ignore-listing-checksum` will disable this behavior, which may speed things up considerably,
-especially on backends (such as [local](https://rclone.org/local/)) where hashes must be computed on the fly instead of retrieved.
-Please note the following:
+Remote or path to alias.
-* While checksums are (by default) generated and stored in the listing files,
-they are NOT currently used for determining diffs (deltas).
-It is anticipated that full checksum support will be added in a future version.
-* `--ignore-listing-checksum` is NOT the same as [`--ignore-checksum`](https://rclone.org/docs/#ignore-checksum),
-and you may wish to use one or the other, or both. In a nutshell:
-`--ignore-listing-checksum` controls whether checksums are considered when scanning for diffs,
-while `--ignore-checksum` controls whether checksums are considered during the copy/sync operations that follow,
-if there ARE diffs.
-* Unless `--ignore-listing-checksum` is passed, bisync currently computes hashes for one path
-*even when there's no common hash with the other path*
-(for example, a [crypt](https://rclone.org/crypt/#modified-time-and-hashes) remote.)
-* If both paths support checksums and have a common hash,
-AND `--ignore-listing-checksum` was not specified when creating the listings,
-`--check-sync=only` can be used to compare Path1 vs. Path2 checksums (as of the time the previous listings were created.)
-However, `--check-sync=only` will NOT include checksums if the previous listings
-were generated on a run using `--ignore-listing-checksum`. For a more robust integrity check of the current state,
-consider using [`check`](commands/rclone_check/)
-(or [`cryptcheck`](https://rclone.org/commands/rclone_cryptcheck/), if at least one path is a `crypt` remote.)
+Can be "myremote:path/to/dir", "myremote:bucket", "myremote:" or "/local/path".
-#### --resilient
+Properties:
-***Caution: this is an experimental feature. Use at your own risk!***
+- Config: remote
+- Env Var: RCLONE_ALIAS_REMOTE
+- Type: string
+- Required: true
-By default, most errors or interruptions will cause bisync to abort and
-require [`--resync`](#resync) to recover. This is a safety feature,
-to prevent bisync from running again until a user checks things out.
-However, in some cases, bisync can go too far and enforce a lockout when one isn't actually necessary,
-like for certain less-serious errors that might resolve themselves on the next run.
-When `--resilient` is specified, bisync tries its best to recover and self-correct,
-and only requires `--resync` as a last resort when a human's involvement is absolutely necessary.
-The intended use case is for running bisync as a background process (such as via scheduled [cron](#cron)).
-When using `--resilient` mode, bisync will still report the error and abort,
-however it will not lock out future runs -- allowing the possibility of retrying at the next normally scheduled time,
-without requiring a `--resync` first. Examples of such retryable errors include
-access test failures, missing listing files, and filter change detections.
-These safety features will still prevent the *current* run from proceeding --
-the difference is that if conditions have improved by the time of the *next* run,
-that next run will be allowed to proceed.
-Certain more serious errors will still enforce a `--resync` lockout, even in `--resilient` mode, to prevent data loss.
-Behavior of `--resilient` may change in a future version.
+# Amazon Drive
-## Operation
+Amazon Drive, formerly known as Amazon Cloud Drive, is a cloud storage
+service run by Amazon for consumers.
-### Runtime flow details
+## Status
-bisync retains the listings of the `Path1` and `Path2` filesystems
-from the prior run.
-On each successive run it will:
+**Important:** rclone supports Amazon Drive only if you have your own
+set of API keys. Unfortunately the [Amazon Drive developer
+program](https://developer.amazon.com/amazon-drive) is now closed to
+new entries so if you don't already have your own set of keys you will
+not be able to use rclone with Amazon Drive.
-- list files on `path1` and `path2`, and check for changes on each side.
- Changes include `New`, `Newer`, `Older`, and `Deleted` files.
-- Propagate changes on `path1` to `path2`, and vice-versa.
+For the history on why rclone no longer has a set of Amazon Drive API
+keys see [the forum](https://forum.rclone.org/t/rclone-has-been-banned-from-amazon-drive/2314).
-### Safety measures
+If you happen to know anyone who works at Amazon then please ask them
+to re-instate rclone into the Amazon Drive developer program - thanks!
-- Lock file prevents multiple simultaneous runs when taking a while.
- This can be particularly useful if bisync is run by cron scheduler.
-- Handle change conflicts non-destructively by creating
- `..path1` and `..path2` file versions.
-- File system access health check using `RCLONE_TEST` files
- (see the `--check-access` flag).
-- Abort on excessive deletes - protects against a failed listing
- being interpreted as all the files were deleted.
- See the `--max-delete` and `--force` flags.
-- If something evil happens, bisync goes into a safe state to block
- damage by later runs. (See [Error Handling](#error-handling))
+## Configuration
-### Normal sync checks
+The initial setup for Amazon Drive involves getting a token from
+Amazon which you need to do in your browser. `rclone config` walks
+you through it.
- Type | Description | Result | Implementation
---------------|-----------------------------------------------|--------------------------|-----------------------------
-Path2 new | File is new on Path2, does not exist on Path1 | Path2 version survives | `rclone copy` Path2 to Path1
-Path2 newer | File is newer on Path2, unchanged on Path1 | Path2 version survives | `rclone copy` Path2 to Path1
-Path2 deleted | File is deleted on Path2, unchanged on Path1 | File is deleted | `rclone delete` Path1
-Path1 new | File is new on Path1, does not exist on Path2 | Path1 version survives | `rclone copy` Path1 to Path2
-Path1 newer | File is newer on Path1, unchanged on Path2 | Path1 version survives | `rclone copy` Path1 to Path2
-Path1 older | File is older on Path1, unchanged on Path2 | _Path1 version survives_ | `rclone copy` Path1 to Path2
-Path2 older | File is older on Path2, unchanged on Path1 | _Path2 version survives_ | `rclone copy` Path2 to Path1
-Path1 deleted | File no longer exists on Path1 | File is deleted | `rclone delete` Path2
+The configuration process for Amazon Drive may involve using an [oauth
+proxy](https://github.com/ncw/oauthproxy). This is used to keep the
+Amazon credentials out of the source code. The proxy runs in Google's
+very secure App Engine environment and doesn't store any credentials
+which pass through it.
-### Unusual sync checks
+Since rclone doesn't currently have its own Amazon Drive credentials
+so you will either need to have your own `client_id` and
+`client_secret` with Amazon Drive, or use a third-party oauth proxy
+in which case you will need to enter `client_id`, `client_secret`,
+`auth_url` and `token_url`.
- Type | Description | Result | Implementation
---------------------------------|---------------------------------------|------------------------------------|-----------------------
-Path1 new/changed AND Path2 new/changed AND Path1 == Path2 | File is new/changed on Path1 AND new/changed on Path2 AND Path1 version is currently identical to Path2 | No change | None
-Path1 new AND Path2 new | File is new on Path1 AND new on Path2 (and Path1 version is NOT identical to Path2) | Files renamed to _Path1 and _Path2 | `rclone copy` _Path2 file to Path1, `rclone copy` _Path1 file to Path2
-Path2 newer AND Path1 changed | File is newer on Path2 AND also changed (newer/older/size) on Path1 (and Path1 version is NOT identical to Path2) | Files renamed to _Path1 and _Path2 | `rclone copy` _Path2 file to Path1, `rclone copy` _Path1 file to Path2
-Path2 newer AND Path1 deleted | File is newer on Path2 AND also deleted on Path1 | Path2 version survives | `rclone copy` Path2 to Path1
-Path2 deleted AND Path1 changed | File is deleted on Path2 AND changed (newer/older/size) on Path1 | Path1 version survives |`rclone copy` Path1 to Path2
-Path1 deleted AND Path2 changed | File is deleted on Path1 AND changed (newer/older/size) on Path2 | Path2 version survives | `rclone copy` Path2 to Path1
+Note also if you are not using Amazon's `auth_url` and `token_url`,
+(ie you filled in something for those) then if setting up on a remote
+machine you can only use the [copying the config method of
+configuration](https://rclone.org/remote_setup/#configuring-by-copying-the-config-file)
+- `rclone authorize` will not work.
-As of `rclone v1.64`, bisync is now better at detecting *false positive* sync conflicts,
-which would previously have resulted in unnecessary renames and duplicates.
-Now, when bisync comes to a file that it wants to rename (because it is new/changed on both sides),
-it first checks whether the Path1 and Path2 versions are currently *identical*
-(using the same underlying function as [`check`](commands/rclone_check/).)
-If bisync concludes that the files are identical, it will skip them and move on.
-Otherwise, it will create renamed `..Path1` and `..Path2` duplicates, as before.
-This behavior also [improves the experience of renaming directories](https://forum.rclone.org/t/bisync-bugs-and-feature-requests/37636#:~:text=Renamed%20directories),
-as a `--resync` is no longer required, so long as the same change has been made on both sides.
+Here is an example of how to make a remote called `remote`. First run:
-### All files changed check {#all-files-changed}
+ rclone config
-If _all_ prior existing files on either of the filesystems have changed
-(e.g. timestamps have changed due to changing the system's timezone)
-then bisync will abort without making any changes.
-Any new files are not considered for this check. You could use `--force`
-to force the sync (whichever side has the changed timestamp files wins).
-Alternately, a `--resync` may be used (Path1 versions will be pushed
-to Path2). Consider the situation carefully and perhaps use `--dry-run`
-before you commit to the changes.
+This will guide you through an interactive setup process:
-### Modification time
+```
+No remotes found, make a new one?
+n) New remote
+r) Rename remote
+c) Copy remote
+s) Set configuration password
+q) Quit config
+n/r/c/s/q> n
+name> remote
+Type of storage to configure.
+Choose a number from below, or type in your own value
+[snip]
+XX / Amazon Drive
+ \ "amazon cloud drive"
+[snip]
+Storage> amazon cloud drive
+Amazon Application Client Id - required.
+client_id> your client ID goes here
+Amazon Application Client Secret - required.
+client_secret> your client secret goes here
+Auth server URL - leave blank to use Amazon's.
+auth_url> Optional auth URL
+Token server url - leave blank to use Amazon's.
+token_url> Optional token URL
+Remote config
+Make sure your Redirect URL is set to "http://127.0.0.1:53682/" in your custom config.
+Use web browser to automatically authenticate rclone with remote?
+ * Say Y if the machine running rclone has a web browser you can use
+ * Say N if running rclone on a (remote) machine without web browser access
+If not sure try Y. If Y failed, try N.
+y) Yes
+n) No
+y/n> y
+If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth
+Log in and authorize rclone for access
+Waiting for code...
+Got code
+--------------------
+[remote]
+client_id = your client ID goes here
+client_secret = your client secret goes here
+auth_url = Optional auth URL
+token_url = Optional token URL
+token = {"access_token":"xxxxxxxxxxxxxxxxxxxxxxx","token_type":"bearer","refresh_token":"xxxxxxxxxxxxxxxxxx","expiry":"2015-09-06T16:07:39.658438471+01:00"}
+--------------------
+y) Yes this is OK
+e) Edit this remote
+d) Delete this remote
+y/e/d> y
+```
-Bisync relies on file timestamps to identify changed files and will
-_refuse_ to operate if backend lacks the modification time support.
+See the [remote setup docs](https://rclone.org/remote_setup/) for how to set it up on a
+machine with no Internet browser available.
-If you or your application should change the content of a file
-without changing the modification time then bisync will _not_
-notice the change, and thus will not copy it to the other side.
+Note that rclone runs a webserver on your local machine to collect the
+token as returned from Amazon. This only runs from the moment it
+opens your browser to the moment you get back the verification
+code. This is on `http://127.0.0.1:53682/` and this it may require
+you to unblock it temporarily if you are running a host firewall.
-Note that on some cloud storage systems it is not possible to have file
-timestamps that match _precisely_ between the local and other filesystems.
+Once configured you can then use `rclone` like this,
-Bisync's approach to this problem is by tracking the changes on each side
-_separately_ over time with a local database of files in that side then
-applying the resulting changes on the other side.
+List directories in top level of your Amazon Drive
-### Error handling {#error-handling}
+ rclone lsd remote:
-Certain bisync critical errors, such as file copy/move failing, will result in
-a bisync lockout of following runs. The lockout is asserted because the sync
-status and history of the Path1 and Path2 filesystems cannot be trusted,
-so it is safer to block any further changes until someone checks things out.
-The recovery is to do a `--resync` again.
+List all the files in your Amazon Drive
-It is recommended to use `--resync --dry-run --verbose` initially and
-_carefully_ review what changes will be made before running the `--resync`
-without `--dry-run`.
+ rclone ls remote:
-Most of these events come up due to an error status from an internal call.
-On such a critical error the `{...}.path1.lst` and `{...}.path2.lst`
-listing files are renamed to extension `.lst-err`, which blocks any future
-bisync runs (since the normal `.lst` files are not found).
-Bisync keeps them under `bisync` subdirectory of the rclone cache directory,
-typically at `${HOME}/.cache/rclone/bisync/` on Linux.
+To copy a local directory to an Amazon Drive directory called backup
-Some errors are considered temporary and re-running the bisync is not blocked.
-The _critical return_ blocks further bisync runs.
+ rclone copy /home/source remote:backup
-See also: [`--resilient`](#resilient)
+### Modification times and hashes
-### Lock file
+Amazon Drive doesn't allow modification times to be changed via
+the API so these won't be accurate or used for syncing.
-When bisync is running, a lock file is created in the bisync working directory,
-typically at `~/.cache/rclone/bisync/PATH1..PATH2.lck` on Linux.
-If bisync should crash or hang, the lock file will remain in place and block
-any further runs of bisync _for the same paths_.
-Delete the lock file as part of debugging the situation.
-The lock file effectively blocks follow-on (e.g., scheduled by _cron_) runs
-when the prior invocation is taking a long time.
-The lock file contains _PID_ of the blocking process, which may help in debug.
+It does support the MD5 hash algorithm, so for a more accurate sync,
+you can use the `--checksum` flag.
-**Note**
-that while concurrent bisync runs are allowed, _be very cautious_
-that there is no overlap in the trees being synched between concurrent runs,
-lest there be replicated files, deleted files and general mayhem.
+### Restricted filename characters
-### Return codes
+| Character | Value | Replacement |
+| --------- |:-----:|:-----------:|
+| NUL | 0x00 | †|
+| / | 0x2F | ï¼ |
-`rclone bisync` returns the following codes to calling program:
-- `0` on a successful run,
-- `1` for a non-critical failing run (a rerun may be successful),
-- `2` for a critically aborted run (requires a `--resync` to recover).
+Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8),
+as they can't be used in JSON strings.
-## Limitations
+### Deleting files
-### Supported backends
+Any files you delete with rclone will end up in the trash. Amazon
+don't provide an API to permanently delete files, nor to empty the
+trash, so you will have to do that with one of Amazon's apps or via
+the Amazon Drive website. As of November 17, 2016, files are
+automatically deleted by Amazon from the trash after 30 days.
-Bisync is considered _BETA_ and has been tested with the following backends:
-- Local filesystem
-- Google Drive
-- Dropbox
-- OneDrive
-- S3
-- SFTP
-- Yandex Disk
+### Using with non `.com` Amazon accounts
-It has not been fully tested with other services yet.
-If it works, or sorta works, please let us know and we'll update the list.
-Run the test suite to check for proper operation as described below.
+Let's say you usually use `amazon.co.uk`. When you authenticate with
+rclone it will take you to an `amazon.com` page to log in. Your
+`amazon.co.uk` email and password should work here just fine.
-First release of `rclone bisync` requires that underlying backend supports
-the modification time feature and will refuse to run otherwise.
-This limitation will be lifted in a future `rclone bisync` release.
-### Concurrent modifications
+### Standard options
-When using **Local, FTP or SFTP** remotes rclone does not create _temporary_
-files at the destination when copying, and thus if the connection is lost
-the created file may be corrupt, which will likely propagate back to the
-original path on the next sync, resulting in data loss.
-This will be solved in a future release, there is no workaround at the moment.
+Here are the Standard options specific to amazon cloud drive (Amazon Drive).
-Files that **change during** a bisync run may result in data loss.
-This has been seen in a highly dynamic environment, where the filesystem
-is getting hammered by running processes during the sync.
-The currently recommended solution is to sync at quiet times or [filter out](#filtering)
-unnecessary directories and files.
+#### --acd-client-id
-As an [alternative approach](https://forum.rclone.org/t/bisync-bugs-and-feature-requests/37636#:~:text=scans%2C%20to%20avoid-,errors%20if%20files%20changed%20during%20sync,-Given%20the%20number),
-consider using `--check-sync=false` (and possibly `--resilient`) to make bisync more forgiving
-of filesystems that change during the sync.
-Be advised that this may cause bisync to miss events that occur during a bisync run,
-so it is a good idea to supplement this with a periodic independent integrity check,
-and corrective sync if diffs are found. For example, a possible sequence could look like this:
+OAuth Client Id.
-1. Normally scheduled bisync run:
+Leave blank normally.
-```
-rclone bisync Path1 Path2 -MPc --check-access --max-delete 10 --filters-file /path/to/filters.txt -v --check-sync=false --no-cleanup --ignore-listing-checksum --disable ListR --checkers=16 --drive-pacer-min-sleep=10ms --create-empty-src-dirs --resilient
-```
+Properties:
-2. Periodic independent integrity check (perhaps scheduled nightly or weekly):
+- Config: client_id
+- Env Var: RCLONE_ACD_CLIENT_ID
+- Type: string
+- Required: false
-```
-rclone check -MvPc Path1 Path2 --filter-from /path/to/filters.txt
-```
+#### --acd-client-secret
-3. If diffs are found, you have some choices to correct them.
-If one side is more up-to-date and you want to make the other side match it, you could run:
+OAuth Client Secret.
-```
-rclone sync Path1 Path2 --filter-from /path/to/filters.txt --create-empty-src-dirs -MPc -v
-```
-(or switch Path1 and Path2 to make Path2 the source-of-truth)
+Leave blank normally.
-Or, if neither side is totally up-to-date, you could run a `--resync` to bring them back into agreement
-(but remember that this could cause deleted files to re-appear.)
+Properties:
-*Note also that `rclone check` does not currently include empty directories,
-so if you want to know if any empty directories are out of sync,
-consider alternatively running the above `rclone sync` command with `--dry-run` added.
+- Config: client_secret
+- Env Var: RCLONE_ACD_CLIENT_SECRET
+- Type: string
+- Required: false
-### Empty directories
+### Advanced options
-By default, new/deleted empty directories on one path are _not_ propagated to the other side.
-This is because bisync (and rclone) natively works on files, not directories.
-However, this can be changed with the `--create-empty-src-dirs` flag, which works in
-much the same way as in [`sync`](https://rclone.org/commands/rclone_sync/) and [`copy`](https://rclone.org/commands/rclone_copy/).
-When used, empty directories created or deleted on one side will also be created or deleted on the other side.
-The following should be noted:
-* `--create-empty-src-dirs` is not compatible with `--remove-empty-dirs`. Use only one or the other (or neither).
-* It is not recommended to switch back and forth between `--create-empty-src-dirs`
-and the default (no `--create-empty-src-dirs`) without running `--resync`.
-This is because it may appear as though all directories (not just the empty ones) were created/deleted,
-when actually you've just toggled between making them visible/invisible to bisync.
-It looks scarier than it is, but it's still probably best to stick to one or the other,
-and use `--resync` when you need to switch.
+Here are the Advanced options specific to amazon cloud drive (Amazon Drive).
-### Renamed directories
+#### --acd-token
-Renaming a folder on the Path1 side results in deleting all files on
-the Path2 side and then copying all files again from Path1 to Path2.
-Bisync sees this as all files in the old directory name as deleted and all
-files in the new directory name as new.
-Currently, the most effective and efficient method of renaming a directory
-is to rename it to the same name on both sides. (As of `rclone v1.64`,
-a `--resync` is no longer required after doing so, as bisync will automatically
-detect that Path1 and Path2 are in agreement.)
+OAuth Access Token as a JSON blob.
-### `--fast-list` used by default
+Properties:
-Unlike most other rclone commands, bisync uses [`--fast-list`](https://rclone.org/docs/#fast-list) by default,
-for backends that support it. In many cases this is desirable, however,
-there are some scenarios in which bisync could be faster *without* `--fast-list`,
-and there is also a [known issue concerning Google Drive users with many empty directories](https://github.com/rclone/rclone/commit/cbf3d4356135814921382dd3285d859d15d0aa77).
-For now, the recommended way to avoid using `--fast-list` is to add `--disable ListR`
-to all bisync commands. The default behavior may change in a future version.
+- Config: token
+- Env Var: RCLONE_ACD_TOKEN
+- Type: string
+- Required: false
-### Overridden Configs
+#### --acd-auth-url
-When rclone detects an overridden config, it adds a suffix like `{ABCDE}` on the fly
-to the internal name of the remote. Bisync follows suit by including this suffix in its listing filenames.
-However, this suffix does not necessarily persist from run to run, especially if different flags are provided.
-So if next time the suffix assigned is `{FGHIJ}`, bisync will get confused,
-because it's looking for a listing file with `{FGHIJ}`, when the file it wants has `{ABCDE}`.
-As a result, it throws
-`Bisync critical error: cannot find prior Path1 or Path2 listings, likely due to critical error on prior run`
-and refuses to run again until the user runs a `--resync` (unless using `--resilient`).
-The best workaround at the moment is to set any backend-specific flags in the [config file](https://rclone.org/commands/rclone_config/)
-instead of specifying them with command flags. (You can still override them as needed for other rclone commands.)
+Auth server URL.
-### Case sensitivity
+Leave blank to use the provider defaults.
-Synching with **case-insensitive** filesystems, such as Windows or `Box`,
-can result in file name conflicts. This will be fixed in a future release.
-The near-term workaround is to make sure that files on both sides
-don't have spelling case differences (`Smile.jpg` vs. `smile.jpg`).
+Properties:
-## Windows support {#windows}
+- Config: auth_url
+- Env Var: RCLONE_ACD_AUTH_URL
+- Type: string
+- Required: false
-Bisync has been tested on Windows 8.1, Windows 10 Pro 64-bit and on Windows
-GitHub runners.
+#### --acd-token-url
-Drive letters are allowed, including drive letters mapped to network drives
-(`rclone bisync J:\localsync GDrive:`).
-If a drive letter is omitted, the shell current drive is the default.
-Drive letters are a single character follows by `:`, so cloud names
-must be more than one character long.
+Token server url.
-Absolute paths (with or without a drive letter), and relative paths
-(with or without a drive letter) are supported.
+Leave blank to use the provider defaults.
-Working directory is created at `C:\Users\MyLogin\AppData\Local\rclone\bisync`.
+Properties:
-Note that bisync output may show a mix of forward `/` and back `\` slashes.
+- Config: token_url
+- Env Var: RCLONE_ACD_TOKEN_URL
+- Type: string
+- Required: false
-Be careful of case independent directory and file naming on Windows
-vs. case dependent Linux
+#### --acd-checkpoint
-## Filtering {#filtering}
+Checkpoint for internal polling (debug).
-See [filtering documentation](https://rclone.org/filtering/)
-for how filter rules are written and interpreted.
+Properties:
-Bisync's [`--filters-file`](#filters-file) flag slightly extends the rclone's
-[--filter-from](https://rclone.org/filtering/#filter-from-read-filtering-patterns-from-a-file)
-filtering mechanism.
-For a given bisync run you may provide _only one_ `--filters-file`.
-The `--include*`, `--exclude*`, and `--filter` flags are also supported.
+- Config: checkpoint
+- Env Var: RCLONE_ACD_CHECKPOINT
+- Type: string
+- Required: false
-### How to filter directories
+#### --acd-upload-wait-per-gb
-Filtering portions of the directory tree is a critical feature for synching.
+Additional time per GiB to wait after a failed complete upload to see if it appears.
-Examples of directory trees (always beneath the Path1/Path2 root level)
-you may want to exclude from your sync:
-- Directory trees containing only software build intermediate files.
-- Directory trees containing application temporary files and data
- such as the Windows `C:\Users\MyLogin\AppData\` tree.
-- Directory trees containing files that are large, less important,
- or are getting thrashed continuously by ongoing processes.
+Sometimes Amazon Drive gives an error when a file has been fully
+uploaded but the file appears anyway after a little while. This
+happens sometimes for files over 1 GiB in size and nearly every time for
+files bigger than 10 GiB. This parameter controls the time rclone waits
+for the file to appear.
-On the other hand, there may be only select directories that you
-actually want to sync, and exclude all others. See the
-[Example include-style filters for Windows user directories](#include-filters)
-below.
+The default value for this parameter is 3 minutes per GiB, so by
+default it will wait 3 minutes for every GiB uploaded to see if the
+file appears.
-### Filters file writing guidelines
+You can disable this feature by setting it to 0. This may cause
+conflict errors as rclone retries the failed upload but the file will
+most likely appear correctly eventually.
-1. Begin with excluding directory trees:
- - e.g. `- /AppData/`
- - `**` on the end is not necessary. Once a given directory level
- is excluded then everything beneath it won't be looked at by rclone.
- - Exclude such directories that are unneeded, are big, dynamically thrashed,
- or where there may be access permission issues.
- - Excluding such dirs first will make rclone operations (much) faster.
- - Specific files may also be excluded, as with the Dropbox exclusions
- example below.
-2. Decide if it's easier (or cleaner) to:
- - Include select directories and therefore _exclude everything else_ -- or --
- - Exclude select directories and therefore _include everything else_
-3. Include select directories:
- - Add lines like: `+ /Documents/PersonalFiles/**` to select which
- directories to include in the sync.
- - `**` on the end specifies to include the full depth of the specified tree.
- - With Include-style filters, files at the Path1/Path2 root are not included.
- They may be included with `+ /*`.
- - Place RCLONE_TEST files within these included directory trees.
- They will only be looked for in these directory trees.
- - Finish by excluding everything else by adding `- **` at the end
- of the filters file.
- - Disregard step 4.
-4. Exclude select directories:
- - Add more lines like in step 1.
- For example: `-/Desktop/tempfiles/`, or `- /testdir/`.
- Again, a `**` on the end is not necessary.
- - Do _not_ add a `- **` in the file. Without this line, everything
- will be included that has not been explicitly excluded.
- - Disregard step 3.
+These values were determined empirically by observing lots of uploads
+of big files for a range of file sizes.
-A few rules for the syntax of a filter file expanding on
-[filtering documentation](https://rclone.org/filtering/):
+Upload with the "-v" flag to see more info about what rclone is doing
+in this situation.
-- Lines may start with spaces and tabs - rclone strips leading whitespace.
-- If the first non-whitespace character is a `#` then the line is a comment
- and will be ignored.
-- Blank lines are ignored.
-- The first non-whitespace character on a filter line must be a `+` or `-`.
-- Exactly 1 space is allowed between the `+/-` and the path term.
-- Only forward slashes (`/`) are used in path terms, even on Windows.
-- The rest of the line is taken as the path term.
- Trailing whitespace is taken literally, and probably is an error.
+Properties:
-### Example include-style filters for Windows user directories {#include-filters}
+- Config: upload_wait_per_gb
+- Env Var: RCLONE_ACD_UPLOAD_WAIT_PER_GB
+- Type: Duration
+- Default: 3m0s
-This Windows _include-style_ example is based on the sync root (Path1)
-set to `C:\Users\MyLogin`. The strategy is to select specific directories
-to be synched with a network drive (Path2).
+#### --acd-templink-threshold
-- `- /AppData/` excludes an entire tree of Windows stored stuff
- that need not be synched.
- In my case, AppData has >11 GB of stuff I don't care about, and there are
- some subdirectories beneath AppData that are not accessible to my
- user login, resulting in bisync critical aborts.
-- Windows creates cache files starting with both upper and
- lowercase `NTUSER` at `C:\Users\MyLogin`. These files may be dynamic,
- locked, and are generally _don't care_.
-- There are just a few directories with _my_ data that I do want synched,
- in the form of `+ /`. By selecting only the directory trees I
- want to avoid the dozen plus directories that various apps make
- at `C:\Users\MyLogin\Documents`.
-- Include files in the root of the sync point, `C:\Users\MyLogin`,
- by adding the `+ /*` line.
-- This is an Include-style filters file, therefore it ends with `- **`
- which excludes everything not explicitly included.
+Files >= this size will be downloaded via their tempLink.
-```
-- /AppData/
-- NTUSER*
-- ntuser*
-+ /Documents/Family/**
-+ /Documents/Sketchup/**
-+ /Documents/Microcapture_Photo/**
-+ /Documents/Microcapture_Video/**
-+ /Desktop/**
-+ /Pictures/**
-+ /*
-- **
-```
+Files this size or more will be downloaded via their "tempLink". This
+is to work around a problem with Amazon Drive which blocks downloads
+of files bigger than about 10 GiB. The default for this is 9 GiB which
+shouldn't need to be changed.
-Note also that Windows implements several "library" links such as
-`C:\Users\MyLogin\My Documents\My Music` pointing to `C:\Users\MyLogin\Music`.
-rclone sees these as links, so you must add `--links` to the
-bisync command line if you which to follow these links. I find that I get
-permission errors in trying to follow the links, so I don't include the
-rclone `--links` flag, but then you get lots of `Can't follow symlink…`
-noise from rclone about not following the links. This noise can be
-quashed by adding `--quiet` to the bisync command line.
+To download files above this threshold, rclone requests a "tempLink"
+which downloads the file through a temporary URL directly from the
+underlying S3 storage.
-## Example exclude-style filters files for use with Dropbox {#exclude-filters}
+Properties:
-- Dropbox disallows synching the listed temporary and configuration/data files.
- The `- ` filters exclude these files where ever they may occur
- in the sync tree. Consider adding similar exclusions for file types
- you don't need to sync, such as core dump and software build files.
-- bisync testing creates `/testdir/` at the top level of the sync tree,
- and usually deletes the tree after the test. If a normal sync should run
- while the `/testdir/` tree exists the `--check-access` phase may fail
- due to unbalanced RCLONE_TEST files.
- The `- /testdir/` filter blocks this tree from being synched.
- You don't need this exclusion if you are not doing bisync development testing.
-- Everything else beneath the Path1/Path2 root will be synched.
-- RCLONE_TEST files may be placed anywhere within the tree, including the root.
+- Config: templink_threshold
+- Env Var: RCLONE_ACD_TEMPLINK_THRESHOLD
+- Type: SizeSuffix
+- Default: 9Gi
-### Example filters file for Dropbox {#example-filters-file}
+#### --acd-encoding
-```
-# Filter file for use with bisync
-# See https://rclone.org/filtering/ for filtering rules
-# NOTICE: If you make changes to this file you MUST do a --resync run.
-# Run with --dry-run to see what changes will be made.
+The encoding for the backend.
-# Dropbox won't sync some files so filter them away here.
-# See https://help.dropbox.com/installs-integrations/sync-uploads/files-not-syncing
-- .dropbox.attr
-- ~*.tmp
-- ~$*
-- .~*
-- desktop.ini
-- .dropbox
+See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info.
-# Used for bisync testing, so excluded from normal runs
-- /testdir/
+Properties:
-# Other example filters
-#- /TiBU/
-#- /Photos/
-```
+- Config: encoding
+- Env Var: RCLONE_ACD_ENCODING
+- Type: Encoding
+- Default: Slash,InvalidUtf8,Dot
-### How --check-access handles filters
-At the start of a bisync run, listings are gathered for Path1 and Path2
-while using the user's `--filters-file`. During the check access phase,
-bisync scans these listings for `RCLONE_TEST` files.
-Any `RCLONE_TEST` files hidden by the `--filters-file` are _not_ in the
-listings and thus not checked during the check access phase.
-## Troubleshooting {#troubleshooting}
+## Limitations
-### Reading bisync logs
+Note that Amazon Drive is case insensitive so you can't have a
+file called "Hello.doc" and one called "hello.doc".
-Here are two normal runs. The first one has a newer file on the remote.
-The second has no deltas between local and remote.
+Amazon Drive has rate limiting so you may notice errors in the
+sync (429 errors). rclone will automatically retry the sync up to 3
+times by default (see `--retries` flag) which should hopefully work
+around this problem.
-```
-2021/05/16 00:24:38 INFO : Synching Path1 "/path/to/local/tree/" with Path2 "dropbox:/"
-2021/05/16 00:24:38 INFO : Path1 checking for diffs
-2021/05/16 00:24:38 INFO : - Path1 File is new - file.txt
-2021/05/16 00:24:38 INFO : Path1: 1 changes: 1 new, 0 newer, 0 older, 0 deleted
-2021/05/16 00:24:38 INFO : Path2 checking for diffs
-2021/05/16 00:24:38 INFO : Applying changes
-2021/05/16 00:24:38 INFO : - Path1 Queue copy to Path2 - dropbox:/file.txt
-2021/05/16 00:24:38 INFO : - Path1 Do queued copies to - Path2
-2021/05/16 00:24:38 INFO : Updating listings
-2021/05/16 00:24:38 INFO : Validating listings for Path1 "/path/to/local/tree/" vs Path2 "dropbox:/"
-2021/05/16 00:24:38 INFO : Bisync successful
+Amazon Drive has an internal limit of file sizes that can be uploaded
+to the service. This limit is not officially published, but all files
+larger than this will fail.
-2021/05/16 00:36:52 INFO : Synching Path1 "/path/to/local/tree/" with Path2 "dropbox:/"
-2021/05/16 00:36:52 INFO : Path1 checking for diffs
-2021/05/16 00:36:52 INFO : Path2 checking for diffs
-2021/05/16 00:36:52 INFO : No changes found
-2021/05/16 00:36:52 INFO : Updating listings
-2021/05/16 00:36:52 INFO : Validating listings for Path1 "/path/to/local/tree/" vs Path2 "dropbox:/"
-2021/05/16 00:36:52 INFO : Bisync successful
-```
+At the time of writing (Jan 2016) is in the area of 50 GiB per file.
+This means that larger files are likely to fail.
-### Dry run oddity
+Unfortunately there is no way for rclone to see that this failure is
+because of file size, so it will retry the operation, as any other
+failure. To avoid this problem, use `--max-size 50000M` option to limit
+the maximum size of uploaded files. Note that `--max-size` does not split
+files into segments, it only ignores files over this size.
-The `--dry-run` messages may indicate that it would try to delete some files.
-For example, if a file is new on Path2 and does not exist on Path1 then
-it would normally be copied to Path1, but with `--dry-run` enabled those
-copies don't happen, which leads to the attempted delete on Path2,
-blocked again by --dry-run: `... Not deleting as --dry-run`.
+`rclone about` is not supported by the Amazon Drive backend. Backends without
+this capability cannot determine free space for an rclone mount or
+use policy `mfs` (most free space) as a member of an rclone union
+remote.
-This whole confusing situation is an artifact of the `--dry-run` flag.
-Scrutinize the proposed deletes carefully, and if the files would have been
-copied to Path1 then the threatened deletes on Path2 may be disregarded.
+See [List of backends that do not support rclone about](https://rclone.org/overview/#optional-features) and [rclone about](https://rclone.org/commands/rclone_about/)
-### Retries
+# Amazon S3 Storage Providers
-Rclone has built-in retries. If you run with `--verbose` you'll see
-error and retry messages such as shown below. This is usually not a bug.
-If at the end of the run, you see `Bisync successful` and not
-`Bisync critical error` or `Bisync aborted` then the run was successful,
-and you can ignore the error messages.
+The S3 backend can be used with a number of different providers:
-The following run shows an intermittent fail. Lines _5_ and _6- are
-low-level messages. Line _6_ is a bubbled-up _warning_ message, conveying
-the error. Rclone normally retries failing commands, so there may be
-numerous such messages in the log.
-Since there are no final error/warning messages on line _7_, rclone has
-recovered from failure after a retry, and the overall sync was successful.
+- AWS S3
+- Alibaba Cloud (Aliyun) Object Storage System (OSS)
+- Ceph
+- China Mobile Ecloud Elastic Object Storage (EOS)
+- Cloudflare R2
+- Arvan Cloud Object Storage (AOS)
+- DigitalOcean Spaces
+- Dreamhost
+- GCS
+- Huawei OBS
+- IBM COS S3
+- IDrive e2
+- IONOS Cloud
+- Leviia Object Storage
+- Liara Object Storage
+- Linode Object Storage
+- Minio
+- Petabox
+- Qiniu Cloud Object Storage (Kodo)
+- RackCorp Object Storage
+- Rclone Serve S3
+- Scaleway
+- Seagate Lyve Cloud
+- SeaweedFS
+- StackPath
+- Storj
+- Synology C2 Object Storage
+- Tencent Cloud Object Storage (COS)
+- Wasabi
-```
-1: 2021/05/14 00:44:12 INFO : Synching Path1 "/path/to/local/tree" with Path2 "dropbox:"
-2: 2021/05/14 00:44:12 INFO : Path1 checking for diffs
-3: 2021/05/14 00:44:12 INFO : Path2 checking for diffs
-4: 2021/05/14 00:44:12 INFO : Path2: 113 changes: 22 new, 0 newer, 0 older, 91 deleted
-5: 2021/05/14 00:44:12 ERROR : /path/to/local/tree/objects/af: error listing: unexpected end of JSON input
-6: 2021/05/14 00:44:12 NOTICE: WARNING listing try 1 failed. - dropbox:
-7: 2021/05/14 00:44:12 INFO : Bisync successful
-```
-This log shows a _Critical failure_ which requires a `--resync` to recover from.
-See the [Runtime Error Handling](#error-handling) section.
+Paths are specified as `remote:bucket` (or `remote:` for the `lsd`
+command.) You may put subdirectories in too, e.g. `remote:bucket/path/to/dir`.
-```
-2021/05/12 00:49:40 INFO : Google drive root '': Waiting for checks to finish
-2021/05/12 00:49:40 INFO : Google drive root '': Waiting for transfers to finish
-2021/05/12 00:49:40 INFO : Google drive root '': not deleting files as there were IO errors
-2021/05/12 00:49:40 ERROR : Attempt 3/3 failed with 3 errors and: not deleting files as there were IO errors
-2021/05/12 00:49:40 ERROR : Failed to sync: not deleting files as there were IO errors
-2021/05/12 00:49:40 NOTICE: WARNING rclone sync try 3 failed. - /path/to/local/tree/
-2021/05/12 00:49:40 ERROR : Bisync aborted. Must run --resync to recover.
-```
+Once you have made a remote (see the provider specific section above)
+you can use it like this:
-### Denied downloads of "infected" or "abusive" files
+See all buckets
-Google Drive has a filter for certain file types (`.exe`, `.apk`, et cetera)
-that by default cannot be copied from Google Drive to the local filesystem.
-If you are having problems, run with `--verbose` to see specifically which
-files are generating complaints. If the error is
-`This file has been identified as malware or spam and cannot be downloaded`,
-consider using the flag
-[--drive-acknowledge-abuse](https://rclone.org/drive/#drive-acknowledge-abuse).
+ rclone lsd remote:
-### Google Doc files
+Make a new bucket
-Google docs exist as virtual files on Google Drive and cannot be transferred
-to other filesystems natively. While it is possible to export a Google doc to
-a normal file (with `.xlsx` extension, for example), it is not possible
-to import a normal file back into a Google document.
+ rclone mkdir remote:bucket
-Bisync's handling of Google Doc files is to flag them in the run log output
-for user's attention and ignore them for any file transfers, deletes, or syncs.
-They will show up with a length of `-1` in the listings.
-This bisync run is otherwise successful:
+List the contents of a bucket
-```
-2021/05/11 08:23:15 INFO : Synching Path1 "/path/to/local/tree/base/" with Path2 "GDrive:"
-2021/05/11 08:23:15 INFO : ...path2.lst-new: Ignoring incorrect line: "- -1 - - 2018-07-29T08:49:30.136000000+0000 GoogleDoc.docx"
-2021/05/11 08:23:15 INFO : Bisync successful
-```
+ rclone ls remote:bucket
-## Usage examples
+Sync `/home/local/directory` to the remote bucket, deleting any excess
+files in the bucket.
-### Cron {#cron}
+ rclone sync --interactive /home/local/directory remote:bucket
-Rclone does not yet have a built-in capability to monitor the local file
-system for changes and must be blindly run periodically.
-On Windows this can be done using a _Task Scheduler_,
-on Linux you can use _Cron_ which is described below.
+## Configuration
-The 1st example runs a sync every 5 minutes between a local directory
-and an OwnCloud server, with output logged to a runlog file:
+Here is an example of making an s3 configuration for the AWS S3 provider.
+Most applies to the other providers as well, any differences are described [below](#providers).
-```
-# Minute (0-59)
-# Hour (0-23)
-# Day of Month (1-31)
-# Month (1-12 or Jan-Dec)
-# Day of Week (0-6 or Sun-Sat)
-# Command
- */5 * * * * /path/to/rclone bisync /local/files MyCloud: --check-access --filters-file /path/to/bysync-filters.txt --log-file /path/to//bisync.log
-```
+First run
-See [crontab syntax](https://www.man7.org/linux/man-pages/man1/crontab.1p.html#INPUT_FILES)
-for the details of crontab time interval expressions.
+ rclone config
-If you run `rclone bisync` as a cron job, redirect stdout/stderr to a file.
-The 2nd example runs a sync to Dropbox every hour and logs all stdout (via the `>>`)
-and stderr (via `2>&1`) to a log file.
+This will guide you through an interactive setup process.
```
-0 * * * * /path/to/rclone bisync /path/to/local/dropbox Dropbox: --check-access --filters-file /home/user/filters.txt >> /path/to/logs/dropbox-run.log 2>&1
+No remotes found, make a new one?
+n) New remote
+s) Set configuration password
+q) Quit config
+n/s/q> n
+name> remote
+Type of storage to configure.
+Choose a number from below, or type in your own value
+[snip]
+XX / Amazon S3 Compliant Storage Providers including AWS, Ceph, ChinaMobile, ArvanCloud, Dreamhost, IBM COS, Liara, Minio, and Tencent COS
+ \ "s3"
+[snip]
+Storage> s3
+Choose your S3 provider.
+Choose a number from below, or type in your own value
+ 1 / Amazon Web Services (AWS) S3
+ \ "AWS"
+ 2 / Ceph Object Storage
+ \ "Ceph"
+ 3 / DigitalOcean Spaces
+ \ "DigitalOcean"
+ 4 / Dreamhost DreamObjects
+ \ "Dreamhost"
+ 5 / IBM COS S3
+ \ "IBMCOS"
+ 6 / Minio Object Storage
+ \ "Minio"
+ 7 / Wasabi Object Storage
+ \ "Wasabi"
+ 8 / Any other S3 compatible provider
+ \ "Other"
+provider> 1
+Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars). Only applies if access_key_id and secret_access_key is blank.
+Choose a number from below, or type in your own value
+ 1 / Enter AWS credentials in the next step
+ \ "false"
+ 2 / Get AWS credentials from the environment (env vars or IAM)
+ \ "true"
+env_auth> 1
+AWS Access Key ID - leave blank for anonymous access or runtime credentials.
+access_key_id> XXX
+AWS Secret Access Key (password) - leave blank for anonymous access or runtime credentials.
+secret_access_key> YYY
+Region to connect to.
+Choose a number from below, or type in your own value
+ / The default endpoint - a good choice if you are unsure.
+ 1 | US Region, Northern Virginia, or Pacific Northwest.
+ | Leave location constraint empty.
+ \ "us-east-1"
+ / US East (Ohio) Region
+ 2 | Needs location constraint us-east-2.
+ \ "us-east-2"
+ / US West (Oregon) Region
+ 3 | Needs location constraint us-west-2.
+ \ "us-west-2"
+ / US West (Northern California) Region
+ 4 | Needs location constraint us-west-1.
+ \ "us-west-1"
+ / Canada (Central) Region
+ 5 | Needs location constraint ca-central-1.
+ \ "ca-central-1"
+ / EU (Ireland) Region
+ 6 | Needs location constraint EU or eu-west-1.
+ \ "eu-west-1"
+ / EU (London) Region
+ 7 | Needs location constraint eu-west-2.
+ \ "eu-west-2"
+ / EU (Frankfurt) Region
+ 8 | Needs location constraint eu-central-1.
+ \ "eu-central-1"
+ / Asia Pacific (Singapore) Region
+ 9 | Needs location constraint ap-southeast-1.
+ \ "ap-southeast-1"
+ / Asia Pacific (Sydney) Region
+10 | Needs location constraint ap-southeast-2.
+ \ "ap-southeast-2"
+ / Asia Pacific (Tokyo) Region
+11 | Needs location constraint ap-northeast-1.
+ \ "ap-northeast-1"
+ / Asia Pacific (Seoul)
+12 | Needs location constraint ap-northeast-2.
+ \ "ap-northeast-2"
+ / Asia Pacific (Mumbai)
+13 | Needs location constraint ap-south-1.
+ \ "ap-south-1"
+ / Asia Pacific (Hong Kong) Region
+14 | Needs location constraint ap-east-1.
+ \ "ap-east-1"
+ / South America (Sao Paulo) Region
+15 | Needs location constraint sa-east-1.
+ \ "sa-east-1"
+region> 1
+Endpoint for S3 API.
+Leave blank if using AWS to use the default endpoint for the region.
+endpoint>
+Location constraint - must be set to match the Region. Used when creating buckets only.
+Choose a number from below, or type in your own value
+ 1 / Empty for US Region, Northern Virginia, or Pacific Northwest.
+ \ ""
+ 2 / US East (Ohio) Region.
+ \ "us-east-2"
+ 3 / US West (Oregon) Region.
+ \ "us-west-2"
+ 4 / US West (Northern California) Region.
+ \ "us-west-1"
+ 5 / Canada (Central) Region.
+ \ "ca-central-1"
+ 6 / EU (Ireland) Region.
+ \ "eu-west-1"
+ 7 / EU (London) Region.
+ \ "eu-west-2"
+ 8 / EU Region.
+ \ "EU"
+ 9 / Asia Pacific (Singapore) Region.
+ \ "ap-southeast-1"
+10 / Asia Pacific (Sydney) Region.
+ \ "ap-southeast-2"
+11 / Asia Pacific (Tokyo) Region.
+ \ "ap-northeast-1"
+12 / Asia Pacific (Seoul)
+ \ "ap-northeast-2"
+13 / Asia Pacific (Mumbai)
+ \ "ap-south-1"
+14 / Asia Pacific (Hong Kong)
+ \ "ap-east-1"
+15 / South America (Sao Paulo) Region.
+ \ "sa-east-1"
+location_constraint> 1
+Canned ACL used when creating buckets and/or storing objects in S3.
+For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
+Choose a number from below, or type in your own value
+ 1 / Owner gets FULL_CONTROL. No one else has access rights (default).
+ \ "private"
+ 2 / Owner gets FULL_CONTROL. The AllUsers group gets READ access.
+ \ "public-read"
+ / Owner gets FULL_CONTROL. The AllUsers group gets READ and WRITE access.
+ 3 | Granting this on a bucket is generally not recommended.
+ \ "public-read-write"
+ 4 / Owner gets FULL_CONTROL. The AuthenticatedUsers group gets READ access.
+ \ "authenticated-read"
+ / Object owner gets FULL_CONTROL. Bucket owner gets READ access.
+ 5 | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
+ \ "bucket-owner-read"
+ / Both the object owner and the bucket owner get FULL_CONTROL over the object.
+ 6 | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
+ \ "bucket-owner-full-control"
+acl> 1
+The server-side encryption algorithm used when storing this object in S3.
+Choose a number from below, or type in your own value
+ 1 / None
+ \ ""
+ 2 / AES256
+ \ "AES256"
+server_side_encryption> 1
+The storage class to use when storing objects in S3.
+Choose a number from below, or type in your own value
+ 1 / Default
+ \ ""
+ 2 / Standard storage class
+ \ "STANDARD"
+ 3 / Reduced redundancy storage class
+ \ "REDUCED_REDUNDANCY"
+ 4 / Standard Infrequent Access storage class
+ \ "STANDARD_IA"
+ 5 / One Zone Infrequent Access storage class
+ \ "ONEZONE_IA"
+ 6 / Glacier storage class
+ \ "GLACIER"
+ 7 / Glacier Deep Archive storage class
+ \ "DEEP_ARCHIVE"
+ 8 / Intelligent-Tiering storage class
+ \ "INTELLIGENT_TIERING"
+ 9 / Glacier Instant Retrieval storage class
+ \ "GLACIER_IR"
+storage_class> 1
+Remote config
+--------------------
+[remote]
+type = s3
+provider = AWS
+env_auth = false
+access_key_id = XXX
+secret_access_key = YYY
+region = us-east-1
+endpoint =
+location_constraint =
+acl = private
+server_side_encryption =
+storage_class =
+--------------------
+y) Yes this is OK
+e) Edit this remote
+d) Delete this remote
+y/e/d>
```
-### Sharing an encrypted folder tree between hosts
+### Modification times and hashes
-bisync can keep a local folder in sync with a cloud service,
-but what if you have some highly sensitive files to be synched?
+#### Modification times
-Usage of a cloud service is for exchanging both routine and sensitive
-personal files between one's home network, one's personal notebook when on the
-road, and with one's work computer. The routine data is not sensitive.
-For the sensitive data, configure an rclone [crypt remote](https://rclone.org/crypt/) to point to
-a subdirectory within the local disk tree that is bisync'd to Dropbox,
-and then set up an bisync for this local crypt directory to a directory
-outside of the main sync tree.
+The modified time is stored as metadata on the object as
+`X-Amz-Meta-Mtime` as floating point since the epoch, accurate to 1 ns.
-### Linux server setup
+If the modification time needs to be updated rclone will attempt to perform a server
+side copy to update the modification if the object can be copied in a single part.
+In the case the object is larger than 5Gb or is in Glacier or Glacier Deep Archive
+storage the object will be uploaded rather than copied.
-- `/path/to/DBoxroot` is the root of my local sync tree.
- There are numerous subdirectories.
-- `/path/to/DBoxroot/crypt` is the root subdirectory for files
- that are encrypted. This local directory target is setup as an
- rclone crypt remote named `Dropcrypt:`.
- See [rclone.conf](#rclone-conf-snippet) snippet below.
-- `/path/to/my/unencrypted/files` is the root of my sensitive
- files - not encrypted, not within the tree synched to Dropbox.
-- To sync my local unencrypted files with the encrypted Dropbox versions
- I manually run `bisync /path/to/my/unencrypted/files DropCrypt:`.
- This step could be bundled into a script to run before and after
- the full Dropbox tree sync in the last step,
- thus actively keeping the sensitive files in sync.
-- `bisync /path/to/DBoxroot Dropbox:` runs periodically via cron,
- keeping my full local sync tree in sync with Dropbox.
+Note that reading this from the object takes an additional `HEAD`
+request as the metadata isn't returned in object listings.
-### Windows notebook setup
+#### Hashes
-- The Dropbox client runs keeping the local tree `C:\Users\MyLogin\Dropbox`
- always in sync with Dropbox. I could have used `rclone bisync` instead.
-- A separate directory tree at `C:\Users\MyLogin\Documents\DropLocal`
- hosts the tree of unencrypted files/folders.
-- To sync my local unencrypted files with the encrypted
- Dropbox versions I manually run the following command:
- `rclone bisync C:\Users\MyLogin\Documents\DropLocal Dropcrypt:`.
-- The Dropbox client then syncs the changes with Dropbox.
+For small objects which weren't uploaded as multipart uploads (objects
+sized below `--s3-upload-cutoff` if uploaded with rclone) rclone uses
+the `ETag:` header as an MD5 checksum.
-### rclone.conf snippet {#rclone-conf-snippet}
+However for objects which were uploaded as multipart uploads or with
+server side encryption (SSE-AWS or SSE-C) the `ETag` header is no
+longer the MD5 sum of the data, so rclone adds an additional piece of
+metadata `X-Amz-Meta-Md5chksum` which is a base64 encoded MD5 hash (in
+the same format as is required for `Content-MD5`). You can use base64 -d and hexdump to check this value manually:
-```
-[Dropbox]
-type = dropbox
-...
+ echo 'VWTGdNx3LyXQDfA0e2Edxw==' | base64 -d | hexdump
-[Dropcrypt]
-type = crypt
-remote = /path/to/DBoxroot/crypt # on the Linux server
-remote = C:\Users\MyLogin\Dropbox\crypt # on the Windows notebook
-filename_encryption = standard
-directory_name_encryption = true
-password = ...
-...
-```
+or you can use `rclone check` to verify the hashes are OK.
-## Testing {#testing}
+For large objects, calculating this hash can take some time so the
+addition of this hash can be disabled with `--s3-disable-checksum`.
+This will mean that these objects do not have an MD5 checksum.
-You should read this section only if you are developing for rclone.
-You need to have rclone source code locally to work with bisync tests.
+Note that reading this from the object takes an additional `HEAD`
+request as the metadata isn't returned in object listings.
-Bisync has a dedicated test framework implemented in the `bisync_test.go`
-file located in the rclone source tree. The test suite is based on the
-`go test` command. Series of tests are stored in subdirectories below the
-`cmd/bisync/testdata` directory. Individual tests can be invoked by their
-directory name, e.g.
-`go test . -case basic -remote local -remote2 gdrive: -v`
+### Reducing costs
-Tests will make a temporary folder on remote and purge it afterwards.
-If during test run there are intermittent errors and rclone retries,
-these errors will be captured and flagged as invalid MISCOMPAREs.
-Rerunning the test will let it pass. Consider such failures as noise.
+#### Avoiding HEAD requests to read the modification time
-### Test command syntax
+By default, rclone will use the modification time of objects stored in
+S3 for syncing. This is stored in object metadata which unfortunately
+takes an extra HEAD request to read which can be expensive (in time
+and money).
-```
-usage: go test ./cmd/bisync [options...]
+The modification time is used by default for all operations that
+require checking the time a file was last updated. It allows rclone to
+treat the remote more like a true filesystem, but it is inefficient on
+S3 because it requires an extra API call to retrieve the metadata.
-Options:
- -case NAME Name(s) of the test case(s) to run. Multiple names should
- be separated by commas. You can remove the `test_` prefix
- and replace `_` by `-` in test name for convenience.
- If not `all`, the name(s) should map to a directory under
- `./cmd/bisync/testdata`.
- Use `all` to run all tests (default: all)
- -remote PATH1 `local` or name of cloud service with `:` (default: local)
- -remote2 PATH2 `local` or name of cloud service with `:` (default: local)
- -no-compare Disable comparing test results with the golden directory
- (default: compare)
- -no-cleanup Disable cleanup of Path1 and Path2 testdirs.
- Useful for troubleshooting. (default: cleanup)
- -golden Store results in the golden directory (default: false)
- This flag can be used with multiple tests.
- -debug Print debug messages
- -stop-at NUM Stop test after given step number. (default: run to the end)
- Implies `-no-compare` and `-no-cleanup`, if the test really
- ends prematurely. Only meaningful for a single test case.
- -refresh-times Force refreshing the target modtime, useful for Dropbox
- (default: false)
- -verbose Run tests verbosely
-```
+The extra API calls can be avoided when syncing (using `rclone sync`
+or `rclone copy`) in a few different ways, each with its own
+tradeoffs.
-Note: unlike rclone flags which must be prefixed by double dash (`--`), the
-test command flags can be equally prefixed by a single `-` or double dash.
+- `--size-only`
+ - Only checks the size of files.
+ - Uses no extra transactions.
+ - If the file doesn't change size then rclone won't detect it has
+ changed.
+ - `rclone sync --size-only /path/to/source s3:bucket`
+- `--checksum`
+ - Checks the size and MD5 checksum of files.
+ - Uses no extra transactions.
+ - The most accurate detection of changes possible.
+ - Will cause the source to read an MD5 checksum which, if it is a
+ local disk, will cause lots of disk activity.
+ - If the source and destination are both S3 this is the
+ **recommended** flag to use for maximum efficiency.
+ - `rclone sync --checksum /path/to/source s3:bucket`
+- `--update --use-server-modtime`
+ - Uses no extra transactions.
+ - Modification time becomes the time the object was uploaded.
+ - For many operations this is sufficient to determine if it needs
+ uploading.
+ - Using `--update` along with `--use-server-modtime`, avoids the
+ extra API call and uploads files whose local modification time
+ is newer than the time it was last uploaded.
+ - Files created with timestamps in the past will be missed by the sync.
+ - `rclone sync --update --use-server-modtime /path/to/source s3:bucket`
-### Running tests
+These flags can and should be used in combination with `--fast-list` -
+see below.
-- `go test . -case basic -remote local -remote2 local`
- runs the `test_basic` test case using only the local filesystem,
- synching one local directory with another local directory.
- Test script output is to the console, while commands within scenario.txt
- have their output sent to the `.../workdir/test.log` file,
- which is finally compared to the golden copy.
-- The first argument after `go test` should be a relative name of the
- directory containing bisync source code. If you run tests right from there,
- the argument will be `.` (current directory) as in most examples below.
- If you run bisync tests from the rclone source directory, the command
- should be `go test ./cmd/bisync ...`.
-- The test engine will mangle rclone output to ensure comparability
- with golden listings and logs.
-- Test scenarios are located in `./cmd/bisync/testdata`. The test `-case`
- argument should match the full name of a subdirectory under that
- directory. Every test subdirectory name on disk must start with `test_`,
- this prefix can be omitted on command line for brevity. Also, underscores
- in the name can be replaced by dashes for convenience.
-- `go test . -remote local -remote2 local -case all` runs all tests.
-- Path1 and Path2 may either be the keyword `local`
- or may be names of configured cloud services.
- `go test . -remote gdrive: -remote2 dropbox: -case basic`
- will run the test between these two services, without transferring
- any files to the local filesystem.
-- Test run stdout and stderr console output may be directed to a file, e.g.
- `go test . -remote gdrive: -remote2 local -case all > runlog.txt 2>&1`
+If using `rclone mount` or any command using the VFS (eg `rclone
+serve`) commands then you might want to consider using the VFS flag
+`--no-modtime` which will stop rclone reading the modification time
+for every object. You could also use `--use-server-modtime` if you are
+happy with the modification times of the objects being the time of
+upload.
-### Test execution flow
+#### Avoiding GET requests to read directory listings
-1. The base setup in the `initial` directory of the testcase is applied
- on the Path1 and Path2 filesystems (via rclone copy the initial directory
- to Path1, then rclone sync Path1 to Path2).
-2. The commands in the scenario.txt file are applied, with output directed
- to the `test.log` file in the test working directory.
- Typically, the first actual command in the `scenario.txt` file is
- to do a `--resync`, which establishes the baseline
- `{...}.path1.lst` and `{...}.path2.lst` files in the test working
- directory (`.../workdir/` relative to the temporary test directory).
- Various commands and listing snapshots are done within the test.
-3. Finally, the contents of the test working directory are compared
- to the contents of the testcase's golden directory.
+Rclone's default directory traversal is to process each directory
+individually. This takes one API call per directory. Using the
+`--fast-list` flag will read all info about the objects into
+memory first using a smaller number of API calls (one per 1000
+objects). See the [rclone docs](https://rclone.org/docs/#fast-list) for more details.
-### Notes about testing
+ rclone sync --fast-list --checksum /path/to/source s3:bucket
-- Test cases are in individual directories beneath `./cmd/bisync/testdata`.
- A command line reference to a test is understood to reference a directory
- beneath `testdata`. For example,
- `go test ./cmd/bisync -case dry-run -remote gdrive: -remote2 local`
- refers to the test case in `./cmd/bisync/testdata/test_dry_run`.
-- The test working directory is located at `.../workdir` relative to a
- temporary test directory, usually under `/tmp` on Linux.
-- The local test sync tree is created at a temporary directory named
- like `bisync.XXX` under system temporary directory.
-- The remote test sync tree is located at a temporary directory
- under `/bisync.XXX/`.
-- `path1` and/or `path2` subdirectories are created in a temporary
- directory under the respective local or cloud test remote.
-- By default, the Path1 and Path2 test dirs and workdir will be deleted
- after each test run. The `-no-cleanup` flag disables purging these
- directories when validating and debugging a given test.
- These directories will be flushed before running another test,
- independent of the `-no-cleanup` usage.
-- You will likely want to add `- /testdir/` to your normal
- bisync `--filters-file` so that normal syncs do not attempt to sync
- the test temporary directories, which may have `RCLONE_TEST` miscompares
- in some testcases which would otherwise trip the `--check-access` system.
- The `--check-access` mechanism is hard-coded to ignore `RCLONE_TEST`
- files beneath `bisync/testdata`, so the test cases may reside on the
- synched tree even if there are check file mismatches in the test tree.
-- Some Dropbox tests can fail, notably printing the following message:
- `src and dst identical but can't set mod time without deleting and re-uploading`
- This is expected and happens due to the way Dropbox handles modification times.
- You should use the `-refresh-times` test flag to make up for this.
-- If Dropbox tests hit request limit for you and print error message
- `too_many_requests/...: Too many requests or write operations.`
- then follow the
- [Dropbox App ID instructions](https://rclone.org/dropbox/#get-your-own-dropbox-app-id).
+`--fast-list` trades off API transactions for memory use. As a rough
+guide rclone uses 1k of memory per object stored, so using
+`--fast-list` on a sync of a million objects will use roughly 1 GiB of
+RAM.
-### Updating golden results
+If you are only copying a small number of files into a big repository
+then using `--no-traverse` is a good idea. This finds objects directly
+instead of through directory listings. You can do a "top-up" sync very
+cheaply by using `--max-age` and `--no-traverse` to copy only recent
+files, eg
-Sometimes even a slight change in the bisync source can cause little changes
-spread around many log files. Updating them manually would be a nightmare.
+ rclone copy --max-age 24h --no-traverse /path/to/source s3:bucket
-The `-golden` flag will store the `test.log` and `*.lst` listings from each
-test case into respective golden directories. Golden results will
-automatically contain generic strings instead of local or cloud paths which
-means that they should match when run with a different cloud service.
+You'd then do a full `rclone sync` less often.
-Your normal workflow might be as follows:
-1. Git-clone the rclone sources locally
-2. Modify bisync source and check that it builds
-3. Run the whole test suite `go test ./cmd/bisync -remote local`
-4. If some tests show log difference, recheck them individually, e.g.:
- `go test ./cmd/bisync -remote local -case basic`
-5. If you are convinced with the difference, goldenize all tests at once:
- `go test ./cmd/bisync -remote local -golden`
-6. Use word diff: `git diff --word-diff ./cmd/bisync/testdata/`.
- Please note that normal line-level diff is generally useless here.
-7. Check the difference _carefully_!
-8. Commit the change (`git commit`) _only_ if you are sure.
- If unsure, save your code changes then wipe the log diffs from git:
- `git reset [--hard]`.
+Note that `--fast-list` isn't required in the top-up sync.
-### Structure of test scenarios
+#### Avoiding HEAD requests after PUT
-- `/initial/` contains a tree of files that will be set
- as the initial condition on both Path1 and Path2 testdirs.
-- `/modfiles/` contains files that will be used to
- modify the Path1 and/or Path2 filesystems.
-- `/golden/` contains the expected content of the test
- working directory (`workdir`) at the completion of the testcase.
-- `/scenario.txt` contains the body of the test, in the form of
- various commands to modify files, run bisync, and snapshot listings.
- Output from these commands is captured to `.../workdir/test.log`
- for comparison to the golden files.
+By default, rclone will HEAD every object it uploads. It does this to
+check the object got uploaded correctly.
-### Supported test commands
+You can disable this with the [--s3-no-head](#s3-no-head) option - see
+there for more details.
-- `test `
- Print the line to the console and to the `test.log`:
- `test sync is working correctly with options x, y, z`
-- `copy-listings `
- Save a copy of all `.lst` listings in the test working directory
- with the specified prefix:
- `save-listings exclude-pass-run`
-- `move-listings `
- Similar to `copy-listings` but removes the source
-- `purge-children `
- This will delete all child files and purge all child subdirs under given
- directory but keep the parent intact. This behavior is important for tests
- with Google Drive because removing and re-creating the parent would change
- its ID.
-- `delete-file `
- Delete a single file.
-- `delete-glob `
- Delete a group of files located one level deep in the given directory
- with names matching a given glob pattern.
-- `touch-glob YYYY-MM-DD `
- Change modification time on a group of files.
-- `touch-copy YYYY-MM-DD `
- Change file modification time then copy it to destination.
-- `copy-file `
- Copy a single file to given directory.
-- `copy-as `
- Similar to above but destination must include both directory
- and the new file name at destination.
-- `copy-dir ` and `sync-dir `
- Copy/sync a directory. Equivalent of `rclone copy` and `rclone sync`.
-- `list-dirs `
- Equivalent to `rclone lsf -R --dirs-only `
-- `bisync [options]`
- Runs bisync against `-remote` and `-remote2`.
+Setting this flag increases the chance for undetected upload failures.
-### Supported substitution terms
+### Versions
-- `{testdir/}` - the root dir of the testcase
-- `{datadir/}` - the `modfiles` dir under the testcase root
-- `{workdir/}` - the temporary test working directory
-- `{path1/}` - the root of the Path1 test directory tree
-- `{path2/}` - the root of the Path2 test directory tree
-- `{session}` - base name of the test listings
-- `{/}` - OS-specific path separator
-- `{spc}`, `{tab}`, `{eol}` - whitespace
-- `{chr:HH}` - raw byte with given hexadecimal code
+When bucket versioning is enabled (this can be done with rclone with
+the [`rclone backend versioning`](#versioning) command) when rclone
+uploads a new version of a file it creates a
+[new version of it](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Versioning.html)
+Likewise when you delete a file, the old version will be marked hidden
+and still be available.
-Substitution results of the terms named like `{dir/}` will end with
-`/` (or backslash on Windows), so it is not necessary to include
-slash in the usage, for example `delete-file {path1/}file1.txt`.
+Old versions of files, where available, are visible using the
+[`--s3-versions`](#s3-versions) flag.
-## Benchmarks
+It is also possible to view a bucket as it was at a certain point in
+time, using the [`--s3-version-at`](#s3-version-at) flag. This will
+show the file versions as they were at that time, showing files that
+have been deleted afterwards, and hiding files that were created
+since.
-_This section is work in progress._
+If you wish to remove all the old versions then you can use the
+[`rclone backend cleanup-hidden remote:bucket`](#cleanup-hidden)
+command which will delete all the old hidden versions of files,
+leaving the current ones intact. You can also supply a path and only
+old versions under that path will be deleted, e.g.
+`rclone backend cleanup-hidden remote:bucket/path/to/stuff`.
-Here are a few data points for scale, execution times, and memory usage.
+When you `purge` a bucket, the current and the old versions will be
+deleted then the bucket will be deleted.
-The first set of data was taken between a local disk to Dropbox.
-The [speedtest.net](https://speedtest.net) download speed was ~170 Mbps,
-and upload speed was ~10 Mbps. 500 files (~9.5 MB each) had been already
-synched. 50 files were added in a new directory, each ~9.5 MB, ~475 MB total.
+However `delete` will cause the current versions of the files to
+become hidden old versions.
-Change | Operations and times | Overall run time
---------------------------------------|--------------------------------------------------------|------------------
-500 files synched (nothing to move) | 1x listings for Path1 & Path2 | 1.5 sec
-500 files synched with --check-access | 1x listings for Path1 & Path2 | 1.5 sec
-50 new files on remote | Queued 50 copies down: 27 sec | 29 sec
-Moved local dir | Queued 50 copies up: 410 sec, 50 deletes up: 9 sec | 421 sec
-Moved remote dir | Queued 50 copies down: 31 sec, 50 deletes down: <1 sec | 33 sec
-Delete local dir | Queued 50 deletes up: 9 sec | 13 sec
+Here is a session showing the listing and retrieval of an old
+version followed by a `cleanup` of the old versions.
-This next data is from a user's application. They had ~400GB of data
-over 1.96 million files being sync'ed between a Windows local disk and some
-remote cloud. The file full path length was on average 35 characters
-(which factors into load time and RAM required).
+Show current version and all the versions with `--s3-versions` flag.
-- Loading the prior listing into memory (1.96 million files, listing file
- size 140 MB) took ~30 sec and occupied about 1 GB of RAM.
-- Getting a fresh listing of the local file system (producing the
- 140 MB output file) took about XXX sec.
-- Getting a fresh listing of the remote file system (producing the 140 MB
- output file) took about XXX sec. The network download speed was measured
- at XXX Mb/s.
-- Once the prior and current Path1 and Path2 listings were loaded (a total
- of four to be loaded, two at a time), determining the deltas was pretty
- quick (a few seconds for this test case), and the transfer time for any
- files to be copied was dominated by the network bandwidth.
+```
+$ rclone -q ls s3:cleanup-test
+ 9 one.txt
-## References
+$ rclone -q --s3-versions ls s3:cleanup-test
+ 9 one.txt
+ 8 one-v2016-07-04-141032-000.txt
+ 16 one-v2016-07-04-141003-000.txt
+ 15 one-v2016-07-02-155621-000.txt
+```
-rclone's bisync implementation was derived from
-the [rclonesync-V2](https://github.com/cjnaz/rclonesync-V2) project,
-including documentation and test mechanisms,
-with [@cjnaz](https://github.com/cjnaz)'s full support and encouragement.
+Retrieve an old version
-`rclone bisync` is similar in nature to a range of other projects:
+```
+$ rclone -q --s3-versions copy s3:cleanup-test/one-v2016-07-04-141003-000.txt /tmp
-- [unison](https://github.com/bcpierce00/unison)
-- [syncthing](https://github.com/syncthing/syncthing)
-- [cjnaz/rclonesync](https://github.com/cjnaz/rclonesync-V2)
-- [ConorWilliams/rsinc](https://github.com/ConorWilliams/rsinc)
-- [jwink3101/syncrclone](https://github.com/Jwink3101/syncrclone)
-- [DavideRossi/upback](https://github.com/DavideRossi/upback)
+$ ls -l /tmp/one-v2016-07-04-141003-000.txt
+-rw-rw-r-- 1 ncw ncw 16 Jul 2 17:46 /tmp/one-v2016-07-04-141003-000.txt
+```
-Bisync adopts the differential synchronization technique, which is
-based on keeping history of changes performed by both synchronizing sides.
-See the _Dual Shadow Method_ section in
-[Neil Fraser's article](https://neil.fraser.name/writing/sync/).
+Clean up all the old versions and show that they've gone.
-Also note a number of academic publications by
-[Benjamin Pierce](http://www.cis.upenn.edu/%7Ebcpierce/papers/index.shtml#File%20Synchronization)
-about _Unison_ and synchronization in general.
+```
+$ rclone -q backend cleanup-hidden s3:cleanup-test
-## Changelog
+$ rclone -q ls s3:cleanup-test
+ 9 one.txt
-### `v1.64`
-* Fixed an [issue](https://forum.rclone.org/t/bisync-bugs-and-feature-requests/37636#:~:text=1.%20Dry%20runs%20are%20not%20completely%20dry)
-causing dry runs to inadvertently commit filter changes
-* Fixed an [issue](https://forum.rclone.org/t/bisync-bugs-and-feature-requests/37636#:~:text=2.%20%2D%2Dresync%20deletes%20data%2C%20contrary%20to%20docs)
-causing `--resync` to erroneously delete empty folders and duplicate files unique to Path2
-* `--check-access` is now enforced during `--resync`, preventing data loss in [certain user error scenarios](https://forum.rclone.org/t/bisync-bugs-and-feature-requests/37636#:~:text=%2D%2Dcheck%2Daccess%20doesn%27t%20always%20fail%20when%20it%20should)
-* Fixed an [issue](https://forum.rclone.org/t/bisync-bugs-and-feature-requests/37636#:~:text=5.%20Bisync%20reads%20files%20in%20excluded%20directories%20during%20delete%20operations)
-causing bisync to consider more files than necessary due to overbroad filters during delete operations
-* [Improved detection of false positive change conflicts](https://forum.rclone.org/t/bisync-bugs-and-feature-requests/37636#:~:text=1.%20Identical%20files%20should%20be%20left%20alone%2C%20even%20if%20new/newer/changed%20on%20both%20sides)
-(identical files are now left alone instead of renamed)
-* Added [support for `--create-empty-src-dirs`](https://forum.rclone.org/t/bisync-bugs-and-feature-requests/37636#:~:text=3.%20Bisync%20should%20create/delete%20empty%20directories%20as%20sync%20does%2C%20when%20%2D%2Dcreate%2Dempty%2Dsrc%2Ddirs%20is%20passed)
-* Added experimental `--resilient` mode to allow [recovery from self-correctable errors](https://forum.rclone.org/t/bisync-bugs-and-feature-requests/37636#:~:text=2.%20Bisync%20should%20be%20more%20resilient%20to%20self%2Dcorrectable%20errors)
-* Added [new `--ignore-listing-checksum` flag](https://forum.rclone.org/t/bisync-bugs-and-feature-requests/37636#:~:text=6.%20%2D%2Dignore%2Dchecksum%20should%20be%20split%20into%20two%20flags%20for%20separate%20purposes)
-to distinguish from `--ignore-checksum`
-* [Performance improvements](https://forum.rclone.org/t/bisync-bugs-and-feature-requests/37636#:~:text=6.%20Deletes%20take%20several%20times%20longer%20than%20copies) for large remotes
-* Documentation and testing improvements
+$ rclone -q --s3-versions ls s3:cleanup-test
+ 9 one.txt
+```
-# Release signing
+#### Versions naming caveat
-The hashes of the binary artefacts of the rclone release are signed
-with a public PGP/GPG key. This can be verified manually as described
-below.
+When using `--s3-versions` flag rclone is relying on the file name
+to work out whether the objects are versions or not. Versions' names
+are created by inserting timestamp between file name and its extension.
+```
+ 9 file.txt
+ 8 file-v2023-07-17-161032-000.txt
+ 16 file-v2023-06-15-141003-000.txt
+```
+If there are real files present with the same names as versions, then
+behaviour of `--s3-versions` can be unpredictable.
-The same mechanism is also used by [rclone selfupdate](https://rclone.org/commands/rclone_selfupdate/)
-to verify that the release has not been tampered with before the new
-update is installed. This checks the SHA256 hash and the signature
-with a public key compiled into the rclone binary.
+### Cleanup
-## Release signing key
+If you run `rclone cleanup s3:bucket` then it will remove all pending
+multipart uploads older than 24 hours. You can use the `--interactive`/`i`
+or `--dry-run` flag to see exactly what it will do. If you want more control over the
+expiry date then run `rclone backend cleanup s3:bucket -o max-age=1h`
+to expire all uploads older than one hour. You can use `rclone backend
+list-multipart-uploads s3:bucket` to see the pending multipart
+uploads.
-You may obtain the release signing key from:
+### Restricted filename characters
-- From [KEYS](/KEYS) on this website - this file contains all past signing keys also.
-- The git repository hosted on GitHub - https://github.com/rclone/rclone/blob/master/docs/content/KEYS
-- `gpg --keyserver hkps://keys.openpgp.org --search nick@craig-wood.com`
-- `gpg --keyserver hkps://keyserver.ubuntu.com --search nick@craig-wood.com`
-- https://www.craig-wood.com/nick/pub/pgp-key.txt
+S3 allows any valid UTF-8 string as a key.
-After importing the key, verify that the fingerprint of one of the
-keys matches: `FBF737ECE9F8AB18604BD2AC93935E02FF3B54FA` as this key is used for signing.
+Invalid UTF-8 bytes will be [replaced](https://rclone.org/overview/#invalid-utf8), as
+they can't be used in XML.
-We recommend that you cross-check the fingerprint shown above through
-the domains listed below. By cross-checking the integrity of the
-fingerprint across multiple domains you can be confident that you
-obtained the correct key.
+The following characters are replaced since these are problematic when
+dealing with the REST API:
-- The [source for this page on GitHub](https://github.com/rclone/rclone/blob/master/docs/content/release_signing.md).
-- Through DNS `dig key.rclone.org txt`
+| Character | Value | Replacement |
+| --------- |:-----:|:-----------:|
+| NUL | 0x00 | †|
+| / | 0x2F | ï¼ |
-If you find anything that doesn't not match, please contact the
-developers at once.
+The encoding will also encode these file names as they don't seem to
+work with the SDK properly:
-## How to verify the release
+| File name | Replacement |
+| --------- |:-----------:|
+| . | ． |
+| .. | ．． |
-In the release directory you will see the release files and some files called `MD5SUMS`, `SHA1SUMS` and `SHA256SUMS`.
+### Multipart uploads
-```
-$ rclone lsf --http-url https://downloads.rclone.org/v1.63.1 :http:
-MD5SUMS
-SHA1SUMS
-SHA256SUMS
-rclone-v1.63.1-freebsd-386.zip
-rclone-v1.63.1-freebsd-amd64.zip
-...
-rclone-v1.63.1-windows-arm64.zip
-rclone-v1.63.1.tar.gz
-version.txt
-```
+rclone supports multipart uploads with S3 which means that it can
+upload files bigger than 5 GiB.
-The `MD5SUMS`, `SHA1SUMS` and `SHA256SUMS` contain hashes of the
-binary files in the release directory along with a signature.
+Note that files uploaded *both* with multipart upload *and* through
+crypt remotes do not have MD5 sums.
-For example:
+rclone switches from single part uploads to multipart uploads at the
+point specified by `--s3-upload-cutoff`. This can be a maximum of 5 GiB
+and a minimum of 0 (ie always upload multipart files).
-```
-$ rclone cat --http-url https://downloads.rclone.org/v1.63.1 :http:SHA256SUMS
------BEGIN PGP SIGNED MESSAGE-----
-Hash: SHA1
+The chunk sizes used in the multipart upload are specified by
+`--s3-chunk-size` and the number of chunks uploaded concurrently is
+specified by `--s3-upload-concurrency`.
-f6d1b2d7477475ce681bdce8cb56f7870f174cb6b2a9ac5d7b3764296ea4a113 rclone-v1.63.1-freebsd-386.zip
-7266febec1f01a25d6575de51c44ddf749071a4950a6384e4164954dff7ac37e rclone-v1.63.1-freebsd-amd64.zip
-...
-66ca083757fb22198309b73879831ed2b42309892394bf193ff95c75dff69c73 rclone-v1.63.1-windows-amd64.zip
-bbb47c16882b6c5f2e8c1b04229378e28f68734c613321ef0ea2263760f74cd0 rclone-v1.63.1-windows-arm64.zip
------BEGIN PGP SIGNATURE-----
+Multipart uploads will use `--transfers` * `--s3-upload-concurrency` *
+`--s3-chunk-size` extra memory. Single part uploads to not use extra
+memory.
-iF0EARECAB0WIQT79zfs6firGGBL0qyTk14C/ztU+gUCZLVKJQAKCRCTk14C/ztU
-+pZuAJ0XJ+QWLP/3jCtkmgcgc4KAwd/rrwCcCRZQ7E+oye1FPY46HOVzCFU3L7g=
-=8qrL
------END PGP SIGNATURE-----
-```
+Single part transfers can be faster than multipart transfers or slower
+depending on your latency from S3 - the more latency, the more likely
+single part transfers will be faster.
-### Download the files
+Increasing `--s3-upload-concurrency` will increase throughput (8 would
+be a sensible value) and increasing `--s3-chunk-size` also increases
+throughput (16M would be sensible). Increasing either of these will
+use more memory. The default values are high enough to gain most of
+the possible performance without using too much memory.
-The first step is to download the binary and SUMs file and verify that
-the SUMs you have downloaded match. Here we download
-`rclone-v1.63.1-windows-amd64.zip` - choose the binary (or binaries)
-appropriate to your architecture. We've also chosen the `SHA256SUMS`
-as these are the most secure. You could verify the other types of hash
-also for extra security. `rclone selfupdate` verifies just the
-`SHA256SUMS`.
-```
-$ mkdir /tmp/check
-$ cd /tmp/check
-$ rclone copy --http-url https://downloads.rclone.org/v1.63.1 :http:SHA256SUMS .
-$ rclone copy --http-url https://downloads.rclone.org/v1.63.1 :http:rclone-v1.63.1-windows-amd64.zip .
-```
+### Buckets and Regions
-### Verify the signatures
+With Amazon S3 you can list buckets (`rclone lsd`) using any region,
+but you can only access the content of a bucket from the region it was
+created in. If you attempt to access a bucket from the wrong region,
+you will get an error, `incorrect region, the bucket is not in 'XXX'
+region`.
-First verify the signatures on the SHA256 file.
+### Authentication
-Import the key. See above for ways to verify this key is correct.
+There are a number of ways to supply `rclone` with a set of AWS
+credentials, with and without using the environment.
-```
-$ gpg --keyserver keyserver.ubuntu.com --receive-keys FBF737ECE9F8AB18604BD2AC93935E02FF3B54FA
-gpg: key 93935E02FF3B54FA: public key "Nick Craig-Wood " imported
-gpg: Total number processed: 1
-gpg: imported: 1
-```
+The different authentication methods are tried in this order:
-Then check the signature:
+ - Directly in the rclone configuration file (`env_auth = false` in the config file):
+ - `access_key_id` and `secret_access_key` are required.
+ - `session_token` can be optionally set when using AWS STS.
+ - Runtime configuration (`env_auth = true` in the config file):
+ - Export the following environment variables before running `rclone`:
+ - Access Key ID: `AWS_ACCESS_KEY_ID` or `AWS_ACCESS_KEY`
+ - Secret Access Key: `AWS_SECRET_ACCESS_KEY` or `AWS_SECRET_KEY`
+ - Session Token: `AWS_SESSION_TOKEN` (optional)
+ - Or, use a [named profile](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html):
+ - Profile files are standard files used by AWS CLI tools
+ - By default it will use the profile in your home directory (e.g. `~/.aws/credentials` on unix based systems) file and the "default" profile, to change set these environment variables:
+ - `AWS_SHARED_CREDENTIALS_FILE` to control which file.
+ - `AWS_PROFILE` to control which profile to use.
+ - Or, run `rclone` in an ECS task with an IAM role (AWS only).
+ - Or, run `rclone` on an EC2 instance with an IAM role (AWS only).
+ - Or, run `rclone` in an EKS pod with an IAM role that is associated with a service account (AWS only).
-```
-$ gpg --verify SHA256SUMS
-gpg: Signature made Mon 17 Jul 2023 15:03:17 BST
-gpg: using DSA key FBF737ECE9F8AB18604BD2AC93935E02FF3B54FA
-gpg: Good signature from "Nick Craig-Wood " [ultimate]
-```
+If none of these option actually end up providing `rclone` with AWS
+credentials then S3 interaction will be non-authenticated (see below).
-Verify the signature was good and is using the fingerprint shown above.
+### S3 Permissions
-Repeat for `MD5SUMS` and `SHA1SUMS` if desired.
+When using the `sync` subcommand of `rclone` the following minimum
+permissions are required to be available on the bucket being written to:
-### Verify the hashes
+* `ListBucket`
+* `DeleteObject`
+* `GetObject`
+* `PutObject`
+* `PutObjectACL`
-Now that we know the signatures on the hashes are OK we can verify the
-binaries match the hashes, completing the verification.
+When using the `lsd` subcommand, the `ListAllMyBuckets` permission is required.
+
+Example policy:
```
-$ sha256sum -c SHA256SUMS 2>&1 | grep OK
-rclone-v1.63.1-windows-amd64.zip: OK
+{
+ "Version": "2012-10-17",
+ "Statement": [
+ {
+ "Effect": "Allow",
+ "Principal": {
+ "AWS": "arn:aws:iam::USER_SID:user/USER_NAME"
+ },
+ "Action": [
+ "s3:ListBucket",
+ "s3:DeleteObject",
+ "s3:GetObject",
+ "s3:PutObject",
+ "s3:PutObjectAcl"
+ ],
+ "Resource": [
+ "arn:aws:s3:::BUCKET_NAME/*",
+ "arn:aws:s3:::BUCKET_NAME"
+ ]
+ },
+ {
+ "Effect": "Allow",
+ "Action": "s3:ListAllMyBuckets",
+ "Resource": "arn:aws:s3:::*"
+ }
+ ]
+}
```
-Or do the check with rclone
+Notes on above:
-```
-$ rclone hashsum sha256 -C SHA256SUMS rclone-v1.63.1-windows-amd64.zip
-2023/09/11 10:53:58 NOTICE: SHA256SUMS: improperly formatted checksum line 0
-2023/09/11 10:53:58 NOTICE: SHA256SUMS: improperly formatted checksum line 1
-2023/09/11 10:53:58 NOTICE: SHA256SUMS: improperly formatted checksum line 49
-2023/09/11 10:53:58 NOTICE: SHA256SUMS: 4 warning(s) suppressed...
-= rclone-v1.63.1-windows-amd64.zip
-2023/09/11 10:53:58 NOTICE: Local file system at /tmp/check: 0 differences found
-2023/09/11 10:53:58 NOTICE: Local file system at /tmp/check: 1 matching files
-```
+1. This is a policy that can be used when creating bucket. It assumes
+ that `USER_NAME` has been created.
+2. The Resource entry must include both resource ARNs, as one implies
+ the bucket and the other implies the bucket's objects.
-### Verify signatures and hashes together
+For reference, [here's an Ansible script](https://gist.github.com/ebridges/ebfc9042dd7c756cd101cfa807b7ae2b)
+that will generate one or more buckets that will work with `rclone sync`.
-You can verify the signatures and hashes in one command line like this:
+### Key Management System (KMS)
-```
-$ gpg --decrypt SHA256SUMS | sha256sum -c --ignore-missing
-gpg: Signature made Mon 17 Jul 2023 15:03:17 BST
-gpg: using DSA key FBF737ECE9F8AB18604BD2AC93935E02FF3B54FA
-gpg: Good signature from "Nick Craig-Wood " [ultimate]
-gpg: aka "Nick Craig-Wood " [unknown]
-rclone-v1.63.1-windows-amd64.zip: OK
-```
+If you are using server-side encryption with KMS then you must make
+sure rclone is configured with `server_side_encryption = aws:kms`
+otherwise you will find you can't transfer small objects - these will
+create checksum errors.
-# 1Fichier
+### Glacier and Glacier Deep Archive
-This is a backend for the [1fichier](https://1fichier.com) cloud
-storage service. Note that a Premium subscription is required to use
-the API.
+You can upload objects using the glacier storage class or transition them to glacier using a [lifecycle policy](http://docs.aws.amazon.com/AmazonS3/latest/user-guide/create-lifecycle.html).
+The bucket can still be synced or copied into normally, but if rclone
+tries to access data from the glacier storage class you will see an error like below.
-Paths are specified as `remote:path`
+ 2017/09/11 19:07:43 Failed to sync: failed to open source object: Object in GLACIER, restore first: path/to/file
-Paths may be as deep as required, e.g. `remote:directory/subdirectory`.
+In this case you need to [restore](http://docs.aws.amazon.com/AmazonS3/latest/user-guide/restore-archived-objects.html)
+the object(s) in question before using rclone.
-## Configuration
+Note that rclone only speaks the S3 API it does not speak the Glacier
+Vault API, so rclone cannot directly access Glacier Vaults.
-The initial setup for 1Fichier involves getting the API key from the website which you
-need to do in your browser.
+### Object-lock enabled S3 bucket
-Here is an example of how to make a remote called `remote`. First run:
+According to AWS's [documentation on S3 Object Lock](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock-overview.html#object-lock-permission):
- rclone config
+> If you configure a default retention period on a bucket, requests to upload objects in such a bucket must include the Content-MD5 header.
-This will guide you through an interactive setup process:
+As mentioned in the [Modification times and hashes](#modification-times-and-hashes) section,
+small files that are not uploaded as multipart, use a different tag, causing the upload to fail.
+A simple solution is to set the `--s3-upload-cutoff 0` and force all the files to be uploaded as multipart.
-```
-No remotes found, make a new one?
-n) New remote
-s) Set configuration password
-q) Quit config
-n/s/q> n
-name> remote
-Type of storage to configure.
-Enter a string value. Press Enter for the default ("").
-Choose a number from below, or type in your own value
-[snip]
-XX / 1Fichier
- \ "fichier"
-[snip]
-Storage> fichier
-** See help for fichier backend at: https://rclone.org/fichier/ **
-Your API Key, get it from https://1fichier.com/console/params.pl
-Enter a string value. Press Enter for the default ("").
-api_key> example_key
+### Standard options
-Edit advanced config? (y/n)
-y) Yes
-n) No
-y/n>
-Remote config
---------------------
-[remote]
-type = fichier
-api_key = example_key
---------------------
-y) Yes this is OK
-e) Edit this remote
-d) Delete this remote
-y/e/d> y
-```
+Here are the Standard options specific to s3 (Amazon S3 Compliant Storage Providers including AWS, Alibaba, ArvanCloud, Ceph, ChinaMobile, Cloudflare, DigitalOcean, Dreamhost, GCS, HuaweiOBS, IBMCOS, IDrive, IONOS, LyveCloud, Leviia, Liara, Linode, Minio, Netease, Petabox, RackCorp, Rclone, Scaleway, SeaweedFS, StackPath, Storj, Synology, TencentCOS, Wasabi, Qiniu and others).
-Once configured you can then use `rclone` like this,
+#### --s3-provider
-List directories in top level of your 1Fichier account
+Choose your S3 provider.
- rclone lsd remote:
+Properties:
-List all the files in your 1Fichier account
+- Config: provider
+- Env Var: RCLONE_S3_PROVIDER
+- Type: string
+- Required: false
+- Examples:
+ - "AWS"
+ - Amazon Web Services (AWS) S3
+ - "Alibaba"
+ - Alibaba Cloud Object Storage System (OSS) formerly Aliyun
+ - "ArvanCloud"
+ - Arvan Cloud Object Storage (AOS)
+ - "Ceph"
+ - Ceph Object Storage
+ - "ChinaMobile"
+ - China Mobile Ecloud Elastic Object Storage (EOS)
+ - "Cloudflare"
+ - Cloudflare R2 Storage
+ - "DigitalOcean"
+ - DigitalOcean Spaces
+ - "Dreamhost"
+ - Dreamhost DreamObjects
+ - "GCS"
+ - Google Cloud Storage
+ - "HuaweiOBS"
+ - Huawei Object Storage Service
+ - "IBMCOS"
+ - IBM COS S3
+ - "IDrive"
+ - IDrive e2
+ - "IONOS"
+ - IONOS Cloud
+ - "LyveCloud"
+ - Seagate Lyve Cloud
+ - "Leviia"
+ - Leviia Object Storage
+ - "Liara"
+ - Liara Object Storage
+ - "Linode"
+ - Linode Object Storage
+ - "Minio"
+ - Minio Object Storage
+ - "Netease"
+ - Netease Object Storage (NOS)
+ - "Petabox"
+ - Petabox Object Storage
+ - "RackCorp"
+ - RackCorp Object Storage
+ - "Rclone"
+ - Rclone S3 Server
+ - "Scaleway"
+ - Scaleway Object Storage
+ - "SeaweedFS"
+ - SeaweedFS S3
+ - "StackPath"
+ - StackPath Object Storage
+ - "Storj"
+ - Storj (S3 Compatible Gateway)
+ - "Synology"
+ - Synology C2 Object Storage
+ - "TencentCOS"
+ - Tencent Cloud Object Storage (COS)
+ - "Wasabi"
+ - Wasabi Object Storage
+ - "Qiniu"
+ - Qiniu Object Storage (Kodo)
+ - "Other"
+ - Any other S3 compatible provider
- rclone ls remote:
+#### --s3-env-auth
-To copy a local directory to a 1Fichier directory called backup
+Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
- rclone copy /home/source remote:backup
+Only applies if access_key_id and secret_access_key is blank.
-### Modified time and hashes ###
+Properties:
-1Fichier does not support modification times. It supports the Whirlpool hash algorithm.
+- Config: env_auth
+- Env Var: RCLONE_S3_ENV_AUTH
+- Type: bool
+- Default: false
+- Examples:
+ - "false"
+ - Enter AWS credentials in the next step.
+ - "true"
+ - Get AWS credentials from the environment (env vars or IAM).
+
+#### --s3-access-key-id
-### Duplicated files ###
+AWS Access Key ID.
-1Fichier can have two files with exactly the same name and path (unlike a
-normal file system).
+Leave blank for anonymous access or runtime credentials.
-Duplicated files cause problems with the syncing and you will see
-messages in the log about duplicates.
+Properties:
-### Restricted filename characters
+- Config: access_key_id
+- Env Var: RCLONE_S3_ACCESS_KEY_ID
+- Type: string
+- Required: false
-In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters)
-the following characters are also replaced:
+#### --s3-secret-access-key
-| Character | Value | Replacement |
-| --------- |:-----:|:-----------:|
-| \ | 0x5C | ï¼¼ |
-| < | 0x3C | ＜ |
-| > | 0x3E | ＞ |
-| " | 0x22 | ＂ |
-| $ | 0x24 | ＄ |
-| ` | 0x60 | ï½€ |
-| ' | 0x27 | ＇ |
+AWS Secret Access Key (password).
-File names can also not start or end with the following characters.
-These only get replaced if they are the first or last character in the
-name:
+Leave blank for anonymous access or runtime credentials.
-| Character | Value | Replacement |
-| --------- |:-----:|:-----------:|
-| SP | 0x20 | â |
+Properties:
-Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8),
-as they can't be used in JSON strings.
+- Config: secret_access_key
+- Env Var: RCLONE_S3_SECRET_ACCESS_KEY
+- Type: string
+- Required: false
+#### --s3-region
-### Standard options
+Region to connect to.
+
+Properties:
+
+- Config: region
+- Env Var: RCLONE_S3_REGION
+- Provider: AWS
+- Type: string
+- Required: false
+- Examples:
+ - "us-east-1"
+ - The default endpoint - a good choice if you are unsure.
+ - US Region, Northern Virginia, or Pacific Northwest.
+ - Leave location constraint empty.
+ - "us-east-2"
+ - US East (Ohio) Region.
+ - Needs location constraint us-east-2.
+ - "us-west-1"
+ - US West (Northern California) Region.
+ - Needs location constraint us-west-1.
+ - "us-west-2"
+ - US West (Oregon) Region.
+ - Needs location constraint us-west-2.
+ - "ca-central-1"
+ - Canada (Central) Region.
+ - Needs location constraint ca-central-1.
+ - "eu-west-1"
+ - EU (Ireland) Region.
+ - Needs location constraint EU or eu-west-1.
+ - "eu-west-2"
+ - EU (London) Region.
+ - Needs location constraint eu-west-2.
+ - "eu-west-3"
+ - EU (Paris) Region.
+ - Needs location constraint eu-west-3.
+ - "eu-north-1"
+ - EU (Stockholm) Region.
+ - Needs location constraint eu-north-1.
+ - "eu-south-1"
+ - EU (Milan) Region.
+ - Needs location constraint eu-south-1.
+ - "eu-central-1"
+ - EU (Frankfurt) Region.
+ - Needs location constraint eu-central-1.
+ - "ap-southeast-1"
+ - Asia Pacific (Singapore) Region.
+ - Needs location constraint ap-southeast-1.
+ - "ap-southeast-2"
+ - Asia Pacific (Sydney) Region.
+ - Needs location constraint ap-southeast-2.
+ - "ap-northeast-1"
+ - Asia Pacific (Tokyo) Region.
+ - Needs location constraint ap-northeast-1.
+ - "ap-northeast-2"
+ - Asia Pacific (Seoul).
+ - Needs location constraint ap-northeast-2.
+ - "ap-northeast-3"
+ - Asia Pacific (Osaka-Local).
+ - Needs location constraint ap-northeast-3.
+ - "ap-south-1"
+ - Asia Pacific (Mumbai).
+ - Needs location constraint ap-south-1.
+ - "ap-east-1"
+ - Asia Pacific (Hong Kong) Region.
+ - Needs location constraint ap-east-1.
+ - "sa-east-1"
+ - South America (Sao Paulo) Region.
+ - Needs location constraint sa-east-1.
+ - "me-south-1"
+ - Middle East (Bahrain) Region.
+ - Needs location constraint me-south-1.
+ - "af-south-1"
+ - Africa (Cape Town) Region.
+ - Needs location constraint af-south-1.
+ - "cn-north-1"
+ - China (Beijing) Region.
+ - Needs location constraint cn-north-1.
+ - "cn-northwest-1"
+ - China (Ningxia) Region.
+ - Needs location constraint cn-northwest-1.
+ - "us-gov-east-1"
+ - AWS GovCloud (US-East) Region.
+ - Needs location constraint us-gov-east-1.
+ - "us-gov-west-1"
+ - AWS GovCloud (US) Region.
+ - Needs location constraint us-gov-west-1.
-Here are the Standard options specific to fichier (1Fichier).
+#### --s3-endpoint
-#### --fichier-api-key
+Endpoint for S3 API.
-Your API Key, get it from https://1fichier.com/console/params.pl.
+Leave blank if using AWS to use the default endpoint for the region.
Properties:
-- Config: api_key
-- Env Var: RCLONE_FICHIER_API_KEY
+- Config: endpoint
+- Env Var: RCLONE_S3_ENDPOINT
+- Provider: AWS
- Type: string
- Required: false
-### Advanced options
-
-Here are the Advanced options specific to fichier (1Fichier).
+#### --s3-location-constraint
-#### --fichier-shared-folder
+Location constraint - must be set to match the Region.
-If you want to download a shared folder, add this parameter.
+Used when creating buckets only.
Properties:
-- Config: shared_folder
-- Env Var: RCLONE_FICHIER_SHARED_FOLDER
+- Config: location_constraint
+- Env Var: RCLONE_S3_LOCATION_CONSTRAINT
+- Provider: AWS
- Type: string
- Required: false
+- Examples:
+ - ""
+ - Empty for US Region, Northern Virginia, or Pacific Northwest
+ - "us-east-2"
+ - US East (Ohio) Region
+ - "us-west-1"
+ - US West (Northern California) Region
+ - "us-west-2"
+ - US West (Oregon) Region
+ - "ca-central-1"
+ - Canada (Central) Region
+ - "eu-west-1"
+ - EU (Ireland) Region
+ - "eu-west-2"
+ - EU (London) Region
+ - "eu-west-3"
+ - EU (Paris) Region
+ - "eu-north-1"
+ - EU (Stockholm) Region
+ - "eu-south-1"
+ - EU (Milan) Region
+ - "EU"
+ - EU Region
+ - "ap-southeast-1"
+ - Asia Pacific (Singapore) Region
+ - "ap-southeast-2"
+ - Asia Pacific (Sydney) Region
+ - "ap-northeast-1"
+ - Asia Pacific (Tokyo) Region
+ - "ap-northeast-2"
+ - Asia Pacific (Seoul) Region
+ - "ap-northeast-3"
+ - Asia Pacific (Osaka-Local) Region
+ - "ap-south-1"
+ - Asia Pacific (Mumbai) Region
+ - "ap-east-1"
+ - Asia Pacific (Hong Kong) Region
+ - "sa-east-1"
+ - South America (Sao Paulo) Region
+ - "me-south-1"
+ - Middle East (Bahrain) Region
+ - "af-south-1"
+ - Africa (Cape Town) Region
+ - "cn-north-1"
+ - China (Beijing) Region
+ - "cn-northwest-1"
+ - China (Ningxia) Region
+ - "us-gov-east-1"
+ - AWS GovCloud (US-East) Region
+ - "us-gov-west-1"
+ - AWS GovCloud (US) Region
-#### --fichier-file-password
-
-If you want to download a shared file that is password protected, add this parameter.
+#### --s3-acl
-**NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/).
+Canned ACL used when creating buckets and storing or copying objects.
-Properties:
+This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.
-- Config: file_password
-- Env Var: RCLONE_FICHIER_FILE_PASSWORD
-- Type: string
-- Required: false
+For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
-#### --fichier-folder-password
+Note that this ACL is applied when server-side copying objects as S3
+doesn't copy the ACL from the source but rather writes a fresh one.
-If you want to list the files in a shared folder that is password protected, add this parameter.
+If the acl is an empty string then no X-Amz-Acl: header is added and
+the default (private) will be used.
-**NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/).
Properties:
-- Config: folder_password
-- Env Var: RCLONE_FICHIER_FOLDER_PASSWORD
+- Config: acl
+- Env Var: RCLONE_S3_ACL
+- Provider: !Storj,Synology,Cloudflare
- Type: string
- Required: false
+- Examples:
+ - "default"
+ - Owner gets Full_CONTROL.
+ - No one else has access rights (default).
+ - "private"
+ - Owner gets FULL_CONTROL.
+ - No one else has access rights (default).
+ - "public-read"
+ - Owner gets FULL_CONTROL.
+ - The AllUsers group gets READ access.
+ - "public-read-write"
+ - Owner gets FULL_CONTROL.
+ - The AllUsers group gets READ and WRITE access.
+ - Granting this on a bucket is generally not recommended.
+ - "authenticated-read"
+ - Owner gets FULL_CONTROL.
+ - The AuthenticatedUsers group gets READ access.
+ - "bucket-owner-read"
+ - Object owner gets FULL_CONTROL.
+ - Bucket owner gets READ access.
+ - If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
+ - "bucket-owner-full-control"
+ - Both the object owner and the bucket owner get FULL_CONTROL over the object.
+ - If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
+ - "private"
+ - Owner gets FULL_CONTROL.
+ - No one else has access rights (default).
+ - This acl is available on IBM Cloud (Infra), IBM Cloud (Storage), On-Premise COS.
+ - "public-read"
+ - Owner gets FULL_CONTROL.
+ - The AllUsers group gets READ access.
+ - This acl is available on IBM Cloud (Infra), IBM Cloud (Storage), On-Premise IBM COS.
+ - "public-read-write"
+ - Owner gets FULL_CONTROL.
+ - The AllUsers group gets READ and WRITE access.
+ - This acl is available on IBM Cloud (Infra), On-Premise IBM COS.
+ - "authenticated-read"
+ - Owner gets FULL_CONTROL.
+ - The AuthenticatedUsers group gets READ access.
+ - Not supported on Buckets.
+ - This acl is available on IBM Cloud (Infra) and On-Premise IBM COS.
-#### --fichier-cdn
+#### --s3-server-side-encryption
-Set if you wish to use CDN download links.
+The server-side encryption algorithm used when storing this object in S3.
Properties:
-- Config: cdn
-- Env Var: RCLONE_FICHIER_CDN
-- Type: bool
-- Default: false
-
-#### --fichier-encoding
+- Config: server_side_encryption
+- Env Var: RCLONE_S3_SERVER_SIDE_ENCRYPTION
+- Provider: AWS,Ceph,ChinaMobile,Minio
+- Type: string
+- Required: false
+- Examples:
+ - ""
+ - None
+ - "AES256"
+ - AES256
+ - "aws:kms"
+ - aws:kms
-The encoding for the backend.
+#### --s3-sse-kms-key-id
-See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info.
+If using KMS ID you must provide the ARN of Key.
Properties:
-- Config: encoding
-- Env Var: RCLONE_FICHIER_ENCODING
-- Type: MultiEncoder
-- Default: Slash,LtGt,DoubleQuote,SingleQuote,BackQuote,Dollar,BackSlash,Del,Ctl,LeftSpace,RightSpace,InvalidUtf8,Dot
-
-
-
-## Limitations
-
-`rclone about` is not supported by the 1Fichier backend. Backends without
-this capability cannot determine free space for an rclone mount or
-use policy `mfs` (most free space) as a member of an rclone union
-remote.
-
-See [List of backends that do not support rclone about](https://rclone.org/overview/#optional-features) and [rclone about](https://rclone.org/commands/rclone_about/)
-
-# Alias
-
-The `alias` remote provides a new name for another remote.
+- Config: sse_kms_key_id
+- Env Var: RCLONE_S3_SSE_KMS_KEY_ID
+- Provider: AWS,Ceph,Minio
+- Type: string
+- Required: false
+- Examples:
+ - ""
+ - None
+ - "arn:aws:kms:us-east-1:*"
+ - arn:aws:kms:*
-Paths may be as deep as required or a local path,
-e.g. `remote:directory/subdirectory` or `/directory/subdirectory`.
+#### --s3-storage-class
-During the initial setup with `rclone config` you will specify the target
-remote. The target remote can either be a local path or another remote.
+The storage class to use when storing new objects in S3.
-Subfolders can be used in target remote. Assume an alias remote named `backup`
-with the target `mydrive:private/backup`. Invoking `rclone mkdir backup:desktop`
-is exactly the same as invoking `rclone mkdir mydrive:private/backup/desktop`.
+Properties:
-There will be no special handling of paths containing `..` segments.
-Invoking `rclone mkdir backup:../desktop` is exactly the same as invoking
-`rclone mkdir mydrive:private/backup/../desktop`.
-The empty path is not allowed as a remote. To alias the current directory
-use `.` instead.
+- Config: storage_class
+- Env Var: RCLONE_S3_STORAGE_CLASS
+- Provider: AWS
+- Type: string
+- Required: false
+- Examples:
+ - ""
+ - Default
+ - "STANDARD"
+ - Standard storage class
+ - "REDUCED_REDUNDANCY"
+ - Reduced redundancy storage class
+ - "STANDARD_IA"
+ - Standard Infrequent Access storage class
+ - "ONEZONE_IA"
+ - One Zone Infrequent Access storage class
+ - "GLACIER"
+ - Glacier storage class
+ - "DEEP_ARCHIVE"
+ - Glacier Deep Archive storage class
+ - "INTELLIGENT_TIERING"
+ - Intelligent-Tiering storage class
+ - "GLACIER_IR"
+ - Glacier Instant Retrieval storage class
-The target remote can also be a [connection string](https://rclone.org/docs/#connection-strings).
-This can be used to modify the config of a remote for different uses, e.g.
-the alias `myDriveTrash` with the target remote `myDrive,trashed_only:`
-can be used to only show the trashed files in `myDrive`.
+### Advanced options
-## Configuration
+Here are the Advanced options specific to s3 (Amazon S3 Compliant Storage Providers including AWS, Alibaba, ArvanCloud, Ceph, ChinaMobile, Cloudflare, DigitalOcean, Dreamhost, GCS, HuaweiOBS, IBMCOS, IDrive, IONOS, LyveCloud, Leviia, Liara, Linode, Minio, Netease, Petabox, RackCorp, Rclone, Scaleway, SeaweedFS, StackPath, Storj, Synology, TencentCOS, Wasabi, Qiniu and others).
-Here is an example of how to make an alias called `remote` for local folder.
-First run:
+#### --s3-bucket-acl
- rclone config
+Canned ACL used when creating buckets.
-This will guide you through an interactive setup process:
+For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
-```
-No remotes found, make a new one?
-n) New remote
-s) Set configuration password
-q) Quit config
-n/s/q> n
-name> remote
-Type of storage to configure.
-Choose a number from below, or type in your own value
-[snip]
-XX / Alias for an existing remote
- \ "alias"
-[snip]
-Storage> alias
-Remote or path to alias.
-Can be "myremote:path/to/dir", "myremote:bucket", "myremote:" or "/local/path".
-remote> /mnt/storage/backup
-Remote config
---------------------
-[remote]
-remote = /mnt/storage/backup
---------------------
-y) Yes this is OK
-e) Edit this remote
-d) Delete this remote
-y/e/d> y
-Current remotes:
+Note that this ACL is applied when only when creating buckets. If it
+isn't set then "acl" is used instead.
-Name Type
-==== ====
-remote alias
+If the "acl" and "bucket_acl" are empty strings then no X-Amz-Acl:
+header is added and the default (private) will be used.
-e) Edit existing remote
-n) New remote
-d) Delete remote
-r) Rename remote
-c) Copy remote
-s) Set configuration password
-q) Quit config
-e/n/d/r/c/s/q> q
-```
-Once configured you can then use `rclone` like this,
+Properties:
-List directories in top level in `/mnt/storage/backup`
+- Config: bucket_acl
+- Env Var: RCLONE_S3_BUCKET_ACL
+- Type: string
+- Required: false
+- Examples:
+ - "private"
+ - Owner gets FULL_CONTROL.
+ - No one else has access rights (default).
+ - "public-read"
+ - Owner gets FULL_CONTROL.
+ - The AllUsers group gets READ access.
+ - "public-read-write"
+ - Owner gets FULL_CONTROL.
+ - The AllUsers group gets READ and WRITE access.
+ - Granting this on a bucket is generally not recommended.
+ - "authenticated-read"
+ - Owner gets FULL_CONTROL.
+ - The AuthenticatedUsers group gets READ access.
- rclone lsd remote:
+#### --s3-requester-pays
-List all the files in `/mnt/storage/backup`
+Enables requester pays option when interacting with S3 bucket.
- rclone ls remote:
+Properties:
-Copy another local directory to the alias directory called source
+- Config: requester_pays
+- Env Var: RCLONE_S3_REQUESTER_PAYS
+- Provider: AWS
+- Type: bool
+- Default: false
- rclone copy /home/source remote:source
+#### --s3-sse-customer-algorithm
+If using SSE-C, the server-side encryption algorithm used when storing this object in S3.
-### Standard options
+Properties:
-Here are the Standard options specific to alias (Alias for an existing remote).
+- Config: sse_customer_algorithm
+- Env Var: RCLONE_S3_SSE_CUSTOMER_ALGORITHM
+- Provider: AWS,Ceph,ChinaMobile,Minio
+- Type: string
+- Required: false
+- Examples:
+ - ""
+ - None
+ - "AES256"
+ - AES256
-#### --alias-remote
+#### --s3-sse-customer-key
-Remote or path to alias.
+To use SSE-C you may provide the secret encryption key used to encrypt/decrypt your data.
-Can be "myremote:path/to/dir", "myremote:bucket", "myremote:" or "/local/path".
+Alternatively you can provide --sse-customer-key-base64.
Properties:
-- Config: remote
-- Env Var: RCLONE_ALIAS_REMOTE
+- Config: sse_customer_key
+- Env Var: RCLONE_S3_SSE_CUSTOMER_KEY
+- Provider: AWS,Ceph,ChinaMobile,Minio
- Type: string
-- Required: true
-
-
-
-# Amazon Drive
+- Required: false
+- Examples:
+ - ""
+ - None
-Amazon Drive, formerly known as Amazon Cloud Drive, is a cloud storage
-service run by Amazon for consumers.
+#### --s3-sse-customer-key-base64
-## Status
+If using SSE-C you must provide the secret encryption key encoded in base64 format to encrypt/decrypt your data.
-**Important:** rclone supports Amazon Drive only if you have your own
-set of API keys. Unfortunately the [Amazon Drive developer
-program](https://developer.amazon.com/amazon-drive) is now closed to
-new entries so if you don't already have your own set of keys you will
-not be able to use rclone with Amazon Drive.
+Alternatively you can provide --sse-customer-key.
-For the history on why rclone no longer has a set of Amazon Drive API
-keys see [the forum](https://forum.rclone.org/t/rclone-has-been-banned-from-amazon-drive/2314).
+Properties:
-If you happen to know anyone who works at Amazon then please ask them
-to re-instate rclone into the Amazon Drive developer program - thanks!
+- Config: sse_customer_key_base64
+- Env Var: RCLONE_S3_SSE_CUSTOMER_KEY_BASE64
+- Provider: AWS,Ceph,ChinaMobile,Minio
+- Type: string
+- Required: false
+- Examples:
+ - ""
+ - None
-## Configuration
+#### --s3-sse-customer-key-md5
-The initial setup for Amazon Drive involves getting a token from
-Amazon which you need to do in your browser. `rclone config` walks
-you through it.
+If using SSE-C you may provide the secret encryption key MD5 checksum (optional).
-The configuration process for Amazon Drive may involve using an [oauth
-proxy](https://github.com/ncw/oauthproxy). This is used to keep the
-Amazon credentials out of the source code. The proxy runs in Google's
-very secure App Engine environment and doesn't store any credentials
-which pass through it.
+If you leave it blank, this is calculated automatically from the sse_customer_key provided.
-Since rclone doesn't currently have its own Amazon Drive credentials
-so you will either need to have your own `client_id` and
-`client_secret` with Amazon Drive, or use a third-party oauth proxy
-in which case you will need to enter `client_id`, `client_secret`,
-`auth_url` and `token_url`.
-Note also if you are not using Amazon's `auth_url` and `token_url`,
-(ie you filled in something for those) then if setting up on a remote
-machine you can only use the [copying the config method of
-configuration](https://rclone.org/remote_setup/#configuring-by-copying-the-config-file)
-- `rclone authorize` will not work.
+Properties:
-Here is an example of how to make a remote called `remote`. First run:
+- Config: sse_customer_key_md5
+- Env Var: RCLONE_S3_SSE_CUSTOMER_KEY_MD5
+- Provider: AWS,Ceph,ChinaMobile,Minio
+- Type: string
+- Required: false
+- Examples:
+ - ""
+ - None
- rclone config
+#### --s3-upload-cutoff
-This will guide you through an interactive setup process:
+Cutoff for switching to chunked upload.
-```
-No remotes found, make a new one?
-n) New remote
-r) Rename remote
-c) Copy remote
-s) Set configuration password
-q) Quit config
-n/r/c/s/q> n
-name> remote
-Type of storage to configure.
-Choose a number from below, or type in your own value
-[snip]
-XX / Amazon Drive
- \ "amazon cloud drive"
-[snip]
-Storage> amazon cloud drive
-Amazon Application Client Id - required.
-client_id> your client ID goes here
-Amazon Application Client Secret - required.
-client_secret> your client secret goes here
-Auth server URL - leave blank to use Amazon's.
-auth_url> Optional auth URL
-Token server url - leave blank to use Amazon's.
-token_url> Optional token URL
-Remote config
-Make sure your Redirect URL is set to "http://127.0.0.1:53682/" in your custom config.
-Use web browser to automatically authenticate rclone with remote?
- * Say Y if the machine running rclone has a web browser you can use
- * Say N if running rclone on a (remote) machine without web browser access
-If not sure try Y. If Y failed, try N.
-y) Yes
-n) No
-y/n> y
-If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth
-Log in and authorize rclone for access
-Waiting for code...
-Got code
---------------------
-[remote]
-client_id = your client ID goes here
-client_secret = your client secret goes here
-auth_url = Optional auth URL
-token_url = Optional token URL
-token = {"access_token":"xxxxxxxxxxxxxxxxxxxxxxx","token_type":"bearer","refresh_token":"xxxxxxxxxxxxxxxxxx","expiry":"2015-09-06T16:07:39.658438471+01:00"}
---------------------
-y) Yes this is OK
-e) Edit this remote
-d) Delete this remote
-y/e/d> y
-```
+Any files larger than this will be uploaded in chunks of chunk_size.
+The minimum is 0 and the maximum is 5 GiB.
-See the [remote setup docs](https://rclone.org/remote_setup/) for how to set it up on a
-machine with no Internet browser available.
+Properties:
-Note that rclone runs a webserver on your local machine to collect the
-token as returned from Amazon. This only runs from the moment it
-opens your browser to the moment you get back the verification
-code. This is on `http://127.0.0.1:53682/` and this it may require
-you to unblock it temporarily if you are running a host firewall.
+- Config: upload_cutoff
+- Env Var: RCLONE_S3_UPLOAD_CUTOFF
+- Type: SizeSuffix
+- Default: 200Mi
-Once configured you can then use `rclone` like this,
+#### --s3-chunk-size
-List directories in top level of your Amazon Drive
+Chunk size to use for uploading.
- rclone lsd remote:
+When uploading files larger than upload_cutoff or files with unknown
+size (e.g. from "rclone rcat" or uploaded with "rclone mount" or google
+photos or google docs) they will be uploaded as multipart uploads
+using this chunk size.
-List all the files in your Amazon Drive
+Note that "--s3-upload-concurrency" chunks of this size are buffered
+in memory per transfer.
- rclone ls remote:
+If you are transferring large files over high-speed links and you have
+enough memory, then increasing this will speed up the transfers.
-To copy a local directory to an Amazon Drive directory called backup
+Rclone will automatically increase the chunk size when uploading a
+large file of known size to stay below the 10,000 chunks limit.
- rclone copy /home/source remote:backup
+Files of unknown size are uploaded with the configured
+chunk_size. Since the default chunk size is 5 MiB and there can be at
+most 10,000 chunks, this means that by default the maximum size of
+a file you can stream upload is 48 GiB. If you wish to stream upload
+larger files then you will need to increase chunk_size.
-### Modified time and MD5SUMs
+Increasing the chunk size decreases the accuracy of the progress
+statistics displayed with "-P" flag. Rclone treats chunk as sent when
+it's buffered by the AWS SDK, when in fact it may still be uploading.
+A bigger chunk size means a bigger AWS SDK buffer and progress
+reporting more deviating from the truth.
-Amazon Drive doesn't allow modification times to be changed via
-the API so these won't be accurate or used for syncing.
-It does store MD5SUMs so for a more accurate sync, you can use the
-`--checksum` flag.
+Properties:
-### Restricted filename characters
+- Config: chunk_size
+- Env Var: RCLONE_S3_CHUNK_SIZE
+- Type: SizeSuffix
+- Default: 5Mi
-| Character | Value | Replacement |
-| --------- |:-----:|:-----------:|
-| NUL | 0x00 | †|
-| / | 0x2F | ï¼ |
+#### --s3-max-upload-parts
-Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8),
-as they can't be used in JSON strings.
+Maximum number of parts in a multipart upload.
-### Deleting files
+This option defines the maximum number of multipart chunks to use
+when doing a multipart upload.
-Any files you delete with rclone will end up in the trash. Amazon
-don't provide an API to permanently delete files, nor to empty the
-trash, so you will have to do that with one of Amazon's apps or via
-the Amazon Drive website. As of November 17, 2016, files are
-automatically deleted by Amazon from the trash after 30 days.
+This can be useful if a service does not support the AWS S3
+specification of 10,000 chunks.
-### Using with non `.com` Amazon accounts
+Rclone will automatically increase the chunk size when uploading a
+large file of a known size to stay below this number of chunks limit.
-Let's say you usually use `amazon.co.uk`. When you authenticate with
-rclone it will take you to an `amazon.com` page to log in. Your
-`amazon.co.uk` email and password should work here just fine.
+Properties:
-### Standard options
+- Config: max_upload_parts
+- Env Var: RCLONE_S3_MAX_UPLOAD_PARTS
+- Type: int
+- Default: 10000
-Here are the Standard options specific to amazon cloud drive (Amazon Drive).
+#### --s3-copy-cutoff
-#### --acd-client-id
+Cutoff for switching to multipart copy.
-OAuth Client Id.
+Any files larger than this that need to be server-side copied will be
+copied in chunks of this size.
-Leave blank normally.
+The minimum is 0 and the maximum is 5 GiB.
Properties:
-- Config: client_id
-- Env Var: RCLONE_ACD_CLIENT_ID
-- Type: string
-- Required: false
+- Config: copy_cutoff
+- Env Var: RCLONE_S3_COPY_CUTOFF
+- Type: SizeSuffix
+- Default: 4.656Gi
-#### --acd-client-secret
+#### --s3-disable-checksum
-OAuth Client Secret.
+Don't store MD5 checksum with object metadata.
-Leave blank normally.
+Normally rclone will calculate the MD5 checksum of the input before
+uploading it so it can add it to metadata on the object. This is great
+for data integrity checking but can cause long delays for large files
+to start uploading.
Properties:
-- Config: client_secret
-- Env Var: RCLONE_ACD_CLIENT_SECRET
-- Type: string
-- Required: false
-
-### Advanced options
-
-Here are the Advanced options specific to amazon cloud drive (Amazon Drive).
-
-#### --acd-token
+- Config: disable_checksum
+- Env Var: RCLONE_S3_DISABLE_CHECKSUM
+- Type: bool
+- Default: false
-OAuth Access Token as a JSON blob.
+#### --s3-shared-credentials-file
-Properties:
+Path to the shared credentials file.
-- Config: token
-- Env Var: RCLONE_ACD_TOKEN
-- Type: string
-- Required: false
+If env_auth = true then rclone can use a shared credentials file.
-#### --acd-auth-url
+If this variable is empty rclone will look for the
+"AWS_SHARED_CREDENTIALS_FILE" env variable. If the env value is empty
+it will default to the current user's home directory.
-Auth server URL.
+ Linux/OSX: "$HOME/.aws/credentials"
+ Windows: "%USERPROFILE%\.aws\credentials"
-Leave blank to use the provider defaults.
Properties:
-- Config: auth_url
-- Env Var: RCLONE_ACD_AUTH_URL
+- Config: shared_credentials_file
+- Env Var: RCLONE_S3_SHARED_CREDENTIALS_FILE
- Type: string
- Required: false
-#### --acd-token-url
+#### --s3-profile
-Token server url.
+Profile to use in the shared credentials file.
+
+If env_auth = true then rclone can use a shared credentials file. This
+variable controls which profile is used in that file.
+
+If empty it will default to the environment variable "AWS_PROFILE" or
+"default" if that environment variable is also not set.
-Leave blank to use the provider defaults.
Properties:
-- Config: token_url
-- Env Var: RCLONE_ACD_TOKEN_URL
+- Config: profile
+- Env Var: RCLONE_S3_PROFILE
- Type: string
- Required: false
-#### --acd-checkpoint
+#### --s3-session-token
-Checkpoint for internal polling (debug).
+An AWS session token.
Properties:
-- Config: checkpoint
-- Env Var: RCLONE_ACD_CHECKPOINT
+- Config: session_token
+- Env Var: RCLONE_S3_SESSION_TOKEN
- Type: string
- Required: false
-#### --acd-upload-wait-per-gb
-
-Additional time per GiB to wait after a failed complete upload to see if it appears.
-
-Sometimes Amazon Drive gives an error when a file has been fully
-uploaded but the file appears anyway after a little while. This
-happens sometimes for files over 1 GiB in size and nearly every time for
-files bigger than 10 GiB. This parameter controls the time rclone waits
-for the file to appear.
-
-The default value for this parameter is 3 minutes per GiB, so by
-default it will wait 3 minutes for every GiB uploaded to see if the
-file appears.
+#### --s3-upload-concurrency
-You can disable this feature by setting it to 0. This may cause
-conflict errors as rclone retries the failed upload but the file will
-most likely appear correctly eventually.
+Concurrency for multipart uploads.
-These values were determined empirically by observing lots of uploads
-of big files for a range of file sizes.
+This is the number of chunks of the same file that are uploaded
+concurrently.
-Upload with the "-v" flag to see more info about what rclone is doing
-in this situation.
+If you are uploading small numbers of large files over high-speed links
+and these uploads do not fully utilize your bandwidth, then increasing
+this may help to speed up the transfers.
Properties:
-- Config: upload_wait_per_gb
-- Env Var: RCLONE_ACD_UPLOAD_WAIT_PER_GB
-- Type: Duration
-- Default: 3m0s
+- Config: upload_concurrency
+- Env Var: RCLONE_S3_UPLOAD_CONCURRENCY
+- Type: int
+- Default: 4
-#### --acd-templink-threshold
+#### --s3-force-path-style
-Files >= this size will be downloaded via their tempLink.
+If true use path style access if false use virtual hosted style.
-Files this size or more will be downloaded via their "tempLink". This
-is to work around a problem with Amazon Drive which blocks downloads
-of files bigger than about 10 GiB. The default for this is 9 GiB which
-shouldn't need to be changed.
+If this is true (the default) then rclone will use path style access,
+if false then rclone will use virtual path style. See [the AWS S3
+docs](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html#access-bucket-intro)
+for more info.
-To download files above this threshold, rclone requests a "tempLink"
-which downloads the file through a temporary URL directly from the
-underlying S3 storage.
+Some providers (e.g. AWS, Aliyun OSS, Netease COS, or Tencent COS) require this set to
+false - rclone will do this automatically based on the provider
+setting.
Properties:
-- Config: templink_threshold
-- Env Var: RCLONE_ACD_TEMPLINK_THRESHOLD
-- Type: SizeSuffix
-- Default: 9Gi
-
-#### --acd-encoding
-
-The encoding for the backend.
+- Config: force_path_style
+- Env Var: RCLONE_S3_FORCE_PATH_STYLE
+- Type: bool
+- Default: true
-See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info.
+#### --s3-v2-auth
-Properties:
+If true use v2 authentication.
-- Config: encoding
-- Env Var: RCLONE_ACD_ENCODING
-- Type: MultiEncoder
-- Default: Slash,InvalidUtf8,Dot
+If this is false (the default) then rclone will use v4 authentication.
+If it is set then rclone will use v2 authentication.
+Use this only if v4 signatures don't work, e.g. pre Jewel/v10 CEPH.
+Properties:
-## Limitations
+- Config: v2_auth
+- Env Var: RCLONE_S3_V2_AUTH
+- Type: bool
+- Default: false
-Note that Amazon Drive is case insensitive so you can't have a
-file called "Hello.doc" and one called "hello.doc".
+#### --s3-use-accelerate-endpoint
-Amazon Drive has rate limiting so you may notice errors in the
-sync (429 errors). rclone will automatically retry the sync up to 3
-times by default (see `--retries` flag) which should hopefully work
-around this problem.
+If true use the AWS S3 accelerated endpoint.
-Amazon Drive has an internal limit of file sizes that can be uploaded
-to the service. This limit is not officially published, but all files
-larger than this will fail.
+See: [AWS S3 Transfer acceleration](https://docs.aws.amazon.com/AmazonS3/latest/dev/transfer-acceleration-examples.html)
-At the time of writing (Jan 2016) is in the area of 50 GiB per file.
-This means that larger files are likely to fail.
+Properties:
-Unfortunately there is no way for rclone to see that this failure is
-because of file size, so it will retry the operation, as any other
-failure. To avoid this problem, use `--max-size 50000M` option to limit
-the maximum size of uploaded files. Note that `--max-size` does not split
-files into segments, it only ignores files over this size.
+- Config: use_accelerate_endpoint
+- Env Var: RCLONE_S3_USE_ACCELERATE_ENDPOINT
+- Provider: AWS
+- Type: bool
+- Default: false
-`rclone about` is not supported by the Amazon Drive backend. Backends without
-this capability cannot determine free space for an rclone mount or
-use policy `mfs` (most free space) as a member of an rclone union
-remote.
+#### --s3-leave-parts-on-error
-See [List of backends that do not support rclone about](https://rclone.org/overview/#optional-features) and [rclone about](https://rclone.org/commands/rclone_about/)
+If true avoid calling abort upload on a failure, leaving all successfully uploaded parts on S3 for manual recovery.
-# Amazon S3 Storage Providers
+It should be set to true for resuming uploads across different sessions.
-The S3 backend can be used with a number of different providers:
+WARNING: Storing parts of an incomplete multipart upload counts towards space usage on S3 and will add additional costs if not cleaned up.
-- AWS S3
-- Alibaba Cloud (Aliyun) Object Storage System (OSS)
-- Ceph
-- China Mobile Ecloud Elastic Object Storage (EOS)
-- Cloudflare R2
-- Arvan Cloud Object Storage (AOS)
-- DigitalOcean Spaces
-- Dreamhost
-- GCS
-- Huawei OBS
-- IBM COS S3
-- IDrive e2
-- IONOS Cloud
- - Leviia Object Storage
-- Liara Object Storage
-- Minio
-- Petabox
-- Qiniu Cloud Object Storage (Kodo)
-- RackCorp Object Storage
-- Scaleway
-- Seagate Lyve Cloud
-- SeaweedFS
-- StackPath
-- Storj
-- Synology C2 Object Storage
-- Tencent Cloud Object Storage (COS)
-- Wasabi
+Properties:
+- Config: leave_parts_on_error
+- Env Var: RCLONE_S3_LEAVE_PARTS_ON_ERROR
+- Provider: AWS
+- Type: bool
+- Default: false
-Paths are specified as `remote:bucket` (or `remote:` for the `lsd`
-command.) You may put subdirectories in too, e.g. `remote:bucket/path/to/dir`.
+#### --s3-list-chunk
-Once you have made a remote (see the provider specific section above)
-you can use it like this:
+Size of listing chunk (response list for each ListObject S3 request).
-See all buckets
+This option is also known as "MaxKeys", "max-items", or "page-size" from the AWS S3 specification.
+Most services truncate the response list to 1000 objects even if requested more than that.
+In AWS S3 this is a global maximum and cannot be changed, see [AWS S3](https://docs.aws.amazon.com/cli/latest/reference/s3/ls.html).
+In Ceph, this can be increased with the "rgw list buckets max chunk" option.
- rclone lsd remote:
-Make a new bucket
+Properties:
- rclone mkdir remote:bucket
+- Config: list_chunk
+- Env Var: RCLONE_S3_LIST_CHUNK
+- Type: int
+- Default: 1000
-List the contents of a bucket
+#### --s3-list-version
- rclone ls remote:bucket
+Version of ListObjects to use: 1,2 or 0 for auto.
-Sync `/home/local/directory` to the remote bucket, deleting any excess
-files in the bucket.
+When S3 originally launched it only provided the ListObjects call to
+enumerate objects in a bucket.
- rclone sync --interactive /home/local/directory remote:bucket
+However in May 2016 the ListObjectsV2 call was introduced. This is
+much higher performance and should be used if at all possible.
-## Configuration
+If set to the default, 0, rclone will guess according to the provider
+set which list objects method to call. If it guesses wrong, then it
+may be set manually here.
-Here is an example of making an s3 configuration for the AWS S3 provider.
-Most applies to the other providers as well, any differences are described [below](#providers).
-First run
+Properties:
- rclone config
+- Config: list_version
+- Env Var: RCLONE_S3_LIST_VERSION
+- Type: int
+- Default: 0
-This will guide you through an interactive setup process.
+#### --s3-list-url-encode
-```
-No remotes found, make a new one?
-n) New remote
-s) Set configuration password
-q) Quit config
-n/s/q> n
-name> remote
-Type of storage to configure.
-Choose a number from below, or type in your own value
-[snip]
-XX / Amazon S3 Compliant Storage Providers including AWS, Ceph, ChinaMobile, ArvanCloud, Dreamhost, IBM COS, Liara, Minio, and Tencent COS
- \ "s3"
-[snip]
-Storage> s3
-Choose your S3 provider.
-Choose a number from below, or type in your own value
- 1 / Amazon Web Services (AWS) S3
- \ "AWS"
- 2 / Ceph Object Storage
- \ "Ceph"
- 3 / DigitalOcean Spaces
- \ "DigitalOcean"
- 4 / Dreamhost DreamObjects
- \ "Dreamhost"
- 5 / IBM COS S3
- \ "IBMCOS"
- 6 / Minio Object Storage
- \ "Minio"
- 7 / Wasabi Object Storage
- \ "Wasabi"
- 8 / Any other S3 compatible provider
- \ "Other"
-provider> 1
-Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars). Only applies if access_key_id and secret_access_key is blank.
-Choose a number from below, or type in your own value
- 1 / Enter AWS credentials in the next step
- \ "false"
- 2 / Get AWS credentials from the environment (env vars or IAM)
- \ "true"
-env_auth> 1
-AWS Access Key ID - leave blank for anonymous access or runtime credentials.
-access_key_id> XXX
-AWS Secret Access Key (password) - leave blank for anonymous access or runtime credentials.
-secret_access_key> YYY
-Region to connect to.
-Choose a number from below, or type in your own value
- / The default endpoint - a good choice if you are unsure.
- 1 | US Region, Northern Virginia, or Pacific Northwest.
- | Leave location constraint empty.
- \ "us-east-1"
- / US East (Ohio) Region
- 2 | Needs location constraint us-east-2.
- \ "us-east-2"
- / US West (Oregon) Region
- 3 | Needs location constraint us-west-2.
- \ "us-west-2"
- / US West (Northern California) Region
- 4 | Needs location constraint us-west-1.
- \ "us-west-1"
- / Canada (Central) Region
- 5 | Needs location constraint ca-central-1.
- \ "ca-central-1"
- / EU (Ireland) Region
- 6 | Needs location constraint EU or eu-west-1.
- \ "eu-west-1"
- / EU (London) Region
- 7 | Needs location constraint eu-west-2.
- \ "eu-west-2"
- / EU (Frankfurt) Region
- 8 | Needs location constraint eu-central-1.
- \ "eu-central-1"
- / Asia Pacific (Singapore) Region
- 9 | Needs location constraint ap-southeast-1.
- \ "ap-southeast-1"
- / Asia Pacific (Sydney) Region
-10 | Needs location constraint ap-southeast-2.
- \ "ap-southeast-2"
- / Asia Pacific (Tokyo) Region
-11 | Needs location constraint ap-northeast-1.
- \ "ap-northeast-1"
- / Asia Pacific (Seoul)
-12 | Needs location constraint ap-northeast-2.
- \ "ap-northeast-2"
- / Asia Pacific (Mumbai)
-13 | Needs location constraint ap-south-1.
- \ "ap-south-1"
- / Asia Pacific (Hong Kong) Region
-14 | Needs location constraint ap-east-1.
- \ "ap-east-1"
- / South America (Sao Paulo) Region
-15 | Needs location constraint sa-east-1.
- \ "sa-east-1"
-region> 1
-Endpoint for S3 API.
-Leave blank if using AWS to use the default endpoint for the region.
-endpoint>
-Location constraint - must be set to match the Region. Used when creating buckets only.
-Choose a number from below, or type in your own value
- 1 / Empty for US Region, Northern Virginia, or Pacific Northwest.
- \ ""
- 2 / US East (Ohio) Region.
- \ "us-east-2"
- 3 / US West (Oregon) Region.
- \ "us-west-2"
- 4 / US West (Northern California) Region.
- \ "us-west-1"
- 5 / Canada (Central) Region.
- \ "ca-central-1"
- 6 / EU (Ireland) Region.
- \ "eu-west-1"
- 7 / EU (London) Region.
- \ "eu-west-2"
- 8 / EU Region.
- \ "EU"
- 9 / Asia Pacific (Singapore) Region.
- \ "ap-southeast-1"
-10 / Asia Pacific (Sydney) Region.
- \ "ap-southeast-2"
-11 / Asia Pacific (Tokyo) Region.
- \ "ap-northeast-1"
-12 / Asia Pacific (Seoul)
- \ "ap-northeast-2"
-13 / Asia Pacific (Mumbai)
- \ "ap-south-1"
-14 / Asia Pacific (Hong Kong)
- \ "ap-east-1"
-15 / South America (Sao Paulo) Region.
- \ "sa-east-1"
-location_constraint> 1
-Canned ACL used when creating buckets and/or storing objects in S3.
-For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
-Choose a number from below, or type in your own value
- 1 / Owner gets FULL_CONTROL. No one else has access rights (default).
- \ "private"
- 2 / Owner gets FULL_CONTROL. The AllUsers group gets READ access.
- \ "public-read"
- / Owner gets FULL_CONTROL. The AllUsers group gets READ and WRITE access.
- 3 | Granting this on a bucket is generally not recommended.
- \ "public-read-write"
- 4 / Owner gets FULL_CONTROL. The AuthenticatedUsers group gets READ access.
- \ "authenticated-read"
- / Object owner gets FULL_CONTROL. Bucket owner gets READ access.
- 5 | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
- \ "bucket-owner-read"
- / Both the object owner and the bucket owner get FULL_CONTROL over the object.
- 6 | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
- \ "bucket-owner-full-control"
-acl> 1
-The server-side encryption algorithm used when storing this object in S3.
-Choose a number from below, or type in your own value
- 1 / None
- \ ""
- 2 / AES256
- \ "AES256"
-server_side_encryption> 1
-The storage class to use when storing objects in S3.
-Choose a number from below, or type in your own value
- 1 / Default
- \ ""
- 2 / Standard storage class
- \ "STANDARD"
- 3 / Reduced redundancy storage class
- \ "REDUCED_REDUNDANCY"
- 4 / Standard Infrequent Access storage class
- \ "STANDARD_IA"
- 5 / One Zone Infrequent Access storage class
- \ "ONEZONE_IA"
- 6 / Glacier storage class
- \ "GLACIER"
- 7 / Glacier Deep Archive storage class
- \ "DEEP_ARCHIVE"
- 8 / Intelligent-Tiering storage class
- \ "INTELLIGENT_TIERING"
- 9 / Glacier Instant Retrieval storage class
- \ "GLACIER_IR"
-storage_class> 1
-Remote config
---------------------
-[remote]
-type = s3
-provider = AWS
-env_auth = false
-access_key_id = XXX
-secret_access_key = YYY
-region = us-east-1
-endpoint =
-location_constraint =
-acl = private
-server_side_encryption =
-storage_class =
---------------------
-y) Yes this is OK
-e) Edit this remote
-d) Delete this remote
-y/e/d>
-```
+Whether to url encode listings: true/false/unset
-### Modified time
+Some providers support URL encoding listings and where this is
+available this is more reliable when using control characters in file
+names. If this is set to unset (the default) then rclone will choose
+according to the provider setting what to apply, but you can override
+rclone's choice here.
-The modified time is stored as metadata on the object as
-`X-Amz-Meta-Mtime` as floating point since the epoch, accurate to 1 ns.
-If the modification time needs to be updated rclone will attempt to perform a server
-side copy to update the modification if the object can be copied in a single part.
-In the case the object is larger than 5Gb or is in Glacier or Glacier Deep Archive
-storage the object will be uploaded rather than copied.
+Properties:
-Note that reading this from the object takes an additional `HEAD`
-request as the metadata isn't returned in object listings.
+- Config: list_url_encode
+- Env Var: RCLONE_S3_LIST_URL_ENCODE
+- Type: Tristate
+- Default: unset
-### Reducing costs
+#### --s3-no-check-bucket
-#### Avoiding HEAD requests to read the modification time
+If set, don't attempt to check the bucket exists or create it.
-By default, rclone will use the modification time of objects stored in
-S3 for syncing. This is stored in object metadata which unfortunately
-takes an extra HEAD request to read which can be expensive (in time
-and money).
+This can be useful when trying to minimise the number of transactions
+rclone does if you know the bucket exists already.
-The modification time is used by default for all operations that
-require checking the time a file was last updated. It allows rclone to
-treat the remote more like a true filesystem, but it is inefficient on
-S3 because it requires an extra API call to retrieve the metadata.
+It can also be needed if the user you are using does not have bucket
+creation permissions. Before v1.52.0 this would have passed silently
+due to a bug.
-The extra API calls can be avoided when syncing (using `rclone sync`
-or `rclone copy`) in a few different ways, each with its own
-tradeoffs.
-- `--size-only`
- - Only checks the size of files.
- - Uses no extra transactions.
- - If the file doesn't change size then rclone won't detect it has
- changed.
- - `rclone sync --size-only /path/to/source s3:bucket`
-- `--checksum`
- - Checks the size and MD5 checksum of files.
- - Uses no extra transactions.
- - The most accurate detection of changes possible.
- - Will cause the source to read an MD5 checksum which, if it is a
- local disk, will cause lots of disk activity.
- - If the source and destination are both S3 this is the
- **recommended** flag to use for maximum efficiency.
- - `rclone sync --checksum /path/to/source s3:bucket`
-- `--update --use-server-modtime`
- - Uses no extra transactions.
- - Modification time becomes the time the object was uploaded.
- - For many operations this is sufficient to determine if it needs
- uploading.
- - Using `--update` along with `--use-server-modtime`, avoids the
- extra API call and uploads files whose local modification time
- is newer than the time it was last uploaded.
- - Files created with timestamps in the past will be missed by the sync.
- - `rclone sync --update --use-server-modtime /path/to/source s3:bucket`
+Properties:
-These flags can and should be used in combination with `--fast-list` -
-see below.
+- Config: no_check_bucket
+- Env Var: RCLONE_S3_NO_CHECK_BUCKET
+- Type: bool
+- Default: false
-If using `rclone mount` or any command using the VFS (eg `rclone
-serve`) commands then you might want to consider using the VFS flag
-`--no-modtime` which will stop rclone reading the modification time
-for every object. You could also use `--use-server-modtime` if you are
-happy with the modification times of the objects being the time of
-upload.
+#### --s3-no-head
-#### Avoiding GET requests to read directory listings
+If set, don't HEAD uploaded objects to check integrity.
-Rclone's default directory traversal is to process each directory
-individually. This takes one API call per directory. Using the
-`--fast-list` flag will read all info about the objects into
-memory first using a smaller number of API calls (one per 1000
-objects). See the [rclone docs](https://rclone.org/docs/#fast-list) for more details.
+This can be useful when trying to minimise the number of transactions
+rclone does.
- rclone sync --fast-list --checksum /path/to/source s3:bucket
+Setting it means that if rclone receives a 200 OK message after
+uploading an object with PUT then it will assume that it got uploaded
+properly.
-`--fast-list` trades off API transactions for memory use. As a rough
-guide rclone uses 1k of memory per object stored, so using
-`--fast-list` on a sync of a million objects will use roughly 1 GiB of
-RAM.
+In particular it will assume:
-If you are only copying a small number of files into a big repository
-then using `--no-traverse` is a good idea. This finds objects directly
-instead of through directory listings. You can do a "top-up" sync very
-cheaply by using `--max-age` and `--no-traverse` to copy only recent
-files, eg
+- the metadata, including modtime, storage class and content type was as uploaded
+- the size was as uploaded
- rclone copy --max-age 24h --no-traverse /path/to/source s3:bucket
+It reads the following items from the response for a single part PUT:
-You'd then do a full `rclone sync` less often.
+- the MD5SUM
+- The uploaded date
-Note that `--fast-list` isn't required in the top-up sync.
+For multipart uploads these items aren't read.
-#### Avoiding HEAD requests after PUT
+If an source object of unknown length is uploaded then rclone **will** do a
+HEAD request.
-By default, rclone will HEAD every object it uploads. It does this to
-check the object got uploaded correctly.
+Setting this flag increases the chance for undetected upload failures,
+in particular an incorrect size, so it isn't recommended for normal
+operation. In practice the chance of an undetected upload failure is
+very small even with this flag.
-You can disable this with the [--s3-no-head](#s3-no-head) option - see
-there for more details.
-Setting this flag increases the chance for undetected upload failures.
+Properties:
-### Hashes
+- Config: no_head
+- Env Var: RCLONE_S3_NO_HEAD
+- Type: bool
+- Default: false
-For small objects which weren't uploaded as multipart uploads (objects
-sized below `--s3-upload-cutoff` if uploaded with rclone) rclone uses
-the `ETag:` header as an MD5 checksum.
+#### --s3-no-head-object
-However for objects which were uploaded as multipart uploads or with
-server side encryption (SSE-AWS or SSE-C) the `ETag` header is no
-longer the MD5 sum of the data, so rclone adds an additional piece of
-metadata `X-Amz-Meta-Md5chksum` which is a base64 encoded MD5 hash (in
-the same format as is required for `Content-MD5`). You can use base64 -d and hexdump to check this value manually:
+If set, do not do HEAD before GET when getting objects.
- echo 'VWTGdNx3LyXQDfA0e2Edxw==' | base64 -d | hexdump
+Properties:
-or you can use `rclone check` to verify the hashes are OK.
+- Config: no_head_object
+- Env Var: RCLONE_S3_NO_HEAD_OBJECT
+- Type: bool
+- Default: false
-For large objects, calculating this hash can take some time so the
-addition of this hash can be disabled with `--s3-disable-checksum`.
-This will mean that these objects do not have an MD5 checksum.
+#### --s3-encoding
-Note that reading this from the object takes an additional `HEAD`
-request as the metadata isn't returned in object listings.
+The encoding for the backend.
-### Versions
+See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info.
-When bucket versioning is enabled (this can be done with rclone with
-the [`rclone backend versioning`](#versioning) command) when rclone
-uploads a new version of a file it creates a
-[new version of it](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Versioning.html)
-Likewise when you delete a file, the old version will be marked hidden
-and still be available.
+Properties:
-Old versions of files, where available, are visible using the
-[`--s3-versions`](#s3-versions) flag.
+- Config: encoding
+- Env Var: RCLONE_S3_ENCODING
+- Type: Encoding
+- Default: Slash,InvalidUtf8,Dot
-It is also possible to view a bucket as it was at a certain point in
-time, using the [`--s3-version-at`](#s3-version-at) flag. This will
-show the file versions as they were at that time, showing files that
-have been deleted afterwards, and hiding files that were created
-since.
+#### --s3-memory-pool-flush-time
-If you wish to remove all the old versions then you can use the
-[`rclone backend cleanup-hidden remote:bucket`](#cleanup-hidden)
-command which will delete all the old hidden versions of files,
-leaving the current ones intact. You can also supply a path and only
-old versions under that path will be deleted, e.g.
-`rclone backend cleanup-hidden remote:bucket/path/to/stuff`.
+How often internal memory buffer pools will be flushed. (no longer used)
-When you `purge` a bucket, the current and the old versions will be
-deleted then the bucket will be deleted.
+Properties:
-However `delete` will cause the current versions of the files to
-become hidden old versions.
+- Config: memory_pool_flush_time
+- Env Var: RCLONE_S3_MEMORY_POOL_FLUSH_TIME
+- Type: Duration
+- Default: 1m0s
-Here is a session showing the listing and retrieval of an old
-version followed by a `cleanup` of the old versions.
+#### --s3-memory-pool-use-mmap
-Show current version and all the versions with `--s3-versions` flag.
+Whether to use mmap buffers in internal memory pool. (no longer used)
-```
-$ rclone -q ls s3:cleanup-test
- 9 one.txt
+Properties:
-$ rclone -q --s3-versions ls s3:cleanup-test
- 9 one.txt
- 8 one-v2016-07-04-141032-000.txt
- 16 one-v2016-07-04-141003-000.txt
- 15 one-v2016-07-02-155621-000.txt
-```
+- Config: memory_pool_use_mmap
+- Env Var: RCLONE_S3_MEMORY_POOL_USE_MMAP
+- Type: bool
+- Default: false
-Retrieve an old version
+#### --s3-disable-http2
-```
-$ rclone -q --s3-versions copy s3:cleanup-test/one-v2016-07-04-141003-000.txt /tmp
+Disable usage of http2 for S3 backends.
-$ ls -l /tmp/one-v2016-07-04-141003-000.txt
--rw-rw-r-- 1 ncw ncw 16 Jul 2 17:46 /tmp/one-v2016-07-04-141003-000.txt
-```
+There is currently an unsolved issue with the s3 (specifically minio) backend
+and HTTP/2. HTTP/2 is enabled by default for the s3 backend but can be
+disabled here. When the issue is solved this flag will be removed.
-Clean up all the old versions and show that they've gone.
+See: https://github.com/rclone/rclone/issues/4673, https://github.com/rclone/rclone/issues/3631
-```
-$ rclone -q backend cleanup-hidden s3:cleanup-test
-$ rclone -q ls s3:cleanup-test
- 9 one.txt
-$ rclone -q --s3-versions ls s3:cleanup-test
- 9 one.txt
-```
+Properties:
-#### Versions naming caveat
+- Config: disable_http2
+- Env Var: RCLONE_S3_DISABLE_HTTP2
+- Type: bool
+- Default: false
-When using `--s3-versions` flag rclone is relying on the file name
-to work out whether the objects are versions or not. Versions' names
-are created by inserting timestamp between file name and its extension.
-```
- 9 file.txt
- 8 file-v2023-07-17-161032-000.txt
- 16 file-v2023-06-15-141003-000.txt
-```
-If there are real files present with the same names as versions, then
-behaviour of `--s3-versions` can be unpredictable.
+#### --s3-download-url
-### Cleanup
+Custom endpoint for downloads.
+This is usually set to a CloudFront CDN URL as AWS S3 offers
+cheaper egress for data downloaded through the CloudFront network.
-If you run `rclone cleanup s3:bucket` then it will remove all pending
-multipart uploads older than 24 hours. You can use the `--interactive`/`i`
-or `--dry-run` flag to see exactly what it will do. If you want more control over the
-expiry date then run `rclone backend cleanup s3:bucket -o max-age=1h`
-to expire all uploads older than one hour. You can use `rclone backend
-list-multipart-uploads s3:bucket` to see the pending multipart
-uploads.
+Properties:
-### Restricted filename characters
+- Config: download_url
+- Env Var: RCLONE_S3_DOWNLOAD_URL
+- Type: string
+- Required: false
-S3 allows any valid UTF-8 string as a key.
+#### --s3-directory-markers
-Invalid UTF-8 bytes will be [replaced](https://rclone.org/overview/#invalid-utf8), as
-they can't be used in XML.
+Upload an empty object with a trailing slash when a new directory is created
-The following characters are replaced since these are problematic when
-dealing with the REST API:
+Empty folders are unsupported for bucket based remotes, this option creates an empty
+object ending with "/", to persist the folder.
-| Character | Value | Replacement |
-| --------- |:-----:|:-----------:|
-| NUL | 0x00 | †|
-| / | 0x2F | ï¼ |
-The encoding will also encode these file names as they don't seem to
-work with the SDK properly:
+Properties:
-| File name | Replacement |
-| --------- |:-----------:|
-| . | ． |
-| .. | ．． |
+- Config: directory_markers
+- Env Var: RCLONE_S3_DIRECTORY_MARKERS
+- Type: bool
+- Default: false
-### Multipart uploads
+#### --s3-use-multipart-etag
-rclone supports multipart uploads with S3 which means that it can
-upload files bigger than 5 GiB.
+Whether to use ETag in multipart uploads for verification
-Note that files uploaded *both* with multipart upload *and* through
-crypt remotes do not have MD5 sums.
+This should be true, false or left unset to use the default for the provider.
-rclone switches from single part uploads to multipart uploads at the
-point specified by `--s3-upload-cutoff`. This can be a maximum of 5 GiB
-and a minimum of 0 (ie always upload multipart files).
-The chunk sizes used in the multipart upload are specified by
-`--s3-chunk-size` and the number of chunks uploaded concurrently is
-specified by `--s3-upload-concurrency`.
+Properties:
-Multipart uploads will use `--transfers` * `--s3-upload-concurrency` *
-`--s3-chunk-size` extra memory. Single part uploads to not use extra
-memory.
+- Config: use_multipart_etag
+- Env Var: RCLONE_S3_USE_MULTIPART_ETAG
+- Type: Tristate
+- Default: unset
-Single part transfers can be faster than multipart transfers or slower
-depending on your latency from S3 - the more latency, the more likely
-single part transfers will be faster.
+#### --s3-use-presigned-request
-Increasing `--s3-upload-concurrency` will increase throughput (8 would
-be a sensible value) and increasing `--s3-chunk-size` also increases
-throughput (16M would be sensible). Increasing either of these will
-use more memory. The default values are high enough to gain most of
-the possible performance without using too much memory.
+Whether to use a presigned request or PutObject for single part uploads
+
+If this is false rclone will use PutObject from the AWS SDK to upload
+an object.
+Versions of rclone < 1.59 use presigned requests to upload a single
+part object and setting this flag to true will re-enable that
+functionality. This shouldn't be necessary except in exceptional
+circumstances or for testing.
-### Buckets and Regions
-With Amazon S3 you can list buckets (`rclone lsd`) using any region,
-but you can only access the content of a bucket from the region it was
-created in. If you attempt to access a bucket from the wrong region,
-you will get an error, `incorrect region, the bucket is not in 'XXX'
-region`.
+Properties:
-### Authentication
+- Config: use_presigned_request
+- Env Var: RCLONE_S3_USE_PRESIGNED_REQUEST
+- Type: bool
+- Default: false
-There are a number of ways to supply `rclone` with a set of AWS
-credentials, with and without using the environment.
+#### --s3-versions
-The different authentication methods are tried in this order:
+Include old versions in directory listings.
- - Directly in the rclone configuration file (`env_auth = false` in the config file):
- - `access_key_id` and `secret_access_key` are required.
- - `session_token` can be optionally set when using AWS STS.
- - Runtime configuration (`env_auth = true` in the config file):
- - Export the following environment variables before running `rclone`:
- - Access Key ID: `AWS_ACCESS_KEY_ID` or `AWS_ACCESS_KEY`
- - Secret Access Key: `AWS_SECRET_ACCESS_KEY` or `AWS_SECRET_KEY`
- - Session Token: `AWS_SESSION_TOKEN` (optional)
- - Or, use a [named profile](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html):
- - Profile files are standard files used by AWS CLI tools
- - By default it will use the profile in your home directory (e.g. `~/.aws/credentials` on unix based systems) file and the "default" profile, to change set these environment variables:
- - `AWS_SHARED_CREDENTIALS_FILE` to control which file.
- - `AWS_PROFILE` to control which profile to use.
- - Or, run `rclone` in an ECS task with an IAM role (AWS only).
- - Or, run `rclone` on an EC2 instance with an IAM role (AWS only).
- - Or, run `rclone` in an EKS pod with an IAM role that is associated with a service account (AWS only).
+Properties:
+
+- Config: versions
+- Env Var: RCLONE_S3_VERSIONS
+- Type: bool
+- Default: false
+
+#### --s3-version-at
+
+Show file versions as they were at the specified time.
+
+The parameter should be a date, "2006-01-02", datetime "2006-01-02
+15:04:05" or a duration for that long ago, eg "100d" or "1h".
+
+Note that when using this no file write operations are permitted,
+so you can't upload files or delete them.
+
+See [the time option docs](https://rclone.org/docs/#time-option) for valid formats.
+
+
+Properties:
+
+- Config: version_at
+- Env Var: RCLONE_S3_VERSION_AT
+- Type: Time
+- Default: off
+
+#### --s3-decompress
+
+If set this will decompress gzip encoded objects.
+
+It is possible to upload objects to S3 with "Content-Encoding: gzip"
+set. Normally rclone will download these files as compressed objects.
+
+If this flag is set then rclone will decompress these files with
+"Content-Encoding: gzip" as they are received. This means that rclone
+can't check the size and hash but the file contents will be decompressed.
+
+
+Properties:
+
+- Config: decompress
+- Env Var: RCLONE_S3_DECOMPRESS
+- Type: bool
+- Default: false
+
+#### --s3-might-gzip
+
+Set this if the backend might gzip objects.
+
+Normally providers will not alter objects when they are downloaded. If
+an object was not uploaded with `Content-Encoding: gzip` then it won't
+be set on download.
+
+However some providers may gzip objects even if they weren't uploaded
+with `Content-Encoding: gzip` (eg Cloudflare).
+
+A symptom of this would be receiving errors like
+
+ ERROR corrupted on transfer: sizes differ NNN vs MMM
+
+If you set this flag and rclone downloads an object with
+Content-Encoding: gzip set and chunked transfer encoding, then rclone
+will decompress the object on the fly.
+
+If this is set to unset (the default) then rclone will choose
+according to the provider setting what to apply, but you can override
+rclone's choice here.
+
+
+Properties:
+
+- Config: might_gzip
+- Env Var: RCLONE_S3_MIGHT_GZIP
+- Type: Tristate
+- Default: unset
+
+#### --s3-use-accept-encoding-gzip
+
+Whether to send `Accept-Encoding: gzip` header.
+
+By default, rclone will append `Accept-Encoding: gzip` to the request to download
+compressed objects whenever possible.
+
+However some providers such as Google Cloud Storage may alter the HTTP headers, breaking
+the signature of the request.
+
+A symptom of this would be receiving errors like
+
+ SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided.
+
+In this case, you might want to try disabling this option.
+
+
+Properties:
+
+- Config: use_accept_encoding_gzip
+- Env Var: RCLONE_S3_USE_ACCEPT_ENCODING_GZIP
+- Type: Tristate
+- Default: unset
+
+#### --s3-no-system-metadata
+
+Suppress setting and reading of system metadata
+
+Properties:
+
+- Config: no_system_metadata
+- Env Var: RCLONE_S3_NO_SYSTEM_METADATA
+- Type: bool
+- Default: false
+
+#### --s3-sts-endpoint
+
+Endpoint for STS.
+
+Leave blank if using AWS to use the default endpoint for the region.
+
+Properties:
+
+- Config: sts_endpoint
+- Env Var: RCLONE_S3_STS_ENDPOINT
+- Provider: AWS
+- Type: string
+- Required: false
+
+#### --s3-use-already-exists
+
+Set if rclone should report BucketAlreadyExists errors on bucket creation.
+
+At some point during the evolution of the s3 protocol, AWS started
+returning an `AlreadyOwnedByYou` error when attempting to create a
+bucket that the user already owned, rather than a
+`BucketAlreadyExists` error.
+
+Unfortunately exactly what has been implemented by s3 clones is a
+little inconsistent, some return `AlreadyOwnedByYou`, some return
+`BucketAlreadyExists` and some return no error at all.
+
+This is important to rclone because it ensures the bucket exists by
+creating it on quite a lot of operations (unless
+`--s3-no-check-bucket` is used).
+
+If rclone knows the provider can return `AlreadyOwnedByYou` or returns
+no error then it can report `BucketAlreadyExists` errors when the user
+attempts to create a bucket not owned by them. Otherwise rclone
+ignores the `BucketAlreadyExists` error which can lead to confusion.
+
+This should be automatically set correctly for all providers rclone
+knows about - please make a bug report if not.
+
+
+Properties:
+
+- Config: use_already_exists
+- Env Var: RCLONE_S3_USE_ALREADY_EXISTS
+- Type: Tristate
+- Default: unset
+
+#### --s3-use-multipart-uploads
+
+Set if rclone should use multipart uploads.
+
+You can change this if you want to disable the use of multipart uploads.
+This shouldn't be necessary in normal operation.
+
+This should be automatically set correctly for all providers rclone
+knows about - please make a bug report if not.
+
+
+Properties:
+
+- Config: use_multipart_uploads
+- Env Var: RCLONE_S3_USE_MULTIPART_UPLOADS
+- Type: Tristate
+- Default: unset
+
+### Metadata
+
+User metadata is stored as x-amz-meta- keys. S3 metadata keys are case insensitive and are always returned in lower case.
+
+Here are the possible system metadata items for the s3 backend.
-If none of these option actually end up providing `rclone` with AWS
-credentials then S3 interaction will be non-authenticated (see below).
+| Name | Help | Type | Example | Read Only |
+|------|------|------|---------|-----------|
+| btime | Time of file birth (creation) read from Last-Modified header | RFC 3339 | 2006-01-02T15:04:05.999999999Z07:00 | **Y** |
+| cache-control | Cache-Control header | string | no-cache | N |
+| content-disposition | Content-Disposition header | string | inline | N |
+| content-encoding | Content-Encoding header | string | gzip | N |
+| content-language | Content-Language header | string | en-US | N |
+| content-type | Content-Type header | string | text/plain | N |
+| mtime | Time of last modification, read from rclone metadata | RFC 3339 | 2006-01-02T15:04:05.999999999Z07:00 | N |
+| tier | Tier of the object | string | GLACIER | **Y** |
-### S3 Permissions
+See the [metadata](https://rclone.org/docs/#metadata) docs for more info.
-When using the `sync` subcommand of `rclone` the following minimum
-permissions are required to be available on the bucket being written to:
+## Backend commands
-* `ListBucket`
-* `DeleteObject`
-* `GetObject`
-* `PutObject`
-* `PutObjectACL`
+Here are the commands specific to the s3 backend.
-When using the `lsd` subcommand, the `ListAllMyBuckets` permission is required.
+Run them with
-Example policy:
+ rclone backend COMMAND remote:
-```
-{
- "Version": "2012-10-17",
- "Statement": [
- {
- "Effect": "Allow",
- "Principal": {
- "AWS": "arn:aws:iam::USER_SID:user/USER_NAME"
- },
- "Action": [
- "s3:ListBucket",
- "s3:DeleteObject",
- "s3:GetObject",
- "s3:PutObject",
- "s3:PutObjectAcl"
- ],
- "Resource": [
- "arn:aws:s3:::BUCKET_NAME/*",
- "arn:aws:s3:::BUCKET_NAME"
- ]
- },
- {
- "Effect": "Allow",
- "Action": "s3:ListAllMyBuckets",
- "Resource": "arn:aws:s3:::*"
- }
- ]
-}
-```
+The help below will explain what arguments each command takes.
-Notes on above:
+See the [backend](https://rclone.org/commands/rclone_backend/) command for more
+info on how to pass options and arguments.
-1. This is a policy that can be used when creating bucket. It assumes
- that `USER_NAME` has been created.
-2. The Resource entry must include both resource ARNs, as one implies
- the bucket and the other implies the bucket's objects.
+These can be run on a running backend using the rc command
+[backend/command](https://rclone.org/rc/#backend-command).
-For reference, [here's an Ansible script](https://gist.github.com/ebridges/ebfc9042dd7c756cd101cfa807b7ae2b)
-that will generate one or more buckets that will work with `rclone sync`.
+### restore
-### Key Management System (KMS)
+Restore objects from GLACIER to normal storage
-If you are using server-side encryption with KMS then you must make
-sure rclone is configured with `server_side_encryption = aws:kms`
-otherwise you will find you can't transfer small objects - these will
-create checksum errors.
+ rclone backend restore remote: [options] [+]
-### Glacier and Glacier Deep Archive
+This command can be used to restore one or more objects from GLACIER
+to normal storage.
-You can upload objects using the glacier storage class or transition them to glacier using a [lifecycle policy](http://docs.aws.amazon.com/AmazonS3/latest/user-guide/create-lifecycle.html).
-The bucket can still be synced or copied into normally, but if rclone
-tries to access data from the glacier storage class you will see an error like below.
+Usage Examples:
- 2017/09/11 19:07:43 Failed to sync: failed to open source object: Object in GLACIER, restore first: path/to/file
+ rclone backend restore s3:bucket/path/to/object -o priority=PRIORITY -o lifetime=DAYS
+ rclone backend restore s3:bucket/path/to/directory -o priority=PRIORITY -o lifetime=DAYS
+ rclone backend restore s3:bucket -o priority=PRIORITY -o lifetime=DAYS
-In this case you need to [restore](http://docs.aws.amazon.com/AmazonS3/latest/user-guide/restore-archived-objects.html)
-the object(s) in question before using rclone.
+This flag also obeys the filters. Test first with --interactive/-i or --dry-run flags
-Note that rclone only speaks the S3 API it does not speak the Glacier
-Vault API, so rclone cannot directly access Glacier Vaults.
+ rclone --interactive backend restore --include "*.txt" s3:bucket/path -o priority=Standard -o lifetime=1
-### Object-lock enabled S3 bucket
+All the objects shown will be marked for restore, then
-According to AWS's [documentation on S3 Object Lock](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock-overview.html#object-lock-permission):
+ rclone backend restore --include "*.txt" s3:bucket/path -o priority=Standard -o lifetime=1
-> If you configure a default retention period on a bucket, requests to upload objects in such a bucket must include the Content-MD5 header.
+It returns a list of status dictionaries with Remote and Status
+keys. The Status will be OK if it was successful or an error message
+if not.
-As mentioned in the [Hashes](#hashes) section, small files that are not uploaded as multipart, use a different tag, causing the upload to fail.
-A simple solution is to set the `--s3-upload-cutoff 0` and force all the files to be uploaded as multipart.
+ [
+ {
+ "Status": "OK",
+ "Remote": "test.txt"
+ },
+ {
+ "Status": "OK",
+ "Remote": "test/file4.txt"
+ }
+ ]
-### Standard options
-Here are the Standard options specific to s3 (Amazon S3 Compliant Storage Providers including AWS, Alibaba, ArvanCloud, Ceph, China Mobile, Cloudflare, GCS, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, IDrive e2, IONOS Cloud, Leviia, Liara, Lyve Cloud, Minio, Netease, Petabox, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Synology, Tencent COS, Qiniu and Wasabi).
+Options:
-#### --s3-provider
+- "description": The optional description for the job.
+- "lifetime": Lifetime of the active copy in days
+- "priority": Priority of restore: Standard|Expedited|Bulk
-Choose your S3 provider.
+### restore-status
-Properties:
+Show the restore status for objects being restored from GLACIER to normal storage
-- Config: provider
-- Env Var: RCLONE_S3_PROVIDER
-- Type: string
-- Required: false
-- Examples:
- - "AWS"
- - Amazon Web Services (AWS) S3
- - "Alibaba"
- - Alibaba Cloud Object Storage System (OSS) formerly Aliyun
- - "ArvanCloud"
- - Arvan Cloud Object Storage (AOS)
- - "Ceph"
- - Ceph Object Storage
- - "ChinaMobile"
- - China Mobile Ecloud Elastic Object Storage (EOS)
- - "Cloudflare"
- - Cloudflare R2 Storage
- - "DigitalOcean"
- - DigitalOcean Spaces
- - "Dreamhost"
- - Dreamhost DreamObjects
- - "GCS"
- - Google Cloud Storage
- - "HuaweiOBS"
- - Huawei Object Storage Service
- - "IBMCOS"
- - IBM COS S3
- - "IDrive"
- - IDrive e2
- - "IONOS"
- - IONOS Cloud
- - "LyveCloud"
- - Seagate Lyve Cloud
- - "Leviia"
- - Leviia Object Storage
- - "Liara"
- - Liara Object Storage
- - "Minio"
- - Minio Object Storage
- - "Netease"
- - Netease Object Storage (NOS)
- - "Petabox"
- - Petabox Object Storage
- - "RackCorp"
- - RackCorp Object Storage
- - "Scaleway"
- - Scaleway Object Storage
- - "SeaweedFS"
- - SeaweedFS S3
- - "StackPath"
- - StackPath Object Storage
- - "Storj"
- - Storj (S3 Compatible Gateway)
- - "Synology"
- - Synology C2 Object Storage
- - "TencentCOS"
- - Tencent Cloud Object Storage (COS)
- - "Wasabi"
- - Wasabi Object Storage
- - "Qiniu"
- - Qiniu Object Storage (Kodo)
- - "Other"
- - Any other S3 compatible provider
+ rclone backend restore-status remote: [options] [+]
-#### --s3-env-auth
+This command can be used to show the status for objects being restored from GLACIER
+to normal storage.
-Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
+Usage Examples:
-Only applies if access_key_id and secret_access_key is blank.
+ rclone backend restore-status s3:bucket/path/to/object
+ rclone backend restore-status s3:bucket/path/to/directory
+ rclone backend restore-status -o all s3:bucket/path/to/directory
-Properties:
+This command does not obey the filters.
-- Config: env_auth
-- Env Var: RCLONE_S3_ENV_AUTH
-- Type: bool
-- Default: false
-- Examples:
- - "false"
- - Enter AWS credentials in the next step.
- - "true"
- - Get AWS credentials from the environment (env vars or IAM).
+It returns a list of status dictionaries.
-#### --s3-access-key-id
+ [
+ {
+ "Remote": "file.txt",
+ "VersionID": null,
+ "RestoreStatus": {
+ "IsRestoreInProgress": true,
+ "RestoreExpiryDate": "2023-09-06T12:29:19+01:00"
+ },
+ "StorageClass": "GLACIER"
+ },
+ {
+ "Remote": "test.pdf",
+ "VersionID": null,
+ "RestoreStatus": {
+ "IsRestoreInProgress": false,
+ "RestoreExpiryDate": "2023-09-06T12:29:19+01:00"
+ },
+ "StorageClass": "DEEP_ARCHIVE"
+ }
+ ]
-AWS Access Key ID.
-Leave blank for anonymous access or runtime credentials.
+Options:
-Properties:
+- "all": if set then show all objects, not just ones with restore status
-- Config: access_key_id
-- Env Var: RCLONE_S3_ACCESS_KEY_ID
-- Type: string
-- Required: false
+### list-multipart-uploads
-#### --s3-secret-access-key
+List the unfinished multipart uploads
-AWS Secret Access Key (password).
+ rclone backend list-multipart-uploads remote: [options] [+]
-Leave blank for anonymous access or runtime credentials.
+This command lists the unfinished multipart uploads in JSON format.
-Properties:
+ rclone backend list-multipart s3:bucket/path/to/object
-- Config: secret_access_key
-- Env Var: RCLONE_S3_SECRET_ACCESS_KEY
-- Type: string
-- Required: false
+It returns a dictionary of buckets with values as lists of unfinished
+multipart uploads.
-#### --s3-region
+You can call it with no bucket in which case it lists all bucket, with
+a bucket or with a bucket and path.
-Region to connect to.
+ {
+ "rclone": [
+ {
+ "Initiated": "2020-06-26T14:20:36Z",
+ "Initiator": {
+ "DisplayName": "XXX",
+ "ID": "arn:aws:iam::XXX:user/XXX"
+ },
+ "Key": "KEY",
+ "Owner": {
+ "DisplayName": null,
+ "ID": "XXX"
+ },
+ "StorageClass": "STANDARD",
+ "UploadId": "XXX"
+ }
+ ],
+ "rclone-1000files": [],
+ "rclone-dst": []
+ }
-Properties:
-- Config: region
-- Env Var: RCLONE_S3_REGION
-- Provider: AWS
-- Type: string
-- Required: false
-- Examples:
- - "us-east-1"
- - The default endpoint - a good choice if you are unsure.
- - US Region, Northern Virginia, or Pacific Northwest.
- - Leave location constraint empty.
- - "us-east-2"
- - US East (Ohio) Region.
- - Needs location constraint us-east-2.
- - "us-west-1"
- - US West (Northern California) Region.
- - Needs location constraint us-west-1.
- - "us-west-2"
- - US West (Oregon) Region.
- - Needs location constraint us-west-2.
- - "ca-central-1"
- - Canada (Central) Region.
- - Needs location constraint ca-central-1.
- - "eu-west-1"
- - EU (Ireland) Region.
- - Needs location constraint EU or eu-west-1.
- - "eu-west-2"
- - EU (London) Region.
- - Needs location constraint eu-west-2.
- - "eu-west-3"
- - EU (Paris) Region.
- - Needs location constraint eu-west-3.
- - "eu-north-1"
- - EU (Stockholm) Region.
- - Needs location constraint eu-north-1.
- - "eu-south-1"
- - EU (Milan) Region.
- - Needs location constraint eu-south-1.
- - "eu-central-1"
- - EU (Frankfurt) Region.
- - Needs location constraint eu-central-1.
- - "ap-southeast-1"
- - Asia Pacific (Singapore) Region.
- - Needs location constraint ap-southeast-1.
- - "ap-southeast-2"
- - Asia Pacific (Sydney) Region.
- - Needs location constraint ap-southeast-2.
- - "ap-northeast-1"
- - Asia Pacific (Tokyo) Region.
- - Needs location constraint ap-northeast-1.
- - "ap-northeast-2"
- - Asia Pacific (Seoul).
- - Needs location constraint ap-northeast-2.
- - "ap-northeast-3"
- - Asia Pacific (Osaka-Local).
- - Needs location constraint ap-northeast-3.
- - "ap-south-1"
- - Asia Pacific (Mumbai).
- - Needs location constraint ap-south-1.
- - "ap-east-1"
- - Asia Pacific (Hong Kong) Region.
- - Needs location constraint ap-east-1.
- - "sa-east-1"
- - South America (Sao Paulo) Region.
- - Needs location constraint sa-east-1.
- - "me-south-1"
- - Middle East (Bahrain) Region.
- - Needs location constraint me-south-1.
- - "af-south-1"
- - Africa (Cape Town) Region.
- - Needs location constraint af-south-1.
- - "cn-north-1"
- - China (Beijing) Region.
- - Needs location constraint cn-north-1.
- - "cn-northwest-1"
- - China (Ningxia) Region.
- - Needs location constraint cn-northwest-1.
- - "us-gov-east-1"
- - AWS GovCloud (US-East) Region.
- - Needs location constraint us-gov-east-1.
- - "us-gov-west-1"
- - AWS GovCloud (US) Region.
- - Needs location constraint us-gov-west-1.
-#### --s3-region
+### cleanup
-region - the location where your bucket will be created and your data stored.
+Remove unfinished multipart uploads.
+ rclone backend cleanup remote: [options] [+]
-Properties:
+This command removes unfinished multipart uploads of age greater than
+max-age which defaults to 24 hours.
-- Config: region
-- Env Var: RCLONE_S3_REGION
-- Provider: RackCorp
-- Type: string
-- Required: false
-- Examples:
- - "global"
- - Global CDN (All locations) Region
- - "au"
- - Australia (All states)
- - "au-nsw"
- - NSW (Australia) Region
- - "au-qld"
- - QLD (Australia) Region
- - "au-vic"
- - VIC (Australia) Region
- - "au-wa"
- - Perth (Australia) Region
- - "ph"
- - Manila (Philippines) Region
- - "th"
- - Bangkok (Thailand) Region
- - "hk"
- - HK (Hong Kong) Region
- - "mn"
- - Ulaanbaatar (Mongolia) Region
- - "kg"
- - Bishkek (Kyrgyzstan) Region
- - "id"
- - Jakarta (Indonesia) Region
- - "jp"
- - Tokyo (Japan) Region
- - "sg"
- - SG (Singapore) Region
- - "de"
- - Frankfurt (Germany) Region
- - "us"
- - USA (AnyCast) Region
- - "us-east-1"
- - New York (USA) Region
- - "us-west-1"
- - Freemont (USA) Region
- - "nz"
- - Auckland (New Zealand) Region
+Note that you can use --interactive/-i or --dry-run with this command to see what
+it would do.
-#### --s3-region
+ rclone backend cleanup s3:bucket/path/to/object
+ rclone backend cleanup -o max-age=7w s3:bucket/path/to/object
-Region to connect to.
+Durations are parsed as per the rest of rclone, 2h, 7d, 7w etc.
-Properties:
-- Config: region
-- Env Var: RCLONE_S3_REGION
-- Provider: Scaleway
-- Type: string
-- Required: false
-- Examples:
- - "nl-ams"
- - Amsterdam, The Netherlands
- - "fr-par"
- - Paris, France
- - "pl-waw"
- - Warsaw, Poland
+Options:
-#### --s3-region
+- "max-age": Max age of upload to delete
-Region to connect to. - the location where your bucket will be created and your data stored. Need bo be same with your endpoint.
+### cleanup-hidden
+Remove old versions of files.
-Properties:
+ rclone backend cleanup-hidden remote: [options] [+]
-- Config: region
-- Env Var: RCLONE_S3_REGION
-- Provider: HuaweiOBS
-- Type: string
-- Required: false
-- Examples:
- - "af-south-1"
- - AF-Johannesburg
- - "ap-southeast-2"
- - AP-Bangkok
- - "ap-southeast-3"
- - AP-Singapore
- - "cn-east-3"
- - CN East-Shanghai1
- - "cn-east-2"
- - CN East-Shanghai2
- - "cn-north-1"
- - CN North-Beijing1
- - "cn-north-4"
- - CN North-Beijing4
- - "cn-south-1"
- - CN South-Guangzhou
- - "ap-southeast-1"
- - CN-Hong Kong
- - "sa-argentina-1"
- - LA-Buenos Aires1
- - "sa-peru-1"
- - LA-Lima1
- - "na-mexico-1"
- - LA-Mexico City1
- - "sa-chile-1"
- - LA-Santiago2
- - "sa-brazil-1"
- - LA-Sao Paulo1
- - "ru-northwest-2"
- - RU-Moscow2
+This command removes any old hidden versions of files
+on a versions enabled bucket.
-#### --s3-region
+Note that you can use --interactive/-i or --dry-run with this command to see what
+it would do.
-Region to connect to.
+ rclone backend cleanup-hidden s3:bucket/path/to/dir
-Properties:
-- Config: region
-- Env Var: RCLONE_S3_REGION
-- Provider: Cloudflare
-- Type: string
-- Required: false
-- Examples:
- - "auto"
- - R2 buckets are automatically distributed across Cloudflare's data centers for low latency.
+### versioning
-#### --s3-region
+Set/get versioning support for a bucket.
-Region to connect to.
+ rclone backend versioning remote: [options] [+]
-Properties:
+This command sets versioning support if a parameter is
+passed and then returns the current versioning status for the bucket
+supplied.
-- Config: region
-- Env Var: RCLONE_S3_REGION
-- Provider: Qiniu
-- Type: string
-- Required: false
-- Examples:
- - "cn-east-1"
- - The default endpoint - a good choice if you are unsure.
- - East China Region 1.
- - Needs location constraint cn-east-1.
- - "cn-east-2"
- - East China Region 2.
- - Needs location constraint cn-east-2.
- - "cn-north-1"
- - North China Region 1.
- - Needs location constraint cn-north-1.
- - "cn-south-1"
- - South China Region 1.
- - Needs location constraint cn-south-1.
- - "us-north-1"
- - North America Region.
- - Needs location constraint us-north-1.
- - "ap-southeast-1"
- - Southeast Asia Region 1.
- - Needs location constraint ap-southeast-1.
- - "ap-northeast-1"
- - Northeast Asia Region 1.
- - Needs location constraint ap-northeast-1.
+ rclone backend versioning s3:bucket # read status only
+ rclone backend versioning s3:bucket Enabled
+ rclone backend versioning s3:bucket Suspended
-#### --s3-region
+It may return "Enabled", "Suspended" or "Unversioned". Note that once versioning
+has been enabled the status can't be set back to "Unversioned".
-Region where your bucket will be created and your data stored.
+### set
-Properties:
+Set command for updating the config parameters.
-- Config: region
-- Env Var: RCLONE_S3_REGION
-- Provider: IONOS
-- Type: string
-- Required: false
-- Examples:
- - "de"
- - Frankfurt, Germany
- - "eu-central-2"
- - Berlin, Germany
- - "eu-south-2"
- - Logrono, Spain
+ rclone backend set remote: [options] [+]
-#### --s3-region
+This set command can be used to update the config parameters
+for a running s3 backend.
-Region where your bucket will be created and your data stored.
+Usage Examples:
+ rclone backend set s3: [-o opt_name=opt_value] [-o opt_name2=opt_value2]
+ rclone rc backend/command command=set fs=s3: [-o opt_name=opt_value] [-o opt_name2=opt_value2]
+ rclone rc backend/command command=set fs=s3: -o session_token=X -o access_key_id=X -o secret_access_key=X
-Properties:
+The option keys are named as they are in the config file.
-- Config: region
-- Env Var: RCLONE_S3_REGION
-- Provider: Petabox
-- Type: string
-- Required: false
-- Examples:
- - "us-east-1"
- - US East (N. Virginia)
- - "eu-central-1"
- - Europe (Frankfurt)
- - "ap-southeast-1"
- - Asia Pacific (Singapore)
- - "me-south-1"
- - Middle East (Bahrain)
- - "sa-east-1"
- - South America (São Paulo)
+This rebuilds the connection to the s3 backend when it is called with
+the new parameters. Only new parameters need be passed as the values
+will default to those currently in use.
-#### --s3-region
+It doesn't return anything.
-Region where your data stored.
-Properties:
-- Config: region
-- Env Var: RCLONE_S3_REGION
-- Provider: Synology
-- Type: string
-- Required: false
-- Examples:
- - "eu-001"
- - Europe Region 1
- - "eu-002"
- - Europe Region 2
- - "us-001"
- - US Region 1
- - "us-002"
- - US Region 2
- - "tw-001"
- - Asia (Taiwan)
+### Anonymous access to public buckets
+
+If you want to use rclone to access a public bucket, configure with a
+blank `access_key_id` and `secret_access_key`. Your config should end
+up looking like this:
+
+```
+[anons3]
+type = s3
+provider = AWS
+env_auth = false
+access_key_id =
+secret_access_key =
+region = us-east-1
+endpoint =
+location_constraint =
+acl = private
+server_side_encryption =
+storage_class =
+```
+
+Then use it as normal with the name of the public bucket, e.g.
-#### --s3-region
+ rclone lsd anons3:1000genomes
-Region to connect to.
+You will be able to list and copy data but not upload it.
-Leave blank if you are using an S3 clone and you don't have a region.
+## Providers
-Properties:
+### AWS S3
-- Config: region
-- Env Var: RCLONE_S3_REGION
-- Provider: !AWS,Alibaba,ArvanCloud,ChinaMobile,Cloudflare,IONOS,Petabox,Liara,Qiniu,RackCorp,Scaleway,Storj,Synology,TencentCOS,HuaweiOBS,IDrive
-- Type: string
-- Required: false
-- Examples:
- - ""
- - Use this if unsure.
- - Will use v4 signatures and an empty region.
- - "other-v2-signature"
- - Use this only if v4 signatures don't work.
- - E.g. pre Jewel/v10 CEPH.
+This is the provider used as main example and described in the [configuration](#configuration) section above.
-#### --s3-endpoint
+### AWS Snowball Edge
-Endpoint for S3 API.
+[AWS Snowball](https://aws.amazon.com/snowball/) is a hardware
+appliance used for transferring bulk data back to AWS. Its main
+software interface is S3 object storage.
-Leave blank if using AWS to use the default endpoint for the region.
+To use rclone with AWS Snowball Edge devices, configure as standard
+for an 'S3 Compatible Service'.
-Properties:
+If using rclone pre v1.59 be sure to set `upload_cutoff = 0` otherwise
+you will run into authentication header issues as the snowball device
+does not support query parameter based authentication.
-- Config: endpoint
-- Env Var: RCLONE_S3_ENDPOINT
-- Provider: AWS
-- Type: string
-- Required: false
+With rclone v1.59 or later setting `upload_cutoff` should not be necessary.
-#### --s3-endpoint
+eg.
+```
+[snowball]
+type = s3
+provider = Other
+access_key_id = YOUR_ACCESS_KEY
+secret_access_key = YOUR_SECRET_KEY
+endpoint = http://[IP of Snowball]:8080
+upload_cutoff = 0
+```
-Endpoint for China Mobile Ecloud Elastic Object Storage (EOS) API.
+### Ceph
-Properties:
+[Ceph](https://ceph.com/) is an open-source, unified, distributed
+storage system designed for excellent performance, reliability and
+scalability. It has an S3 compatible object storage interface.
-- Config: endpoint
-- Env Var: RCLONE_S3_ENDPOINT
-- Provider: ChinaMobile
-- Type: string
-- Required: false
-- Examples:
- - "eos-wuxi-1.cmecloud.cn"
- - The default endpoint - a good choice if you are unsure.
- - East China (Suzhou)
- - "eos-jinan-1.cmecloud.cn"
- - East China (Jinan)
- - "eos-ningbo-1.cmecloud.cn"
- - East China (Hangzhou)
- - "eos-shanghai-1.cmecloud.cn"
- - East China (Shanghai-1)
- - "eos-zhengzhou-1.cmecloud.cn"
- - Central China (Zhengzhou)
- - "eos-hunan-1.cmecloud.cn"
- - Central China (Changsha-1)
- - "eos-zhuzhou-1.cmecloud.cn"
- - Central China (Changsha-2)
- - "eos-guangzhou-1.cmecloud.cn"
- - South China (Guangzhou-2)
- - "eos-dongguan-1.cmecloud.cn"
- - South China (Guangzhou-3)
- - "eos-beijing-1.cmecloud.cn"
- - North China (Beijing-1)
- - "eos-beijing-2.cmecloud.cn"
- - North China (Beijing-2)
- - "eos-beijing-4.cmecloud.cn"
- - North China (Beijing-3)
- - "eos-huhehaote-1.cmecloud.cn"
- - North China (Huhehaote)
- - "eos-chengdu-1.cmecloud.cn"
- - Southwest China (Chengdu)
- - "eos-chongqing-1.cmecloud.cn"
- - Southwest China (Chongqing)
- - "eos-guiyang-1.cmecloud.cn"
- - Southwest China (Guiyang)
- - "eos-xian-1.cmecloud.cn"
- - Nouthwest China (Xian)
- - "eos-yunnan.cmecloud.cn"
- - Yunnan China (Kunming)
- - "eos-yunnan-2.cmecloud.cn"
- - Yunnan China (Kunming-2)
- - "eos-tianjin-1.cmecloud.cn"
- - Tianjin China (Tianjin)
- - "eos-jilin-1.cmecloud.cn"
- - Jilin China (Changchun)
- - "eos-hubei-1.cmecloud.cn"
- - Hubei China (Xiangyan)
- - "eos-jiangxi-1.cmecloud.cn"
- - Jiangxi China (Nanchang)
- - "eos-gansu-1.cmecloud.cn"
- - Gansu China (Lanzhou)
- - "eos-shanxi-1.cmecloud.cn"
- - Shanxi China (Taiyuan)
- - "eos-liaoning-1.cmecloud.cn"
- - Liaoning China (Shenyang)
- - "eos-hebei-1.cmecloud.cn"
- - Hebei China (Shijiazhuang)
- - "eos-fujian-1.cmecloud.cn"
- - Fujian China (Xiamen)
- - "eos-guangxi-1.cmecloud.cn"
- - Guangxi China (Nanning)
- - "eos-anhui-1.cmecloud.cn"
- - Anhui China (Huainan)
+To use rclone with Ceph, configure as above but leave the region blank
+and set the endpoint. You should end up with something like this in
+your config:
-#### --s3-endpoint
-Endpoint for Arvan Cloud Object Storage (AOS) API.
+```
+[ceph]
+type = s3
+provider = Ceph
+env_auth = false
+access_key_id = XXX
+secret_access_key = YYY
+region =
+endpoint = https://ceph.endpoint.example.com
+location_constraint =
+acl =
+server_side_encryption =
+storage_class =
+```
-Properties:
+If you are using an older version of CEPH (e.g. 10.2.x Jewel) and a
+version of rclone before v1.59 then you may need to supply the
+parameter `--s3-upload-cutoff 0` or put this in the config file as
+`upload_cutoff 0` to work around a bug which causes uploading of small
+files to fail.
-- Config: endpoint
-- Env Var: RCLONE_S3_ENDPOINT
-- Provider: ArvanCloud
-- Type: string
-- Required: false
-- Examples:
- - "s3.ir-thr-at1.arvanstorage.ir"
- - The default endpoint - a good choice if you are unsure.
- - Tehran Iran (Simin)
- - "s3.ir-tbz-sh1.arvanstorage.ir"
- - Tabriz Iran (Shahriar)
+Note also that Ceph sometimes puts `/` in the passwords it gives
+users. If you read the secret access key using the command line tools
+you will get a JSON blob with the `/` escaped as `\/`. Make sure you
+only write `/` in the secret access key.
-#### --s3-endpoint
+Eg the dump from Ceph looks something like this (irrelevant keys
+removed).
-Endpoint for IBM COS S3 API.
+```
+{
+ "user_id": "xxx",
+ "display_name": "xxxx",
+ "keys": [
+ {
+ "user": "xxx",
+ "access_key": "xxxxxx",
+ "secret_key": "xxxxxx\/xxxx"
+ }
+ ],
+}
+```
-Specify if using an IBM COS On Premise.
+Because this is a json dump, it is encoding the `/` as `\/`, so if you
+use the secret key as `xxxxxx/xxxx` it will work fine.
-Properties:
+### Cloudflare R2 {#cloudflare-r2}
-- Config: endpoint
-- Env Var: RCLONE_S3_ENDPOINT
-- Provider: IBMCOS
-- Type: string
-- Required: false
-- Examples:
- - "s3.us.cloud-object-storage.appdomain.cloud"
- - US Cross Region Endpoint
- - "s3.dal.us.cloud-object-storage.appdomain.cloud"
- - US Cross Region Dallas Endpoint
- - "s3.wdc.us.cloud-object-storage.appdomain.cloud"
- - US Cross Region Washington DC Endpoint
- - "s3.sjc.us.cloud-object-storage.appdomain.cloud"
- - US Cross Region San Jose Endpoint
- - "s3.private.us.cloud-object-storage.appdomain.cloud"
- - US Cross Region Private Endpoint
- - "s3.private.dal.us.cloud-object-storage.appdomain.cloud"
- - US Cross Region Dallas Private Endpoint
- - "s3.private.wdc.us.cloud-object-storage.appdomain.cloud"
- - US Cross Region Washington DC Private Endpoint
- - "s3.private.sjc.us.cloud-object-storage.appdomain.cloud"
- - US Cross Region San Jose Private Endpoint
- - "s3.us-east.cloud-object-storage.appdomain.cloud"
- - US Region East Endpoint
- - "s3.private.us-east.cloud-object-storage.appdomain.cloud"
- - US Region East Private Endpoint
- - "s3.us-south.cloud-object-storage.appdomain.cloud"
- - US Region South Endpoint
- - "s3.private.us-south.cloud-object-storage.appdomain.cloud"
- - US Region South Private Endpoint
- - "s3.eu.cloud-object-storage.appdomain.cloud"
- - EU Cross Region Endpoint
- - "s3.fra.eu.cloud-object-storage.appdomain.cloud"
- - EU Cross Region Frankfurt Endpoint
- - "s3.mil.eu.cloud-object-storage.appdomain.cloud"
- - EU Cross Region Milan Endpoint
- - "s3.ams.eu.cloud-object-storage.appdomain.cloud"
- - EU Cross Region Amsterdam Endpoint
- - "s3.private.eu.cloud-object-storage.appdomain.cloud"
- - EU Cross Region Private Endpoint
- - "s3.private.fra.eu.cloud-object-storage.appdomain.cloud"
- - EU Cross Region Frankfurt Private Endpoint
- - "s3.private.mil.eu.cloud-object-storage.appdomain.cloud"
- - EU Cross Region Milan Private Endpoint
- - "s3.private.ams.eu.cloud-object-storage.appdomain.cloud"
- - EU Cross Region Amsterdam Private Endpoint
- - "s3.eu-gb.cloud-object-storage.appdomain.cloud"
- - Great Britain Endpoint
- - "s3.private.eu-gb.cloud-object-storage.appdomain.cloud"
- - Great Britain Private Endpoint
- - "s3.eu-de.cloud-object-storage.appdomain.cloud"
- - EU Region DE Endpoint
- - "s3.private.eu-de.cloud-object-storage.appdomain.cloud"
- - EU Region DE Private Endpoint
- - "s3.ap.cloud-object-storage.appdomain.cloud"
- - APAC Cross Regional Endpoint
- - "s3.tok.ap.cloud-object-storage.appdomain.cloud"
- - APAC Cross Regional Tokyo Endpoint
- - "s3.hkg.ap.cloud-object-storage.appdomain.cloud"
- - APAC Cross Regional HongKong Endpoint
- - "s3.seo.ap.cloud-object-storage.appdomain.cloud"
- - APAC Cross Regional Seoul Endpoint
- - "s3.private.ap.cloud-object-storage.appdomain.cloud"
- - APAC Cross Regional Private Endpoint
- - "s3.private.tok.ap.cloud-object-storage.appdomain.cloud"
- - APAC Cross Regional Tokyo Private Endpoint
- - "s3.private.hkg.ap.cloud-object-storage.appdomain.cloud"
- - APAC Cross Regional HongKong Private Endpoint
- - "s3.private.seo.ap.cloud-object-storage.appdomain.cloud"
- - APAC Cross Regional Seoul Private Endpoint
- - "s3.jp-tok.cloud-object-storage.appdomain.cloud"
- - APAC Region Japan Endpoint
- - "s3.private.jp-tok.cloud-object-storage.appdomain.cloud"
- - APAC Region Japan Private Endpoint
- - "s3.au-syd.cloud-object-storage.appdomain.cloud"
- - APAC Region Australia Endpoint
- - "s3.private.au-syd.cloud-object-storage.appdomain.cloud"
- - APAC Region Australia Private Endpoint
- - "s3.ams03.cloud-object-storage.appdomain.cloud"
- - Amsterdam Single Site Endpoint
- - "s3.private.ams03.cloud-object-storage.appdomain.cloud"
- - Amsterdam Single Site Private Endpoint
- - "s3.che01.cloud-object-storage.appdomain.cloud"
- - Chennai Single Site Endpoint
- - "s3.private.che01.cloud-object-storage.appdomain.cloud"
- - Chennai Single Site Private Endpoint
- - "s3.mel01.cloud-object-storage.appdomain.cloud"
- - Melbourne Single Site Endpoint
- - "s3.private.mel01.cloud-object-storage.appdomain.cloud"
- - Melbourne Single Site Private Endpoint
- - "s3.osl01.cloud-object-storage.appdomain.cloud"
- - Oslo Single Site Endpoint
- - "s3.private.osl01.cloud-object-storage.appdomain.cloud"
- - Oslo Single Site Private Endpoint
- - "s3.tor01.cloud-object-storage.appdomain.cloud"
- - Toronto Single Site Endpoint
- - "s3.private.tor01.cloud-object-storage.appdomain.cloud"
- - Toronto Single Site Private Endpoint
- - "s3.seo01.cloud-object-storage.appdomain.cloud"
- - Seoul Single Site Endpoint
- - "s3.private.seo01.cloud-object-storage.appdomain.cloud"
- - Seoul Single Site Private Endpoint
- - "s3.mon01.cloud-object-storage.appdomain.cloud"
- - Montreal Single Site Endpoint
- - "s3.private.mon01.cloud-object-storage.appdomain.cloud"
- - Montreal Single Site Private Endpoint
- - "s3.mex01.cloud-object-storage.appdomain.cloud"
- - Mexico Single Site Endpoint
- - "s3.private.mex01.cloud-object-storage.appdomain.cloud"
- - Mexico Single Site Private Endpoint
- - "s3.sjc04.cloud-object-storage.appdomain.cloud"
- - San Jose Single Site Endpoint
- - "s3.private.sjc04.cloud-object-storage.appdomain.cloud"
- - San Jose Single Site Private Endpoint
- - "s3.mil01.cloud-object-storage.appdomain.cloud"
- - Milan Single Site Endpoint
- - "s3.private.mil01.cloud-object-storage.appdomain.cloud"
- - Milan Single Site Private Endpoint
- - "s3.hkg02.cloud-object-storage.appdomain.cloud"
- - Hong Kong Single Site Endpoint
- - "s3.private.hkg02.cloud-object-storage.appdomain.cloud"
- - Hong Kong Single Site Private Endpoint
- - "s3.par01.cloud-object-storage.appdomain.cloud"
- - Paris Single Site Endpoint
- - "s3.private.par01.cloud-object-storage.appdomain.cloud"
- - Paris Single Site Private Endpoint
- - "s3.sng01.cloud-object-storage.appdomain.cloud"
- - Singapore Single Site Endpoint
- - "s3.private.sng01.cloud-object-storage.appdomain.cloud"
- - Singapore Single Site Private Endpoint
+[Cloudflare R2](https://blog.cloudflare.com/r2-open-beta/) Storage
+allows developers to store large amounts of unstructured data without
+the costly egress bandwidth fees associated with typical cloud storage
+services.
-#### --s3-endpoint
+Here is an example of making a Cloudflare R2 configuration. First run:
-Endpoint for IONOS S3 Object Storage.
+ rclone config
-Specify the endpoint from the same region.
+This will guide you through an interactive setup process.
-Properties:
+Note that all buckets are private, and all are stored in the same
+"auto" region. It is necessary to use Cloudflare workers to share the
+content of a bucket publicly.
-- Config: endpoint
-- Env Var: RCLONE_S3_ENDPOINT
-- Provider: IONOS
-- Type: string
-- Required: false
-- Examples:
- - "s3-eu-central-1.ionoscloud.com"
- - Frankfurt, Germany
- - "s3-eu-central-2.ionoscloud.com"
- - Berlin, Germany
- - "s3-eu-south-2.ionoscloud.com"
- - Logrono, Spain
+```
+No remotes found, make a new one?
+n) New remote
+s) Set configuration password
+q) Quit config
+n/s/q> n
+name> r2
+Option Storage.
+Type of storage to configure.
+Choose a number from below, or type in your own value.
+...
+XX / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, China Mobile, Cloudflare, ArvanCloud, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, Lyve Cloud, Minio, Netease, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Synology, Tencent COS and Wasabi
+ \ (s3)
+...
+Storage> s3
+Option provider.
+Choose your S3 provider.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+...
+XX / Cloudflare R2 Storage
+ \ (Cloudflare)
+...
+provider> Cloudflare
+Option env_auth.
+Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
+Only applies if access_key_id and secret_access_key is blank.
+Choose a number from below, or type in your own boolean value (true or false).
+Press Enter for the default (false).
+ 1 / Enter AWS credentials in the next step.
+ \ (false)
+ 2 / Get AWS credentials from the environment (env vars or IAM).
+ \ (true)
+env_auth> 1
+Option access_key_id.
+AWS Access Key ID.
+Leave blank for anonymous access or runtime credentials.
+Enter a value. Press Enter to leave empty.
+access_key_id> ACCESS_KEY
+Option secret_access_key.
+AWS Secret Access Key (password).
+Leave blank for anonymous access or runtime credentials.
+Enter a value. Press Enter to leave empty.
+secret_access_key> SECRET_ACCESS_KEY
+Option region.
+Region to connect to.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+ 1 / R2 buckets are automatically distributed across Cloudflare's data centers for low latency.
+ \ (auto)
+region> 1
+Option endpoint.
+Endpoint for S3 API.
+Required when using an S3 clone.
+Enter a value. Press Enter to leave empty.
+endpoint> https://ACCOUNT_ID.r2.cloudflarestorage.com
+Edit advanced config?
+y) Yes
+n) No (default)
+y/n> n
+--------------------
+y) Yes this is OK (default)
+e) Edit this remote
+d) Delete this remote
+y/e/d> y
+```
-#### --s3-endpoint
+This will leave your config looking something like:
-Endpoint for Petabox S3 Object Storage.
+```
+[r2]
+type = s3
+provider = Cloudflare
+access_key_id = ACCESS_KEY
+secret_access_key = SECRET_ACCESS_KEY
+region = auto
+endpoint = https://ACCOUNT_ID.r2.cloudflarestorage.com
+acl = private
+```
-Specify the endpoint from the same region.
+Now run `rclone lsf r2:` to see your buckets and `rclone lsf
+r2:bucket` to look within a bucket.
-Properties:
+### Dreamhost
-- Config: endpoint
-- Env Var: RCLONE_S3_ENDPOINT
-- Provider: Petabox
-- Type: string
-- Required: true
-- Examples:
- - "s3.petabox.io"
- - US East (N. Virginia)
- - "s3.us-east-1.petabox.io"
- - US East (N. Virginia)
- - "s3.eu-central-1.petabox.io"
- - Europe (Frankfurt)
- - "s3.ap-southeast-1.petabox.io"
- - Asia Pacific (Singapore)
- - "s3.me-south-1.petabox.io"
- - Middle East (Bahrain)
- - "s3.sa-east-1.petabox.io"
- - South America (São Paulo)
+Dreamhost [DreamObjects](https://www.dreamhost.com/cloud/storage/) is
+an object storage system based on CEPH.
-#### --s3-endpoint
+To use rclone with Dreamhost, configure as above but leave the region blank
+and set the endpoint. You should end up with something like this in
+your config:
-Endpoint for Leviia Object Storage API.
+```
+[dreamobjects]
+type = s3
+provider = DreamHost
+env_auth = false
+access_key_id = your_access_key
+secret_access_key = your_secret_key
+region =
+endpoint = objects-us-west-1.dream.io
+location_constraint =
+acl = private
+server_side_encryption =
+storage_class =
+```
-Properties:
+### Google Cloud Storage
-- Config: endpoint
-- Env Var: RCLONE_S3_ENDPOINT
-- Provider: Leviia
-- Type: string
-- Required: false
-- Examples:
- - "s3.leviia.com"
- - The default endpoint
- - Leviia
+[GoogleCloudStorage](https://cloud.google.com/storage/docs) is an [S3-interoperable](https://cloud.google.com/storage/docs/interoperability) object storage service from Google Cloud Platform.
-#### --s3-endpoint
+To connect to Google Cloud Storage you will need an access key and secret key. These can be retrieved by creating an [HMAC key](https://cloud.google.com/storage/docs/authentication/managing-hmackeys).
-Endpoint for Liara Object Storage API.
+```
+[gs]
+type = s3
+provider = GCS
+access_key_id = your_access_key
+secret_access_key = your_secret_key
+endpoint = https://storage.googleapis.com
+```
-Properties:
+**Note** that `--s3-versions` does not work with GCS when it needs to do directory paging. Rclone will return the error:
-- Config: endpoint
-- Env Var: RCLONE_S3_ENDPOINT
-- Provider: Liara
-- Type: string
-- Required: false
-- Examples:
- - "storage.iran.liara.space"
- - The default endpoint
- - Iran
+ s3 protocol error: received versions listing with IsTruncated set with no NextKeyMarker
-#### --s3-endpoint
+This is Google bug [#312292516](https://issuetracker.google.com/u/0/issues/312292516).
-Endpoint for OSS API.
+### DigitalOcean Spaces
-Properties:
+[Spaces](https://www.digitalocean.com/products/object-storage/) is an [S3-interoperable](https://developers.digitalocean.com/documentation/spaces/) object storage service from cloud provider DigitalOcean.
-- Config: endpoint
-- Env Var: RCLONE_S3_ENDPOINT
-- Provider: Alibaba
-- Type: string
-- Required: false
-- Examples:
- - "oss-accelerate.aliyuncs.com"
- - Global Accelerate
- - "oss-accelerate-overseas.aliyuncs.com"
- - Global Accelerate (outside mainland China)
- - "oss-cn-hangzhou.aliyuncs.com"
- - East China 1 (Hangzhou)
- - "oss-cn-shanghai.aliyuncs.com"
- - East China 2 (Shanghai)
- - "oss-cn-qingdao.aliyuncs.com"
- - North China 1 (Qingdao)
- - "oss-cn-beijing.aliyuncs.com"
- - North China 2 (Beijing)
- - "oss-cn-zhangjiakou.aliyuncs.com"
- - North China 3 (Zhangjiakou)
- - "oss-cn-huhehaote.aliyuncs.com"
- - North China 5 (Hohhot)
- - "oss-cn-wulanchabu.aliyuncs.com"
- - North China 6 (Ulanqab)
- - "oss-cn-shenzhen.aliyuncs.com"
- - South China 1 (Shenzhen)
- - "oss-cn-heyuan.aliyuncs.com"
- - South China 2 (Heyuan)
- - "oss-cn-guangzhou.aliyuncs.com"
- - South China 3 (Guangzhou)
- - "oss-cn-chengdu.aliyuncs.com"
- - West China 1 (Chengdu)
- - "oss-cn-hongkong.aliyuncs.com"
- - Hong Kong (Hong Kong)
- - "oss-us-west-1.aliyuncs.com"
- - US West 1 (Silicon Valley)
- - "oss-us-east-1.aliyuncs.com"
- - US East 1 (Virginia)
- - "oss-ap-southeast-1.aliyuncs.com"
- - Southeast Asia Southeast 1 (Singapore)
- - "oss-ap-southeast-2.aliyuncs.com"
- - Asia Pacific Southeast 2 (Sydney)
- - "oss-ap-southeast-3.aliyuncs.com"
- - Southeast Asia Southeast 3 (Kuala Lumpur)
- - "oss-ap-southeast-5.aliyuncs.com"
- - Asia Pacific Southeast 5 (Jakarta)
- - "oss-ap-northeast-1.aliyuncs.com"
- - Asia Pacific Northeast 1 (Japan)
- - "oss-ap-south-1.aliyuncs.com"
- - Asia Pacific South 1 (Mumbai)
- - "oss-eu-central-1.aliyuncs.com"
- - Central Europe 1 (Frankfurt)
- - "oss-eu-west-1.aliyuncs.com"
- - West Europe (London)
- - "oss-me-east-1.aliyuncs.com"
- - Middle East 1 (Dubai)
+To connect to DigitalOcean Spaces you will need an access key and secret key. These can be retrieved on the "[Applications & API](https://cloud.digitalocean.com/settings/api/tokens)" page of the DigitalOcean control panel. They will be needed when prompted by `rclone config` for your `access_key_id` and `secret_access_key`.
-#### --s3-endpoint
+When prompted for a `region` or `location_constraint`, press enter to use the default value. The region must be included in the `endpoint` setting (e.g. `nyc3.digitaloceanspaces.com`). The default values can be used for other settings.
-Endpoint for OBS API.
+Going through the whole process of creating a new remote by running `rclone config`, each prompt should be answered as shown below:
-Properties:
+```
+Storage> s3
+env_auth> 1
+access_key_id> YOUR_ACCESS_KEY
+secret_access_key> YOUR_SECRET_KEY
+region>
+endpoint> nyc3.digitaloceanspaces.com
+location_constraint>
+acl>
+storage_class>
+```
-- Config: endpoint
-- Env Var: RCLONE_S3_ENDPOINT
-- Provider: HuaweiOBS
-- Type: string
-- Required: false
-- Examples:
- - "obs.af-south-1.myhuaweicloud.com"
- - AF-Johannesburg
- - "obs.ap-southeast-2.myhuaweicloud.com"
- - AP-Bangkok
- - "obs.ap-southeast-3.myhuaweicloud.com"
- - AP-Singapore
- - "obs.cn-east-3.myhuaweicloud.com"
- - CN East-Shanghai1
- - "obs.cn-east-2.myhuaweicloud.com"
- - CN East-Shanghai2
- - "obs.cn-north-1.myhuaweicloud.com"
- - CN North-Beijing1
- - "obs.cn-north-4.myhuaweicloud.com"
- - CN North-Beijing4
- - "obs.cn-south-1.myhuaweicloud.com"
- - CN South-Guangzhou
- - "obs.ap-southeast-1.myhuaweicloud.com"
- - CN-Hong Kong
- - "obs.sa-argentina-1.myhuaweicloud.com"
- - LA-Buenos Aires1
- - "obs.sa-peru-1.myhuaweicloud.com"
- - LA-Lima1
- - "obs.na-mexico-1.myhuaweicloud.com"
- - LA-Mexico City1
- - "obs.sa-chile-1.myhuaweicloud.com"
- - LA-Santiago2
- - "obs.sa-brazil-1.myhuaweicloud.com"
- - LA-Sao Paulo1
- - "obs.ru-northwest-2.myhuaweicloud.com"
- - RU-Moscow2
+The resulting configuration file should look like:
-#### --s3-endpoint
+```
+[spaces]
+type = s3
+provider = DigitalOcean
+env_auth = false
+access_key_id = YOUR_ACCESS_KEY
+secret_access_key = YOUR_SECRET_KEY
+region =
+endpoint = nyc3.digitaloceanspaces.com
+location_constraint =
+acl =
+server_side_encryption =
+storage_class =
+```
-Endpoint for Scaleway Object Storage.
+Once configured, you can create a new Space and begin copying files. For example:
-Properties:
+```
+rclone mkdir spaces:my-new-space
+rclone copy /path/to/files spaces:my-new-space
+```
+### Huawei OBS {#huawei-obs}
-- Config: endpoint
-- Env Var: RCLONE_S3_ENDPOINT
-- Provider: Scaleway
-- Type: string
-- Required: false
-- Examples:
- - "s3.nl-ams.scw.cloud"
- - Amsterdam Endpoint
- - "s3.fr-par.scw.cloud"
- - Paris Endpoint
- - "s3.pl-waw.scw.cloud"
- - Warsaw Endpoint
+Object Storage Service (OBS) provides stable, secure, efficient, and easy-to-use cloud storage that lets you store virtually any volume of unstructured data in any format and access it from anywhere.
-#### --s3-endpoint
+OBS provides an S3 interface, you can copy and modify the following configuration and add it to your rclone configuration file.
+```
+[obs]
+type = s3
+provider = HuaweiOBS
+access_key_id = your-access-key-id
+secret_access_key = your-secret-access-key
+region = af-south-1
+endpoint = obs.af-south-1.myhuaweicloud.com
+acl = private
+```
-Endpoint for StackPath Object Storage.
+Or you can also configure via the interactive command line:
+```
+No remotes found, make a new one?
+n) New remote
+s) Set configuration password
+q) Quit config
+n/s/q> n
+name> obs
+Option Storage.
+Type of storage to configure.
+Choose a number from below, or type in your own value.
+[snip]
+ 5 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, China Mobile, Cloudflare, ArvanCloud, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, Lyve Cloud, Minio, Netease, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Synology, Tencent COS and Wasabi
+ \ (s3)
+[snip]
+Storage> 5
+Option provider.
+Choose your S3 provider.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+[snip]
+ 9 / Huawei Object Storage Service
+ \ (HuaweiOBS)
+[snip]
+provider> 9
+Option env_auth.
+Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
+Only applies if access_key_id and secret_access_key is blank.
+Choose a number from below, or type in your own boolean value (true or false).
+Press Enter for the default (false).
+ 1 / Enter AWS credentials in the next step.
+ \ (false)
+ 2 / Get AWS credentials from the environment (env vars or IAM).
+ \ (true)
+env_auth> 1
+Option access_key_id.
+AWS Access Key ID.
+Leave blank for anonymous access or runtime credentials.
+Enter a value. Press Enter to leave empty.
+access_key_id> your-access-key-id
+Option secret_access_key.
+AWS Secret Access Key (password).
+Leave blank for anonymous access or runtime credentials.
+Enter a value. Press Enter to leave empty.
+secret_access_key> your-secret-access-key
+Option region.
+Region to connect to.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+ 1 / AF-Johannesburg
+ \ (af-south-1)
+ 2 / AP-Bangkok
+ \ (ap-southeast-2)
+[snip]
+region> 1
+Option endpoint.
+Endpoint for OBS API.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+ 1 / AF-Johannesburg
+ \ (obs.af-south-1.myhuaweicloud.com)
+ 2 / AP-Bangkok
+ \ (obs.ap-southeast-2.myhuaweicloud.com)
+[snip]
+endpoint> 1
+Option acl.
+Canned ACL used when creating buckets and storing or copying objects.
+This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.
+For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
+Note that this ACL is applied when server-side copying objects as S3
+doesn't copy the ACL from the source but rather writes a fresh one.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+ / Owner gets FULL_CONTROL.
+ 1 | No one else has access rights (default).
+ \ (private)
+[snip]
+acl> 1
+Edit advanced config?
+y) Yes
+n) No (default)
+y/n>
+--------------------
+[obs]
+type = s3
+provider = HuaweiOBS
+access_key_id = your-access-key-id
+secret_access_key = your-secret-access-key
+region = af-south-1
+endpoint = obs.af-south-1.myhuaweicloud.com
+acl = private
+--------------------
+y) Yes this is OK (default)
+e) Edit this remote
+d) Delete this remote
+y/e/d> y
+Current remotes:
-Properties:
+Name Type
+==== ====
+obs s3
-- Config: endpoint
-- Env Var: RCLONE_S3_ENDPOINT
-- Provider: StackPath
-- Type: string
-- Required: false
-- Examples:
- - "s3.us-east-2.stackpathstorage.com"
- - US East Endpoint
- - "s3.us-west-1.stackpathstorage.com"
- - US West Endpoint
- - "s3.eu-central-1.stackpathstorage.com"
- - EU Endpoint
+e) Edit existing remote
+n) New remote
+d) Delete remote
+r) Rename remote
+c) Copy remote
+s) Set configuration password
+q) Quit config
+e/n/d/r/c/s/q> q
+```
-#### --s3-endpoint
+### IBM COS (S3)
-Endpoint for Google Cloud Storage.
+Information stored with IBM Cloud Object Storage is encrypted and dispersed across multiple geographic locations, and accessed through an implementation of the S3 API. This service makes use of the distributed storage technologies provided by IBM’s Cloud Object Storage System (formerly Cleversafe). For more information visit: (http://www.ibm.com/cloud/object-storage)
-Properties:
+To configure access to IBM COS S3, follow the steps below:
-- Config: endpoint
-- Env Var: RCLONE_S3_ENDPOINT
-- Provider: GCS
-- Type: string
-- Required: false
-- Examples:
- - "https://storage.googleapis.com"
- - Google Cloud Storage endpoint
+1. Run rclone config and select n for a new remote.
+```
+ 2018/02/14 14:13:11 NOTICE: Config file "C:\\Users\\a\\.config\\rclone\\rclone.conf" not found - using defaults
+ No remotes found, make a new one?
+ n) New remote
+ s) Set configuration password
+ q) Quit config
+ n/s/q> n
+```
-#### --s3-endpoint
+2. Enter the name for the configuration
+```
+ name>
+```
-Endpoint for Storj Gateway.
+3. Select "s3" storage.
+```
+Choose a number from below, or type in your own value
+ 1 / Alias for an existing remote
+ \ "alias"
+ 2 / Amazon Drive
+ \ "amazon cloud drive"
+ 3 / Amazon S3 Complaint Storage Providers (Dreamhost, Ceph, ChinaMobile, Liara, ArvanCloud, Minio, IBM COS)
+ \ "s3"
+ 4 / Backblaze B2
+ \ "b2"
+[snip]
+ 23 / HTTP
+ \ "http"
+Storage> 3
+```
-Properties:
+4. Select IBM COS as the S3 Storage Provider.
+```
+Choose the S3 provider.
+Choose a number from below, or type in your own value
+ 1 / Choose this option to configure Storage to AWS S3
+ \ "AWS"
+ 2 / Choose this option to configure Storage to Ceph Systems
+ \ "Ceph"
+ 3 / Choose this option to configure Storage to Dreamhost
+ \ "Dreamhost"
+ 4 / Choose this option to the configure Storage to IBM COS S3
+ \ "IBMCOS"
+ 5 / Choose this option to the configure Storage to Minio
+ \ "Minio"
+ Provider>4
+```
-- Config: endpoint
-- Env Var: RCLONE_S3_ENDPOINT
-- Provider: Storj
-- Type: string
-- Required: false
-- Examples:
- - "gateway.storjshare.io"
- - Global Hosted Gateway
+5. Enter the Access Key and Secret.
+```
+ AWS Access Key ID - leave blank for anonymous access or runtime credentials.
+ access_key_id> <>
+ AWS Secret Access Key (password) - leave blank for anonymous access or runtime credentials.
+ secret_access_key> <>
+```
-#### --s3-endpoint
+6. Specify the endpoint for IBM COS. For Public IBM COS, choose from the option below. For On Premise IBM COS, enter an endpoint address.
+```
+ Endpoint for IBM COS S3 API.
+ Specify if using an IBM COS On Premise.
+ Choose a number from below, or type in your own value
+ 1 / US Cross Region Endpoint
+ \ "s3-api.us-geo.objectstorage.softlayer.net"
+ 2 / US Cross Region Dallas Endpoint
+ \ "s3-api.dal.us-geo.objectstorage.softlayer.net"
+ 3 / US Cross Region Washington DC Endpoint
+ \ "s3-api.wdc-us-geo.objectstorage.softlayer.net"
+ 4 / US Cross Region San Jose Endpoint
+ \ "s3-api.sjc-us-geo.objectstorage.softlayer.net"
+ 5 / US Cross Region Private Endpoint
+ \ "s3-api.us-geo.objectstorage.service.networklayer.com"
+ 6 / US Cross Region Dallas Private Endpoint
+ \ "s3-api.dal-us-geo.objectstorage.service.networklayer.com"
+ 7 / US Cross Region Washington DC Private Endpoint
+ \ "s3-api.wdc-us-geo.objectstorage.service.networklayer.com"
+ 8 / US Cross Region San Jose Private Endpoint
+ \ "s3-api.sjc-us-geo.objectstorage.service.networklayer.com"
+ 9 / US Region East Endpoint
+ \ "s3.us-east.objectstorage.softlayer.net"
+ 10 / US Region East Private Endpoint
+ \ "s3.us-east.objectstorage.service.networklayer.com"
+ 11 / US Region South Endpoint
+[snip]
+ 34 / Toronto Single Site Private Endpoint
+ \ "s3.tor01.objectstorage.service.networklayer.com"
+ endpoint>1
+```
-Endpoint for Synology C2 Object Storage API.
-Properties:
+7. Specify a IBM COS Location Constraint. The location constraint must match endpoint when using IBM Cloud Public. For on-prem COS, do not make a selection from this list, hit enter
+```
+ 1 / US Cross Region Standard
+ \ "us-standard"
+ 2 / US Cross Region Vault
+ \ "us-vault"
+ 3 / US Cross Region Cold
+ \ "us-cold"
+ 4 / US Cross Region Flex
+ \ "us-flex"
+ 5 / US East Region Standard
+ \ "us-east-standard"
+ 6 / US East Region Vault
+ \ "us-east-vault"
+ 7 / US East Region Cold
+ \ "us-east-cold"
+ 8 / US East Region Flex
+ \ "us-east-flex"
+ 9 / US South Region Standard
+ \ "us-south-standard"
+ 10 / US South Region Vault
+ \ "us-south-vault"
+[snip]
+ 32 / Toronto Flex
+ \ "tor01-flex"
+location_constraint>1
+```
-- Config: endpoint
-- Env Var: RCLONE_S3_ENDPOINT
-- Provider: Synology
-- Type: string
-- Required: false
-- Examples:
- - "eu-001.s3.synologyc2.net"
- - EU Endpoint 1
- - "eu-002.s3.synologyc2.net"
- - EU Endpoint 2
- - "us-001.s3.synologyc2.net"
- - US Endpoint 1
- - "us-002.s3.synologyc2.net"
- - US Endpoint 2
- - "tw-001.s3.synologyc2.net"
- - TW Endpoint 1
+9. Specify a canned ACL. IBM Cloud (Storage) supports "public-read" and "private". IBM Cloud(Infra) supports all the canned ACLs. On-Premise COS supports all the canned ACLs.
+```
+Canned ACL used when creating buckets and/or storing objects in S3.
+For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
+Choose a number from below, or type in your own value
+ 1 / Owner gets FULL_CONTROL. No one else has access rights (default). This acl is available on IBM Cloud (Infra), IBM Cloud (Storage), On-Premise COS
+ \ "private"
+ 2 / Owner gets FULL_CONTROL. The AllUsers group gets READ access. This acl is available on IBM Cloud (Infra), IBM Cloud (Storage), On-Premise IBM COS
+ \ "public-read"
+ 3 / Owner gets FULL_CONTROL. The AllUsers group gets READ and WRITE access. This acl is available on IBM Cloud (Infra), On-Premise IBM COS
+ \ "public-read-write"
+ 4 / Owner gets FULL_CONTROL. The AuthenticatedUsers group gets READ access. Not supported on Buckets. This acl is available on IBM Cloud (Infra) and On-Premise IBM COS
+ \ "authenticated-read"
+acl> 1
+```
-#### --s3-endpoint
-Endpoint for Tencent COS API.
+12. Review the displayed configuration and accept to save the "remote" then quit. The config file should look like this
+```
+ [xxx]
+ type = s3
+ Provider = IBMCOS
+ access_key_id = xxx
+ secret_access_key = yyy
+ endpoint = s3-api.us-geo.objectstorage.softlayer.net
+ location_constraint = us-standard
+ acl = private
+```
-Properties:
+13. Execute rclone commands
+```
+ 1) Create a bucket.
+ rclone mkdir IBM-COS-XREGION:newbucket
+ 2) List available buckets.
+ rclone lsd IBM-COS-XREGION:
+ -1 2017-11-08 21:16:22 -1 test
+ -1 2018-02-14 20:16:39 -1 newbucket
+ 3) List contents of a bucket.
+ rclone ls IBM-COS-XREGION:newbucket
+ 18685952 test.exe
+ 4) Copy a file from local to remote.
+ rclone copy /Users/file.txt IBM-COS-XREGION:newbucket
+ 5) Copy a file from remote to local.
+ rclone copy IBM-COS-XREGION:newbucket/file.txt .
+ 6) Delete a file on remote.
+ rclone delete IBM-COS-XREGION:newbucket/file.txt
+```
-- Config: endpoint
-- Env Var: RCLONE_S3_ENDPOINT
-- Provider: TencentCOS
-- Type: string
-- Required: false
-- Examples:
- - "cos.ap-beijing.myqcloud.com"
- - Beijing Region
- - "cos.ap-nanjing.myqcloud.com"
- - Nanjing Region
- - "cos.ap-shanghai.myqcloud.com"
- - Shanghai Region
- - "cos.ap-guangzhou.myqcloud.com"
- - Guangzhou Region
- - "cos.ap-nanjing.myqcloud.com"
- - Nanjing Region
- - "cos.ap-chengdu.myqcloud.com"
- - Chengdu Region
- - "cos.ap-chongqing.myqcloud.com"
- - Chongqing Region
- - "cos.ap-hongkong.myqcloud.com"
- - Hong Kong (China) Region
- - "cos.ap-singapore.myqcloud.com"
- - Singapore Region
- - "cos.ap-mumbai.myqcloud.com"
- - Mumbai Region
- - "cos.ap-seoul.myqcloud.com"
- - Seoul Region
- - "cos.ap-bangkok.myqcloud.com"
- - Bangkok Region
- - "cos.ap-tokyo.myqcloud.com"
- - Tokyo Region
- - "cos.na-siliconvalley.myqcloud.com"
- - Silicon Valley Region
- - "cos.na-ashburn.myqcloud.com"
- - Virginia Region
- - "cos.na-toronto.myqcloud.com"
- - Toronto Region
- - "cos.eu-frankfurt.myqcloud.com"
- - Frankfurt Region
- - "cos.eu-moscow.myqcloud.com"
- - Moscow Region
- - "cos.accelerate.myqcloud.com"
- - Use Tencent COS Accelerate Endpoint
+### IDrive e2 {#idrive-e2}
-#### --s3-endpoint
+Here is an example of making an [IDrive e2](https://www.idrive.com/e2/)
+configuration. First run:
-Endpoint for RackCorp Object Storage.
+ rclone config
-Properties:
+This will guide you through an interactive setup process.
-- Config: endpoint
-- Env Var: RCLONE_S3_ENDPOINT
-- Provider: RackCorp
-- Type: string
-- Required: false
-- Examples:
- - "s3.rackcorp.com"
- - Global (AnyCast) Endpoint
- - "au.s3.rackcorp.com"
- - Australia (Anycast) Endpoint
- - "au-nsw.s3.rackcorp.com"
- - Sydney (Australia) Endpoint
- - "au-qld.s3.rackcorp.com"
- - Brisbane (Australia) Endpoint
- - "au-vic.s3.rackcorp.com"
- - Melbourne (Australia) Endpoint
- - "au-wa.s3.rackcorp.com"
- - Perth (Australia) Endpoint
- - "ph.s3.rackcorp.com"
- - Manila (Philippines) Endpoint
- - "th.s3.rackcorp.com"
- - Bangkok (Thailand) Endpoint
- - "hk.s3.rackcorp.com"
- - HK (Hong Kong) Endpoint
- - "mn.s3.rackcorp.com"
- - Ulaanbaatar (Mongolia) Endpoint
- - "kg.s3.rackcorp.com"
- - Bishkek (Kyrgyzstan) Endpoint
- - "id.s3.rackcorp.com"
- - Jakarta (Indonesia) Endpoint
- - "jp.s3.rackcorp.com"
- - Tokyo (Japan) Endpoint
- - "sg.s3.rackcorp.com"
- - SG (Singapore) Endpoint
- - "de.s3.rackcorp.com"
- - Frankfurt (Germany) Endpoint
- - "us.s3.rackcorp.com"
- - USA (AnyCast) Endpoint
- - "us-east-1.s3.rackcorp.com"
- - New York (USA) Endpoint
- - "us-west-1.s3.rackcorp.com"
- - Freemont (USA) Endpoint
- - "nz.s3.rackcorp.com"
- - Auckland (New Zealand) Endpoint
+```
+No remotes found, make a new one?
+n) New remote
+s) Set configuration password
+q) Quit config
+n/s/q> n
-#### --s3-endpoint
+Enter name for new remote.
+name> e2
-Endpoint for Qiniu Object Storage.
+Option Storage.
+Type of storage to configure.
+Choose a number from below, or type in your own value.
+[snip]
+XX / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, China Mobile, Cloudflare, ArvanCloud, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, IDrive e2, Lyve Cloud, Minio, Netease, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Synology, Tencent COS and Wasabi
+ \ (s3)
+[snip]
+Storage> s3
-Properties:
+Option provider.
+Choose your S3 provider.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+[snip]
+XX / IDrive e2
+ \ (IDrive)
+[snip]
+provider> IDrive
-- Config: endpoint
-- Env Var: RCLONE_S3_ENDPOINT
-- Provider: Qiniu
-- Type: string
-- Required: false
-- Examples:
- - "s3-cn-east-1.qiniucs.com"
- - East China Endpoint 1
- - "s3-cn-east-2.qiniucs.com"
- - East China Endpoint 2
- - "s3-cn-north-1.qiniucs.com"
- - North China Endpoint 1
- - "s3-cn-south-1.qiniucs.com"
- - South China Endpoint 1
- - "s3-us-north-1.qiniucs.com"
- - North America Endpoint 1
- - "s3-ap-southeast-1.qiniucs.com"
- - Southeast Asia Endpoint 1
- - "s3-ap-northeast-1.qiniucs.com"
- - Northeast Asia Endpoint 1
+Option env_auth.
+Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
+Only applies if access_key_id and secret_access_key is blank.
+Choose a number from below, or type in your own boolean value (true or false).
+Press Enter for the default (false).
+ 1 / Enter AWS credentials in the next step.
+ \ (false)
+ 2 / Get AWS credentials from the environment (env vars or IAM).
+ \ (true)
+env_auth>
-#### --s3-endpoint
+Option access_key_id.
+AWS Access Key ID.
+Leave blank for anonymous access or runtime credentials.
+Enter a value. Press Enter to leave empty.
+access_key_id> YOUR_ACCESS_KEY
-Endpoint for S3 API.
+Option secret_access_key.
+AWS Secret Access Key (password).
+Leave blank for anonymous access or runtime credentials.
+Enter a value. Press Enter to leave empty.
+secret_access_key> YOUR_SECRET_KEY
-Required when using an S3 clone.
+Option acl.
+Canned ACL used when creating buckets and storing or copying objects.
+This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.
+For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
+Note that this ACL is applied when server-side copying objects as S3
+doesn't copy the ACL from the source but rather writes a fresh one.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+ / Owner gets FULL_CONTROL.
+ 1 | No one else has access rights (default).
+ \ (private)
+ / Owner gets FULL_CONTROL.
+ 2 | The AllUsers group gets READ access.
+ \ (public-read)
+ / Owner gets FULL_CONTROL.
+ 3 | The AllUsers group gets READ and WRITE access.
+ | Granting this on a bucket is generally not recommended.
+ \ (public-read-write)
+ / Owner gets FULL_CONTROL.
+ 4 | The AuthenticatedUsers group gets READ access.
+ \ (authenticated-read)
+ / Object owner gets FULL_CONTROL.
+ 5 | Bucket owner gets READ access.
+ | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
+ \ (bucket-owner-read)
+ / Both the object owner and the bucket owner get FULL_CONTROL over the object.
+ 6 | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
+ \ (bucket-owner-full-control)
+acl>
-Properties:
+Edit advanced config?
+y) Yes
+n) No (default)
+y/n>
-- Config: endpoint
-- Env Var: RCLONE_S3_ENDPOINT
-- Provider: !AWS,ArvanCloud,IBMCOS,IDrive,IONOS,TencentCOS,HuaweiOBS,Alibaba,ChinaMobile,GCS,Liara,Scaleway,StackPath,Storj,Synology,RackCorp,Qiniu,Petabox
-- Type: string
-- Required: false
-- Examples:
- - "objects-us-east-1.dream.io"
- - Dream Objects endpoint
- - "syd1.digitaloceanspaces.com"
- - DigitalOcean Spaces Sydney 1
- - "sfo3.digitaloceanspaces.com"
- - DigitalOcean Spaces San Francisco 3
- - "fra1.digitaloceanspaces.com"
- - DigitalOcean Spaces Frankfurt 1
- - "nyc3.digitaloceanspaces.com"
- - DigitalOcean Spaces New York 3
- - "ams3.digitaloceanspaces.com"
- - DigitalOcean Spaces Amsterdam 3
- - "sgp1.digitaloceanspaces.com"
- - DigitalOcean Spaces Singapore 1
- - "localhost:8333"
- - SeaweedFS S3 localhost
- - "s3.us-east-1.lyvecloud.seagate.com"
- - Seagate Lyve Cloud US East 1 (Virginia)
- - "s3.us-west-1.lyvecloud.seagate.com"
- - Seagate Lyve Cloud US West 1 (California)
- - "s3.ap-southeast-1.lyvecloud.seagate.com"
- - Seagate Lyve Cloud AP Southeast 1 (Singapore)
- - "s3.wasabisys.com"
- - Wasabi US East 1 (N. Virginia)
- - "s3.us-east-2.wasabisys.com"
- - Wasabi US East 2 (N. Virginia)
- - "s3.us-central-1.wasabisys.com"
- - Wasabi US Central 1 (Texas)
- - "s3.us-west-1.wasabisys.com"
- - Wasabi US West 1 (Oregon)
- - "s3.ca-central-1.wasabisys.com"
- - Wasabi CA Central 1 (Toronto)
- - "s3.eu-central-1.wasabisys.com"
- - Wasabi EU Central 1 (Amsterdam)
- - "s3.eu-central-2.wasabisys.com"
- - Wasabi EU Central 2 (Frankfurt)
- - "s3.eu-west-1.wasabisys.com"
- - Wasabi EU West 1 (London)
- - "s3.eu-west-2.wasabisys.com"
- - Wasabi EU West 2 (Paris)
- - "s3.ap-northeast-1.wasabisys.com"
- - Wasabi AP Northeast 1 (Tokyo) endpoint
- - "s3.ap-northeast-2.wasabisys.com"
- - Wasabi AP Northeast 2 (Osaka) endpoint
- - "s3.ap-southeast-1.wasabisys.com"
- - Wasabi AP Southeast 1 (Singapore)
- - "s3.ap-southeast-2.wasabisys.com"
- - Wasabi AP Southeast 2 (Sydney)
- - "storage.iran.liara.space"
- - Liara Iran endpoint
- - "s3.ir-thr-at1.arvanstorage.ir"
- - ArvanCloud Tehran Iran (Simin) endpoint
- - "s3.ir-tbz-sh1.arvanstorage.ir"
- - ArvanCloud Tabriz Iran (Shahriar) endpoint
+Configuration complete.
+Options:
+- type: s3
+- provider: IDrive
+- access_key_id: YOUR_ACCESS_KEY
+- secret_access_key: YOUR_SECRET_KEY
+- endpoint: q9d9.la12.idrivee2-5.com
+Keep this "e2" remote?
+y) Yes this is OK (default)
+e) Edit this remote
+d) Delete this remote
+y/e/d> y
+```
-#### --s3-location-constraint
+### IONOS Cloud {#ionos}
-Location constraint - must be set to match the Region.
+[IONOS S3 Object Storage](https://cloud.ionos.com/storage/object-storage) is a service offered by IONOS for storing and accessing unstructured data.
+To connect to the service, you will need an access key and a secret key. These can be found in the [Data Center Designer](https://dcd.ionos.com/), by selecting **Manager resources** > **Object Storage Key Manager**.
-Used when creating buckets only.
-Properties:
+Here is an example of a configuration. First, run `rclone config`. This will walk you through an interactive setup process. Type `n` to add the new remote, and then enter a name:
-- Config: location_constraint
-- Env Var: RCLONE_S3_LOCATION_CONSTRAINT
-- Provider: AWS
-- Type: string
-- Required: false
-- Examples:
- - ""
- - Empty for US Region, Northern Virginia, or Pacific Northwest
- - "us-east-2"
- - US East (Ohio) Region
- - "us-west-1"
- - US West (Northern California) Region
- - "us-west-2"
- - US West (Oregon) Region
- - "ca-central-1"
- - Canada (Central) Region
- - "eu-west-1"
- - EU (Ireland) Region
- - "eu-west-2"
- - EU (London) Region
- - "eu-west-3"
- - EU (Paris) Region
- - "eu-north-1"
- - EU (Stockholm) Region
- - "eu-south-1"
- - EU (Milan) Region
- - "EU"
- - EU Region
- - "ap-southeast-1"
- - Asia Pacific (Singapore) Region
- - "ap-southeast-2"
- - Asia Pacific (Sydney) Region
- - "ap-northeast-1"
- - Asia Pacific (Tokyo) Region
- - "ap-northeast-2"
- - Asia Pacific (Seoul) Region
- - "ap-northeast-3"
- - Asia Pacific (Osaka-Local) Region
- - "ap-south-1"
- - Asia Pacific (Mumbai) Region
- - "ap-east-1"
- - Asia Pacific (Hong Kong) Region
- - "sa-east-1"
- - South America (Sao Paulo) Region
- - "me-south-1"
- - Middle East (Bahrain) Region
- - "af-south-1"
- - Africa (Cape Town) Region
- - "cn-north-1"
- - China (Beijing) Region
- - "cn-northwest-1"
- - China (Ningxia) Region
- - "us-gov-east-1"
- - AWS GovCloud (US-East) Region
- - "us-gov-west-1"
- - AWS GovCloud (US) Region
+```
+Enter name for new remote.
+name> ionos-fra
+```
-#### --s3-location-constraint
+Type `s3` to choose the connection type:
+```
+Option Storage.
+Type of storage to configure.
+Choose a number from below, or type in your own value.
+[snip]
+XX / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, China Mobile, Cloudflare, ArvanCloud, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, IDrive e2, IONOS Cloud, Lyve Cloud, Minio, Netease, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Synology, Tencent COS and Wasabi
+ \ (s3)
+[snip]
+Storage> s3
+```
-Location constraint - must match endpoint.
+Type `IONOS`:
+```
+Option provider.
+Choose your S3 provider.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+[snip]
+XX / IONOS Cloud
+ \ (IONOS)
+[snip]
+provider> IONOS
+```
-Used when creating buckets only.
+Press Enter to choose the default option `Enter AWS credentials in the next step`:
+```
+Option env_auth.
+Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
+Only applies if access_key_id and secret_access_key is blank.
+Choose a number from below, or type in your own boolean value (true or false).
+Press Enter for the default (false).
+ 1 / Enter AWS credentials in the next step.
+ \ (false)
+ 2 / Get AWS credentials from the environment (env vars or IAM).
+ \ (true)
+env_auth>
+```
-Properties:
+Enter your Access Key and Secret key. These can be retrieved in the [Data Center Designer](https://dcd.ionos.com/), click on the menu “Manager resources†/ "Object Storage Key Manager".
+```
+Option access_key_id.
+AWS Access Key ID.
+Leave blank for anonymous access or runtime credentials.
+Enter a value. Press Enter to leave empty.
+access_key_id> YOUR_ACCESS_KEY
-- Config: location_constraint
-- Env Var: RCLONE_S3_LOCATION_CONSTRAINT
-- Provider: ChinaMobile
-- Type: string
-- Required: false
-- Examples:
- - "wuxi1"
- - East China (Suzhou)
- - "jinan1"
- - East China (Jinan)
- - "ningbo1"
- - East China (Hangzhou)
- - "shanghai1"
- - East China (Shanghai-1)
- - "zhengzhou1"
- - Central China (Zhengzhou)
- - "hunan1"
- - Central China (Changsha-1)
- - "zhuzhou1"
- - Central China (Changsha-2)
- - "guangzhou1"
- - South China (Guangzhou-2)
- - "dongguan1"
- - South China (Guangzhou-3)
- - "beijing1"
- - North China (Beijing-1)
- - "beijing2"
- - North China (Beijing-2)
- - "beijing4"
- - North China (Beijing-3)
- - "huhehaote1"
- - North China (Huhehaote)
- - "chengdu1"
- - Southwest China (Chengdu)
- - "chongqing1"
- - Southwest China (Chongqing)
- - "guiyang1"
- - Southwest China (Guiyang)
- - "xian1"
- - Nouthwest China (Xian)
- - "yunnan"
- - Yunnan China (Kunming)
- - "yunnan2"
- - Yunnan China (Kunming-2)
- - "tianjin1"
- - Tianjin China (Tianjin)
- - "jilin1"
- - Jilin China (Changchun)
- - "hubei1"
- - Hubei China (Xiangyan)
- - "jiangxi1"
- - Jiangxi China (Nanchang)
- - "gansu1"
- - Gansu China (Lanzhou)
- - "shanxi1"
- - Shanxi China (Taiyuan)
- - "liaoning1"
- - Liaoning China (Shenyang)
- - "hebei1"
- - Hebei China (Shijiazhuang)
- - "fujian1"
- - Fujian China (Xiamen)
- - "guangxi1"
- - Guangxi China (Nanning)
- - "anhui1"
- - Anhui China (Huainan)
+Option secret_access_key.
+AWS Secret Access Key (password).
+Leave blank for anonymous access or runtime credentials.
+Enter a value. Press Enter to leave empty.
+secret_access_key> YOUR_SECRET_KEY
+```
-#### --s3-location-constraint
+Choose the region where your bucket is located:
+```
+Option region.
+Region where your bucket will be created and your data stored.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+ 1 / Frankfurt, Germany
+ \ (de)
+ 2 / Berlin, Germany
+ \ (eu-central-2)
+ 3 / Logrono, Spain
+ \ (eu-south-2)
+region> 2
+```
-Location constraint - must match endpoint.
+Choose the endpoint from the same region:
+```
+Option endpoint.
+Endpoint for IONOS S3 Object Storage.
+Specify the endpoint from the same region.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+ 1 / Frankfurt, Germany
+ \ (s3-eu-central-1.ionoscloud.com)
+ 2 / Berlin, Germany
+ \ (s3-eu-central-2.ionoscloud.com)
+ 3 / Logrono, Spain
+ \ (s3-eu-south-2.ionoscloud.com)
+endpoint> 1
+```
-Used when creating buckets only.
+Press Enter to choose the default option or choose the desired ACL setting:
+```
+Option acl.
+Canned ACL used when creating buckets and storing or copying objects.
+This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.
+For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
+Note that this ACL is applied when server-side copying objects as S3
+doesn't copy the ACL from the source but rather writes a fresh one.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+ / Owner gets FULL_CONTROL.
+ 1 | No one else has access rights (default).
+ \ (private)
+ / Owner gets FULL_CONTROL.
+[snip]
+acl>
+```
-Properties:
+Press Enter to skip the advanced config:
+```
+Edit advanced config?
+y) Yes
+n) No (default)
+y/n>
+```
-- Config: location_constraint
-- Env Var: RCLONE_S3_LOCATION_CONSTRAINT
-- Provider: ArvanCloud
-- Type: string
-- Required: false
-- Examples:
- - "ir-thr-at1"
- - Tehran Iran (Simin)
- - "ir-tbz-sh1"
- - Tabriz Iran (Shahriar)
+Press Enter to save the configuration, and then `q` to quit the configuration process:
+```
+Configuration complete.
+Options:
+- type: s3
+- provider: IONOS
+- access_key_id: YOUR_ACCESS_KEY
+- secret_access_key: YOUR_SECRET_KEY
+- endpoint: s3-eu-central-1.ionoscloud.com
+Keep this "ionos-fra" remote?
+y) Yes this is OK (default)
+e) Edit this remote
+d) Delete this remote
+y/e/d> y
+```
-#### --s3-location-constraint
+Done! Now you can try some commands (for macOS, use `./rclone` instead of `rclone`).
-Location constraint - must match endpoint when using IBM Cloud Public.
+1) Create a bucket (the name must be unique within the whole IONOS S3)
+```
+rclone mkdir ionos-fra:my-bucket
+```
+2) List available buckets
+```
+rclone lsd ionos-fra:
+```
+4) Copy a file from local to remote
+```
+rclone copy /Users/file.txt ionos-fra:my-bucket
+```
+3) List contents of a bucket
+```
+rclone ls ionos-fra:my-bucket
+```
+5) Copy a file from remote to local
+```
+rclone copy ionos-fra:my-bucket/file.txt
+```
-For on-prem COS, do not make a selection from this list, hit enter.
+### Minio
-Properties:
+[Minio](https://minio.io/) is an object storage server built for cloud application developers and devops.
-- Config: location_constraint
-- Env Var: RCLONE_S3_LOCATION_CONSTRAINT
-- Provider: IBMCOS
-- Type: string
-- Required: false
-- Examples:
- - "us-standard"
- - US Cross Region Standard
- - "us-vault"
- - US Cross Region Vault
- - "us-cold"
- - US Cross Region Cold
- - "us-flex"
- - US Cross Region Flex
- - "us-east-standard"
- - US East Region Standard
- - "us-east-vault"
- - US East Region Vault
- - "us-east-cold"
- - US East Region Cold
- - "us-east-flex"
- - US East Region Flex
- - "us-south-standard"
- - US South Region Standard
- - "us-south-vault"
- - US South Region Vault
- - "us-south-cold"
- - US South Region Cold
- - "us-south-flex"
- - US South Region Flex
- - "eu-standard"
- - EU Cross Region Standard
- - "eu-vault"
- - EU Cross Region Vault
- - "eu-cold"
- - EU Cross Region Cold
- - "eu-flex"
- - EU Cross Region Flex
- - "eu-gb-standard"
- - Great Britain Standard
- - "eu-gb-vault"
- - Great Britain Vault
- - "eu-gb-cold"
- - Great Britain Cold
- - "eu-gb-flex"
- - Great Britain Flex
- - "ap-standard"
- - APAC Standard
- - "ap-vault"
- - APAC Vault
- - "ap-cold"
- - APAC Cold
- - "ap-flex"
- - APAC Flex
- - "mel01-standard"
- - Melbourne Standard
- - "mel01-vault"
- - Melbourne Vault
- - "mel01-cold"
- - Melbourne Cold
- - "mel01-flex"
- - Melbourne Flex
- - "tor01-standard"
- - Toronto Standard
- - "tor01-vault"
- - Toronto Vault
- - "tor01-cold"
- - Toronto Cold
- - "tor01-flex"
- - Toronto Flex
+It is very easy to install and provides an S3 compatible server which can be used by rclone.
-#### --s3-location-constraint
+To use it, install Minio following the instructions [here](https://docs.minio.io/docs/minio-quickstart-guide).
-Location constraint - the location where your bucket will be located and your data stored.
+When it configures itself Minio will print something like this
+```
+Endpoint: http://192.168.1.106:9000 http://172.23.0.1:9000
+AccessKey: USWUXHGYZQYFYFFIT3RE
+SecretKey: MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03
+Region: us-east-1
+SQS ARNs: arn:minio:sqs:us-east-1:1:redis arn:minio:sqs:us-east-1:2:redis
-Properties:
+Browser Access:
+ http://192.168.1.106:9000 http://172.23.0.1:9000
-- Config: location_constraint
-- Env Var: RCLONE_S3_LOCATION_CONSTRAINT
-- Provider: RackCorp
-- Type: string
-- Required: false
-- Examples:
- - "global"
- - Global CDN Region
- - "au"
- - Australia (All locations)
- - "au-nsw"
- - NSW (Australia) Region
- - "au-qld"
- - QLD (Australia) Region
- - "au-vic"
- - VIC (Australia) Region
- - "au-wa"
- - Perth (Australia) Region
- - "ph"
- - Manila (Philippines) Region
- - "th"
- - Bangkok (Thailand) Region
- - "hk"
- - HK (Hong Kong) Region
- - "mn"
- - Ulaanbaatar (Mongolia) Region
- - "kg"
- - Bishkek (Kyrgyzstan) Region
- - "id"
- - Jakarta (Indonesia) Region
- - "jp"
- - Tokyo (Japan) Region
- - "sg"
- - SG (Singapore) Region
- - "de"
- - Frankfurt (Germany) Region
- - "us"
- - USA (AnyCast) Region
- - "us-east-1"
- - New York (USA) Region
- - "us-west-1"
- - Freemont (USA) Region
- - "nz"
- - Auckland (New Zealand) Region
+Command-line Access: https://docs.minio.io/docs/minio-client-quickstart-guide
+ $ mc config host add myminio http://192.168.1.106:9000 USWUXHGYZQYFYFFIT3RE MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03
-#### --s3-location-constraint
+Object API (Amazon S3 compatible):
+ Go: https://docs.minio.io/docs/golang-client-quickstart-guide
+ Java: https://docs.minio.io/docs/java-client-quickstart-guide
+ Python: https://docs.minio.io/docs/python-client-quickstart-guide
+ JavaScript: https://docs.minio.io/docs/javascript-client-quickstart-guide
+ .NET: https://docs.minio.io/docs/dotnet-client-quickstart-guide
-Location constraint - must be set to match the Region.
+Drive Capacity: 26 GiB Free, 165 GiB Total
+```
-Used when creating buckets only.
+These details need to go into `rclone config` like this. Note that it
+is important to put the region in as stated above.
-Properties:
+```
+env_auth> 1
+access_key_id> USWUXHGYZQYFYFFIT3RE
+secret_access_key> MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03
+region> us-east-1
+endpoint> http://192.168.1.106:9000
+location_constraint>
+server_side_encryption>
+```
-- Config: location_constraint
-- Env Var: RCLONE_S3_LOCATION_CONSTRAINT
-- Provider: Qiniu
-- Type: string
-- Required: false
-- Examples:
- - "cn-east-1"
- - East China Region 1
- - "cn-east-2"
- - East China Region 2
- - "cn-north-1"
- - North China Region 1
- - "cn-south-1"
- - South China Region 1
- - "us-north-1"
- - North America Region 1
- - "ap-southeast-1"
- - Southeast Asia Region 1
- - "ap-northeast-1"
- - Northeast Asia Region 1
+Which makes the config file look like this
-#### --s3-location-constraint
+```
+[minio]
+type = s3
+provider = Minio
+env_auth = false
+access_key_id = USWUXHGYZQYFYFFIT3RE
+secret_access_key = MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03
+region = us-east-1
+endpoint = http://192.168.1.106:9000
+location_constraint =
+server_side_encryption =
+```
-Location constraint - must be set to match the Region.
+So once set up, for example, to copy files into a bucket
-Leave blank if not sure. Used when creating buckets only.
+```
+rclone copy /path/to/files minio:bucket
+```
-Properties:
+### Qiniu Cloud Object Storage (Kodo) {#qiniu}
-- Config: location_constraint
-- Env Var: RCLONE_S3_LOCATION_CONSTRAINT
-- Provider: !AWS,Alibaba,ArvanCloud,HuaweiOBS,ChinaMobile,Cloudflare,IBMCOS,IDrive,IONOS,Leviia,Liara,Qiniu,RackCorp,Scaleway,StackPath,Storj,TencentCOS,Petabox
-- Type: string
-- Required: false
+[Qiniu Cloud Object Storage (Kodo)](https://www.qiniu.com/en/products/kodo), a completely independent-researched core technology which is proven by repeated customer experience has occupied absolute leading market leader position. Kodo can be widely applied to mass data management.
-#### --s3-acl
+To configure access to Qiniu Kodo, follow the steps below:
-Canned ACL used when creating buckets and storing or copying objects.
+1. Run `rclone config` and select `n` for a new remote.
-This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.
+```
+rclone config
+No remotes found, make a new one?
+n) New remote
+s) Set configuration password
+q) Quit config
+n/s/q> n
+```
-For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
+2. Give the name of the configuration. For example, name it 'qiniu'.
-Note that this ACL is applied when server-side copying objects as S3
-doesn't copy the ACL from the source but rather writes a fresh one.
+```
+name> qiniu
+```
-If the acl is an empty string then no X-Amz-Acl: header is added and
-the default (private) will be used.
+3. Select `s3` storage.
+```
+Choose a number from below, or type in your own value
+ 1 / 1Fichier
+ \ (fichier)
+ 2 / Akamai NetStorage
+ \ (netstorage)
+ 3 / Alias for an existing remote
+ \ (alias)
+ 4 / Amazon Drive
+ \ (amazon cloud drive)
+ 5 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, China Mobile, Cloudflare, ArvanCloud, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, IDrive e2, Liara, Lyve Cloud, Minio, Netease, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Synology, Tencent COS, Qiniu and Wasabi
+ \ (s3)
+[snip]
+Storage> s3
+```
-Properties:
+4. Select `Qiniu` provider.
+```
+Choose a number from below, or type in your own value
+1 / Amazon Web Services (AWS) S3
+ \ "AWS"
+[snip]
+22 / Qiniu Object Storage (Kodo)
+ \ (Qiniu)
+[snip]
+provider> Qiniu
+```
-- Config: acl
-- Env Var: RCLONE_S3_ACL
-- Provider: !Storj,Synology,Cloudflare
-- Type: string
-- Required: false
-- Examples:
- - "default"
- - Owner gets Full_CONTROL.
- - No one else has access rights (default).
- - "private"
- - Owner gets FULL_CONTROL.
- - No one else has access rights (default).
- - "public-read"
- - Owner gets FULL_CONTROL.
- - The AllUsers group gets READ access.
- - "public-read-write"
- - Owner gets FULL_CONTROL.
- - The AllUsers group gets READ and WRITE access.
- - Granting this on a bucket is generally not recommended.
- - "authenticated-read"
- - Owner gets FULL_CONTROL.
- - The AuthenticatedUsers group gets READ access.
- - "bucket-owner-read"
- - Object owner gets FULL_CONTROL.
- - Bucket owner gets READ access.
- - If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
- - "bucket-owner-full-control"
- - Both the object owner and the bucket owner get FULL_CONTROL over the object.
- - If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
- - "private"
- - Owner gets FULL_CONTROL.
- - No one else has access rights (default).
- - This acl is available on IBM Cloud (Infra), IBM Cloud (Storage), On-Premise COS.
- - "public-read"
- - Owner gets FULL_CONTROL.
- - The AllUsers group gets READ access.
- - This acl is available on IBM Cloud (Infra), IBM Cloud (Storage), On-Premise IBM COS.
- - "public-read-write"
- - Owner gets FULL_CONTROL.
- - The AllUsers group gets READ and WRITE access.
- - This acl is available on IBM Cloud (Infra), On-Premise IBM COS.
- - "authenticated-read"
- - Owner gets FULL_CONTROL.
- - The AuthenticatedUsers group gets READ access.
- - Not supported on Buckets.
- - This acl is available on IBM Cloud (Infra) and On-Premise IBM COS.
+5. Enter your SecretId and SecretKey of Qiniu Kodo.
-#### --s3-server-side-encryption
+```
+Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
+Only applies if access_key_id and secret_access_key is blank.
+Enter a boolean value (true or false). Press Enter for the default ("false").
+Choose a number from below, or type in your own value
+ 1 / Enter AWS credentials in the next step
+ \ "false"
+ 2 / Get AWS credentials from the environment (env vars or IAM)
+ \ "true"
+env_auth> 1
+AWS Access Key ID.
+Leave blank for anonymous access or runtime credentials.
+Enter a string value. Press Enter for the default ("").
+access_key_id> AKIDxxxxxxxxxx
+AWS Secret Access Key (password)
+Leave blank for anonymous access or runtime credentials.
+Enter a string value. Press Enter for the default ("").
+secret_access_key> xxxxxxxxxxx
+```
-The server-side encryption algorithm used when storing this object in S3.
+6. Select endpoint for Qiniu Kodo. This is the standard endpoint for different region.
-Properties:
+```
+ / The default endpoint - a good choice if you are unsure.
+ 1 | East China Region 1.
+ | Needs location constraint cn-east-1.
+ \ (cn-east-1)
+ / East China Region 2.
+ 2 | Needs location constraint cn-east-2.
+ \ (cn-east-2)
+ / North China Region 1.
+ 3 | Needs location constraint cn-north-1.
+ \ (cn-north-1)
+ / South China Region 1.
+ 4 | Needs location constraint cn-south-1.
+ \ (cn-south-1)
+ / North America Region.
+ 5 | Needs location constraint us-north-1.
+ \ (us-north-1)
+ / Southeast Asia Region 1.
+ 6 | Needs location constraint ap-southeast-1.
+ \ (ap-southeast-1)
+ / Northeast Asia Region 1.
+ 7 | Needs location constraint ap-northeast-1.
+ \ (ap-northeast-1)
+[snip]
+endpoint> 1
-- Config: server_side_encryption
-- Env Var: RCLONE_S3_SERVER_SIDE_ENCRYPTION
-- Provider: AWS,Ceph,ChinaMobile,Minio
-- Type: string
-- Required: false
-- Examples:
- - ""
- - None
- - "AES256"
- - AES256
- - "aws:kms"
- - aws:kms
+Option endpoint.
+Endpoint for Qiniu Object Storage.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+ 1 / East China Endpoint 1
+ \ (s3-cn-east-1.qiniucs.com)
+ 2 / East China Endpoint 2
+ \ (s3-cn-east-2.qiniucs.com)
+ 3 / North China Endpoint 1
+ \ (s3-cn-north-1.qiniucs.com)
+ 4 / South China Endpoint 1
+ \ (s3-cn-south-1.qiniucs.com)
+ 5 / North America Endpoint 1
+ \ (s3-us-north-1.qiniucs.com)
+ 6 / Southeast Asia Endpoint 1
+ \ (s3-ap-southeast-1.qiniucs.com)
+ 7 / Northeast Asia Endpoint 1
+ \ (s3-ap-northeast-1.qiniucs.com)
+endpoint> 1
-#### --s3-sse-kms-key-id
+Option location_constraint.
+Location constraint - must be set to match the Region.
+Used when creating buckets only.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+ 1 / East China Region 1
+ \ (cn-east-1)
+ 2 / East China Region 2
+ \ (cn-east-2)
+ 3 / North China Region 1
+ \ (cn-north-1)
+ 4 / South China Region 1
+ \ (cn-south-1)
+ 5 / North America Region 1
+ \ (us-north-1)
+ 6 / Southeast Asia Region 1
+ \ (ap-southeast-1)
+ 7 / Northeast Asia Region 1
+ \ (ap-northeast-1)
+location_constraint> 1
+```
-If using KMS ID you must provide the ARN of Key.
+7. Choose acl and storage class.
-Properties:
+```
+Note that this ACL is applied when server-side copying objects as S3
+doesn't copy the ACL from the source but rather writes a fresh one.
+Enter a string value. Press Enter for the default ("").
+Choose a number from below, or type in your own value
+ / Owner gets FULL_CONTROL.
+ 1 | No one else has access rights (default).
+ \ (private)
+ / Owner gets FULL_CONTROL.
+ 2 | The AllUsers group gets READ access.
+ \ (public-read)
+[snip]
+acl> 2
+The storage class to use when storing new objects in Tencent COS.
+Enter a string value. Press Enter for the default ("").
+Choose a number from below, or type in your own value
+ 1 / Standard storage class
+ \ (STANDARD)
+ 2 / Infrequent access storage mode
+ \ (LINE)
+ 3 / Archive storage mode
+ \ (GLACIER)
+ 4 / Deep archive storage mode
+ \ (DEEP_ARCHIVE)
+[snip]
+storage_class> 1
+Edit advanced config? (y/n)
+y) Yes
+n) No (default)
+y/n> n
+Remote config
+--------------------
+[qiniu]
+- type: s3
+- provider: Qiniu
+- access_key_id: xxx
+- secret_access_key: xxx
+- region: cn-east-1
+- endpoint: s3-cn-east-1.qiniucs.com
+- location_constraint: cn-east-1
+- acl: public-read
+- storage_class: STANDARD
+--------------------
+y) Yes this is OK (default)
+e) Edit this remote
+d) Delete this remote
+y/e/d> y
+Current remotes:
-- Config: sse_kms_key_id
-- Env Var: RCLONE_S3_SSE_KMS_KEY_ID
-- Provider: AWS,Ceph,Minio
-- Type: string
-- Required: false
-- Examples:
- - ""
- - None
- - "arn:aws:kms:us-east-1:*"
- - arn:aws:kms:*
+Name Type
+==== ====
+qiniu s3
+```
-#### --s3-storage-class
+### RackCorp {#RackCorp}
-The storage class to use when storing new objects in S3.
+[RackCorp Object Storage](https://www.rackcorp.com/storage/s3storage) is an S3 compatible object storage platform from your friendly cloud provider RackCorp.
+The service is fast, reliable, well priced and located in many strategic locations unserviced by others, to ensure you can maintain data sovereignty.
-Properties:
+Before you can use RackCorp Object Storage, you'll need to "[sign up](https://www.rackcorp.com/signup)" for an account on our "[portal](https://portal.rackcorp.com)".
+Next you can create an `access key`, a `secret key` and `buckets`, in your location of choice with ease.
+These details are required for the next steps of configuration, when `rclone config` asks for your `access_key_id` and `secret_access_key`.
-- Config: storage_class
-- Env Var: RCLONE_S3_STORAGE_CLASS
-- Provider: AWS
-- Type: string
-- Required: false
-- Examples:
- - ""
- - Default
- - "STANDARD"
- - Standard storage class
- - "REDUCED_REDUNDANCY"
- - Reduced redundancy storage class
- - "STANDARD_IA"
- - Standard Infrequent Access storage class
- - "ONEZONE_IA"
- - One Zone Infrequent Access storage class
- - "GLACIER"
- - Glacier storage class
- - "DEEP_ARCHIVE"
- - Glacier Deep Archive storage class
- - "INTELLIGENT_TIERING"
- - Intelligent-Tiering storage class
- - "GLACIER_IR"
- - Glacier Instant Retrieval storage class
+Your config should end up looking a bit like this:
-#### --s3-storage-class
+```
+[RCS3-demo-config]
+type = s3
+provider = RackCorp
+env_auth = true
+access_key_id = YOURACCESSKEY
+secret_access_key = YOURSECRETACCESSKEY
+region = au-nsw
+endpoint = s3.rackcorp.com
+location_constraint = au-nsw
+```
-The storage class to use when storing new objects in OSS.
+### Rclone Serve S3 {#rclone}
-Properties:
+Rclone can serve any remote over the S3 protocol. For details see the
+[rclone serve s3](https://rclone.org/commands/rclone_serve_http/) documentation.
-- Config: storage_class
-- Env Var: RCLONE_S3_STORAGE_CLASS
-- Provider: Alibaba
-- Type: string
-- Required: false
-- Examples:
- - ""
- - Default
- - "STANDARD"
- - Standard storage class
- - "GLACIER"
- - Archive storage mode
- - "STANDARD_IA"
- - Infrequent access storage mode
+For example, to serve `remote:path` over s3, run the server like this:
-#### --s3-storage-class
+```
+rclone serve s3 --auth-key ACCESS_KEY_ID,SECRET_ACCESS_KEY remote:path
+```
-The storage class to use when storing new objects in ChinaMobile.
+This will be compatible with an rclone remote which is defined like this:
-Properties:
+```
+[serves3]
+type = s3
+provider = Rclone
+endpoint = http://127.0.0.1:8080/
+access_key_id = ACCESS_KEY_ID
+secret_access_key = SECRET_ACCESS_KEY
+use_multipart_uploads = false
+```
-- Config: storage_class
-- Env Var: RCLONE_S3_STORAGE_CLASS
-- Provider: ChinaMobile
-- Type: string
-- Required: false
-- Examples:
- - ""
- - Default
- - "STANDARD"
- - Standard storage class
- - "GLACIER"
- - Archive storage mode
- - "STANDARD_IA"
- - Infrequent access storage mode
+Note that setting `disable_multipart_uploads = true` is to work around
+[a bug](https://rclone.org/commands/rclone_serve_http/#bugs) which will be fixed in due course.
-#### --s3-storage-class
+### Scaleway
-The storage class to use when storing new objects in Liara
+[Scaleway](https://www.scaleway.com/object-storage/) The Object Storage platform allows you to store anything from backups, logs and web assets to documents and photos.
+Files can be dropped from the Scaleway console or transferred through our API and CLI or using any S3-compatible tool.
-Properties:
+Scaleway provides an S3 interface which can be configured for use with rclone like this:
-- Config: storage_class
-- Env Var: RCLONE_S3_STORAGE_CLASS
-- Provider: Liara
-- Type: string
-- Required: false
-- Examples:
- - "STANDARD"
- - Standard storage class
+```
+[scaleway]
+type = s3
+provider = Scaleway
+env_auth = false
+endpoint = s3.nl-ams.scw.cloud
+access_key_id = SCWXXXXXXXXXXXXXX
+secret_access_key = 1111111-2222-3333-44444-55555555555555
+region = nl-ams
+location_constraint =
+acl = private
+server_side_encryption =
+storage_class =
+```
-#### --s3-storage-class
+[C14 Cold Storage](https://www.online.net/en/storage/c14-cold-storage) is the low-cost S3 Glacier alternative from Scaleway and it works the same way as on S3 by accepting the "GLACIER" `storage_class`.
+So you can configure your remote with the `storage_class = GLACIER` option to upload directly to C14. Don't forget that in this state you can't read files back after, you will need to restore them to "STANDARD" storage_class first before being able to read them (see "restore" section above)
-The storage class to use when storing new objects in ArvanCloud.
+### Seagate Lyve Cloud {#lyve}
-Properties:
+[Seagate Lyve Cloud](https://www.seagate.com/gb/en/services/cloud/storage/) is an S3
+compatible object storage platform from [Seagate](https://seagate.com/) intended for enterprise use.
-- Config: storage_class
-- Env Var: RCLONE_S3_STORAGE_CLASS
-- Provider: ArvanCloud
-- Type: string
-- Required: false
-- Examples:
- - "STANDARD"
- - Standard storage class
+Here is a config run through for a remote called `remote` - you may
+choose a different name of course. Note that to create an access key
+and secret key you will need to create a service account first.
-#### --s3-storage-class
+```
+$ rclone config
+No remotes found, make a new one?
+n) New remote
+s) Set configuration password
+q) Quit config
+n/s/q> n
+name> remote
+```
-The storage class to use when storing new objects in Tencent COS.
+Choose `s3` backend
-Properties:
+```
+Type of storage to configure.
+Choose a number from below, or type in your own value.
+[snip]
+XX / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, ChinaMobile, ArvanCloud, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, Liara, Lyve Cloud, Minio, RackCorp, SeaweedFS, and Tencent COS
+ \ (s3)
+[snip]
+Storage> s3
+```
-- Config: storage_class
-- Env Var: RCLONE_S3_STORAGE_CLASS
-- Provider: TencentCOS
-- Type: string
-- Required: false
-- Examples:
- - ""
- - Default
- - "STANDARD"
- - Standard storage class
- - "ARCHIVE"
- - Archive storage mode
- - "STANDARD_IA"
- - Infrequent access storage mode
+Choose `LyveCloud` as S3 provider
-#### --s3-storage-class
+```
+Choose your S3 provider.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+[snip]
+XX / Seagate Lyve Cloud
+ \ (LyveCloud)
+[snip]
+provider> LyveCloud
+```
-The storage class to use when storing new objects in S3.
+Take the default (just press enter) to enter access key and secret in the config file.
-Properties:
+```
+Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
+Only applies if access_key_id and secret_access_key is blank.
+Choose a number from below, or type in your own boolean value (true or false).
+Press Enter for the default (false).
+ 1 / Enter AWS credentials in the next step.
+ \ (false)
+ 2 / Get AWS credentials from the environment (env vars or IAM).
+ \ (true)
+env_auth>
+```
-- Config: storage_class
-- Env Var: RCLONE_S3_STORAGE_CLASS
-- Provider: Scaleway
-- Type: string
-- Required: false
-- Examples:
- - ""
- - Default.
- - "STANDARD"
- - The Standard class for any upload.
- - Suitable for on-demand content like streaming or CDN.
- - Available in all regions.
- - "GLACIER"
- - Archived storage.
- - Prices are lower, but it needs to be restored first to be accessed.
- - Available in FR-PAR and NL-AMS regions.
- - "ONEZONE_IA"
- - One Zone - Infrequent Access.
- - A good choice for storing secondary backup copies or easily re-creatable data.
- - Available in the FR-PAR region only.
+```
+AWS Access Key ID.
+Leave blank for anonymous access or runtime credentials.
+Enter a value. Press Enter to leave empty.
+access_key_id> XXX
+```
-#### --s3-storage-class
+```
+AWS Secret Access Key (password).
+Leave blank for anonymous access or runtime credentials.
+Enter a value. Press Enter to leave empty.
+secret_access_key> YYY
+```
-The storage class to use when storing new objects in Qiniu.
+Leave region blank
-Properties:
+```
+Region to connect to.
+Leave blank if you are using an S3 clone and you don't have a region.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+ / Use this if unsure.
+ 1 | Will use v4 signatures and an empty region.
+ \ ()
+ / Use this only if v4 signatures don't work.
+ 2 | E.g. pre Jewel/v10 CEPH.
+ \ (other-v2-signature)
+region>
+```
-- Config: storage_class
-- Env Var: RCLONE_S3_STORAGE_CLASS
-- Provider: Qiniu
-- Type: string
-- Required: false
-- Examples:
- - "STANDARD"
- - Standard storage class
- - "LINE"
- - Infrequent access storage mode
- - "GLACIER"
- - Archive storage mode
- - "DEEP_ARCHIVE"
- - Deep archive storage mode
+Choose an endpoint from the list
-### Advanced options
+```
+Endpoint for S3 API.
+Required when using an S3 clone.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+ 1 / Seagate Lyve Cloud US East 1 (Virginia)
+ \ (s3.us-east-1.lyvecloud.seagate.com)
+ 2 / Seagate Lyve Cloud US West 1 (California)
+ \ (s3.us-west-1.lyvecloud.seagate.com)
+ 3 / Seagate Lyve Cloud AP Southeast 1 (Singapore)
+ \ (s3.ap-southeast-1.lyvecloud.seagate.com)
+endpoint> 1
+```
-Here are the Advanced options specific to s3 (Amazon S3 Compliant Storage Providers including AWS, Alibaba, ArvanCloud, Ceph, China Mobile, Cloudflare, GCS, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, IDrive e2, IONOS Cloud, Leviia, Liara, Lyve Cloud, Minio, Netease, Petabox, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Synology, Tencent COS, Qiniu and Wasabi).
+Leave location constraint blank
-#### --s3-bucket-acl
+```
+Location constraint - must be set to match the Region.
+Leave blank if not sure. Used when creating buckets only.
+Enter a value. Press Enter to leave empty.
+location_constraint>
+```
-Canned ACL used when creating buckets.
+Choose default ACL (`private`).
+```
+Canned ACL used when creating buckets and storing or copying objects.
+This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.
For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
+Note that this ACL is applied when server-side copying objects as S3
+doesn't copy the ACL from the source but rather writes a fresh one.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+ / Owner gets FULL_CONTROL.
+ 1 | No one else has access rights (default).
+ \ (private)
+[snip]
+acl>
+```
-Note that this ACL is applied when only when creating buckets. If it
-isn't set then "acl" is used instead.
-
-If the "acl" and "bucket_acl" are empty strings then no X-Amz-Acl:
-header is added and the default (private) will be used.
-
-
-Properties:
+And the config file should end up looking like this:
-- Config: bucket_acl
-- Env Var: RCLONE_S3_BUCKET_ACL
-- Type: string
-- Required: false
-- Examples:
- - "private"
- - Owner gets FULL_CONTROL.
- - No one else has access rights (default).
- - "public-read"
- - Owner gets FULL_CONTROL.
- - The AllUsers group gets READ access.
- - "public-read-write"
- - Owner gets FULL_CONTROL.
- - The AllUsers group gets READ and WRITE access.
- - Granting this on a bucket is generally not recommended.
- - "authenticated-read"
- - Owner gets FULL_CONTROL.
- - The AuthenticatedUsers group gets READ access.
+```
+[remote]
+type = s3
+provider = LyveCloud
+access_key_id = XXX
+secret_access_key = YYY
+endpoint = s3.us-east-1.lyvecloud.seagate.com
+```
-#### --s3-requester-pays
+### SeaweedFS
-Enables requester pays option when interacting with S3 bucket.
+[SeaweedFS](https://github.com/chrislusf/seaweedfs/) is a distributed storage system for
+blobs, objects, files, and data lake, with O(1) disk seek and a scalable file metadata store.
+It has an S3 compatible object storage interface. SeaweedFS can also act as a
+[gateway to remote S3 compatible object store](https://github.com/chrislusf/seaweedfs/wiki/Gateway-to-Remote-Object-Storage)
+to cache data and metadata with asynchronous write back, for fast local speed and minimize access cost.
-Properties:
+Assuming the SeaweedFS are configured with `weed shell` as such:
+```
+> s3.bucket.create -name foo
+> s3.configure -access_key=any -secret_key=any -buckets=foo -user=me -actions=Read,Write,List,Tagging,Admin -apply
+{
+ "identities": [
+ {
+ "name": "me",
+ "credentials": [
+ {
+ "accessKey": "any",
+ "secretKey": "any"
+ }
+ ],
+ "actions": [
+ "Read:foo",
+ "Write:foo",
+ "List:foo",
+ "Tagging:foo",
+ "Admin:foo"
+ ]
+ }
+ ]
+}
+```
-- Config: requester_pays
-- Env Var: RCLONE_S3_REQUESTER_PAYS
-- Provider: AWS
-- Type: bool
-- Default: false
+To use rclone with SeaweedFS, above configuration should end up with something like this in
+your config:
-#### --s3-sse-customer-algorithm
+```
+[seaweedfs_s3]
+type = s3
+provider = SeaweedFS
+access_key_id = any
+secret_access_key = any
+endpoint = localhost:8333
+```
-If using SSE-C, the server-side encryption algorithm used when storing this object in S3.
+So once set up, for example to copy files into a bucket
-Properties:
+```
+rclone copy /path/to/files seaweedfs_s3:foo
+```
-- Config: sse_customer_algorithm
-- Env Var: RCLONE_S3_SSE_CUSTOMER_ALGORITHM
-- Provider: AWS,Ceph,ChinaMobile,Minio
-- Type: string
-- Required: false
-- Examples:
- - ""
- - None
- - "AES256"
- - AES256
+### Wasabi
-#### --s3-sse-customer-key
+[Wasabi](https://wasabi.com) is a cloud-based object storage service for a
+broad range of applications and use cases. Wasabi is designed for
+individuals and organizations that require a high-performance,
+reliable, and secure data storage infrastructure at minimal cost.
-To use SSE-C you may provide the secret encryption key used to encrypt/decrypt your data.
+Wasabi provides an S3 interface which can be configured for use with
+rclone like this.
-Alternatively you can provide --sse-customer-key-base64.
+```
+No remotes found, make a new one?
+n) New remote
+s) Set configuration password
+n/s> n
+name> wasabi
+Type of storage to configure.
+Choose a number from below, or type in your own value
+[snip]
+XX / Amazon S3 (also Dreamhost, Ceph, ChinaMobile, ArvanCloud, Minio, Liara)
+ \ "s3"
+[snip]
+Storage> s3
+Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars). Only applies if access_key_id and secret_access_key is blank.
+Choose a number from below, or type in your own value
+ 1 / Enter AWS credentials in the next step
+ \ "false"
+ 2 / Get AWS credentials from the environment (env vars or IAM)
+ \ "true"
+env_auth> 1
+AWS Access Key ID - leave blank for anonymous access or runtime credentials.
+access_key_id> YOURACCESSKEY
+AWS Secret Access Key (password) - leave blank for anonymous access or runtime credentials.
+secret_access_key> YOURSECRETACCESSKEY
+Region to connect to.
+Choose a number from below, or type in your own value
+ / The default endpoint - a good choice if you are unsure.
+ 1 | US Region, Northern Virginia, or Pacific Northwest.
+ | Leave location constraint empty.
+ \ "us-east-1"
+[snip]
+region> us-east-1
+Endpoint for S3 API.
+Leave blank if using AWS to use the default endpoint for the region.
+Specify if using an S3 clone such as Ceph.
+endpoint> s3.wasabisys.com
+Location constraint - must be set to match the Region. Used when creating buckets only.
+Choose a number from below, or type in your own value
+ 1 / Empty for US Region, Northern Virginia, or Pacific Northwest.
+ \ ""
+[snip]
+location_constraint>
+Canned ACL used when creating buckets and/or storing objects in S3.
+For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
+Choose a number from below, or type in your own value
+ 1 / Owner gets FULL_CONTROL. No one else has access rights (default).
+ \ "private"
+[snip]
+acl>
+The server-side encryption algorithm used when storing this object in S3.
+Choose a number from below, or type in your own value
+ 1 / None
+ \ ""
+ 2 / AES256
+ \ "AES256"
+server_side_encryption>
+The storage class to use when storing objects in S3.
+Choose a number from below, or type in your own value
+ 1 / Default
+ \ ""
+ 2 / Standard storage class
+ \ "STANDARD"
+ 3 / Reduced redundancy storage class
+ \ "REDUCED_REDUNDANCY"
+ 4 / Standard Infrequent Access storage class
+ \ "STANDARD_IA"
+storage_class>
+Remote config
+--------------------
+[wasabi]
+env_auth = false
+access_key_id = YOURACCESSKEY
+secret_access_key = YOURSECRETACCESSKEY
+region = us-east-1
+endpoint = s3.wasabisys.com
+location_constraint =
+acl =
+server_side_encryption =
+storage_class =
+--------------------
+y) Yes this is OK
+e) Edit this remote
+d) Delete this remote
+y/e/d> y
+```
-Properties:
+This will leave the config file looking like this.
-- Config: sse_customer_key
-- Env Var: RCLONE_S3_SSE_CUSTOMER_KEY
-- Provider: AWS,Ceph,ChinaMobile,Minio
-- Type: string
-- Required: false
-- Examples:
- - ""
- - None
+```
+[wasabi]
+type = s3
+provider = Wasabi
+env_auth = false
+access_key_id = YOURACCESSKEY
+secret_access_key = YOURSECRETACCESSKEY
+region =
+endpoint = s3.wasabisys.com
+location_constraint =
+acl =
+server_side_encryption =
+storage_class =
+```
-#### --s3-sse-customer-key-base64
+### Alibaba OSS {#alibaba-oss}
-If using SSE-C you must provide the secret encryption key encoded in base64 format to encrypt/decrypt your data.
+Here is an example of making an [Alibaba Cloud (Aliyun) OSS](https://www.alibabacloud.com/product/oss/)
+configuration. First run:
-Alternatively you can provide --sse-customer-key.
+ rclone config
-Properties:
+This will guide you through an interactive setup process.
-- Config: sse_customer_key_base64
-- Env Var: RCLONE_S3_SSE_CUSTOMER_KEY_BASE64
-- Provider: AWS,Ceph,ChinaMobile,Minio
-- Type: string
-- Required: false
-- Examples:
- - ""
- - None
+```
+No remotes found, make a new one?
+n) New remote
+s) Set configuration password
+q) Quit config
+n/s/q> n
+name> oss
+Type of storage to configure.
+Enter a string value. Press Enter for the default ("").
+Choose a number from below, or type in your own value
+[snip]
+ 4 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, ChinaMobile, ArvanCloud, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, Liara, Minio, and Tencent COS
+ \ "s3"
+[snip]
+Storage> s3
+Choose your S3 provider.
+Enter a string value. Press Enter for the default ("").
+Choose a number from below, or type in your own value
+ 1 / Amazon Web Services (AWS) S3
+ \ "AWS"
+ 2 / Alibaba Cloud Object Storage System (OSS) formerly Aliyun
+ \ "Alibaba"
+ 3 / Ceph Object Storage
+ \ "Ceph"
+[snip]
+provider> Alibaba
+Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
+Only applies if access_key_id and secret_access_key is blank.
+Enter a boolean value (true or false). Press Enter for the default ("false").
+Choose a number from below, or type in your own value
+ 1 / Enter AWS credentials in the next step
+ \ "false"
+ 2 / Get AWS credentials from the environment (env vars or IAM)
+ \ "true"
+env_auth> 1
+AWS Access Key ID.
+Leave blank for anonymous access or runtime credentials.
+Enter a string value. Press Enter for the default ("").
+access_key_id> accesskeyid
+AWS Secret Access Key (password)
+Leave blank for anonymous access or runtime credentials.
+Enter a string value. Press Enter for the default ("").
+secret_access_key> secretaccesskey
+Endpoint for OSS API.
+Enter a string value. Press Enter for the default ("").
+Choose a number from below, or type in your own value
+ 1 / East China 1 (Hangzhou)
+ \ "oss-cn-hangzhou.aliyuncs.com"
+ 2 / East China 2 (Shanghai)
+ \ "oss-cn-shanghai.aliyuncs.com"
+ 3 / North China 1 (Qingdao)
+ \ "oss-cn-qingdao.aliyuncs.com"
+[snip]
+endpoint> 1
+Canned ACL used when creating buckets and storing or copying objects.
-#### --s3-sse-customer-key-md5
+Note that this ACL is applied when server-side copying objects as S3
+doesn't copy the ACL from the source but rather writes a fresh one.
+Enter a string value. Press Enter for the default ("").
+Choose a number from below, or type in your own value
+ 1 / Owner gets FULL_CONTROL. No one else has access rights (default).
+ \ "private"
+ 2 / Owner gets FULL_CONTROL. The AllUsers group gets READ access.
+ \ "public-read"
+ / Owner gets FULL_CONTROL. The AllUsers group gets READ and WRITE access.
+[snip]
+acl> 1
+The storage class to use when storing new objects in OSS.
+Enter a string value. Press Enter for the default ("").
+Choose a number from below, or type in your own value
+ 1 / Default
+ \ ""
+ 2 / Standard storage class
+ \ "STANDARD"
+ 3 / Archive storage mode.
+ \ "GLACIER"
+ 4 / Infrequent access storage mode.
+ \ "STANDARD_IA"
+storage_class> 1
+Edit advanced config? (y/n)
+y) Yes
+n) No
+y/n> n
+Remote config
+--------------------
+[oss]
+type = s3
+provider = Alibaba
+env_auth = false
+access_key_id = accesskeyid
+secret_access_key = secretaccesskey
+endpoint = oss-cn-hangzhou.aliyuncs.com
+acl = private
+storage_class = Standard
+--------------------
+y) Yes this is OK
+e) Edit this remote
+d) Delete this remote
+y/e/d> y
+```
-If using SSE-C you may provide the secret encryption key MD5 checksum (optional).
+### China Mobile Ecloud Elastic Object Storage (EOS) {#china-mobile-ecloud-eos}
-If you leave it blank, this is calculated automatically from the sse_customer_key provided.
+Here is an example of making an [China Mobile Ecloud Elastic Object Storage (EOS)](https:///ecloud.10086.cn/home/product-introduction/eos/)
+configuration. First run:
+ rclone config
-Properties:
+This will guide you through an interactive setup process.
-- Config: sse_customer_key_md5
-- Env Var: RCLONE_S3_SSE_CUSTOMER_KEY_MD5
-- Provider: AWS,Ceph,ChinaMobile,Minio
-- Type: string
-- Required: false
-- Examples:
- - ""
- - None
+```
+No remotes found, make a new one?
+n) New remote
+s) Set configuration password
+q) Quit config
+n/s/q> n
+name> ChinaMobile
+Option Storage.
+Type of storage to configure.
+Choose a number from below, or type in your own value.
+ ...
+ 5 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, ChinaMobile, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, Lyve Cloud, Minio, RackCorp, SeaweedFS, and Tencent COS
+ \ (s3)
+ ...
+Storage> s3
+Option provider.
+Choose your S3 provider.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+ ...
+ 4 / China Mobile Ecloud Elastic Object Storage (EOS)
+ \ (ChinaMobile)
+ ...
+provider> ChinaMobile
+Option env_auth.
+Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
+Only applies if access_key_id and secret_access_key is blank.
+Choose a number from below, or type in your own boolean value (true or false).
+Press Enter for the default (false).
+ 1 / Enter AWS credentials in the next step.
+ \ (false)
+ 2 / Get AWS credentials from the environment (env vars or IAM).
+ \ (true)
+env_auth>
+Option access_key_id.
+AWS Access Key ID.
+Leave blank for anonymous access or runtime credentials.
+Enter a value. Press Enter to leave empty.
+access_key_id> accesskeyid
+Option secret_access_key.
+AWS Secret Access Key (password).
+Leave blank for anonymous access or runtime credentials.
+Enter a value. Press Enter to leave empty.
+secret_access_key> secretaccesskey
+Option endpoint.
+Endpoint for China Mobile Ecloud Elastic Object Storage (EOS) API.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+ / The default endpoint - a good choice if you are unsure.
+ 1 | East China (Suzhou)
+ \ (eos-wuxi-1.cmecloud.cn)
+ 2 / East China (Jinan)
+ \ (eos-jinan-1.cmecloud.cn)
+ 3 / East China (Hangzhou)
+ \ (eos-ningbo-1.cmecloud.cn)
+ 4 / East China (Shanghai-1)
+ \ (eos-shanghai-1.cmecloud.cn)
+ 5 / Central China (Zhengzhou)
+ \ (eos-zhengzhou-1.cmecloud.cn)
+ 6 / Central China (Changsha-1)
+ \ (eos-hunan-1.cmecloud.cn)
+ 7 / Central China (Changsha-2)
+ \ (eos-zhuzhou-1.cmecloud.cn)
+ 8 / South China (Guangzhou-2)
+ \ (eos-guangzhou-1.cmecloud.cn)
+ 9 / South China (Guangzhou-3)
+ \ (eos-dongguan-1.cmecloud.cn)
+10 / North China (Beijing-1)
+ \ (eos-beijing-1.cmecloud.cn)
+11 / North China (Beijing-2)
+ \ (eos-beijing-2.cmecloud.cn)
+12 / North China (Beijing-3)
+ \ (eos-beijing-4.cmecloud.cn)
+13 / North China (Huhehaote)
+ \ (eos-huhehaote-1.cmecloud.cn)
+14 / Southwest China (Chengdu)
+ \ (eos-chengdu-1.cmecloud.cn)
+15 / Southwest China (Chongqing)
+ \ (eos-chongqing-1.cmecloud.cn)
+16 / Southwest China (Guiyang)
+ \ (eos-guiyang-1.cmecloud.cn)
+17 / Nouthwest China (Xian)
+ \ (eos-xian-1.cmecloud.cn)
+18 / Yunnan China (Kunming)
+ \ (eos-yunnan.cmecloud.cn)
+19 / Yunnan China (Kunming-2)
+ \ (eos-yunnan-2.cmecloud.cn)
+20 / Tianjin China (Tianjin)
+ \ (eos-tianjin-1.cmecloud.cn)
+21 / Jilin China (Changchun)
+ \ (eos-jilin-1.cmecloud.cn)
+22 / Hubei China (Xiangyan)
+ \ (eos-hubei-1.cmecloud.cn)
+23 / Jiangxi China (Nanchang)
+ \ (eos-jiangxi-1.cmecloud.cn)
+24 / Gansu China (Lanzhou)
+ \ (eos-gansu-1.cmecloud.cn)
+25 / Shanxi China (Taiyuan)
+ \ (eos-shanxi-1.cmecloud.cn)
+26 / Liaoning China (Shenyang)
+ \ (eos-liaoning-1.cmecloud.cn)
+27 / Hebei China (Shijiazhuang)
+ \ (eos-hebei-1.cmecloud.cn)
+28 / Fujian China (Xiamen)
+ \ (eos-fujian-1.cmecloud.cn)
+29 / Guangxi China (Nanning)
+ \ (eos-guangxi-1.cmecloud.cn)
+30 / Anhui China (Huainan)
+ \ (eos-anhui-1.cmecloud.cn)
+endpoint> 1
+Option location_constraint.
+Location constraint - must match endpoint.
+Used when creating buckets only.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+ 1 / East China (Suzhou)
+ \ (wuxi1)
+ 2 / East China (Jinan)
+ \ (jinan1)
+ 3 / East China (Hangzhou)
+ \ (ningbo1)
+ 4 / East China (Shanghai-1)
+ \ (shanghai1)
+ 5 / Central China (Zhengzhou)
+ \ (zhengzhou1)
+ 6 / Central China (Changsha-1)
+ \ (hunan1)
+ 7 / Central China (Changsha-2)
+ \ (zhuzhou1)
+ 8 / South China (Guangzhou-2)
+ \ (guangzhou1)
+ 9 / South China (Guangzhou-3)
+ \ (dongguan1)
+10 / North China (Beijing-1)
+ \ (beijing1)
+11 / North China (Beijing-2)
+ \ (beijing2)
+12 / North China (Beijing-3)
+ \ (beijing4)
+13 / North China (Huhehaote)
+ \ (huhehaote1)
+14 / Southwest China (Chengdu)
+ \ (chengdu1)
+15 / Southwest China (Chongqing)
+ \ (chongqing1)
+16 / Southwest China (Guiyang)
+ \ (guiyang1)
+17 / Nouthwest China (Xian)
+ \ (xian1)
+18 / Yunnan China (Kunming)
+ \ (yunnan)
+19 / Yunnan China (Kunming-2)
+ \ (yunnan2)
+20 / Tianjin China (Tianjin)
+ \ (tianjin1)
+21 / Jilin China (Changchun)
+ \ (jilin1)
+22 / Hubei China (Xiangyan)
+ \ (hubei1)
+23 / Jiangxi China (Nanchang)
+ \ (jiangxi1)
+24 / Gansu China (Lanzhou)
+ \ (gansu1)
+25 / Shanxi China (Taiyuan)
+ \ (shanxi1)
+26 / Liaoning China (Shenyang)
+ \ (liaoning1)
+27 / Hebei China (Shijiazhuang)
+ \ (hebei1)
+28 / Fujian China (Xiamen)
+ \ (fujian1)
+29 / Guangxi China (Nanning)
+ \ (guangxi1)
+30 / Anhui China (Huainan)
+ \ (anhui1)
+location_constraint> 1
+Option acl.
+Canned ACL used when creating buckets and storing or copying objects.
+This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.
+For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
+Note that this ACL is applied when server-side copying objects as S3
+doesn't copy the ACL from the source but rather writes a fresh one.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+ / Owner gets FULL_CONTROL.
+ 1 | No one else has access rights (default).
+ \ (private)
+ / Owner gets FULL_CONTROL.
+ 2 | The AllUsers group gets READ access.
+ \ (public-read)
+ / Owner gets FULL_CONTROL.
+ 3 | The AllUsers group gets READ and WRITE access.
+ | Granting this on a bucket is generally not recommended.
+ \ (public-read-write)
+ / Owner gets FULL_CONTROL.
+ 4 | The AuthenticatedUsers group gets READ access.
+ \ (authenticated-read)
+ / Object owner gets FULL_CONTROL.
+acl> private
+Option server_side_encryption.
+The server-side encryption algorithm used when storing this object in S3.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+ 1 / None
+ \ ()
+ 2 / AES256
+ \ (AES256)
+server_side_encryption>
+Option storage_class.
+The storage class to use when storing new objects in ChinaMobile.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+ 1 / Default
+ \ ()
+ 2 / Standard storage class
+ \ (STANDARD)
+ 3 / Archive storage mode
+ \ (GLACIER)
+ 4 / Infrequent access storage mode
+ \ (STANDARD_IA)
+storage_class>
+Edit advanced config?
+y) Yes
+n) No (default)
+y/n> n
+--------------------
+[ChinaMobile]
+type = s3
+provider = ChinaMobile
+access_key_id = accesskeyid
+secret_access_key = secretaccesskey
+endpoint = eos-wuxi-1.cmecloud.cn
+location_constraint = wuxi1
+acl = private
+--------------------
+y) Yes this is OK (default)
+e) Edit this remote
+d) Delete this remote
+y/e/d> y
+```
+### Leviia Cloud Object Storage {#leviia}
-#### --s3-upload-cutoff
+[Leviia Object Storage](https://www.leviia.com/object-storage/), backup and secure your data in a 100% French cloud, independent of GAFAM..
-Cutoff for switching to chunked upload.
+To configure access to Leviia, follow the steps below:
-Any files larger than this will be uploaded in chunks of chunk_size.
-The minimum is 0 and the maximum is 5 GiB.
+1. Run `rclone config` and select `n` for a new remote.
-Properties:
+```
+rclone config
+No remotes found, make a new one?
+n) New remote
+s) Set configuration password
+q) Quit config
+n/s/q> n
+```
-- Config: upload_cutoff
-- Env Var: RCLONE_S3_UPLOAD_CUTOFF
-- Type: SizeSuffix
-- Default: 200Mi
+2. Give the name of the configuration. For example, name it 'leviia'.
-#### --s3-chunk-size
+```
+name> leviia
+```
-Chunk size to use for uploading.
+3. Select `s3` storage.
-When uploading files larger than upload_cutoff or files with unknown
-size (e.g. from "rclone rcat" or uploaded with "rclone mount" or google
-photos or google docs) they will be uploaded as multipart uploads
-using this chunk size.
+```
+Choose a number from below, or type in your own value
+ 1 / 1Fichier
+ \ (fichier)
+ 2 / Akamai NetStorage
+ \ (netstorage)
+ 3 / Alias for an existing remote
+ \ (alias)
+ 4 / Amazon Drive
+ \ (amazon cloud drive)
+ 5 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, China Mobile, Cloudflare, ArvanCloud, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, IDrive e2, Liara, Lyve Cloud, Minio, Netease, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Synology, Tencent COS, Qiniu and Wasabi
+ \ (s3)
+[snip]
+Storage> s3
+```
-Note that "--s3-upload-concurrency" chunks of this size are buffered
-in memory per transfer.
+4. Select `Leviia` provider.
+```
+Choose a number from below, or type in your own value
+1 / Amazon Web Services (AWS) S3
+ \ "AWS"
+[snip]
+15 / Leviia Object Storage
+ \ (Leviia)
+[snip]
+provider> Leviia
+```
-If you are transferring large files over high-speed links and you have
-enough memory, then increasing this will speed up the transfers.
+5. Enter your SecretId and SecretKey of Leviia.
-Rclone will automatically increase the chunk size when uploading a
-large file of known size to stay below the 10,000 chunks limit.
+```
+Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
+Only applies if access_key_id and secret_access_key is blank.
+Enter a boolean value (true or false). Press Enter for the default ("false").
+Choose a number from below, or type in your own value
+ 1 / Enter AWS credentials in the next step
+ \ "false"
+ 2 / Get AWS credentials from the environment (env vars or IAM)
+ \ "true"
+env_auth> 1
+AWS Access Key ID.
+Leave blank for anonymous access or runtime credentials.
+Enter a string value. Press Enter for the default ("").
+access_key_id> ZnIx.xxxxxxxxxxxxxxx
+AWS Secret Access Key (password)
+Leave blank for anonymous access or runtime credentials.
+Enter a string value. Press Enter for the default ("").
+secret_access_key> xxxxxxxxxxx
+```
-Files of unknown size are uploaded with the configured
-chunk_size. Since the default chunk size is 5 MiB and there can be at
-most 10,000 chunks, this means that by default the maximum size of
-a file you can stream upload is 48 GiB. If you wish to stream upload
-larger files then you will need to increase chunk_size.
+6. Select endpoint for Leviia.
-Increasing the chunk size decreases the accuracy of the progress
-statistics displayed with "-P" flag. Rclone treats chunk as sent when
-it's buffered by the AWS SDK, when in fact it may still be uploading.
-A bigger chunk size means a bigger AWS SDK buffer and progress
-reporting more deviating from the truth.
+```
+ / The default endpoint
+ 1 | Leviia.
+ \ (s3.leviia.com)
+[snip]
+endpoint> 1
+```
+7. Choose acl.
+```
+Note that this ACL is applied when server-side copying objects as S3
+doesn't copy the ACL from the source but rather writes a fresh one.
+Enter a string value. Press Enter for the default ("").
+Choose a number from below, or type in your own value
+ / Owner gets FULL_CONTROL.
+ 1 | No one else has access rights (default).
+ \ (private)
+ / Owner gets FULL_CONTROL.
+ 2 | The AllUsers group gets READ access.
+ \ (public-read)
+[snip]
+acl> 1
+Edit advanced config? (y/n)
+y) Yes
+n) No (default)
+y/n> n
+Remote config
+--------------------
+[leviia]
+- type: s3
+- provider: Leviia
+- access_key_id: ZnIx.xxxxxxx
+- secret_access_key: xxxxxxxx
+- endpoint: s3.leviia.com
+- acl: private
+--------------------
+y) Yes this is OK (default)
+e) Edit this remote
+d) Delete this remote
+y/e/d> y
+Current remotes:
-Properties:
+Name Type
+==== ====
+leviia s3
+```
-- Config: chunk_size
-- Env Var: RCLONE_S3_CHUNK_SIZE
-- Type: SizeSuffix
-- Default: 5Mi
+### Liara {#liara-cloud}
-#### --s3-max-upload-parts
+Here is an example of making a [Liara Object Storage](https://liara.ir/landing/object-storage)
+configuration. First run:
-Maximum number of parts in a multipart upload.
+ rclone config
-This option defines the maximum number of multipart chunks to use
-when doing a multipart upload.
+This will guide you through an interactive setup process.
-This can be useful if a service does not support the AWS S3
-specification of 10,000 chunks.
+```
+No remotes found, make a new one?
+n) New remote
+s) Set configuration password
+n/s> n
+name> Liara
+Type of storage to configure.
+Choose a number from below, or type in your own value
+[snip]
+XX / Amazon S3 (also Dreamhost, Ceph, ChinaMobile, ArvanCloud, Liara, Minio)
+ \ "s3"
+[snip]
+Storage> s3
+Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars). Only applies if access_key_id and secret_access_key is blank.
+Choose a number from below, or type in your own value
+ 1 / Enter AWS credentials in the next step
+ \ "false"
+ 2 / Get AWS credentials from the environment (env vars or IAM)
+ \ "true"
+env_auth> 1
+AWS Access Key ID - leave blank for anonymous access or runtime credentials.
+access_key_id> YOURACCESSKEY
+AWS Secret Access Key (password) - leave blank for anonymous access or runtime credentials.
+secret_access_key> YOURSECRETACCESSKEY
+Region to connect to.
+Choose a number from below, or type in your own value
+ / The default endpoint
+ 1 | US Region, Northern Virginia, or Pacific Northwest.
+ | Leave location constraint empty.
+ \ "us-east-1"
+[snip]
+region>
+Endpoint for S3 API.
+Leave blank if using Liara to use the default endpoint for the region.
+Specify if using an S3 clone such as Ceph.
+endpoint> storage.iran.liara.space
+Canned ACL used when creating buckets and/or storing objects in S3.
+For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
+Choose a number from below, or type in your own value
+ 1 / Owner gets FULL_CONTROL. No one else has access rights (default).
+ \ "private"
+[snip]
+acl>
+The server-side encryption algorithm used when storing this object in S3.
+Choose a number from below, or type in your own value
+ 1 / None
+ \ ""
+ 2 / AES256
+ \ "AES256"
+server_side_encryption>
+The storage class to use when storing objects in S3.
+Choose a number from below, or type in your own value
+ 1 / Default
+ \ ""
+ 2 / Standard storage class
+ \ "STANDARD"
+storage_class>
+Remote config
+--------------------
+[Liara]
+env_auth = false
+access_key_id = YOURACCESSKEY
+secret_access_key = YOURSECRETACCESSKEY
+endpoint = storage.iran.liara.space
+location_constraint =
+acl =
+server_side_encryption =
+storage_class =
+--------------------
+y) Yes this is OK
+e) Edit this remote
+d) Delete this remote
+y/e/d> y
+```
-Rclone will automatically increase the chunk size when uploading a
-large file of a known size to stay below this number of chunks limit.
+This will leave the config file looking like this.
+```
+[Liara]
+type = s3
+provider = Liara
+env_auth = false
+access_key_id = YOURACCESSKEY
+secret_access_key = YOURSECRETACCESSKEY
+region =
+endpoint = storage.iran.liara.space
+location_constraint =
+acl =
+server_side_encryption =
+storage_class =
+```
-Properties:
+### Linode {#linode}
-- Config: max_upload_parts
-- Env Var: RCLONE_S3_MAX_UPLOAD_PARTS
-- Type: int
-- Default: 10000
+Here is an example of making a [Linode Object Storage](https://www.linode.com/products/object-storage/)
+configuration. First run:
-#### --s3-copy-cutoff
+ rclone config
-Cutoff for switching to multipart copy.
+This will guide you through an interactive setup process.
-Any files larger than this that need to be server-side copied will be
-copied in chunks of this size.
+```
+No remotes found, make a new one?
+n) New remote
+s) Set configuration password
+q) Quit config
+n/s/q> n
-The minimum is 0 and the maximum is 5 GiB.
+Enter name for new remote.
+name> linode
-Properties:
+Option Storage.
+Type of storage to configure.
+Choose a number from below, or type in your own value.
+[snip]
+ X / Amazon S3 Compliant Storage Providers including AWS, ...Linode, ...and others
+ \ (s3)
+[snip]
+Storage> s3
-- Config: copy_cutoff
-- Env Var: RCLONE_S3_COPY_CUTOFF
-- Type: SizeSuffix
-- Default: 4.656Gi
+Option provider.
+Choose your S3 provider.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+[snip]
+XX / Linode Object Storage
+ \ (Linode)
+[snip]
+provider> Linode
-#### --s3-disable-checksum
+Option env_auth.
+Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
+Only applies if access_key_id and secret_access_key is blank.
+Choose a number from below, or type in your own boolean value (true or false).
+Press Enter for the default (false).
+ 1 / Enter AWS credentials in the next step.
+ \ (false)
+ 2 / Get AWS credentials from the environment (env vars or IAM).
+ \ (true)
+env_auth>
-Don't store MD5 checksum with object metadata.
+Option access_key_id.
+AWS Access Key ID.
+Leave blank for anonymous access or runtime credentials.
+Enter a value. Press Enter to leave empty.
+access_key_id> ACCESS_KEY
-Normally rclone will calculate the MD5 checksum of the input before
-uploading it so it can add it to metadata on the object. This is great
-for data integrity checking but can cause long delays for large files
-to start uploading.
+Option secret_access_key.
+AWS Secret Access Key (password).
+Leave blank for anonymous access or runtime credentials.
+Enter a value. Press Enter to leave empty.
+secret_access_key> SECRET_ACCESS_KEY
-Properties:
+Option endpoint.
+Endpoint for Linode Object Storage API.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+ 1 / Atlanta, GA (USA), us-southeast-1
+ \ (us-southeast-1.linodeobjects.com)
+ 2 / Chicago, IL (USA), us-ord-1
+ \ (us-ord-1.linodeobjects.com)
+ 3 / Frankfurt (Germany), eu-central-1
+ \ (eu-central-1.linodeobjects.com)
+ 4 / Milan (Italy), it-mil-1
+ \ (it-mil-1.linodeobjects.com)
+ 5 / Newark, NJ (USA), us-east-1
+ \ (us-east-1.linodeobjects.com)
+ 6 / Paris (France), fr-par-1
+ \ (fr-par-1.linodeobjects.com)
+ 7 / Seattle, WA (USA), us-sea-1
+ \ (us-sea-1.linodeobjects.com)
+ 8 / Singapore ap-south-1
+ \ (ap-south-1.linodeobjects.com)
+ 9 / Stockholm (Sweden), se-sto-1
+ \ (se-sto-1.linodeobjects.com)
+10 / Washington, DC, (USA), us-iad-1
+ \ (us-iad-1.linodeobjects.com)
+endpoint> 3
-- Config: disable_checksum
-- Env Var: RCLONE_S3_DISABLE_CHECKSUM
-- Type: bool
-- Default: false
+Option acl.
+Canned ACL used when creating buckets and storing or copying objects.
+This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.
+For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
+Note that this ACL is applied when server-side copying objects as S3
+doesn't copy the ACL from the source but rather writes a fresh one.
+If the acl is an empty string then no X-Amz-Acl: header is added and
+the default (private) will be used.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+ / Owner gets FULL_CONTROL.
+ 1 | No one else has access rights (default).
+ \ (private)
+[snip]
+acl>
-#### --s3-shared-credentials-file
+Edit advanced config?
+y) Yes
+n) No (default)
+y/n> n
-Path to the shared credentials file.
+Configuration complete.
+Options:
+- type: s3
+- provider: Linode
+- access_key_id: ACCESS_KEY
+- secret_access_key: SECRET_ACCESS_KEY
+- endpoint: eu-central-1.linodeobjects.com
+Keep this "linode" remote?
+y) Yes this is OK (default)
+e) Edit this remote
+d) Delete this remote
+y/e/d> y
+```
-If env_auth = true then rclone can use a shared credentials file.
+This will leave the config file looking like this.
-If this variable is empty rclone will look for the
-"AWS_SHARED_CREDENTIALS_FILE" env variable. If the env value is empty
-it will default to the current user's home directory.
+```
+[linode]
+type = s3
+provider = Linode
+access_key_id = ACCESS_KEY
+secret_access_key = SECRET_ACCESS_KEY
+endpoint = eu-central-1.linodeobjects.com
+```
- Linux/OSX: "$HOME/.aws/credentials"
- Windows: "%USERPROFILE%\.aws\credentials"
+### ArvanCloud {#arvan-cloud}
+[ArvanCloud](https://www.arvancloud.com/en/products/cloud-storage) ArvanCloud Object Storage goes beyond the limited traditional file storage.
+It gives you access to backup and archived files and allows sharing.
+Files like profile image in the app, images sent by users or scanned documents can be stored securely and easily in our Object Storage service.
-Properties:
+ArvanCloud provides an S3 interface which can be configured for use with
+rclone like this.
-- Config: shared_credentials_file
-- Env Var: RCLONE_S3_SHARED_CREDENTIALS_FILE
-- Type: string
-- Required: false
+```
+No remotes found, make a new one?
+n) New remote
+s) Set configuration password
+n/s> n
+name> ArvanCloud
+Type of storage to configure.
+Choose a number from below, or type in your own value
+[snip]
+XX / Amazon S3 (also Dreamhost, Ceph, ChinaMobile, ArvanCloud, Liara, Minio)
+ \ "s3"
+[snip]
+Storage> s3
+Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars). Only applies if access_key_id and secret_access_key is blank.
+Choose a number from below, or type in your own value
+ 1 / Enter AWS credentials in the next step
+ \ "false"
+ 2 / Get AWS credentials from the environment (env vars or IAM)
+ \ "true"
+env_auth> 1
+AWS Access Key ID - leave blank for anonymous access or runtime credentials.
+access_key_id> YOURACCESSKEY
+AWS Secret Access Key (password) - leave blank for anonymous access or runtime credentials.
+secret_access_key> YOURSECRETACCESSKEY
+Region to connect to.
+Choose a number from below, or type in your own value
+ / The default endpoint - a good choice if you are unsure.
+ 1 | US Region, Northern Virginia, or Pacific Northwest.
+ | Leave location constraint empty.
+ \ "us-east-1"
+[snip]
+region>
+Endpoint for S3 API.
+Leave blank if using ArvanCloud to use the default endpoint for the region.
+Specify if using an S3 clone such as Ceph.
+endpoint> s3.arvanstorage.com
+Location constraint - must be set to match the Region. Used when creating buckets only.
+Choose a number from below, or type in your own value
+ 1 / Empty for Iran-Tehran Region.
+ \ ""
+[snip]
+location_constraint>
+Canned ACL used when creating buckets and/or storing objects in S3.
+For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
+Choose a number from below, or type in your own value
+ 1 / Owner gets FULL_CONTROL. No one else has access rights (default).
+ \ "private"
+[snip]
+acl>
+The server-side encryption algorithm used when storing this object in S3.
+Choose a number from below, or type in your own value
+ 1 / None
+ \ ""
+ 2 / AES256
+ \ "AES256"
+server_side_encryption>
+The storage class to use when storing objects in S3.
+Choose a number from below, or type in your own value
+ 1 / Default
+ \ ""
+ 2 / Standard storage class
+ \ "STANDARD"
+storage_class>
+Remote config
+--------------------
+[ArvanCloud]
+env_auth = false
+access_key_id = YOURACCESSKEY
+secret_access_key = YOURSECRETACCESSKEY
+region = ir-thr-at1
+endpoint = s3.arvanstorage.com
+location_constraint =
+acl =
+server_side_encryption =
+storage_class =
+--------------------
+y) Yes this is OK
+e) Edit this remote
+d) Delete this remote
+y/e/d> y
+```
-#### --s3-profile
+This will leave the config file looking like this.
-Profile to use in the shared credentials file.
+```
+[ArvanCloud]
+type = s3
+provider = ArvanCloud
+env_auth = false
+access_key_id = YOURACCESSKEY
+secret_access_key = YOURSECRETACCESSKEY
+region =
+endpoint = s3.arvanstorage.com
+location_constraint =
+acl =
+server_side_encryption =
+storage_class =
+```
-If env_auth = true then rclone can use a shared credentials file. This
-variable controls which profile is used in that file.
+### Tencent COS {#tencent-cos}
-If empty it will default to the environment variable "AWS_PROFILE" or
-"default" if that environment variable is also not set.
+[Tencent Cloud Object Storage (COS)](https://intl.cloud.tencent.com/product/cos) is a distributed storage service offered by Tencent Cloud for unstructured data. It is secure, stable, massive, convenient, low-delay and low-cost.
+To configure access to Tencent COS, follow the steps below:
-Properties:
+1. Run `rclone config` and select `n` for a new remote.
-- Config: profile
-- Env Var: RCLONE_S3_PROFILE
-- Type: string
-- Required: false
+```
+rclone config
+No remotes found, make a new one?
+n) New remote
+s) Set configuration password
+q) Quit config
+n/s/q> n
+```
-#### --s3-session-token
+2. Give the name of the configuration. For example, name it 'cos'.
-An AWS session token.
+```
+name> cos
+```
-Properties:
+3. Select `s3` storage.
-- Config: session_token
-- Env Var: RCLONE_S3_SESSION_TOKEN
-- Type: string
-- Required: false
+```
+Choose a number from below, or type in your own value
+1 / 1Fichier
+ \ "fichier"
+ 2 / Alias for an existing remote
+ \ "alias"
+ 3 / Amazon Drive
+ \ "amazon cloud drive"
+ 4 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, ChinaMobile, ArvanCloud, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, Liara, Minio, and Tencent COS
+ \ "s3"
+[snip]
+Storage> s3
+```
-#### --s3-upload-concurrency
+4. Select `TencentCOS` provider.
+```
+Choose a number from below, or type in your own value
+1 / Amazon Web Services (AWS) S3
+ \ "AWS"
+[snip]
+11 / Tencent Cloud Object Storage (COS)
+ \ "TencentCOS"
+[snip]
+provider> TencentCOS
+```
-Concurrency for multipart uploads.
+5. Enter your SecretId and SecretKey of Tencent Cloud.
-This is the number of chunks of the same file that are uploaded
-concurrently.
+```
+Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
+Only applies if access_key_id and secret_access_key is blank.
+Enter a boolean value (true or false). Press Enter for the default ("false").
+Choose a number from below, or type in your own value
+ 1 / Enter AWS credentials in the next step
+ \ "false"
+ 2 / Get AWS credentials from the environment (env vars or IAM)
+ \ "true"
+env_auth> 1
+AWS Access Key ID.
+Leave blank for anonymous access or runtime credentials.
+Enter a string value. Press Enter for the default ("").
+access_key_id> AKIDxxxxxxxxxx
+AWS Secret Access Key (password)
+Leave blank for anonymous access or runtime credentials.
+Enter a string value. Press Enter for the default ("").
+secret_access_key> xxxxxxxxxxx
+```
-If you are uploading small numbers of large files over high-speed links
-and these uploads do not fully utilize your bandwidth, then increasing
-this may help to speed up the transfers.
+6. Select endpoint for Tencent COS. This is the standard endpoint for different region.
-Properties:
+```
+ 1 / Beijing Region.
+ \ "cos.ap-beijing.myqcloud.com"
+ 2 / Nanjing Region.
+ \ "cos.ap-nanjing.myqcloud.com"
+ 3 / Shanghai Region.
+ \ "cos.ap-shanghai.myqcloud.com"
+ 4 / Guangzhou Region.
+ \ "cos.ap-guangzhou.myqcloud.com"
+[snip]
+endpoint> 4
+```
-- Config: upload_concurrency
-- Env Var: RCLONE_S3_UPLOAD_CONCURRENCY
-- Type: int
-- Default: 4
+7. Choose acl and storage class.
-#### --s3-force-path-style
+```
+Note that this ACL is applied when server-side copying objects as S3
+doesn't copy the ACL from the source but rather writes a fresh one.
+Enter a string value. Press Enter for the default ("").
+Choose a number from below, or type in your own value
+ 1 / Owner gets Full_CONTROL. No one else has access rights (default).
+ \ "default"
+[snip]
+acl> 1
+The storage class to use when storing new objects in Tencent COS.
+Enter a string value. Press Enter for the default ("").
+Choose a number from below, or type in your own value
+ 1 / Default
+ \ ""
+[snip]
+storage_class> 1
+Edit advanced config? (y/n)
+y) Yes
+n) No (default)
+y/n> n
+Remote config
+--------------------
+[cos]
+type = s3
+provider = TencentCOS
+env_auth = false
+access_key_id = xxx
+secret_access_key = xxx
+endpoint = cos.ap-guangzhou.myqcloud.com
+acl = default
+--------------------
+y) Yes this is OK (default)
+e) Edit this remote
+d) Delete this remote
+y/e/d> y
+Current remotes:
-If true use path style access if false use virtual hosted style.
+Name Type
+==== ====
+cos s3
+```
-If this is true (the default) then rclone will use path style access,
-if false then rclone will use virtual path style. See [the AWS S3
-docs](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html#access-bucket-intro)
-for more info.
+### Netease NOS
-Some providers (e.g. AWS, Aliyun OSS, Netease COS, or Tencent COS) require this set to
-false - rclone will do this automatically based on the provider
-setting.
+For Netease NOS configure as per the configurator `rclone config`
+setting the provider `Netease`. This will automatically set
+`force_path_style = false` which is necessary for it to run properly.
-Properties:
+### Petabox
-- Config: force_path_style
-- Env Var: RCLONE_S3_FORCE_PATH_STYLE
-- Type: bool
-- Default: true
+Here is an example of making a [Petabox](https://petabox.io/)
+configuration. First run:
-#### --s3-v2-auth
+```bash
+rclone config
+```
-If true use v2 authentication.
+This will guide you through an interactive setup process.
-If this is false (the default) then rclone will use v4 authentication.
-If it is set then rclone will use v2 authentication.
+```
+No remotes found, make a new one?
+n) New remote
+s) Set configuration password
+n/s> n
-Use this only if v4 signatures don't work, e.g. pre Jewel/v10 CEPH.
+Enter name for new remote.
+name> My Petabox Storage
-Properties:
+Option Storage.
+Type of storage to configure.
+Choose a number from below, or type in your own value.
+[snip]
+XX / Amazon S3 Compliant Storage Providers including AWS, ...
+ \ "s3"
+[snip]
+Storage> s3
-- Config: v2_auth
-- Env Var: RCLONE_S3_V2_AUTH
-- Type: bool
-- Default: false
+Option provider.
+Choose your S3 provider.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+[snip]
+XX / Petabox Object Storage
+ \ (Petabox)
+[snip]
+provider> Petabox
-#### --s3-use-accelerate-endpoint
+Option env_auth.
+Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
+Only applies if access_key_id and secret_access_key is blank.
+Choose a number from below, or type in your own boolean value (true or false).
+Press Enter for the default (false).
+ 1 / Enter AWS credentials in the next step.
+ \ (false)
+ 2 / Get AWS credentials from the environment (env vars or IAM).
+ \ (true)
+env_auth> 1
-If true use the AWS S3 accelerated endpoint.
+Option access_key_id.
+AWS Access Key ID.
+Leave blank for anonymous access or runtime credentials.
+Enter a value. Press Enter to leave empty.
+access_key_id> YOUR_ACCESS_KEY_ID
-See: [AWS S3 Transfer acceleration](https://docs.aws.amazon.com/AmazonS3/latest/dev/transfer-acceleration-examples.html)
+Option secret_access_key.
+AWS Secret Access Key (password).
+Leave blank for anonymous access or runtime credentials.
+Enter a value. Press Enter to leave empty.
+secret_access_key> YOUR_SECRET_ACCESS_KEY
-Properties:
+Option region.
+Region where your bucket will be created and your data stored.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+ 1 / US East (N. Virginia)
+ \ (us-east-1)
+ 2 / Europe (Frankfurt)
+ \ (eu-central-1)
+ 3 / Asia Pacific (Singapore)
+ \ (ap-southeast-1)
+ 4 / Middle East (Bahrain)
+ \ (me-south-1)
+ 5 / South America (São Paulo)
+ \ (sa-east-1)
+region> 1
-- Config: use_accelerate_endpoint
-- Env Var: RCLONE_S3_USE_ACCELERATE_ENDPOINT
-- Provider: AWS
-- Type: bool
-- Default: false
+Option endpoint.
+Endpoint for Petabox S3 Object Storage.
+Specify the endpoint from the same region.
+Choose a number from below, or type in your own value.
+ 1 / US East (N. Virginia)
+ \ (s3.petabox.io)
+ 2 / US East (N. Virginia)
+ \ (s3.us-east-1.petabox.io)
+ 3 / Europe (Frankfurt)
+ \ (s3.eu-central-1.petabox.io)
+ 4 / Asia Pacific (Singapore)
+ \ (s3.ap-southeast-1.petabox.io)
+ 5 / Middle East (Bahrain)
+ \ (s3.me-south-1.petabox.io)
+ 6 / South America (São Paulo)
+ \ (s3.sa-east-1.petabox.io)
+endpoint> 1
-#### --s3-leave-parts-on-error
+Option acl.
+Canned ACL used when creating buckets and storing or copying objects.
+This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.
+For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
+Note that this ACL is applied when server-side copying objects as S3
+doesn't copy the ACL from the source but rather writes a fresh one.
+If the acl is an empty string then no X-Amz-Acl: header is added and
+the default (private) will be used.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+ / Owner gets FULL_CONTROL.
+ 1 | No one else has access rights (default).
+ \ (private)
+ / Owner gets FULL_CONTROL.
+ 2 | The AllUsers group gets READ access.
+ \ (public-read)
+ / Owner gets FULL_CONTROL.
+ 3 | The AllUsers group gets READ and WRITE access.
+ | Granting this on a bucket is generally not recommended.
+ \ (public-read-write)
+ / Owner gets FULL_CONTROL.
+ 4 | The AuthenticatedUsers group gets READ access.
+ \ (authenticated-read)
+ / Object owner gets FULL_CONTROL.
+ 5 | Bucket owner gets READ access.
+ | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
+ \ (bucket-owner-read)
+ / Both the object owner and the bucket owner get FULL_CONTROL over the object.
+ 6 | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
+ \ (bucket-owner-full-control)
+acl> 1
-If true avoid calling abort upload on a failure, leaving all successfully uploaded parts on S3 for manual recovery.
+Edit advanced config?
+y) Yes
+n) No (default)
+y/n> No
-It should be set to true for resuming uploads across different sessions.
+Configuration complete.
+Options:
+- type: s3
+- provider: Petabox
+- access_key_id: YOUR_ACCESS_KEY_ID
+- secret_access_key: YOUR_SECRET_ACCESS_KEY
+- region: us-east-1
+- endpoint: s3.petabox.io
+Keep this "My Petabox Storage" remote?
+y) Yes this is OK (default)
+e) Edit this remote
+d) Delete this remote
+y/e/d> y
+```
-WARNING: Storing parts of an incomplete multipart upload counts towards space usage on S3 and will add additional costs if not cleaned up.
+This will leave the config file looking like this.
+```
+[My Petabox Storage]
+type = s3
+provider = Petabox
+access_key_id = YOUR_ACCESS_KEY_ID
+secret_access_key = YOUR_SECRET_ACCESS_KEY
+region = us-east-1
+endpoint = s3.petabox.io
+```
-Properties:
+### Storj
-- Config: leave_parts_on_error
-- Env Var: RCLONE_S3_LEAVE_PARTS_ON_ERROR
-- Provider: AWS
-- Type: bool
-- Default: false
+Storj is a decentralized cloud storage which can be used through its
+native protocol or an S3 compatible gateway.
-#### --s3-list-chunk
+The S3 compatible gateway is configured using `rclone config` with a
+type of `s3` and with a provider name of `Storj`. Here is an example
+run of the configurator.
-Size of listing chunk (response list for each ListObject S3 request).
+```
+Type of storage to configure.
+Storage> s3
+Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
+Only applies if access_key_id and secret_access_key is blank.
+Choose a number from below, or type in your own boolean value (true or false).
+Press Enter for the default (false).
+ 1 / Enter AWS credentials in the next step.
+ \ (false)
+ 2 / Get AWS credentials from the environment (env vars or IAM).
+ \ (true)
+env_auth> 1
+Option access_key_id.
+AWS Access Key ID.
+Leave blank for anonymous access or runtime credentials.
+Enter a value. Press Enter to leave empty.
+access_key_id> XXXX (as shown when creating the access grant)
+Option secret_access_key.
+AWS Secret Access Key (password).
+Leave blank for anonymous access or runtime credentials.
+Enter a value. Press Enter to leave empty.
+secret_access_key> XXXX (as shown when creating the access grant)
+Option endpoint.
+Endpoint of the Shared Gateway.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+ 1 / EU1 Shared Gateway
+ \ (gateway.eu1.storjshare.io)
+ 2 / US1 Shared Gateway
+ \ (gateway.us1.storjshare.io)
+ 3 / Asia-Pacific Shared Gateway
+ \ (gateway.ap1.storjshare.io)
+endpoint> 1 (as shown when creating the access grant)
+Edit advanced config?
+y) Yes
+n) No (default)
+y/n> n
+```
-This option is also known as "MaxKeys", "max-items", or "page-size" from the AWS S3 specification.
-Most services truncate the response list to 1000 objects even if requested more than that.
-In AWS S3 this is a global maximum and cannot be changed, see [AWS S3](https://docs.aws.amazon.com/cli/latest/reference/s3/ls.html).
-In Ceph, this can be increased with the "rgw list buckets max chunk" option.
+Note that s3 credentials are generated when you [create an access
+grant](https://docs.storj.io/dcs/api-reference/s3-compatible-gateway#usage).
+#### Backend quirks
-Properties:
+- `--chunk-size` is forced to be 64 MiB or greater. This will use more
+ memory than the default of 5 MiB.
+- Server side copy is disabled as it isn't currently supported in the
+ gateway.
+- GetTier and SetTier are not supported.
-- Config: list_chunk
-- Env Var: RCLONE_S3_LIST_CHUNK
-- Type: int
-- Default: 1000
+#### Backend bugs
-#### --s3-list-version
+Due to [issue #39](https://github.com/storj/gateway-mt/issues/39)
+uploading multipart files via the S3 gateway causes them to lose their
+metadata. For rclone's purpose this means that the modification time
+is not stored, nor is any MD5SUM (if one is available from the
+source).
-Version of ListObjects to use: 1,2 or 0 for auto.
+This has the following consequences:
-When S3 originally launched it only provided the ListObjects call to
-enumerate objects in a bucket.
+- Using `rclone rcat` will fail as the medatada doesn't match after upload
+- Uploading files with `rclone mount` will fail for the same reason
+ - This can worked around by using `--vfs-cache-mode writes` or `--vfs-cache-mode full` or setting `--s3-upload-cutoff` large
+- Files uploaded via a multipart upload won't have their modtimes
+ - This will mean that `rclone sync` will likely keep trying to upload files bigger than `--s3-upload-cutoff`
+ - This can be worked around with `--checksum` or `--size-only` or setting `--s3-upload-cutoff` large
+ - The maximum value for `--s3-upload-cutoff` is 5GiB though
-However in May 2016 the ListObjectsV2 call was introduced. This is
-much higher performance and should be used if at all possible.
+One general purpose workaround is to set `--s3-upload-cutoff 5G`. This
+means that rclone will upload files smaller than 5GiB as single parts.
+Note that this can be set in the config file with `upload_cutoff = 5G`
+or configured in the advanced settings. If you regularly transfer
+files larger than 5G then using `--checksum` or `--size-only` in
+`rclone sync` is the recommended workaround.
-If set to the default, 0, rclone will guess according to the provider
-set which list objects method to call. If it guesses wrong, then it
-may be set manually here.
+#### Comparison with the native protocol
+Use the [the native protocol](/storj) to take advantage of
+client-side encryption as well as to achieve the best possible
+download performance. Uploads will be erasure-coded locally, thus a
+1gb upload will result in 2.68gb of data being uploaded to storage
+nodes across the network.
-Properties:
+Use this backend and the S3 compatible Hosted Gateway to increase
+upload performance and reduce the load on your systems and network.
+Uploads will be encrypted and erasure-coded server-side, thus a 1GB
+upload will result in only in 1GB of data being uploaded to storage
+nodes across the network.
-- Config: list_version
-- Env Var: RCLONE_S3_LIST_VERSION
-- Type: int
-- Default: 0
+For more detailed comparison please check the documentation of the
+[storj](/storj) backend.
-#### --s3-list-url-encode
+## Limitations
-Whether to url encode listings: true/false/unset
+`rclone about` is not supported by the S3 backend. Backends without
+this capability cannot determine free space for an rclone mount or
+use policy `mfs` (most free space) as a member of an rclone union
+remote.
-Some providers support URL encoding listings and where this is
-available this is more reliable when using control characters in file
-names. If this is set to unset (the default) then rclone will choose
-according to the provider setting what to apply, but you can override
-rclone's choice here.
+See [List of backends that do not support rclone about](https://rclone.org/overview/#optional-features) and [rclone about](https://rclone.org/commands/rclone_about/)
-Properties:
-- Config: list_url_encode
-- Env Var: RCLONE_S3_LIST_URL_ENCODE
-- Type: Tristate
-- Default: unset
+### Synology C2 Object Storage {#synology-c2}
-#### --s3-no-check-bucket
+[Synology C2 Object Storage](https://c2.synology.com/en-global/object-storage/overview) provides a secure, S3-compatible, and cost-effective cloud storage solution without API request, download fees, and deletion penalty.
-If set, don't attempt to check the bucket exists or create it.
+The S3 compatible gateway is configured using `rclone config` with a
+type of `s3` and with a provider name of `Synology`. Here is an example
+run of the configurator.
-This can be useful when trying to minimise the number of transactions
-rclone does if you know the bucket exists already.
+First run:
-It can also be needed if the user you are using does not have bucket
-creation permissions. Before v1.52.0 this would have passed silently
-due to a bug.
+```
+rclone config
+```
+This will guide you through an interactive setup process.
-Properties:
+```
+No remotes found, make a new one?
+n) New remote
+s) Set configuration password
+q) Quit config
-- Config: no_check_bucket
-- Env Var: RCLONE_S3_NO_CHECK_BUCKET
-- Type: bool
-- Default: false
+n/s/q> n
-#### --s3-no-head
+Enter name for new remote.1
+name> syno
-If set, don't HEAD uploaded objects to check integrity.
+Type of storage to configure.
+Enter a string value. Press Enter for the default ("").
+Choose a number from below, or type in your own value
-This can be useful when trying to minimise the number of transactions
-rclone does.
+ 5 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, China Mobile, Cloudflare, GCS, ArvanCloud, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, IDrive e2, IONOS Cloud, Liara, Lyve Cloud, Minio, Netease, Petabox, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Synology, Tencent COS, Qiniu and Wasabi
+ \ "s3"
-Setting it means that if rclone receives a 200 OK message after
-uploading an object with PUT then it will assume that it got uploaded
-properly.
+Storage> s3
-In particular it will assume:
+Choose your S3 provider.
+Enter a string value. Press Enter for the default ("").
+Choose a number from below, or type in your own value
+ 24 / Synology C2 Object Storage
+ \ (Synology)
-- the metadata, including modtime, storage class and content type was as uploaded
-- the size was as uploaded
+provider> Synology
-It reads the following items from the response for a single part PUT:
+Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
+Only applies if access_key_id and secret_access_key is blank.
+Enter a boolean value (true or false). Press Enter for the default ("false").
+Choose a number from below, or type in your own value
+ 1 / Enter AWS credentials in the next step
+ \ "false"
+ 2 / Get AWS credentials from the environment (env vars or IAM)
+ \ "true"
-- the MD5SUM
-- The uploaded date
+env_auth> 1
-For multipart uploads these items aren't read.
+AWS Access Key ID.
+Leave blank for anonymous access or runtime credentials.
+Enter a string value. Press Enter for the default ("").
-If an source object of unknown length is uploaded then rclone **will** do a
-HEAD request.
+access_key_id> accesskeyid
-Setting this flag increases the chance for undetected upload failures,
-in particular an incorrect size, so it isn't recommended for normal
-operation. In practice the chance of an undetected upload failure is
-very small even with this flag.
+AWS Secret Access Key (password)
+Leave blank for anonymous access or runtime credentials.
+Enter a string value. Press Enter for the default ("").
+secret_access_key> secretaccesskey
-Properties:
+Region where your data stored.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+ 1 / Europe Region 1
+ \ (eu-001)
+ 2 / Europe Region 2
+ \ (eu-002)
+ 3 / US Region 1
+ \ (us-001)
+ 4 / US Region 2
+ \ (us-002)
+ 5 / Asia (Taiwan)
+ \ (tw-001)
-- Config: no_head
-- Env Var: RCLONE_S3_NO_HEAD
-- Type: bool
-- Default: false
+region > 1
-#### --s3-no-head-object
+Option endpoint.
+Endpoint for Synology C2 Object Storage API.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+ 1 / EU Endpoint 1
+ \ (eu-001.s3.synologyc2.net)
+ 2 / US Endpoint 1
+ \ (us-001.s3.synologyc2.net)
+ 3 / TW Endpoint 1
+ \ (tw-001.s3.synologyc2.net)
-If set, do not do HEAD before GET when getting objects.
+endpoint> 1
-Properties:
+Option location_constraint.
+Location constraint - must be set to match the Region.
+Leave blank if not sure. Used when creating buckets only.
+Enter a value. Press Enter to leave empty.
+location_constraint>
-- Config: no_head_object
-- Env Var: RCLONE_S3_NO_HEAD_OBJECT
-- Type: bool
-- Default: false
+Edit advanced config? (y/n)
+y) Yes
+n) No
+y/n> y
-#### --s3-encoding
+Option no_check_bucket.
+If set, don't attempt to check the bucket exists or create it.
+This can be useful when trying to minimise the number of transactions
+rclone does if you know the bucket exists already.
+It can also be needed if the user you are using does not have bucket
+creation permissions. Before v1.52.0 this would have passed silently
+due to a bug.
+Enter a boolean value (true or false). Press Enter for the default (true).
-The encoding for the backend.
+no_check_bucket> true
-See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info.
+Configuration complete.
+Options:
+- type: s3
+- provider: Synology
+- region: eu-001
+- endpoint: eu-001.s3.synologyc2.net
+- no_check_bucket: true
+Keep this "syno" remote?
+y) Yes this is OK (default)
+e) Edit this remote
+d) Delete this remote
-Properties:
+y/e/d> y
-- Config: encoding
-- Env Var: RCLONE_S3_ENCODING
-- Type: MultiEncoder
-- Default: Slash,InvalidUtf8,Dot
+# Backblaze B2
-#### --s3-memory-pool-flush-time
+B2 is [Backblaze's cloud storage system](https://www.backblaze.com/b2/).
-How often internal memory buffer pools will be flushed. (no longer used)
+Paths are specified as `remote:bucket` (or `remote:` for the `lsd`
+command.) You may put subdirectories in too, e.g. `remote:bucket/path/to/dir`.
-Properties:
+## Configuration
-- Config: memory_pool_flush_time
-- Env Var: RCLONE_S3_MEMORY_POOL_FLUSH_TIME
-- Type: Duration
-- Default: 1m0s
+Here is an example of making a b2 configuration. First run
-#### --s3-memory-pool-use-mmap
+ rclone config
-Whether to use mmap buffers in internal memory pool. (no longer used)
+This will guide you through an interactive setup process. To authenticate
+you will either need your Account ID (a short hex number) and Master
+Application Key (a long hex number) OR an Application Key, which is the
+recommended method. See below for further details on generating and using
+an Application Key.
-Properties:
+```
+No remotes found, make a new one?
+n) New remote
+q) Quit config
+n/q> n
+name> remote
+Type of storage to configure.
+Choose a number from below, or type in your own value
+[snip]
+XX / Backblaze B2
+ \ "b2"
+[snip]
+Storage> b2
+Account ID or Application Key ID
+account> 123456789abc
+Application Key
+key> 0123456789abcdef0123456789abcdef0123456789
+Endpoint for the service - leave blank normally.
+endpoint>
+Remote config
+--------------------
+[remote]
+account = 123456789abc
+key = 0123456789abcdef0123456789abcdef0123456789
+endpoint =
+--------------------
+y) Yes this is OK
+e) Edit this remote
+d) Delete this remote
+y/e/d> y
+```
-- Config: memory_pool_use_mmap
-- Env Var: RCLONE_S3_MEMORY_POOL_USE_MMAP
-- Type: bool
-- Default: false
+This remote is called `remote` and can now be used like this
-#### --s3-disable-http2
+See all buckets
-Disable usage of http2 for S3 backends.
+ rclone lsd remote:
-There is currently an unsolved issue with the s3 (specifically minio) backend
-and HTTP/2. HTTP/2 is enabled by default for the s3 backend but can be
-disabled here. When the issue is solved this flag will be removed.
+Create a new bucket
-See: https://github.com/rclone/rclone/issues/4673, https://github.com/rclone/rclone/issues/3631
+ rclone mkdir remote:bucket
+List the contents of a bucket
+ rclone ls remote:bucket
-Properties:
+Sync `/home/local/directory` to the remote bucket, deleting any
+excess files in the bucket.
-- Config: disable_http2
-- Env Var: RCLONE_S3_DISABLE_HTTP2
-- Type: bool
-- Default: false
+ rclone sync --interactive /home/local/directory remote:bucket
-#### --s3-download-url
+### Application Keys
-Custom endpoint for downloads.
-This is usually set to a CloudFront CDN URL as AWS S3 offers
-cheaper egress for data downloaded through the CloudFront network.
+B2 supports multiple [Application Keys for different access permission
+to B2 Buckets](https://www.backblaze.com/b2/docs/application_keys.html).
-Properties:
+You can use these with rclone too; you will need to use rclone version 1.43
+or later.
-- Config: download_url
-- Env Var: RCLONE_S3_DOWNLOAD_URL
-- Type: string
-- Required: false
+Follow Backblaze's docs to create an Application Key with the required
+permission and add the `applicationKeyId` as the `account` and the
+`Application Key` itself as the `key`.
-#### --s3-directory-markers
+Note that you must put the _applicationKeyId_ as the `account` – you
+can't use the master Account ID. If you try then B2 will return 401
+errors.
-Upload an empty object with a trailing slash when a new directory is created
+### --fast-list
-Empty folders are unsupported for bucket based remotes, this option creates an empty
-object ending with "/", to persist the folder.
+This remote supports `--fast-list` which allows you to use fewer
+transactions in exchange for more memory. See the [rclone
+docs](https://rclone.org/docs/#fast-list) for more details.
+### Modification times
-Properties:
+The modification time is stored as metadata on the object as
+`X-Bz-Info-src_last_modified_millis` as milliseconds since 1970-01-01
+in the Backblaze standard. Other tools should be able to use this as
+a modified time.
-- Config: directory_markers
-- Env Var: RCLONE_S3_DIRECTORY_MARKERS
-- Type: bool
-- Default: false
+Modified times are used in syncing and are fully supported. Note that
+if a modification time needs to be updated on an object then it will
+create a new version of the object.
-#### --s3-use-multipart-etag
+### Restricted filename characters
-Whether to use ETag in multipart uploads for verification
+In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters)
+the following characters are also replaced:
-This should be true, false or left unset to use the default for the provider.
+| Character | Value | Replacement |
+| --------- |:-----:|:-----------:|
+| \ | 0x5C | ï¼¼ |
+Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8),
+as they can't be used in JSON strings.
-Properties:
+Note that in 2020-05 Backblaze started allowing \ characters in file
+names. Rclone hasn't changed its encoding as this could cause syncs to
+re-transfer files. If you want rclone not to replace \ then see the
+`--b2-encoding` flag below and remove the `BackSlash` from the
+string. This can be set in the config.
-- Config: use_multipart_etag
-- Env Var: RCLONE_S3_USE_MULTIPART_ETAG
-- Type: Tristate
-- Default: unset
+### SHA1 checksums
-#### --s3-use-presigned-request
+The SHA1 checksums of the files are checked on upload and download and
+will be used in the syncing process.
-Whether to use a presigned request or PutObject for single part uploads
+Large files (bigger than the limit in `--b2-upload-cutoff`) which are
+uploaded in chunks will store their SHA1 on the object as
+`X-Bz-Info-large_file_sha1` as recommended by Backblaze.
-If this is false rclone will use PutObject from the AWS SDK to upload
-an object.
+For a large file to be uploaded with an SHA1 checksum, the source
+needs to support SHA1 checksums. The local disk supports SHA1
+checksums so large file transfers from local disk will have an SHA1.
+See [the overview](https://rclone.org/overview/#features) for exactly which remotes
+support SHA1.
-Versions of rclone < 1.59 use presigned requests to upload a single
-part object and setting this flag to true will re-enable that
-functionality. This shouldn't be necessary except in exceptional
-circumstances or for testing.
+Sources which don't support SHA1, in particular `crypt` will upload
+large files without SHA1 checksums. This may be fixed in the future
+(see [#1767](https://github.com/rclone/rclone/issues/1767)).
+Files sizes below `--b2-upload-cutoff` will always have an SHA1
+regardless of the source.
-Properties:
+### Transfers
-- Config: use_presigned_request
-- Env Var: RCLONE_S3_USE_PRESIGNED_REQUEST
-- Type: bool
-- Default: false
+Backblaze recommends that you do lots of transfers simultaneously for
+maximum speed. In tests from my SSD equipped laptop the optimum
+setting is about `--transfers 32` though higher numbers may be used
+for a slight speed improvement. The optimum number for you may vary
+depending on your hardware, how big the files are, how much you want
+to load your computer, etc. The default of `--transfers 4` is
+definitely too low for Backblaze B2 though.
-#### --s3-versions
+Note that uploading big files (bigger than 200 MiB by default) will use
+a 96 MiB RAM buffer by default. There can be at most `--transfers` of
+these in use at any moment, so this sets the upper limit on the memory
+used.
-Include old versions in directory listings.
+### Versions
-Properties:
+When rclone uploads a new version of a file it creates a [new version
+of it](https://www.backblaze.com/b2/docs/file_versions.html).
+Likewise when you delete a file, the old version will be marked hidden
+and still be available. Conversely, you may opt in to a "hard delete"
+of files with the `--b2-hard-delete` flag which would permanently remove
+the file instead of hiding it.
-- Config: versions
-- Env Var: RCLONE_S3_VERSIONS
-- Type: bool
-- Default: false
+Old versions of files, where available, are visible using the
+`--b2-versions` flag.
-#### --s3-version-at
+It is also possible to view a bucket as it was at a certain point in time,
+using the `--b2-version-at` flag. This will show the file versions as they
+were at that time, showing files that have been deleted afterwards, and
+hiding files that were created since.
-Show file versions as they were at the specified time.
+If you wish to remove all the old versions then you can use the
+`rclone cleanup remote:bucket` command which will delete all the old
+versions of files, leaving the current ones intact. You can also
+supply a path and only old versions under that path will be deleted,
+e.g. `rclone cleanup remote:bucket/path/to/stuff`.
-The parameter should be a date, "2006-01-02", datetime "2006-01-02
-15:04:05" or a duration for that long ago, eg "100d" or "1h".
+Note that `cleanup` will remove partially uploaded files from the bucket
+if they are more than a day old.
-Note that when using this no file write operations are permitted,
-so you can't upload files or delete them.
+When you `purge` a bucket, the current and the old versions will be
+deleted then the bucket will be deleted.
-See [the time option docs](https://rclone.org/docs/#time-option) for valid formats.
+However `delete` will cause the current versions of the files to
+become hidden old versions.
+Here is a session showing the listing and retrieval of an old
+version followed by a `cleanup` of the old versions.
-Properties:
+Show current version and all the versions with `--b2-versions` flag.
-- Config: version_at
-- Env Var: RCLONE_S3_VERSION_AT
-- Type: Time
-- Default: off
+```
+$ rclone -q ls b2:cleanup-test
+ 9 one.txt
-#### --s3-decompress
+$ rclone -q --b2-versions ls b2:cleanup-test
+ 9 one.txt
+ 8 one-v2016-07-04-141032-000.txt
+ 16 one-v2016-07-04-141003-000.txt
+ 15 one-v2016-07-02-155621-000.txt
+```
-If set this will decompress gzip encoded objects.
+Retrieve an old version
-It is possible to upload objects to S3 with "Content-Encoding: gzip"
-set. Normally rclone will download these files as compressed objects.
+```
+$ rclone -q --b2-versions copy b2:cleanup-test/one-v2016-07-04-141003-000.txt /tmp
-If this flag is set then rclone will decompress these files with
-"Content-Encoding: gzip" as they are received. This means that rclone
-can't check the size and hash but the file contents will be decompressed.
+$ ls -l /tmp/one-v2016-07-04-141003-000.txt
+-rw-rw-r-- 1 ncw ncw 16 Jul 2 17:46 /tmp/one-v2016-07-04-141003-000.txt
+```
+Clean up all the old versions and show that they've gone.
-Properties:
+```
+$ rclone -q cleanup b2:cleanup-test
-- Config: decompress
-- Env Var: RCLONE_S3_DECOMPRESS
-- Type: bool
-- Default: false
+$ rclone -q ls b2:cleanup-test
+ 9 one.txt
-#### --s3-might-gzip
+$ rclone -q --b2-versions ls b2:cleanup-test
+ 9 one.txt
+```
-Set this if the backend might gzip objects.
+#### Versions naming caveat
-Normally providers will not alter objects when they are downloaded. If
-an object was not uploaded with `Content-Encoding: gzip` then it won't
-be set on download.
+When using `--b2-versions` flag rclone is relying on the file name
+to work out whether the objects are versions or not. Versions' names
+are created by inserting timestamp between file name and its extension.
+```
+ 9 file.txt
+ 8 file-v2023-07-17-161032-000.txt
+ 16 file-v2023-06-15-141003-000.txt
+```
+If there are real files present with the same names as versions, then
+behaviour of `--b2-versions` can be unpredictable.
-However some providers may gzip objects even if they weren't uploaded
-with `Content-Encoding: gzip` (eg Cloudflare).
+### Data usage
-A symptom of this would be receiving errors like
+It is useful to know how many requests are sent to the server in different scenarios.
- ERROR corrupted on transfer: sizes differ NNN vs MMM
+All copy commands send the following 4 requests:
-If you set this flag and rclone downloads an object with
-Content-Encoding: gzip set and chunked transfer encoding, then rclone
-will decompress the object on the fly.
+```
+/b2api/v1/b2_authorize_account
+/b2api/v1/b2_create_bucket
+/b2api/v1/b2_list_buckets
+/b2api/v1/b2_list_file_names
+```
-If this is set to unset (the default) then rclone will choose
-according to the provider setting what to apply, but you can override
-rclone's choice here.
+The `b2_list_file_names` request will be sent once for every 1k files
+in the remote path, providing the checksum and modification time of
+the listed files. As of version 1.33 issue
+[#818](https://github.com/rclone/rclone/issues/818) causes extra requests
+to be sent when using B2 with Crypt. When a copy operation does not
+require any files to be uploaded, no more requests will be sent.
+Uploading files that do not require chunking, will send 2 requests per
+file upload:
-Properties:
+```
+/b2api/v1/b2_get_upload_url
+/b2api/v1/b2_upload_file/
+```
-- Config: might_gzip
-- Env Var: RCLONE_S3_MIGHT_GZIP
-- Type: Tristate
-- Default: unset
+Uploading files requiring chunking, will send 2 requests (one each to
+start and finish the upload) and another 2 requests for each chunk:
-#### --s3-use-accept-encoding-gzip
+```
+/b2api/v1/b2_start_large_file
+/b2api/v1/b2_get_upload_part_url
+/b2api/v1/b2_upload_part/
+/b2api/v1/b2_finish_large_file
+```
-Whether to send `Accept-Encoding: gzip` header.
+#### Versions
-By default, rclone will append `Accept-Encoding: gzip` to the request to download
-compressed objects whenever possible.
+Versions can be viewed with the `--b2-versions` flag. When it is set
+rclone will show and act on older versions of files. For example
-However some providers such as Google Cloud Storage may alter the HTTP headers, breaking
-the signature of the request.
+Listing without `--b2-versions`
-A symptom of this would be receiving errors like
+```
+$ rclone -q ls b2:cleanup-test
+ 9 one.txt
+```
- SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided.
+And with
-In this case, you might want to try disabling this option.
+```
+$ rclone -q --b2-versions ls b2:cleanup-test
+ 9 one.txt
+ 8 one-v2016-07-04-141032-000.txt
+ 16 one-v2016-07-04-141003-000.txt
+ 15 one-v2016-07-02-155621-000.txt
+```
+Showing that the current version is unchanged but older versions can
+be seen. These have the UTC date that they were uploaded to the
+server to the nearest millisecond appended to them.
-Properties:
+Note that when using `--b2-versions` no file write operations are
+permitted, so you can't upload files or delete them.
-- Config: use_accept_encoding_gzip
-- Env Var: RCLONE_S3_USE_ACCEPT_ENCODING_GZIP
-- Type: Tristate
-- Default: unset
+### B2 and rclone link
-#### --s3-no-system-metadata
+Rclone supports generating file share links for private B2 buckets.
+They can either be for a file for example:
-Suppress setting and reading of system metadata
+```
+./rclone link B2:bucket/path/to/file.txt
+https://f002.backblazeb2.com/file/bucket/path/to/file.txt?Authorization=xxxxxxxx
-Properties:
+```
-- Config: no_system_metadata
-- Env Var: RCLONE_S3_NO_SYSTEM_METADATA
-- Type: bool
-- Default: false
+or if run on a directory you will get:
-#### --s3-sts-endpoint
+```
+./rclone link B2:bucket/path
+https://f002.backblazeb2.com/file/bucket/path?Authorization=xxxxxxxx
+```
-Endpoint for STS.
+you can then use the authorization token (the part of the url from the
+ `?Authorization=` on) on any file path under that directory. For example:
-Leave blank if using AWS to use the default endpoint for the region.
+```
+https://f002.backblazeb2.com/file/bucket/path/to/file1?Authorization=xxxxxxxx
+https://f002.backblazeb2.com/file/bucket/path/file2?Authorization=xxxxxxxx
+https://f002.backblazeb2.com/file/bucket/path/folder/file3?Authorization=xxxxxxxx
-Properties:
+```
-- Config: sts_endpoint
-- Env Var: RCLONE_S3_STS_ENDPOINT
-- Provider: AWS
-- Type: string
-- Required: false
-### Metadata
+### Standard options
-User metadata is stored as x-amz-meta- keys. S3 metadata keys are case insensitive and are always returned in lower case.
+Here are the Standard options specific to b2 (Backblaze B2).
-Here are the possible system metadata items for the s3 backend.
+#### --b2-account
-| Name | Help | Type | Example | Read Only |
-|------|------|------|---------|-----------|
-| btime | Time of file birth (creation) read from Last-Modified header | RFC 3339 | 2006-01-02T15:04:05.999999999Z07:00 | **Y** |
-| cache-control | Cache-Control header | string | no-cache | N |
-| content-disposition | Content-Disposition header | string | inline | N |
-| content-encoding | Content-Encoding header | string | gzip | N |
-| content-language | Content-Language header | string | en-US | N |
-| content-type | Content-Type header | string | text/plain | N |
-| mtime | Time of last modification, read from rclone metadata | RFC 3339 | 2006-01-02T15:04:05.999999999Z07:00 | N |
-| tier | Tier of the object | string | GLACIER | **Y** |
+Account ID or Application Key ID.
-See the [metadata](https://rclone.org/docs/#metadata) docs for more info.
+Properties:
-## Backend commands
+- Config: account
+- Env Var: RCLONE_B2_ACCOUNT
+- Type: string
+- Required: true
-Here are the commands specific to the s3 backend.
+#### --b2-key
-Run them with
+Application Key.
- rclone backend COMMAND remote:
+Properties:
-The help below will explain what arguments each command takes.
+- Config: key
+- Env Var: RCLONE_B2_KEY
+- Type: string
+- Required: true
-See the [backend](https://rclone.org/commands/rclone_backend/) command for more
-info on how to pass options and arguments.
+#### --b2-hard-delete
-These can be run on a running backend using the rc command
-[backend/command](https://rclone.org/rc/#backend-command).
+Permanently delete files on remote removal, otherwise hide files.
-### restore
+Properties:
-Restore objects from GLACIER to normal storage
+- Config: hard_delete
+- Env Var: RCLONE_B2_HARD_DELETE
+- Type: bool
+- Default: false
- rclone backend restore remote: [options] [+]
+### Advanced options
-This command can be used to restore one or more objects from GLACIER
-to normal storage.
+Here are the Advanced options specific to b2 (Backblaze B2).
-Usage Examples:
+#### --b2-endpoint
- rclone backend restore s3:bucket/path/to/object -o priority=PRIORITY -o lifetime=DAYS
- rclone backend restore s3:bucket/path/to/directory -o priority=PRIORITY -o lifetime=DAYS
- rclone backend restore s3:bucket -o priority=PRIORITY -o lifetime=DAYS
+Endpoint for the service.
-This flag also obeys the filters. Test first with --interactive/-i or --dry-run flags
+Leave blank normally.
- rclone --interactive backend restore --include "*.txt" s3:bucket/path -o priority=Standard -o lifetime=1
+Properties:
-All the objects shown will be marked for restore, then
+- Config: endpoint
+- Env Var: RCLONE_B2_ENDPOINT
+- Type: string
+- Required: false
- rclone backend restore --include "*.txt" s3:bucket/path -o priority=Standard -o lifetime=1
+#### --b2-test-mode
-It returns a list of status dictionaries with Remote and Status
-keys. The Status will be OK if it was successful or an error message
-if not.
+A flag string for X-Bz-Test-Mode header for debugging.
- [
- {
- "Status": "OK",
- "Remote": "test.txt"
- },
- {
- "Status": "OK",
- "Remote": "test/file4.txt"
- }
- ]
+This is for debugging purposes only. Setting it to one of the strings
+below will cause b2 to return specific errors:
+ * "fail_some_uploads"
+ * "expire_some_account_authorization_tokens"
+ * "force_cap_exceeded"
+These will be set in the "X-Bz-Test-Mode" header which is documented
+in the [b2 integrations checklist](https://www.backblaze.com/b2/docs/integration_checklist.html).
-Options:
+Properties:
-- "description": The optional description for the job.
-- "lifetime": Lifetime of the active copy in days
-- "priority": Priority of restore: Standard|Expedited|Bulk
+- Config: test_mode
+- Env Var: RCLONE_B2_TEST_MODE
+- Type: string
+- Required: false
-### restore-status
+#### --b2-versions
-Show the restore status for objects being restored from GLACIER to normal storage
+Include old versions in directory listings.
- rclone backend restore-status remote: [options] [+]
+Note that when using this no file write operations are permitted,
+so you can't upload files or delete them.
-This command can be used to show the status for objects being restored from GLACIER
-to normal storage.
+Properties:
-Usage Examples:
+- Config: versions
+- Env Var: RCLONE_B2_VERSIONS
+- Type: bool
+- Default: false
- rclone backend restore-status s3:bucket/path/to/object
- rclone backend restore-status s3:bucket/path/to/directory
- rclone backend restore-status -o all s3:bucket/path/to/directory
+#### --b2-version-at
-This command does not obey the filters.
+Show file versions as they were at the specified time.
-It returns a list of status dictionaries.
+Note that when using this no file write operations are permitted,
+so you can't upload files or delete them.
- [
- {
- "Remote": "file.txt",
- "VersionID": null,
- "RestoreStatus": {
- "IsRestoreInProgress": true,
- "RestoreExpiryDate": "2023-09-06T12:29:19+01:00"
- },
- "StorageClass": "GLACIER"
- },
- {
- "Remote": "test.pdf",
- "VersionID": null,
- "RestoreStatus": {
- "IsRestoreInProgress": false,
- "RestoreExpiryDate": "2023-09-06T12:29:19+01:00"
- },
- "StorageClass": "DEEP_ARCHIVE"
- }
- ]
+Properties:
+- Config: version_at
+- Env Var: RCLONE_B2_VERSION_AT
+- Type: Time
+- Default: off
-Options:
+#### --b2-upload-cutoff
-- "all": if set then show all objects, not just ones with restore status
+Cutoff for switching to chunked upload.
-### list-multipart-uploads
+Files above this size will be uploaded in chunks of "--b2-chunk-size".
-List the unfinished multipart uploads
+This value should be set no larger than 4.657 GiB (== 5 GB).
- rclone backend list-multipart-uploads remote: [options] [+]
+Properties:
-This command lists the unfinished multipart uploads in JSON format.
+- Config: upload_cutoff
+- Env Var: RCLONE_B2_UPLOAD_CUTOFF
+- Type: SizeSuffix
+- Default: 200Mi
- rclone backend list-multipart s3:bucket/path/to/object
+#### --b2-copy-cutoff
-It returns a dictionary of buckets with values as lists of unfinished
-multipart uploads.
+Cutoff for switching to multipart copy.
-You can call it with no bucket in which case it lists all bucket, with
-a bucket or with a bucket and path.
+Any files larger than this that need to be server-side copied will be
+copied in chunks of this size.
- {
- "rclone": [
- {
- "Initiated": "2020-06-26T14:20:36Z",
- "Initiator": {
- "DisplayName": "XXX",
- "ID": "arn:aws:iam::XXX:user/XXX"
- },
- "Key": "KEY",
- "Owner": {
- "DisplayName": null,
- "ID": "XXX"
- },
- "StorageClass": "STANDARD",
- "UploadId": "XXX"
- }
- ],
- "rclone-1000files": [],
- "rclone-dst": []
- }
+The minimum is 0 and the maximum is 4.6 GiB.
+Properties:
+- Config: copy_cutoff
+- Env Var: RCLONE_B2_COPY_CUTOFF
+- Type: SizeSuffix
+- Default: 4Gi
-### cleanup
+#### --b2-chunk-size
-Remove unfinished multipart uploads.
+Upload chunk size.
- rclone backend cleanup remote: [options] [+]
+When uploading large files, chunk the file into this size.
-This command removes unfinished multipart uploads of age greater than
-max-age which defaults to 24 hours.
+Must fit in memory. These chunks are buffered in memory and there
+might a maximum of "--transfers" chunks in progress at once.
-Note that you can use --interactive/-i or --dry-run with this command to see what
-it would do.
+5,000,000 Bytes is the minimum size.
- rclone backend cleanup s3:bucket/path/to/object
- rclone backend cleanup -o max-age=7w s3:bucket/path/to/object
+Properties:
-Durations are parsed as per the rest of rclone, 2h, 7d, 7w etc.
+- Config: chunk_size
+- Env Var: RCLONE_B2_CHUNK_SIZE
+- Type: SizeSuffix
+- Default: 96Mi
+#### --b2-upload-concurrency
-Options:
+Concurrency for multipart uploads.
-- "max-age": Max age of upload to delete
+This is the number of chunks of the same file that are uploaded
+concurrently.
-### cleanup-hidden
+Note that chunks are stored in memory and there may be up to
+"--transfers" * "--b2-upload-concurrency" chunks stored at once
+in memory.
-Remove old versions of files.
+Properties:
- rclone backend cleanup-hidden remote: [options] [+]
+- Config: upload_concurrency
+- Env Var: RCLONE_B2_UPLOAD_CONCURRENCY
+- Type: int
+- Default: 4
-This command removes any old hidden versions of files
-on a versions enabled bucket.
+#### --b2-disable-checksum
-Note that you can use --interactive/-i or --dry-run with this command to see what
-it would do.
+Disable checksums for large (> upload cutoff) files.
- rclone backend cleanup-hidden s3:bucket/path/to/dir
+Normally rclone will calculate the SHA1 checksum of the input before
+uploading it so it can add it to metadata on the object. This is great
+for data integrity checking but can cause long delays for large files
+to start uploading.
+Properties:
-### versioning
+- Config: disable_checksum
+- Env Var: RCLONE_B2_DISABLE_CHECKSUM
+- Type: bool
+- Default: false
-Set/get versioning support for a bucket.
+#### --b2-download-url
- rclone backend versioning remote: [options] [+]
+Custom endpoint for downloads.
-This command sets versioning support if a parameter is
-passed and then returns the current versioning status for the bucket
-supplied.
+This is usually set to a Cloudflare CDN URL as Backblaze offers
+free egress for data downloaded through the Cloudflare network.
+Rclone works with private buckets by sending an "Authorization" header.
+If the custom endpoint rewrites the requests for authentication,
+e.g., in Cloudflare Workers, this header needs to be handled properly.
+Leave blank if you want to use the endpoint provided by Backblaze.
- rclone backend versioning s3:bucket # read status only
- rclone backend versioning s3:bucket Enabled
- rclone backend versioning s3:bucket Suspended
+The URL provided here SHOULD have the protocol and SHOULD NOT have
+a trailing slash or specify the /file/bucket subpath as rclone will
+request files with "{download_url}/file/{bucket_name}/{path}".
-It may return "Enabled", "Suspended" or "Unversioned". Note that once versioning
-has been enabled the status can't be set back to "Unversioned".
+Example:
+> https://mysubdomain.mydomain.tld
+(No trailing "/", "file" or "bucket")
+Properties:
-### set
+- Config: download_url
+- Env Var: RCLONE_B2_DOWNLOAD_URL
+- Type: string
+- Required: false
-Set command for updating the config parameters.
+#### --b2-download-auth-duration
- rclone backend set remote: [options] [+]
+Time before the authorization token will expire in s or suffix ms|s|m|h|d.
-This set command can be used to update the config parameters
-for a running s3 backend.
+The duration before the download authorization token will expire.
+The minimum value is 1 second. The maximum value is one week.
-Usage Examples:
+Properties:
- rclone backend set s3: [-o opt_name=opt_value] [-o opt_name2=opt_value2]
- rclone rc backend/command command=set fs=s3: [-o opt_name=opt_value] [-o opt_name2=opt_value2]
- rclone rc backend/command command=set fs=s3: -o session_token=X -o access_key_id=X -o secret_access_key=X
+- Config: download_auth_duration
+- Env Var: RCLONE_B2_DOWNLOAD_AUTH_DURATION
+- Type: Duration
+- Default: 1w
-The option keys are named as they are in the config file.
+#### --b2-memory-pool-flush-time
-This rebuilds the connection to the s3 backend when it is called with
-the new parameters. Only new parameters need be passed as the values
-will default to those currently in use.
+How often internal memory buffer pools will be flushed. (no longer used)
-It doesn't return anything.
+Properties:
+- Config: memory_pool_flush_time
+- Env Var: RCLONE_B2_MEMORY_POOL_FLUSH_TIME
+- Type: Duration
+- Default: 1m0s
+#### --b2-memory-pool-use-mmap
+Whether to use mmap buffers in internal memory pool. (no longer used)
-### Anonymous access to public buckets
+Properties:
-If you want to use rclone to access a public bucket, configure with a
-blank `access_key_id` and `secret_access_key`. Your config should end
-up looking like this:
+- Config: memory_pool_use_mmap
+- Env Var: RCLONE_B2_MEMORY_POOL_USE_MMAP
+- Type: bool
+- Default: false
-```
-[anons3]
-type = s3
-provider = AWS
-env_auth = false
-access_key_id =
-secret_access_key =
-region = us-east-1
-endpoint =
-location_constraint =
-acl = private
-server_side_encryption =
-storage_class =
-```
+#### --b2-lifecycle
-Then use it as normal with the name of the public bucket, e.g.
+Set the number of days deleted files should be kept when creating a bucket.
- rclone lsd anons3:1000genomes
+On bucket creation, this parameter is used to create a lifecycle rule
+for the entire bucket.
-You will be able to list and copy data but not upload it.
+If lifecycle is 0 (the default) it does not create a lifecycle rule so
+the default B2 behaviour applies. This is to create versions of files
+on delete and overwrite and to keep them indefinitely.
-## Providers
+If lifecycle is >0 then it creates a single rule setting the number of
+days before a file that is deleted or overwritten is deleted
+permanently. This is known as daysFromHidingToDeleting in the b2 docs.
-### AWS S3
+The minimum value for this parameter is 1 day.
-This is the provider used as main example and described in the [configuration](#configuration) section above.
+You can also enable hard_delete in the config also which will mean
+deletions won't cause versions but overwrites will still cause
+versions to be made.
-### AWS Snowball Edge
+See: [rclone backend lifecycle](#lifecycle) for setting lifecycles after bucket creation.
-[AWS Snowball](https://aws.amazon.com/snowball/) is a hardware
-appliance used for transferring bulk data back to AWS. Its main
-software interface is S3 object storage.
-To use rclone with AWS Snowball Edge devices, configure as standard
-for an 'S3 Compatible Service'.
+Properties:
-If using rclone pre v1.59 be sure to set `upload_cutoff = 0` otherwise
-you will run into authentication header issues as the snowball device
-does not support query parameter based authentication.
+- Config: lifecycle
+- Env Var: RCLONE_B2_LIFECYCLE
+- Type: int
+- Default: 0
-With rclone v1.59 or later setting `upload_cutoff` should not be necessary.
+#### --b2-encoding
-eg.
-```
-[snowball]
-type = s3
-provider = Other
-access_key_id = YOUR_ACCESS_KEY
-secret_access_key = YOUR_SECRET_KEY
-endpoint = http://[IP of Snowball]:8080
-upload_cutoff = 0
-```
+The encoding for the backend.
-### Ceph
+See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info.
-[Ceph](https://ceph.com/) is an open-source, unified, distributed
-storage system designed for excellent performance, reliability and
-scalability. It has an S3 compatible object storage interface.
+Properties:
-To use rclone with Ceph, configure as above but leave the region blank
-and set the endpoint. You should end up with something like this in
-your config:
+- Config: encoding
+- Env Var: RCLONE_B2_ENCODING
+- Type: Encoding
+- Default: Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot
+## Backend commands
-```
-[ceph]
-type = s3
-provider = Ceph
-env_auth = false
-access_key_id = XXX
-secret_access_key = YYY
-region =
-endpoint = https://ceph.endpoint.example.com
-location_constraint =
-acl =
-server_side_encryption =
-storage_class =
-```
+Here are the commands specific to the b2 backend.
-If you are using an older version of CEPH (e.g. 10.2.x Jewel) and a
-version of rclone before v1.59 then you may need to supply the
-parameter `--s3-upload-cutoff 0` or put this in the config file as
-`upload_cutoff 0` to work around a bug which causes uploading of small
-files to fail.
+Run them with
-Note also that Ceph sometimes puts `/` in the passwords it gives
-users. If you read the secret access key using the command line tools
-you will get a JSON blob with the `/` escaped as `\/`. Make sure you
-only write `/` in the secret access key.
+ rclone backend COMMAND remote:
-Eg the dump from Ceph looks something like this (irrelevant keys
-removed).
+The help below will explain what arguments each command takes.
-```
-{
- "user_id": "xxx",
- "display_name": "xxxx",
- "keys": [
- {
- "user": "xxx",
- "access_key": "xxxxxx",
- "secret_key": "xxxxxx\/xxxx"
- }
- ],
-}
-```
+See the [backend](https://rclone.org/commands/rclone_backend/) command for more
+info on how to pass options and arguments.
-Because this is a json dump, it is encoding the `/` as `\/`, so if you
-use the secret key as `xxxxxx/xxxx` it will work fine.
+These can be run on a running backend using the rc command
+[backend/command](https://rclone.org/rc/#backend-command).
-### Cloudflare R2 {#cloudflare-r2}
+### lifecycle
-[Cloudflare R2](https://blog.cloudflare.com/r2-open-beta/) Storage
-allows developers to store large amounts of unstructured data without
-the costly egress bandwidth fees associated with typical cloud storage
-services.
+Read or set the lifecycle for a bucket
-Here is an example of making a Cloudflare R2 configuration. First run:
+ rclone backend lifecycle remote: [options] [+]
- rclone config
+This command can be used to read or set the lifecycle for a bucket.
-This will guide you through an interactive setup process.
+Usage Examples:
-Note that all buckets are private, and all are stored in the same
-"auto" region. It is necessary to use Cloudflare workers to share the
-content of a bucket publicly.
+To show the current lifecycle rules:
-```
-No remotes found, make a new one?
-n) New remote
-s) Set configuration password
-q) Quit config
-n/s/q> n
-name> r2
-Option Storage.
-Type of storage to configure.
-Choose a number from below, or type in your own value.
-...
-XX / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, China Mobile, Cloudflare, ArvanCloud, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, Lyve Cloud, Minio, Netease, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Synology, Tencent COS and Wasabi
- \ (s3)
-...
-Storage> s3
-Option provider.
-Choose your S3 provider.
-Choose a number from below, or type in your own value.
-Press Enter to leave empty.
-...
-XX / Cloudflare R2 Storage
- \ (Cloudflare)
-...
-provider> Cloudflare
-Option env_auth.
-Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
-Only applies if access_key_id and secret_access_key is blank.
-Choose a number from below, or type in your own boolean value (true or false).
-Press Enter for the default (false).
- 1 / Enter AWS credentials in the next step.
- \ (false)
- 2 / Get AWS credentials from the environment (env vars or IAM).
- \ (true)
-env_auth> 1
-Option access_key_id.
-AWS Access Key ID.
-Leave blank for anonymous access or runtime credentials.
-Enter a value. Press Enter to leave empty.
-access_key_id> ACCESS_KEY
-Option secret_access_key.
-AWS Secret Access Key (password).
-Leave blank for anonymous access or runtime credentials.
-Enter a value. Press Enter to leave empty.
-secret_access_key> SECRET_ACCESS_KEY
-Option region.
-Region to connect to.
-Choose a number from below, or type in your own value.
-Press Enter to leave empty.
- 1 / R2 buckets are automatically distributed across Cloudflare's data centers for low latency.
- \ (auto)
-region> 1
-Option endpoint.
-Endpoint for S3 API.
-Required when using an S3 clone.
-Enter a value. Press Enter to leave empty.
-endpoint> https://ACCOUNT_ID.r2.cloudflarestorage.com
-Edit advanced config?
-y) Yes
-n) No (default)
-y/n> n
---------------------
-y) Yes this is OK (default)
-e) Edit this remote
-d) Delete this remote
-y/e/d> y
-```
+ rclone backend lifecycle b2:bucket
-This will leave your config looking something like:
+This will dump something like this showing the lifecycle rules.
-```
-[r2]
-type = s3
-provider = Cloudflare
-access_key_id = ACCESS_KEY
-secret_access_key = SECRET_ACCESS_KEY
-region = auto
-endpoint = https://ACCOUNT_ID.r2.cloudflarestorage.com
-acl = private
-```
+ [
+ {
+ "daysFromHidingToDeleting": 1,
+ "daysFromUploadingToHiding": null,
+ "fileNamePrefix": ""
+ }
+ ]
-Now run `rclone lsf r2:` to see your buckets and `rclone lsf
-r2:bucket` to look within a bucket.
+If there are no lifecycle rules (the default) then it will just return [].
-### Dreamhost
+To reset the current lifecycle rules:
-Dreamhost [DreamObjects](https://www.dreamhost.com/cloud/storage/) is
-an object storage system based on CEPH.
+ rclone backend lifecycle b2:bucket -o daysFromHidingToDeleting=30
+ rclone backend lifecycle b2:bucket -o daysFromUploadingToHiding=5 -o daysFromHidingToDeleting=1
-To use rclone with Dreamhost, configure as above but leave the region blank
-and set the endpoint. You should end up with something like this in
-your config:
+This will run and then print the new lifecycle rules as above.
-```
-[dreamobjects]
-type = s3
-provider = DreamHost
-env_auth = false
-access_key_id = your_access_key
-secret_access_key = your_secret_key
-region =
-endpoint = objects-us-west-1.dream.io
-location_constraint =
-acl = private
-server_side_encryption =
-storage_class =
-```
+Rclone only lets you set lifecycles for the whole bucket with the
+fileNamePrefix = "".
-### Google Cloud Storage
+You can't disable versioning with B2. The best you can do is to set
+the daysFromHidingToDeleting to 1 day. You can enable hard_delete in
+the config also which will mean deletions won't cause versions but
+overwrites will still cause versions to be made.
-[GoogleCloudStorage](https://cloud.google.com/storage/docs) is an [S3-interoperable](https://cloud.google.com/storage/docs/interoperability) object storage service from Google Cloud Platform.
+ rclone backend lifecycle b2:bucket -o daysFromHidingToDeleting=1
-To connect to Google Cloud Storage you will need an access key and secret key. These can be retrieved by creating an [HMAC key](https://cloud.google.com/storage/docs/authentication/managing-hmackeys).
+See: https://www.backblaze.com/docs/cloud-storage-lifecycle-rules
-```
-[gs]
-type = s3
-provider = GCS
-access_key_id = your_access_key
-secret_access_key = your_secret_key
-endpoint = https://storage.googleapis.com
-```
-### DigitalOcean Spaces
+Options:
-[Spaces](https://www.digitalocean.com/products/object-storage/) is an [S3-interoperable](https://developers.digitalocean.com/documentation/spaces/) object storage service from cloud provider DigitalOcean.
+- "daysFromHidingToDeleting": After a file has been hidden for this many days it is deleted. 0 is off.
+- "daysFromUploadingToHiding": This many days after uploading a file is hidden
-To connect to DigitalOcean Spaces you will need an access key and secret key. These can be retrieved on the "[Applications & API](https://cloud.digitalocean.com/settings/api/tokens)" page of the DigitalOcean control panel. They will be needed when prompted by `rclone config` for your `access_key_id` and `secret_access_key`.
-When prompted for a `region` or `location_constraint`, press enter to use the default value. The region must be included in the `endpoint` setting (e.g. `nyc3.digitaloceanspaces.com`). The default values can be used for other settings.
-Going through the whole process of creating a new remote by running `rclone config`, each prompt should be answered as shown below:
+## Limitations
-```
-Storage> s3
-env_auth> 1
-access_key_id> YOUR_ACCESS_KEY
-secret_access_key> YOUR_SECRET_KEY
-region>
-endpoint> nyc3.digitaloceanspaces.com
-location_constraint>
-acl>
-storage_class>
-```
+`rclone about` is not supported by the B2 backend. Backends without
+this capability cannot determine free space for an rclone mount or
+use policy `mfs` (most free space) as a member of an rclone union
+remote.
-The resulting configuration file should look like:
+See [List of backends that do not support rclone about](https://rclone.org/overview/#optional-features) and [rclone about](https://rclone.org/commands/rclone_about/)
-```
-[spaces]
-type = s3
-provider = DigitalOcean
-env_auth = false
-access_key_id = YOUR_ACCESS_KEY
-secret_access_key = YOUR_SECRET_KEY
-region =
-endpoint = nyc3.digitaloceanspaces.com
-location_constraint =
-acl =
-server_side_encryption =
-storage_class =
-```
+# Box
-Once configured, you can create a new Space and begin copying files. For example:
+Paths are specified as `remote:path`
-```
-rclone mkdir spaces:my-new-space
-rclone copy /path/to/files spaces:my-new-space
-```
-### Huawei OBS {#huawei-obs}
+Paths may be as deep as required, e.g. `remote:directory/subdirectory`.
-Object Storage Service (OBS) provides stable, secure, efficient, and easy-to-use cloud storage that lets you store virtually any volume of unstructured data in any format and access it from anywhere.
+The initial setup for Box involves getting a token from Box which you
+can do either in your browser, or with a config.json downloaded from Box
+to use JWT authentication. `rclone config` walks you through it.
-OBS provides an S3 interface, you can copy and modify the following configuration and add it to your rclone configuration file.
-```
-[obs]
-type = s3
-provider = HuaweiOBS
-access_key_id = your-access-key-id
-secret_access_key = your-secret-access-key
-region = af-south-1
-endpoint = obs.af-south-1.myhuaweicloud.com
-acl = private
-```
+## Configuration
+
+Here is an example of how to make a remote called `remote`. First run:
+
+ rclone config
+
+This will guide you through an interactive setup process:
-Or you can also configure via the interactive command line:
```
No remotes found, make a new one?
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
-name> obs
-Option Storage.
+name> remote
Type of storage to configure.
-Choose a number from below, or type in your own value.
-[snip]
- 5 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, China Mobile, Cloudflare, ArvanCloud, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, Lyve Cloud, Minio, Netease, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Synology, Tencent COS and Wasabi
- \ (s3)
-[snip]
-Storage> 5
-Option provider.
-Choose your S3 provider.
-Choose a number from below, or type in your own value.
-Press Enter to leave empty.
-[snip]
- 9 / Huawei Object Storage Service
- \ (HuaweiOBS)
-[snip]
-provider> 9
-Option env_auth.
-Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
-Only applies if access_key_id and secret_access_key is blank.
-Choose a number from below, or type in your own boolean value (true or false).
-Press Enter for the default (false).
- 1 / Enter AWS credentials in the next step.
- \ (false)
- 2 / Get AWS credentials from the environment (env vars or IAM).
- \ (true)
-env_auth> 1
-Option access_key_id.
-AWS Access Key ID.
-Leave blank for anonymous access or runtime credentials.
-Enter a value. Press Enter to leave empty.
-access_key_id> your-access-key-id
-Option secret_access_key.
-AWS Secret Access Key (password).
-Leave blank for anonymous access or runtime credentials.
-Enter a value. Press Enter to leave empty.
-secret_access_key> your-secret-access-key
-Option region.
-Region to connect to.
-Choose a number from below, or type in your own value.
-Press Enter to leave empty.
- 1 / AF-Johannesburg
- \ (af-south-1)
- 2 / AP-Bangkok
- \ (ap-southeast-2)
-[snip]
-region> 1
-Option endpoint.
-Endpoint for OBS API.
-Choose a number from below, or type in your own value.
-Press Enter to leave empty.
- 1 / AF-Johannesburg
- \ (obs.af-south-1.myhuaweicloud.com)
- 2 / AP-Bangkok
- \ (obs.ap-southeast-2.myhuaweicloud.com)
+Choose a number from below, or type in your own value
[snip]
-endpoint> 1
-Option acl.
-Canned ACL used when creating buckets and storing or copying objects.
-This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.
-For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
-Note that this ACL is applied when server-side copying objects as S3
-doesn't copy the ACL from the source but rather writes a fresh one.
-Choose a number from below, or type in your own value.
-Press Enter to leave empty.
- / Owner gets FULL_CONTROL.
- 1 | No one else has access rights (default).
- \ (private)
+XX / Box
+ \ "box"
[snip]
-acl> 1
-Edit advanced config?
+Storage> box
+Box App Client Id - leave blank normally.
+client_id>
+Box App Client Secret - leave blank normally.
+client_secret>
+Box App config.json location
+Leave blank normally.
+Enter a string value. Press Enter for the default ("").
+box_config_file>
+Box App Primary Access Token
+Leave blank normally.
+Enter a string value. Press Enter for the default ("").
+access_token>
+
+Enter a string value. Press Enter for the default ("user").
+Choose a number from below, or type in your own value
+ 1 / Rclone should act on behalf of a user
+ \ "user"
+ 2 / Rclone should act on behalf of a service account
+ \ "enterprise"
+box_sub_type>
+Remote config
+Use web browser to automatically authenticate rclone with remote?
+ * Say Y if the machine running rclone has a web browser you can use
+ * Say N if running rclone on a (remote) machine without web browser access
+If not sure try Y. If Y failed, try N.
y) Yes
-n) No (default)
-y/n>
+n) No
+y/n> y
+If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth
+Log in and authorize rclone for access
+Waiting for code...
+Got code
--------------------
-[obs]
-type = s3
-provider = HuaweiOBS
-access_key_id = your-access-key-id
-secret_access_key = your-secret-access-key
-region = af-south-1
-endpoint = obs.af-south-1.myhuaweicloud.com
-acl = private
+[remote]
+client_id =
+client_secret =
+token = {"access_token":"XXX","token_type":"bearer","refresh_token":"XXX","expiry":"XXX"}
--------------------
-y) Yes this is OK (default)
+y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> y
-Current remotes:
-
-Name Type
-==== ====
-obs s3
-
-e) Edit existing remote
-n) New remote
-d) Delete remote
-r) Rename remote
-c) Copy remote
-s) Set configuration password
-q) Quit config
-e/n/d/r/c/s/q> q
-```
-
-### IBM COS (S3)
-
-Information stored with IBM Cloud Object Storage is encrypted and dispersed across multiple geographic locations, and accessed through an implementation of the S3 API. This service makes use of the distributed storage technologies provided by IBM’s Cloud Object Storage System (formerly Cleversafe). For more information visit: (http://www.ibm.com/cloud/object-storage)
-
-To configure access to IBM COS S3, follow the steps below:
-
-1. Run rclone config and select n for a new remote.
-```
- 2018/02/14 14:13:11 NOTICE: Config file "C:\\Users\\a\\.config\\rclone\\rclone.conf" not found - using defaults
- No remotes found, make a new one?
- n) New remote
- s) Set configuration password
- q) Quit config
- n/s/q> n
-```
-
-2. Enter the name for the configuration
-```
- name>
-```
-
-3. Select "s3" storage.
-```
-Choose a number from below, or type in your own value
- 1 / Alias for an existing remote
- \ "alias"
- 2 / Amazon Drive
- \ "amazon cloud drive"
- 3 / Amazon S3 Complaint Storage Providers (Dreamhost, Ceph, ChinaMobile, Liara, ArvanCloud, Minio, IBM COS)
- \ "s3"
- 4 / Backblaze B2
- \ "b2"
-[snip]
- 23 / HTTP
- \ "http"
-Storage> 3
-```
-
-4. Select IBM COS as the S3 Storage Provider.
-```
-Choose the S3 provider.
-Choose a number from below, or type in your own value
- 1 / Choose this option to configure Storage to AWS S3
- \ "AWS"
- 2 / Choose this option to configure Storage to Ceph Systems
- \ "Ceph"
- 3 / Choose this option to configure Storage to Dreamhost
- \ "Dreamhost"
- 4 / Choose this option to the configure Storage to IBM COS S3
- \ "IBMCOS"
- 5 / Choose this option to the configure Storage to Minio
- \ "Minio"
- Provider>4
-```
-
-5. Enter the Access Key and Secret.
-```
- AWS Access Key ID - leave blank for anonymous access or runtime credentials.
- access_key_id> <>
- AWS Secret Access Key (password) - leave blank for anonymous access or runtime credentials.
- secret_access_key> <>
-```
-
-6. Specify the endpoint for IBM COS. For Public IBM COS, choose from the option below. For On Premise IBM COS, enter an endpoint address.
-```
- Endpoint for IBM COS S3 API.
- Specify if using an IBM COS On Premise.
- Choose a number from below, or type in your own value
- 1 / US Cross Region Endpoint
- \ "s3-api.us-geo.objectstorage.softlayer.net"
- 2 / US Cross Region Dallas Endpoint
- \ "s3-api.dal.us-geo.objectstorage.softlayer.net"
- 3 / US Cross Region Washington DC Endpoint
- \ "s3-api.wdc-us-geo.objectstorage.softlayer.net"
- 4 / US Cross Region San Jose Endpoint
- \ "s3-api.sjc-us-geo.objectstorage.softlayer.net"
- 5 / US Cross Region Private Endpoint
- \ "s3-api.us-geo.objectstorage.service.networklayer.com"
- 6 / US Cross Region Dallas Private Endpoint
- \ "s3-api.dal-us-geo.objectstorage.service.networklayer.com"
- 7 / US Cross Region Washington DC Private Endpoint
- \ "s3-api.wdc-us-geo.objectstorage.service.networklayer.com"
- 8 / US Cross Region San Jose Private Endpoint
- \ "s3-api.sjc-us-geo.objectstorage.service.networklayer.com"
- 9 / US Region East Endpoint
- \ "s3.us-east.objectstorage.softlayer.net"
- 10 / US Region East Private Endpoint
- \ "s3.us-east.objectstorage.service.networklayer.com"
- 11 / US Region South Endpoint
-[snip]
- 34 / Toronto Single Site Private Endpoint
- \ "s3.tor01.objectstorage.service.networklayer.com"
- endpoint>1
-```
-
-
-7. Specify a IBM COS Location Constraint. The location constraint must match endpoint when using IBM Cloud Public. For on-prem COS, do not make a selection from this list, hit enter
-```
- 1 / US Cross Region Standard
- \ "us-standard"
- 2 / US Cross Region Vault
- \ "us-vault"
- 3 / US Cross Region Cold
- \ "us-cold"
- 4 / US Cross Region Flex
- \ "us-flex"
- 5 / US East Region Standard
- \ "us-east-standard"
- 6 / US East Region Vault
- \ "us-east-vault"
- 7 / US East Region Cold
- \ "us-east-cold"
- 8 / US East Region Flex
- \ "us-east-flex"
- 9 / US South Region Standard
- \ "us-south-standard"
- 10 / US South Region Vault
- \ "us-south-vault"
-[snip]
- 32 / Toronto Flex
- \ "tor01-flex"
-location_constraint>1
-```
-
-9. Specify a canned ACL. IBM Cloud (Storage) supports "public-read" and "private". IBM Cloud(Infra) supports all the canned ACLs. On-Premise COS supports all the canned ACLs.
-```
-Canned ACL used when creating buckets and/or storing objects in S3.
-For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
-Choose a number from below, or type in your own value
- 1 / Owner gets FULL_CONTROL. No one else has access rights (default). This acl is available on IBM Cloud (Infra), IBM Cloud (Storage), On-Premise COS
- \ "private"
- 2 / Owner gets FULL_CONTROL. The AllUsers group gets READ access. This acl is available on IBM Cloud (Infra), IBM Cloud (Storage), On-Premise IBM COS
- \ "public-read"
- 3 / Owner gets FULL_CONTROL. The AllUsers group gets READ and WRITE access. This acl is available on IBM Cloud (Infra), On-Premise IBM COS
- \ "public-read-write"
- 4 / Owner gets FULL_CONTROL. The AuthenticatedUsers group gets READ access. Not supported on Buckets. This acl is available on IBM Cloud (Infra) and On-Premise IBM COS
- \ "authenticated-read"
-acl> 1
```
+See the [remote setup docs](https://rclone.org/remote_setup/) for how to set it up on a
+machine with no Internet browser available.
-12. Review the displayed configuration and accept to save the "remote" then quit. The config file should look like this
-```
- [xxx]
- type = s3
- Provider = IBMCOS
- access_key_id = xxx
- secret_access_key = yyy
- endpoint = s3-api.us-geo.objectstorage.softlayer.net
- location_constraint = us-standard
- acl = private
-```
-
-13. Execute rclone commands
-```
- 1) Create a bucket.
- rclone mkdir IBM-COS-XREGION:newbucket
- 2) List available buckets.
- rclone lsd IBM-COS-XREGION:
- -1 2017-11-08 21:16:22 -1 test
- -1 2018-02-14 20:16:39 -1 newbucket
- 3) List contents of a bucket.
- rclone ls IBM-COS-XREGION:newbucket
- 18685952 test.exe
- 4) Copy a file from local to remote.
- rclone copy /Users/file.txt IBM-COS-XREGION:newbucket
- 5) Copy a file from remote to local.
- rclone copy IBM-COS-XREGION:newbucket/file.txt .
- 6) Delete a file on remote.
- rclone delete IBM-COS-XREGION:newbucket/file.txt
-```
+Note that rclone runs a webserver on your local machine to collect the
+token as returned from Box. This only runs from the moment it opens
+your browser to the moment you get back the verification code. This
+is on `http://127.0.0.1:53682/` and this it may require you to unblock
+it temporarily if you are running a host firewall.
-### IDrive e2 {#idrive-e2}
+Once configured you can then use `rclone` like this,
-Here is an example of making an [IDrive e2](https://www.idrive.com/e2/)
-configuration. First run:
+List directories in top level of your Box
- rclone config
+ rclone lsd remote:
-This will guide you through an interactive setup process.
+List all the files in your Box
-```
-No remotes found, make a new one?
-n) New remote
-s) Set configuration password
-q) Quit config
-n/s/q> n
+ rclone ls remote:
-Enter name for new remote.
-name> e2
+To copy a local directory to an Box directory called backup
-Option Storage.
-Type of storage to configure.
-Choose a number from below, or type in your own value.
-[snip]
-XX / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, China Mobile, Cloudflare, ArvanCloud, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, IDrive e2, Lyve Cloud, Minio, Netease, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Synology, Tencent COS and Wasabi
- \ (s3)
-[snip]
-Storage> s3
+ rclone copy /home/source remote:backup
-Option provider.
-Choose your S3 provider.
-Choose a number from below, or type in your own value.
-Press Enter to leave empty.
-[snip]
-XX / IDrive e2
- \ (IDrive)
-[snip]
-provider> IDrive
+### Using rclone with an Enterprise account with SSO
-Option env_auth.
-Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
-Only applies if access_key_id and secret_access_key is blank.
-Choose a number from below, or type in your own boolean value (true or false).
-Press Enter for the default (false).
- 1 / Enter AWS credentials in the next step.
- \ (false)
- 2 / Get AWS credentials from the environment (env vars or IAM).
- \ (true)
-env_auth>
+If you have an "Enterprise" account type with Box with single sign on
+(SSO), you need to create a password to use Box with rclone. This can
+be done at your Enterprise Box account by going to Settings, "Account"
+Tab, and then set the password in the "Authentication" field.
-Option access_key_id.
-AWS Access Key ID.
-Leave blank for anonymous access or runtime credentials.
-Enter a value. Press Enter to leave empty.
-access_key_id> YOUR_ACCESS_KEY
+Once you have done this, you can setup your Enterprise Box account
+using the same procedure detailed above in the, using the password you
+have just set.
-Option secret_access_key.
-AWS Secret Access Key (password).
-Leave blank for anonymous access or runtime credentials.
-Enter a value. Press Enter to leave empty.
-secret_access_key> YOUR_SECRET_KEY
+### Invalid refresh token
-Option acl.
-Canned ACL used when creating buckets and storing or copying objects.
-This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.
-For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
-Note that this ACL is applied when server-side copying objects as S3
-doesn't copy the ACL from the source but rather writes a fresh one.
-Choose a number from below, or type in your own value.
-Press Enter to leave empty.
- / Owner gets FULL_CONTROL.
- 1 | No one else has access rights (default).
- \ (private)
- / Owner gets FULL_CONTROL.
- 2 | The AllUsers group gets READ access.
- \ (public-read)
- / Owner gets FULL_CONTROL.
- 3 | The AllUsers group gets READ and WRITE access.
- | Granting this on a bucket is generally not recommended.
- \ (public-read-write)
- / Owner gets FULL_CONTROL.
- 4 | The AuthenticatedUsers group gets READ access.
- \ (authenticated-read)
- / Object owner gets FULL_CONTROL.
- 5 | Bucket owner gets READ access.
- | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
- \ (bucket-owner-read)
- / Both the object owner and the bucket owner get FULL_CONTROL over the object.
- 6 | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
- \ (bucket-owner-full-control)
-acl>
+According to the [box docs](https://developer.box.com/v2.0/docs/oauth-20#section-6-using-the-access-and-refresh-tokens):
-Edit advanced config?
-y) Yes
-n) No (default)
-y/n>
+> Each refresh_token is valid for one use in 60 days.
-Configuration complete.
-Options:
-- type: s3
-- provider: IDrive
-- access_key_id: YOUR_ACCESS_KEY
-- secret_access_key: YOUR_SECRET_KEY
-- endpoint: q9d9.la12.idrivee2-5.com
-Keep this "e2" remote?
-y) Yes this is OK (default)
-e) Edit this remote
-d) Delete this remote
-y/e/d> y
-```
+This means that if you
-### IONOS Cloud {#ionos}
+ * Don't use the box remote for 60 days
+ * Copy the config file with a box refresh token in and use it in two places
+ * Get an error on a token refresh
-[IONOS S3 Object Storage](https://cloud.ionos.com/storage/object-storage) is a service offered by IONOS for storing and accessing unstructured data.
-To connect to the service, you will need an access key and a secret key. These can be found in the [Data Center Designer](https://dcd.ionos.com/), by selecting **Manager resources** > **Object Storage Key Manager**.
+then rclone will return an error which includes the text `Invalid
+refresh token`.
+To fix this you will need to use oauth2 again to update the refresh
+token. You can use the methods in [the remote setup
+docs](https://rclone.org/remote_setup/), bearing in mind that if you use the copy the
+config file method, you should not use that remote on the computer you
+did the authentication on.
-Here is an example of a configuration. First, run `rclone config`. This will walk you through an interactive setup process. Type `n` to add the new remote, and then enter a name:
+Here is how to do it.
```
-Enter name for new remote.
-name> ionos-fra
-```
+$ rclone config
+Current remotes:
-Type `s3` to choose the connection type:
-```
-Option Storage.
-Type of storage to configure.
-Choose a number from below, or type in your own value.
-[snip]
-XX / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, China Mobile, Cloudflare, ArvanCloud, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, IDrive e2, IONOS Cloud, Lyve Cloud, Minio, Netease, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Synology, Tencent COS and Wasabi
- \ (s3)
-[snip]
-Storage> s3
-```
+Name Type
+==== ====
+remote box
-Type `IONOS`:
-```
-Option provider.
-Choose your S3 provider.
-Choose a number from below, or type in your own value.
-Press Enter to leave empty.
-[snip]
-XX / IONOS Cloud
- \ (IONOS)
-[snip]
-provider> IONOS
+e) Edit existing remote
+n) New remote
+d) Delete remote
+r) Rename remote
+c) Copy remote
+s) Set configuration password
+q) Quit config
+e/n/d/r/c/s/q> e
+Choose a number from below, or type in an existing value
+ 1 > remote
+remote> remote
+--------------------
+[remote]
+type = box
+token = {"access_token":"XXX","token_type":"bearer","refresh_token":"XXX","expiry":"2017-07-08T23:40:08.059167677+01:00"}
+--------------------
+Edit remote
+Value "client_id" = ""
+Edit? (y/n)>
+y) Yes
+n) No
+y/n> n
+Value "client_secret" = ""
+Edit? (y/n)>
+y) Yes
+n) No
+y/n> n
+Remote config
+Already have a token - refresh?
+y) Yes
+n) No
+y/n> y
+Use web browser to automatically authenticate rclone with remote?
+ * Say Y if the machine running rclone has a web browser you can use
+ * Say N if running rclone on a (remote) machine without web browser access
+If not sure try Y. If Y failed, try N.
+y) Yes
+n) No
+y/n> y
+If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth
+Log in and authorize rclone for access
+Waiting for code...
+Got code
+--------------------
+[remote]
+type = box
+token = {"access_token":"YYY","token_type":"bearer","refresh_token":"YYY","expiry":"2017-07-23T12:22:29.259137901+01:00"}
+--------------------
+y) Yes this is OK
+e) Edit this remote
+d) Delete this remote
+y/e/d> y
```
-Press Enter to choose the default option `Enter AWS credentials in the next step`:
-```
-Option env_auth.
-Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
-Only applies if access_key_id and secret_access_key is blank.
-Choose a number from below, or type in your own boolean value (true or false).
-Press Enter for the default (false).
- 1 / Enter AWS credentials in the next step.
- \ (false)
- 2 / Get AWS credentials from the environment (env vars or IAM).
- \ (true)
-env_auth>
-```
+### Modification times and hashes
-Enter your Access Key and Secret key. These can be retrieved in the [Data Center Designer](https://dcd.ionos.com/), click on the menu “Manager resources†/ "Object Storage Key Manager".
-```
-Option access_key_id.
-AWS Access Key ID.
-Leave blank for anonymous access or runtime credentials.
-Enter a value. Press Enter to leave empty.
-access_key_id> YOUR_ACCESS_KEY
+Box allows modification times to be set on objects accurate to 1
+second. These will be used to detect whether objects need syncing or
+not.
-Option secret_access_key.
-AWS Secret Access Key (password).
-Leave blank for anonymous access or runtime credentials.
-Enter a value. Press Enter to leave empty.
-secret_access_key> YOUR_SECRET_KEY
-```
+Box supports SHA1 type hashes, so you can use the `--checksum`
+flag.
-Choose the region where your bucket is located:
-```
-Option region.
-Region where your bucket will be created and your data stored.
-Choose a number from below, or type in your own value.
-Press Enter to leave empty.
- 1 / Frankfurt, Germany
- \ (de)
- 2 / Berlin, Germany
- \ (eu-central-2)
- 3 / Logrono, Spain
- \ (eu-south-2)
-region> 2
-```
+### Restricted filename characters
-Choose the endpoint from the same region:
-```
-Option endpoint.
-Endpoint for IONOS S3 Object Storage.
-Specify the endpoint from the same region.
-Choose a number from below, or type in your own value.
-Press Enter to leave empty.
- 1 / Frankfurt, Germany
- \ (s3-eu-central-1.ionoscloud.com)
- 2 / Berlin, Germany
- \ (s3-eu-central-2.ionoscloud.com)
- 3 / Logrono, Spain
- \ (s3-eu-south-2.ionoscloud.com)
-endpoint> 1
-```
+In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters)
+the following characters are also replaced:
-Press Enter to choose the default option or choose the desired ACL setting:
-```
-Option acl.
-Canned ACL used when creating buckets and storing or copying objects.
-This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.
-For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
-Note that this ACL is applied when server-side copying objects as S3
-doesn't copy the ACL from the source but rather writes a fresh one.
-Choose a number from below, or type in your own value.
-Press Enter to leave empty.
- / Owner gets FULL_CONTROL.
- 1 | No one else has access rights (default).
- \ (private)
- / Owner gets FULL_CONTROL.
-[snip]
-acl>
-```
+| Character | Value | Replacement |
+| --------- |:-----:|:-----------:|
+| \ | 0x5C | ï¼¼ |
-Press Enter to skip the advanced config:
-```
-Edit advanced config?
-y) Yes
-n) No (default)
-y/n>
-```
+File names can also not end with the following characters.
+These only get replaced if they are the last character in the name:
-Press Enter to save the configuration, and then `q` to quit the configuration process:
-```
-Configuration complete.
-Options:
-- type: s3
-- provider: IONOS
-- access_key_id: YOUR_ACCESS_KEY
-- secret_access_key: YOUR_SECRET_KEY
-- endpoint: s3-eu-central-1.ionoscloud.com
-Keep this "ionos-fra" remote?
-y) Yes this is OK (default)
-e) Edit this remote
-d) Delete this remote
-y/e/d> y
-```
+| Character | Value | Replacement |
+| --------- |:-----:|:-----------:|
+| SP | 0x20 | â |
-Done! Now you can try some commands (for macOS, use `./rclone` instead of `rclone`).
+Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8),
+as they can't be used in JSON strings.
-1) Create a bucket (the name must be unique within the whole IONOS S3)
-```
-rclone mkdir ionos-fra:my-bucket
-```
-2) List available buckets
-```
-rclone lsd ionos-fra:
-```
-4) Copy a file from local to remote
-```
-rclone copy /Users/file.txt ionos-fra:my-bucket
-```
-3) List contents of a bucket
-```
-rclone ls ionos-fra:my-bucket
-```
-5) Copy a file from remote to local
-```
-rclone copy ionos-fra:my-bucket/file.txt
-```
+### Transfers
-### Minio
+For files above 50 MiB rclone will use a chunked transfer. Rclone will
+upload up to `--transfers` chunks at the same time (shared among all
+the multipart uploads). Chunks are buffered in memory and are
+normally 8 MiB so increasing `--transfers` will increase memory use.
-[Minio](https://minio.io/) is an object storage server built for cloud application developers and devops.
+### Deleting files
-It is very easy to install and provides an S3 compatible server which can be used by rclone.
+Depending on the enterprise settings for your user, the item will
+either be actually deleted from Box or moved to the trash.
-To use it, install Minio following the instructions [here](https://docs.minio.io/docs/minio-quickstart-guide).
+Emptying the trash is supported via the rclone however cleanup command
+however this deletes every trashed file and folder individually so it
+may take a very long time.
+Emptying the trash via the WebUI does not have this limitation
+so it is advised to empty the trash via the WebUI.
-When it configures itself Minio will print something like this
+### Root folder ID
-```
-Endpoint: http://192.168.1.106:9000 http://172.23.0.1:9000
-AccessKey: USWUXHGYZQYFYFFIT3RE
-SecretKey: MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03
-Region: us-east-1
-SQS ARNs: arn:minio:sqs:us-east-1:1:redis arn:minio:sqs:us-east-1:2:redis
+You can set the `root_folder_id` for rclone. This is the directory
+(identified by its `Folder ID`) that rclone considers to be the root
+of your Box drive.
-Browser Access:
- http://192.168.1.106:9000 http://172.23.0.1:9000
+Normally you will leave this blank and rclone will determine the
+correct root to use itself.
-Command-line Access: https://docs.minio.io/docs/minio-client-quickstart-guide
- $ mc config host add myminio http://192.168.1.106:9000 USWUXHGYZQYFYFFIT3RE MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03
+However you can set this to restrict rclone to a specific folder
+hierarchy.
-Object API (Amazon S3 compatible):
- Go: https://docs.minio.io/docs/golang-client-quickstart-guide
- Java: https://docs.minio.io/docs/java-client-quickstart-guide
- Python: https://docs.minio.io/docs/python-client-quickstart-guide
- JavaScript: https://docs.minio.io/docs/javascript-client-quickstart-guide
- .NET: https://docs.minio.io/docs/dotnet-client-quickstart-guide
+In order to do this you will have to find the `Folder ID` of the
+directory you wish rclone to display. This will be the last segment
+of the URL when you open the relevant folder in the Box web
+interface.
-Drive Capacity: 26 GiB Free, 165 GiB Total
-```
+So if the folder you want rclone to use has a URL which looks like
+`https://app.box.com/folder/11xxxxxxxxx8`
+in the browser, then you use `11xxxxxxxxx8` as
+the `root_folder_id` in the config.
-These details need to go into `rclone config` like this. Note that it
-is important to put the region in as stated above.
-```
-env_auth> 1
-access_key_id> USWUXHGYZQYFYFFIT3RE
-secret_access_key> MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03
-region> us-east-1
-endpoint> http://192.168.1.106:9000
-location_constraint>
-server_side_encryption>
-```
+### Standard options
-Which makes the config file look like this
+Here are the Standard options specific to box (Box).
-```
-[minio]
-type = s3
-provider = Minio
-env_auth = false
-access_key_id = USWUXHGYZQYFYFFIT3RE
-secret_access_key = MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03
-region = us-east-1
-endpoint = http://192.168.1.106:9000
-location_constraint =
-server_side_encryption =
-```
+#### --box-client-id
-So once set up, for example, to copy files into a bucket
+OAuth Client Id.
-```
-rclone copy /path/to/files minio:bucket
-```
+Leave blank normally.
-### Qiniu Cloud Object Storage (Kodo) {#qiniu}
+Properties:
-[Qiniu Cloud Object Storage (Kodo)](https://www.qiniu.com/en/products/kodo), a completely independent-researched core technology which is proven by repeated customer experience has occupied absolute leading market leader position. Kodo can be widely applied to mass data management.
+- Config: client_id
+- Env Var: RCLONE_BOX_CLIENT_ID
+- Type: string
+- Required: false
-To configure access to Qiniu Kodo, follow the steps below:
+#### --box-client-secret
-1. Run `rclone config` and select `n` for a new remote.
+OAuth Client Secret.
-```
-rclone config
-No remotes found, make a new one?
-n) New remote
-s) Set configuration password
-q) Quit config
-n/s/q> n
-```
+Leave blank normally.
-2. Give the name of the configuration. For example, name it 'qiniu'.
+Properties:
-```
-name> qiniu
-```
+- Config: client_secret
+- Env Var: RCLONE_BOX_CLIENT_SECRET
+- Type: string
+- Required: false
-3. Select `s3` storage.
+#### --box-box-config-file
-```
-Choose a number from below, or type in your own value
- 1 / 1Fichier
- \ (fichier)
- 2 / Akamai NetStorage
- \ (netstorage)
- 3 / Alias for an existing remote
- \ (alias)
- 4 / Amazon Drive
- \ (amazon cloud drive)
- 5 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, China Mobile, Cloudflare, ArvanCloud, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, IDrive e2, Liara, Lyve Cloud, Minio, Netease, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Synology, Tencent COS, Qiniu and Wasabi
- \ (s3)
-[snip]
-Storage> s3
-```
+Box App config.json location
-4. Select `Qiniu` provider.
-```
-Choose a number from below, or type in your own value
-1 / Amazon Web Services (AWS) S3
- \ "AWS"
-[snip]
-22 / Qiniu Object Storage (Kodo)
- \ (Qiniu)
-[snip]
-provider> Qiniu
-```
+Leave blank normally.
-5. Enter your SecretId and SecretKey of Qiniu Kodo.
+Leading `~` will be expanded in the file name as will environment variables such as `${RCLONE_CONFIG_DIR}`.
-```
-Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
-Only applies if access_key_id and secret_access_key is blank.
-Enter a boolean value (true or false). Press Enter for the default ("false").
-Choose a number from below, or type in your own value
- 1 / Enter AWS credentials in the next step
- \ "false"
- 2 / Get AWS credentials from the environment (env vars or IAM)
- \ "true"
-env_auth> 1
-AWS Access Key ID.
-Leave blank for anonymous access or runtime credentials.
-Enter a string value. Press Enter for the default ("").
-access_key_id> AKIDxxxxxxxxxx
-AWS Secret Access Key (password)
-Leave blank for anonymous access or runtime credentials.
-Enter a string value. Press Enter for the default ("").
-secret_access_key> xxxxxxxxxxx
-```
+Properties:
-6. Select endpoint for Qiniu Kodo. This is the standard endpoint for different region.
+- Config: box_config_file
+- Env Var: RCLONE_BOX_BOX_CONFIG_FILE
+- Type: string
+- Required: false
-```
- / The default endpoint - a good choice if you are unsure.
- 1 | East China Region 1.
- | Needs location constraint cn-east-1.
- \ (cn-east-1)
- / East China Region 2.
- 2 | Needs location constraint cn-east-2.
- \ (cn-east-2)
- / North China Region 1.
- 3 | Needs location constraint cn-north-1.
- \ (cn-north-1)
- / South China Region 1.
- 4 | Needs location constraint cn-south-1.
- \ (cn-south-1)
- / North America Region.
- 5 | Needs location constraint us-north-1.
- \ (us-north-1)
- / Southeast Asia Region 1.
- 6 | Needs location constraint ap-southeast-1.
- \ (ap-southeast-1)
- / Northeast Asia Region 1.
- 7 | Needs location constraint ap-northeast-1.
- \ (ap-northeast-1)
-[snip]
-endpoint> 1
+#### --box-access-token
-Option endpoint.
-Endpoint for Qiniu Object Storage.
-Choose a number from below, or type in your own value.
-Press Enter to leave empty.
- 1 / East China Endpoint 1
- \ (s3-cn-east-1.qiniucs.com)
- 2 / East China Endpoint 2
- \ (s3-cn-east-2.qiniucs.com)
- 3 / North China Endpoint 1
- \ (s3-cn-north-1.qiniucs.com)
- 4 / South China Endpoint 1
- \ (s3-cn-south-1.qiniucs.com)
- 5 / North America Endpoint 1
- \ (s3-us-north-1.qiniucs.com)
- 6 / Southeast Asia Endpoint 1
- \ (s3-ap-southeast-1.qiniucs.com)
- 7 / Northeast Asia Endpoint 1
- \ (s3-ap-northeast-1.qiniucs.com)
-endpoint> 1
+Box App Primary Access Token
-Option location_constraint.
-Location constraint - must be set to match the Region.
-Used when creating buckets only.
-Choose a number from below, or type in your own value.
-Press Enter to leave empty.
- 1 / East China Region 1
- \ (cn-east-1)
- 2 / East China Region 2
- \ (cn-east-2)
- 3 / North China Region 1
- \ (cn-north-1)
- 4 / South China Region 1
- \ (cn-south-1)
- 5 / North America Region 1
- \ (us-north-1)
- 6 / Southeast Asia Region 1
- \ (ap-southeast-1)
- 7 / Northeast Asia Region 1
- \ (ap-northeast-1)
-location_constraint> 1
-```
+Leave blank normally.
-7. Choose acl and storage class.
+Properties:
-```
-Note that this ACL is applied when server-side copying objects as S3
-doesn't copy the ACL from the source but rather writes a fresh one.
-Enter a string value. Press Enter for the default ("").
-Choose a number from below, or type in your own value
- / Owner gets FULL_CONTROL.
- 1 | No one else has access rights (default).
- \ (private)
- / Owner gets FULL_CONTROL.
- 2 | The AllUsers group gets READ access.
- \ (public-read)
-[snip]
-acl> 2
-The storage class to use when storing new objects in Tencent COS.
-Enter a string value. Press Enter for the default ("").
-Choose a number from below, or type in your own value
- 1 / Standard storage class
- \ (STANDARD)
- 2 / Infrequent access storage mode
- \ (LINE)
- 3 / Archive storage mode
- \ (GLACIER)
- 4 / Deep archive storage mode
- \ (DEEP_ARCHIVE)
-[snip]
-storage_class> 1
-Edit advanced config? (y/n)
-y) Yes
-n) No (default)
-y/n> n
-Remote config
---------------------
-[qiniu]
-- type: s3
-- provider: Qiniu
-- access_key_id: xxx
-- secret_access_key: xxx
-- region: cn-east-1
-- endpoint: s3-cn-east-1.qiniucs.com
-- location_constraint: cn-east-1
-- acl: public-read
-- storage_class: STANDARD
---------------------
-y) Yes this is OK (default)
-e) Edit this remote
-d) Delete this remote
-y/e/d> y
-Current remotes:
+- Config: access_token
+- Env Var: RCLONE_BOX_ACCESS_TOKEN
+- Type: string
+- Required: false
-Name Type
-==== ====
-qiniu s3
-```
+#### --box-box-sub-type
-### RackCorp {#RackCorp}
-[RackCorp Object Storage](https://www.rackcorp.com/storage/s3storage) is an S3 compatible object storage platform from your friendly cloud provider RackCorp.
-The service is fast, reliable, well priced and located in many strategic locations unserviced by others, to ensure you can maintain data sovereignty.
-Before you can use RackCorp Object Storage, you'll need to "[sign up](https://www.rackcorp.com/signup)" for an account on our "[portal](https://portal.rackcorp.com)".
-Next you can create an `access key`, a `secret key` and `buckets`, in your location of choice with ease.
-These details are required for the next steps of configuration, when `rclone config` asks for your `access_key_id` and `secret_access_key`.
+Properties:
-Your config should end up looking a bit like this:
+- Config: box_sub_type
+- Env Var: RCLONE_BOX_BOX_SUB_TYPE
+- Type: string
+- Default: "user"
+- Examples:
+ - "user"
+ - Rclone should act on behalf of a user.
+ - "enterprise"
+ - Rclone should act on behalf of a service account.
-```
-[RCS3-demo-config]
-type = s3
-provider = RackCorp
-env_auth = true
-access_key_id = YOURACCESSKEY
-secret_access_key = YOURSECRETACCESSKEY
-region = au-nsw
-endpoint = s3.rackcorp.com
-location_constraint = au-nsw
-```
+### Advanced options
+Here are the Advanced options specific to box (Box).
-### Scaleway
+#### --box-token
-[Scaleway](https://www.scaleway.com/object-storage/) The Object Storage platform allows you to store anything from backups, logs and web assets to documents and photos.
-Files can be dropped from the Scaleway console or transferred through our API and CLI or using any S3-compatible tool.
+OAuth Access Token as a JSON blob.
-Scaleway provides an S3 interface which can be configured for use with rclone like this:
+Properties:
-```
-[scaleway]
-type = s3
-provider = Scaleway
-env_auth = false
-endpoint = s3.nl-ams.scw.cloud
-access_key_id = SCWXXXXXXXXXXXXXX
-secret_access_key = 1111111-2222-3333-44444-55555555555555
-region = nl-ams
-location_constraint =
-acl = private
-server_side_encryption =
-storage_class =
-```
+- Config: token
+- Env Var: RCLONE_BOX_TOKEN
+- Type: string
+- Required: false
-[C14 Cold Storage](https://www.online.net/en/storage/c14-cold-storage) is the low-cost S3 Glacier alternative from Scaleway and it works the same way as on S3 by accepting the "GLACIER" `storage_class`.
-So you can configure your remote with the `storage_class = GLACIER` option to upload directly to C14. Don't forget that in this state you can't read files back after, you will need to restore them to "STANDARD" storage_class first before being able to read them (see "restore" section above)
+#### --box-auth-url
-### Seagate Lyve Cloud {#lyve}
+Auth server URL.
-[Seagate Lyve Cloud](https://www.seagate.com/gb/en/services/cloud/storage/) is an S3
-compatible object storage platform from [Seagate](https://seagate.com/) intended for enterprise use.
+Leave blank to use the provider defaults.
-Here is a config run through for a remote called `remote` - you may
-choose a different name of course. Note that to create an access key
-and secret key you will need to create a service account first.
+Properties:
-```
-$ rclone config
-No remotes found, make a new one?
-n) New remote
-s) Set configuration password
-q) Quit config
-n/s/q> n
-name> remote
-```
+- Config: auth_url
+- Env Var: RCLONE_BOX_AUTH_URL
+- Type: string
+- Required: false
-Choose `s3` backend
+#### --box-token-url
-```
-Type of storage to configure.
-Choose a number from below, or type in your own value.
-[snip]
-XX / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, ChinaMobile, ArvanCloud, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, Liara, Lyve Cloud, Minio, RackCorp, SeaweedFS, and Tencent COS
- \ (s3)
-[snip]
-Storage> s3
-```
+Token server url.
-Choose `LyveCloud` as S3 provider
+Leave blank to use the provider defaults.
-```
-Choose your S3 provider.
-Choose a number from below, or type in your own value.
-Press Enter to leave empty.
-[snip]
-XX / Seagate Lyve Cloud
- \ (LyveCloud)
-[snip]
-provider> LyveCloud
-```
+Properties:
-Take the default (just press enter) to enter access key and secret in the config file.
+- Config: token_url
+- Env Var: RCLONE_BOX_TOKEN_URL
+- Type: string
+- Required: false
-```
-Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
-Only applies if access_key_id and secret_access_key is blank.
-Choose a number from below, or type in your own boolean value (true or false).
-Press Enter for the default (false).
- 1 / Enter AWS credentials in the next step.
- \ (false)
- 2 / Get AWS credentials from the environment (env vars or IAM).
- \ (true)
-env_auth>
-```
+#### --box-root-folder-id
-```
-AWS Access Key ID.
-Leave blank for anonymous access or runtime credentials.
-Enter a value. Press Enter to leave empty.
-access_key_id> XXX
-```
+Fill in for rclone to use a non root folder as its starting point.
-```
-AWS Secret Access Key (password).
-Leave blank for anonymous access or runtime credentials.
-Enter a value. Press Enter to leave empty.
-secret_access_key> YYY
-```
+Properties:
-Leave region blank
+- Config: root_folder_id
+- Env Var: RCLONE_BOX_ROOT_FOLDER_ID
+- Type: string
+- Default: "0"
-```
-Region to connect to.
-Leave blank if you are using an S3 clone and you don't have a region.
-Choose a number from below, or type in your own value.
-Press Enter to leave empty.
- / Use this if unsure.
- 1 | Will use v4 signatures and an empty region.
- \ ()
- / Use this only if v4 signatures don't work.
- 2 | E.g. pre Jewel/v10 CEPH.
- \ (other-v2-signature)
-region>
-```
+#### --box-upload-cutoff
-Choose an endpoint from the list
+Cutoff for switching to multipart upload (>= 50 MiB).
-```
-Endpoint for S3 API.
-Required when using an S3 clone.
-Choose a number from below, or type in your own value.
-Press Enter to leave empty.
- 1 / Seagate Lyve Cloud US East 1 (Virginia)
- \ (s3.us-east-1.lyvecloud.seagate.com)
- 2 / Seagate Lyve Cloud US West 1 (California)
- \ (s3.us-west-1.lyvecloud.seagate.com)
- 3 / Seagate Lyve Cloud AP Southeast 1 (Singapore)
- \ (s3.ap-southeast-1.lyvecloud.seagate.com)
-endpoint> 1
-```
+Properties:
-Leave location constraint blank
+- Config: upload_cutoff
+- Env Var: RCLONE_BOX_UPLOAD_CUTOFF
+- Type: SizeSuffix
+- Default: 50Mi
-```
-Location constraint - must be set to match the Region.
-Leave blank if not sure. Used when creating buckets only.
-Enter a value. Press Enter to leave empty.
-location_constraint>
-```
+#### --box-commit-retries
-Choose default ACL (`private`).
+Max number of times to try committing a multipart file.
-```
-Canned ACL used when creating buckets and storing or copying objects.
-This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.
-For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
-Note that this ACL is applied when server-side copying objects as S3
-doesn't copy the ACL from the source but rather writes a fresh one.
-Choose a number from below, or type in your own value.
-Press Enter to leave empty.
- / Owner gets FULL_CONTROL.
- 1 | No one else has access rights (default).
- \ (private)
-[snip]
-acl>
-```
+Properties:
-And the config file should end up looking like this:
+- Config: commit_retries
+- Env Var: RCLONE_BOX_COMMIT_RETRIES
+- Type: int
+- Default: 100
-```
-[remote]
-type = s3
-provider = LyveCloud
-access_key_id = XXX
-secret_access_key = YYY
-endpoint = s3.us-east-1.lyvecloud.seagate.com
-```
+#### --box-list-chunk
-### SeaweedFS
+Size of listing chunk 1-1000.
-[SeaweedFS](https://github.com/chrislusf/seaweedfs/) is a distributed storage system for
-blobs, objects, files, and data lake, with O(1) disk seek and a scalable file metadata store.
-It has an S3 compatible object storage interface. SeaweedFS can also act as a
-[gateway to remote S3 compatible object store](https://github.com/chrislusf/seaweedfs/wiki/Gateway-to-Remote-Object-Storage)
-to cache data and metadata with asynchronous write back, for fast local speed and minimize access cost.
+Properties:
-Assuming the SeaweedFS are configured with `weed shell` as such:
-```
-> s3.bucket.create -name foo
-> s3.configure -access_key=any -secret_key=any -buckets=foo -user=me -actions=Read,Write,List,Tagging,Admin -apply
-{
- "identities": [
- {
- "name": "me",
- "credentials": [
- {
- "accessKey": "any",
- "secretKey": "any"
- }
- ],
- "actions": [
- "Read:foo",
- "Write:foo",
- "List:foo",
- "Tagging:foo",
- "Admin:foo"
- ]
- }
- ]
-}
-```
+- Config: list_chunk
+- Env Var: RCLONE_BOX_LIST_CHUNK
+- Type: int
+- Default: 1000
-To use rclone with SeaweedFS, above configuration should end up with something like this in
-your config:
+#### --box-owned-by
-```
-[seaweedfs_s3]
-type = s3
-provider = SeaweedFS
-access_key_id = any
-secret_access_key = any
-endpoint = localhost:8333
-```
+Only show items owned by the login (email address) passed in.
-So once set up, for example to copy files into a bucket
+Properties:
-```
-rclone copy /path/to/files seaweedfs_s3:foo
-```
+- Config: owned_by
+- Env Var: RCLONE_BOX_OWNED_BY
+- Type: string
+- Required: false
-### Wasabi
+#### --box-impersonate
-[Wasabi](https://wasabi.com) is a cloud-based object storage service for a
-broad range of applications and use cases. Wasabi is designed for
-individuals and organizations that require a high-performance,
-reliable, and secure data storage infrastructure at minimal cost.
+Impersonate this user ID when using a service account.
-Wasabi provides an S3 interface which can be configured for use with
-rclone like this.
+Setting this flag allows rclone, when using a JWT service account, to
+act on behalf of another user by setting the as-user header.
-```
-No remotes found, make a new one?
-n) New remote
-s) Set configuration password
-n/s> n
-name> wasabi
-Type of storage to configure.
-Choose a number from below, or type in your own value
-[snip]
-XX / Amazon S3 (also Dreamhost, Ceph, ChinaMobile, ArvanCloud, Minio, Liara)
- \ "s3"
-[snip]
-Storage> s3
-Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars). Only applies if access_key_id and secret_access_key is blank.
-Choose a number from below, or type in your own value
- 1 / Enter AWS credentials in the next step
- \ "false"
- 2 / Get AWS credentials from the environment (env vars or IAM)
- \ "true"
-env_auth> 1
-AWS Access Key ID - leave blank for anonymous access or runtime credentials.
-access_key_id> YOURACCESSKEY
-AWS Secret Access Key (password) - leave blank for anonymous access or runtime credentials.
-secret_access_key> YOURSECRETACCESSKEY
-Region to connect to.
-Choose a number from below, or type in your own value
- / The default endpoint - a good choice if you are unsure.
- 1 | US Region, Northern Virginia, or Pacific Northwest.
- | Leave location constraint empty.
- \ "us-east-1"
-[snip]
-region> us-east-1
-Endpoint for S3 API.
-Leave blank if using AWS to use the default endpoint for the region.
-Specify if using an S3 clone such as Ceph.
-endpoint> s3.wasabisys.com
-Location constraint - must be set to match the Region. Used when creating buckets only.
-Choose a number from below, or type in your own value
- 1 / Empty for US Region, Northern Virginia, or Pacific Northwest.
- \ ""
-[snip]
-location_constraint>
-Canned ACL used when creating buckets and/or storing objects in S3.
-For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
-Choose a number from below, or type in your own value
- 1 / Owner gets FULL_CONTROL. No one else has access rights (default).
- \ "private"
-[snip]
-acl>
-The server-side encryption algorithm used when storing this object in S3.
-Choose a number from below, or type in your own value
- 1 / None
- \ ""
- 2 / AES256
- \ "AES256"
-server_side_encryption>
-The storage class to use when storing objects in S3.
-Choose a number from below, or type in your own value
- 1 / Default
- \ ""
- 2 / Standard storage class
- \ "STANDARD"
- 3 / Reduced redundancy storage class
- \ "REDUCED_REDUNDANCY"
- 4 / Standard Infrequent Access storage class
- \ "STANDARD_IA"
-storage_class>
-Remote config
---------------------
-[wasabi]
-env_auth = false
-access_key_id = YOURACCESSKEY
-secret_access_key = YOURSECRETACCESSKEY
-region = us-east-1
-endpoint = s3.wasabisys.com
-location_constraint =
-acl =
-server_side_encryption =
-storage_class =
---------------------
-y) Yes this is OK
-e) Edit this remote
-d) Delete this remote
-y/e/d> y
-```
+The user ID is the Box identifier for a user. User IDs can found for
+any user via the GET /users endpoint, which is only available to
+admins, or by calling the GET /users/me endpoint with an authenticated
+user session.
+
+See: https://developer.box.com/guides/authentication/jwt/as-user/
+
+
+Properties:
+
+- Config: impersonate
+- Env Var: RCLONE_BOX_IMPERSONATE
+- Type: string
+- Required: false
-This will leave the config file looking like this.
+#### --box-encoding
-```
-[wasabi]
-type = s3
-provider = Wasabi
-env_auth = false
-access_key_id = YOURACCESSKEY
-secret_access_key = YOURSECRETACCESSKEY
-region =
-endpoint = s3.wasabisys.com
-location_constraint =
-acl =
-server_side_encryption =
-storage_class =
-```
+The encoding for the backend.
-### Alibaba OSS {#alibaba-oss}
+See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info.
-Here is an example of making an [Alibaba Cloud (Aliyun) OSS](https://www.alibabacloud.com/product/oss/)
-configuration. First run:
+Properties:
- rclone config
+- Config: encoding
+- Env Var: RCLONE_BOX_ENCODING
+- Type: Encoding
+- Default: Slash,BackSlash,Del,Ctl,RightSpace,InvalidUtf8,Dot
-This will guide you through an interactive setup process.
-```
-No remotes found, make a new one?
-n) New remote
-s) Set configuration password
-q) Quit config
-n/s/q> n
-name> oss
-Type of storage to configure.
-Enter a string value. Press Enter for the default ("").
-Choose a number from below, or type in your own value
-[snip]
- 4 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, ChinaMobile, ArvanCloud, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, Liara, Minio, and Tencent COS
- \ "s3"
-[snip]
-Storage> s3
-Choose your S3 provider.
-Enter a string value. Press Enter for the default ("").
-Choose a number from below, or type in your own value
- 1 / Amazon Web Services (AWS) S3
- \ "AWS"
- 2 / Alibaba Cloud Object Storage System (OSS) formerly Aliyun
- \ "Alibaba"
- 3 / Ceph Object Storage
- \ "Ceph"
-[snip]
-provider> Alibaba
-Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
-Only applies if access_key_id and secret_access_key is blank.
-Enter a boolean value (true or false). Press Enter for the default ("false").
-Choose a number from below, or type in your own value
- 1 / Enter AWS credentials in the next step
- \ "false"
- 2 / Get AWS credentials from the environment (env vars or IAM)
- \ "true"
-env_auth> 1
-AWS Access Key ID.
-Leave blank for anonymous access or runtime credentials.
-Enter a string value. Press Enter for the default ("").
-access_key_id> accesskeyid
-AWS Secret Access Key (password)
-Leave blank for anonymous access or runtime credentials.
-Enter a string value. Press Enter for the default ("").
-secret_access_key> secretaccesskey
-Endpoint for OSS API.
-Enter a string value. Press Enter for the default ("").
-Choose a number from below, or type in your own value
- 1 / East China 1 (Hangzhou)
- \ "oss-cn-hangzhou.aliyuncs.com"
- 2 / East China 2 (Shanghai)
- \ "oss-cn-shanghai.aliyuncs.com"
- 3 / North China 1 (Qingdao)
- \ "oss-cn-qingdao.aliyuncs.com"
-[snip]
-endpoint> 1
-Canned ACL used when creating buckets and storing or copying objects.
-Note that this ACL is applied when server-side copying objects as S3
-doesn't copy the ACL from the source but rather writes a fresh one.
-Enter a string value. Press Enter for the default ("").
-Choose a number from below, or type in your own value
- 1 / Owner gets FULL_CONTROL. No one else has access rights (default).
- \ "private"
- 2 / Owner gets FULL_CONTROL. The AllUsers group gets READ access.
- \ "public-read"
- / Owner gets FULL_CONTROL. The AllUsers group gets READ and WRITE access.
-[snip]
-acl> 1
-The storage class to use when storing new objects in OSS.
-Enter a string value. Press Enter for the default ("").
-Choose a number from below, or type in your own value
- 1 / Default
- \ ""
- 2 / Standard storage class
- \ "STANDARD"
- 3 / Archive storage mode.
- \ "GLACIER"
- 4 / Infrequent access storage mode.
- \ "STANDARD_IA"
-storage_class> 1
-Edit advanced config? (y/n)
-y) Yes
-n) No
-y/n> n
-Remote config
---------------------
-[oss]
-type = s3
-provider = Alibaba
-env_auth = false
-access_key_id = accesskeyid
-secret_access_key = secretaccesskey
-endpoint = oss-cn-hangzhou.aliyuncs.com
-acl = private
-storage_class = Standard
---------------------
-y) Yes this is OK
-e) Edit this remote
-d) Delete this remote
-y/e/d> y
-```
+## Limitations
-### China Mobile Ecloud Elastic Object Storage (EOS) {#china-mobile-ecloud-eos}
+Note that Box is case insensitive so you can't have a file called
+"Hello.doc" and one called "hello.doc".
-Here is an example of making an [China Mobile Ecloud Elastic Object Storage (EOS)](https:///ecloud.10086.cn/home/product-introduction/eos/)
-configuration. First run:
+Box file names can't have the `\` character in. rclone maps this to
+and from an identical looking unicode equivalent `ï¼¼` (U+FF3C Fullwidth
+Reverse Solidus).
- rclone config
+Box only supports filenames up to 255 characters in length.
-This will guide you through an interactive setup process.
+Box has [API rate limits](https://developer.box.com/guides/api-calls/permissions-and-errors/rate-limits/) that sometimes reduce the speed of rclone.
-```
-No remotes found, make a new one?
-n) New remote
-s) Set configuration password
-q) Quit config
-n/s/q> n
-name> ChinaMobile
-Option Storage.
-Type of storage to configure.
-Choose a number from below, or type in your own value.
- ...
- 5 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, ChinaMobile, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, Lyve Cloud, Minio, RackCorp, SeaweedFS, and Tencent COS
- \ (s3)
- ...
-Storage> s3
-Option provider.
-Choose your S3 provider.
-Choose a number from below, or type in your own value.
-Press Enter to leave empty.
- ...
- 4 / China Mobile Ecloud Elastic Object Storage (EOS)
- \ (ChinaMobile)
- ...
-provider> ChinaMobile
-Option env_auth.
-Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
-Only applies if access_key_id and secret_access_key is blank.
-Choose a number from below, or type in your own boolean value (true or false).
-Press Enter for the default (false).
- 1 / Enter AWS credentials in the next step.
- \ (false)
- 2 / Get AWS credentials from the environment (env vars or IAM).
- \ (true)
-env_auth>
-Option access_key_id.
-AWS Access Key ID.
-Leave blank for anonymous access or runtime credentials.
-Enter a value. Press Enter to leave empty.
-access_key_id> accesskeyid
-Option secret_access_key.
-AWS Secret Access Key (password).
-Leave blank for anonymous access or runtime credentials.
-Enter a value. Press Enter to leave empty.
-secret_access_key> secretaccesskey
-Option endpoint.
-Endpoint for China Mobile Ecloud Elastic Object Storage (EOS) API.
-Choose a number from below, or type in your own value.
-Press Enter to leave empty.
- / The default endpoint - a good choice if you are unsure.
- 1 | East China (Suzhou)
- \ (eos-wuxi-1.cmecloud.cn)
- 2 / East China (Jinan)
- \ (eos-jinan-1.cmecloud.cn)
- 3 / East China (Hangzhou)
- \ (eos-ningbo-1.cmecloud.cn)
- 4 / East China (Shanghai-1)
- \ (eos-shanghai-1.cmecloud.cn)
- 5 / Central China (Zhengzhou)
- \ (eos-zhengzhou-1.cmecloud.cn)
- 6 / Central China (Changsha-1)
- \ (eos-hunan-1.cmecloud.cn)
- 7 / Central China (Changsha-2)
- \ (eos-zhuzhou-1.cmecloud.cn)
- 8 / South China (Guangzhou-2)
- \ (eos-guangzhou-1.cmecloud.cn)
- 9 / South China (Guangzhou-3)
- \ (eos-dongguan-1.cmecloud.cn)
-10 / North China (Beijing-1)
- \ (eos-beijing-1.cmecloud.cn)
-11 / North China (Beijing-2)
- \ (eos-beijing-2.cmecloud.cn)
-12 / North China (Beijing-3)
- \ (eos-beijing-4.cmecloud.cn)
-13 / North China (Huhehaote)
- \ (eos-huhehaote-1.cmecloud.cn)
-14 / Southwest China (Chengdu)
- \ (eos-chengdu-1.cmecloud.cn)
-15 / Southwest China (Chongqing)
- \ (eos-chongqing-1.cmecloud.cn)
-16 / Southwest China (Guiyang)
- \ (eos-guiyang-1.cmecloud.cn)
-17 / Nouthwest China (Xian)
- \ (eos-xian-1.cmecloud.cn)
-18 / Yunnan China (Kunming)
- \ (eos-yunnan.cmecloud.cn)
-19 / Yunnan China (Kunming-2)
- \ (eos-yunnan-2.cmecloud.cn)
-20 / Tianjin China (Tianjin)
- \ (eos-tianjin-1.cmecloud.cn)
-21 / Jilin China (Changchun)
- \ (eos-jilin-1.cmecloud.cn)
-22 / Hubei China (Xiangyan)
- \ (eos-hubei-1.cmecloud.cn)
-23 / Jiangxi China (Nanchang)
- \ (eos-jiangxi-1.cmecloud.cn)
-24 / Gansu China (Lanzhou)
- \ (eos-gansu-1.cmecloud.cn)
-25 / Shanxi China (Taiyuan)
- \ (eos-shanxi-1.cmecloud.cn)
-26 / Liaoning China (Shenyang)
- \ (eos-liaoning-1.cmecloud.cn)
-27 / Hebei China (Shijiazhuang)
- \ (eos-hebei-1.cmecloud.cn)
-28 / Fujian China (Xiamen)
- \ (eos-fujian-1.cmecloud.cn)
-29 / Guangxi China (Nanning)
- \ (eos-guangxi-1.cmecloud.cn)
-30 / Anhui China (Huainan)
- \ (eos-anhui-1.cmecloud.cn)
-endpoint> 1
-Option location_constraint.
-Location constraint - must match endpoint.
-Used when creating buckets only.
-Choose a number from below, or type in your own value.
-Press Enter to leave empty.
- 1 / East China (Suzhou)
- \ (wuxi1)
- 2 / East China (Jinan)
- \ (jinan1)
- 3 / East China (Hangzhou)
- \ (ningbo1)
- 4 / East China (Shanghai-1)
- \ (shanghai1)
- 5 / Central China (Zhengzhou)
- \ (zhengzhou1)
- 6 / Central China (Changsha-1)
- \ (hunan1)
- 7 / Central China (Changsha-2)
- \ (zhuzhou1)
- 8 / South China (Guangzhou-2)
- \ (guangzhou1)
- 9 / South China (Guangzhou-3)
- \ (dongguan1)
-10 / North China (Beijing-1)
- \ (beijing1)
-11 / North China (Beijing-2)
- \ (beijing2)
-12 / North China (Beijing-3)
- \ (beijing4)
-13 / North China (Huhehaote)
- \ (huhehaote1)
-14 / Southwest China (Chengdu)
- \ (chengdu1)
-15 / Southwest China (Chongqing)
- \ (chongqing1)
-16 / Southwest China (Guiyang)
- \ (guiyang1)
-17 / Nouthwest China (Xian)
- \ (xian1)
-18 / Yunnan China (Kunming)
- \ (yunnan)
-19 / Yunnan China (Kunming-2)
- \ (yunnan2)
-20 / Tianjin China (Tianjin)
- \ (tianjin1)
-21 / Jilin China (Changchun)
- \ (jilin1)
-22 / Hubei China (Xiangyan)
- \ (hubei1)
-23 / Jiangxi China (Nanchang)
- \ (jiangxi1)
-24 / Gansu China (Lanzhou)
- \ (gansu1)
-25 / Shanxi China (Taiyuan)
- \ (shanxi1)
-26 / Liaoning China (Shenyang)
- \ (liaoning1)
-27 / Hebei China (Shijiazhuang)
- \ (hebei1)
-28 / Fujian China (Xiamen)
- \ (fujian1)
-29 / Guangxi China (Nanning)
- \ (guangxi1)
-30 / Anhui China (Huainan)
- \ (anhui1)
-location_constraint> 1
-Option acl.
-Canned ACL used when creating buckets and storing or copying objects.
-This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.
-For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
-Note that this ACL is applied when server-side copying objects as S3
-doesn't copy the ACL from the source but rather writes a fresh one.
-Choose a number from below, or type in your own value.
-Press Enter to leave empty.
- / Owner gets FULL_CONTROL.
- 1 | No one else has access rights (default).
- \ (private)
- / Owner gets FULL_CONTROL.
- 2 | The AllUsers group gets READ access.
- \ (public-read)
- / Owner gets FULL_CONTROL.
- 3 | The AllUsers group gets READ and WRITE access.
- | Granting this on a bucket is generally not recommended.
- \ (public-read-write)
- / Owner gets FULL_CONTROL.
- 4 | The AuthenticatedUsers group gets READ access.
- \ (authenticated-read)
- / Object owner gets FULL_CONTROL.
-acl> private
-Option server_side_encryption.
-The server-side encryption algorithm used when storing this object in S3.
-Choose a number from below, or type in your own value.
-Press Enter to leave empty.
- 1 / None
- \ ()
- 2 / AES256
- \ (AES256)
-server_side_encryption>
-Option storage_class.
-The storage class to use when storing new objects in ChinaMobile.
-Choose a number from below, or type in your own value.
-Press Enter to leave empty.
- 1 / Default
- \ ()
- 2 / Standard storage class
- \ (STANDARD)
- 3 / Archive storage mode
- \ (GLACIER)
- 4 / Infrequent access storage mode
- \ (STANDARD_IA)
-storage_class>
-Edit advanced config?
-y) Yes
-n) No (default)
-y/n> n
---------------------
-[ChinaMobile]
-type = s3
-provider = ChinaMobile
-access_key_id = accesskeyid
-secret_access_key = secretaccesskey
-endpoint = eos-wuxi-1.cmecloud.cn
-location_constraint = wuxi1
-acl = private
---------------------
-y) Yes this is OK (default)
-e) Edit this remote
-d) Delete this remote
-y/e/d> y
-```
-### Leviia Cloud Object Storage {#leviia}
+`rclone about` is not supported by the Box backend. Backends without
+this capability cannot determine free space for an rclone mount or
+use policy `mfs` (most free space) as a member of an rclone union
+remote.
-[Leviia Object Storage](https://www.leviia.com/object-storage/), backup and secure your data in a 100% French cloud, independent of GAFAM..
+See [List of backends that do not support rclone about](https://rclone.org/overview/#optional-features) and [rclone about](https://rclone.org/commands/rclone_about/)
-To configure access to Leviia, follow the steps below:
+## Get your own Box App ID
-1. Run `rclone config` and select `n` for a new remote.
+Here is how to create your own Box App ID for rclone:
-```
-rclone config
-No remotes found, make a new one?
-n) New remote
-s) Set configuration password
-q) Quit config
-n/s/q> n
-```
+1. Go to the [Box Developer Console](https://app.box.com/developers/console)
+and login, then click `My Apps` on the sidebar. Click `Create New App`
+and select `Custom App`.
-2. Give the name of the configuration. For example, name it 'leviia'.
+2. In the first screen on the box that pops up, you can pretty much enter
+whatever you want. The `App Name` can be whatever. For `Purpose` choose
+automation to avoid having to fill out anything else. Click `Next`.
-```
-name> leviia
-```
+3. In the second screen of the creation screen, select
+`User Authentication (OAuth 2.0)`. Then click `Create App`.
-3. Select `s3` storage.
+4. You should now be on the `Configuration` tab of your new app. If not,
+click on it at the top of the webpage. Copy down `Client ID`
+and `Client Secret`, you'll need those for rclone.
-```
-Choose a number from below, or type in your own value
- 1 / 1Fichier
- \ (fichier)
- 2 / Akamai NetStorage
- \ (netstorage)
- 3 / Alias for an existing remote
- \ (alias)
- 4 / Amazon Drive
- \ (amazon cloud drive)
- 5 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, China Mobile, Cloudflare, ArvanCloud, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, IDrive e2, Liara, Lyve Cloud, Minio, Netease, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Synology, Tencent COS, Qiniu and Wasabi
- \ (s3)
-[snip]
-Storage> s3
-```
+5. Under "OAuth 2.0 Redirect URI", add `http://127.0.0.1:53682/`
-4. Select `Leviia` provider.
-```
-Choose a number from below, or type in your own value
-1 / Amazon Web Services (AWS) S3
- \ "AWS"
-[snip]
-15 / Leviia Object Storage
- \ (Leviia)
-[snip]
-provider> Leviia
-```
+6. For `Application Scopes`, select `Read all files and folders stored in Box`
+and `Write all files and folders stored in box` (assuming you want to do both).
+Leave others unchecked. Click `Save Changes` at the top right.
-5. Enter your SecretId and SecretKey of Leviia.
+# Cache
-```
-Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
-Only applies if access_key_id and secret_access_key is blank.
-Enter a boolean value (true or false). Press Enter for the default ("false").
-Choose a number from below, or type in your own value
- 1 / Enter AWS credentials in the next step
- \ "false"
- 2 / Get AWS credentials from the environment (env vars or IAM)
- \ "true"
-env_auth> 1
-AWS Access Key ID.
-Leave blank for anonymous access or runtime credentials.
-Enter a string value. Press Enter for the default ("").
-access_key_id> ZnIx.xxxxxxxxxxxxxxx
-AWS Secret Access Key (password)
-Leave blank for anonymous access or runtime credentials.
-Enter a string value. Press Enter for the default ("").
-secret_access_key> xxxxxxxxxxx
-```
+The `cache` remote wraps another existing remote and stores file structure
+and its data for long running tasks like `rclone mount`.
-6. Select endpoint for Leviia.
+## Status
-```
- / The default endpoint
- 1 | Leviia.
- \ (s3.leviia.com)
-[snip]
-endpoint> 1
-```
-7. Choose acl.
+The cache backend code is working but it currently doesn't
+have a maintainer so there are [outstanding bugs](https://github.com/rclone/rclone/issues?q=is%3Aopen+is%3Aissue+label%3Abug+label%3A%22Remote%3A+Cache%22) which aren't getting fixed.
-```
-Note that this ACL is applied when server-side copying objects as S3
-doesn't copy the ACL from the source but rather writes a fresh one.
-Enter a string value. Press Enter for the default ("").
-Choose a number from below, or type in your own value
- / Owner gets FULL_CONTROL.
- 1 | No one else has access rights (default).
- \ (private)
- / Owner gets FULL_CONTROL.
- 2 | The AllUsers group gets READ access.
- \ (public-read)
-[snip]
-acl> 1
-Edit advanced config? (y/n)
-y) Yes
-n) No (default)
-y/n> n
-Remote config
---------------------
-[leviia]
-- type: s3
-- provider: Leviia
-- access_key_id: ZnIx.xxxxxxx
-- secret_access_key: xxxxxxxx
-- endpoint: s3.leviia.com
-- acl: private
---------------------
-y) Yes this is OK (default)
-e) Edit this remote
-d) Delete this remote
-y/e/d> y
-Current remotes:
+The cache backend is due to be phased out in favour of the VFS caching
+layer eventually which is more tightly integrated into rclone.
-Name Type
-==== ====
-leviia s3
-```
-### Liara {#liara-cloud}
+Until this happens we recommend only using the cache backend if you
+find you can't work without it. There are many docs online describing
+the use of the cache backend to minimize API hits and by-and-large
+these are out of date and the cache backend isn't needed in those
+scenarios any more.
-Here is an example of making a [Liara Object Storage](https://liara.ir/landing/object-storage)
-configuration. First run:
+## Configuration
- rclone config
+To get started you just need to have an existing remote which can be configured
+with `cache`.
-This will guide you through an interactive setup process.
+Here is an example of how to make a remote called `test-cache`. First run:
+
+ rclone config
+
+This will guide you through an interactive setup process:
```
No remotes found, make a new one?
n) New remote
+r) Rename remote
+c) Copy remote
s) Set configuration password
-n/s> n
-name> Liara
+q) Quit config
+n/r/c/s/q> n
+name> test-cache
Type of storage to configure.
Choose a number from below, or type in your own value
[snip]
-XX / Amazon S3 (also Dreamhost, Ceph, ChinaMobile, ArvanCloud, Liara, Minio)
- \ "s3"
-[snip]
-Storage> s3
-Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars). Only applies if access_key_id and secret_access_key is blank.
-Choose a number from below, or type in your own value
- 1 / Enter AWS credentials in the next step
- \ "false"
- 2 / Get AWS credentials from the environment (env vars or IAM)
- \ "true"
-env_auth> 1
-AWS Access Key ID - leave blank for anonymous access or runtime credentials.
-access_key_id> YOURACCESSKEY
-AWS Secret Access Key (password) - leave blank for anonymous access or runtime credentials.
-secret_access_key> YOURSECRETACCESSKEY
-Region to connect to.
-Choose a number from below, or type in your own value
- / The default endpoint
- 1 | US Region, Northern Virginia, or Pacific Northwest.
- | Leave location constraint empty.
- \ "us-east-1"
+XX / Cache a remote
+ \ "cache"
[snip]
-region>
-Endpoint for S3 API.
-Leave blank if using Liara to use the default endpoint for the region.
-Specify if using an S3 clone such as Ceph.
-endpoint> storage.iran.liara.space
-Canned ACL used when creating buckets and/or storing objects in S3.
-For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
+Storage> cache
+Remote to cache.
+Normally should contain a ':' and a path, e.g. "myremote:path/to/dir",
+"myremote:bucket" or maybe "myremote:" (not recommended).
+remote> local:/test
+Optional: The URL of the Plex server
+plex_url> http://127.0.0.1:32400
+Optional: The username of the Plex user
+plex_username> dummyusername
+Optional: The password of the Plex user
+y) Yes type in my own password
+g) Generate random password
+n) No leave this optional password blank
+y/g/n> y
+Enter the password:
+password:
+Confirm the password:
+password:
+The size of a chunk. Lower value good for slow connections but can affect seamless reading.
+Default: 5M
Choose a number from below, or type in your own value
- 1 / Owner gets FULL_CONTROL. No one else has access rights (default).
- \ "private"
-[snip]
-acl>
-The server-side encryption algorithm used when storing this object in S3.
+ 1 / 1 MiB
+ \ "1M"
+ 2 / 5 MiB
+ \ "5M"
+ 3 / 10 MiB
+ \ "10M"
+chunk_size> 2
+How much time should object info (file size, file hashes, etc.) be stored in cache. Use a very high value if you don't plan on changing the source FS from outside the cache.
+Accepted units are: "s", "m", "h".
+Default: 5m
Choose a number from below, or type in your own value
- 1 / None
- \ ""
- 2 / AES256
- \ "AES256"
-server_side_encryption>
-The storage class to use when storing objects in S3.
+ 1 / 1 hour
+ \ "1h"
+ 2 / 24 hours
+ \ "24h"
+ 3 / 24 hours
+ \ "48h"
+info_age> 2
+The maximum size of stored chunks. When the storage grows beyond this size, the oldest chunks will be deleted.
+Default: 10G
Choose a number from below, or type in your own value
- 1 / Default
- \ ""
- 2 / Standard storage class
- \ "STANDARD"
-storage_class>
+ 1 / 500 MiB
+ \ "500M"
+ 2 / 1 GiB
+ \ "1G"
+ 3 / 10 GiB
+ \ "10G"
+chunk_total_size> 3
Remote config
--------------------
-[Liara]
-env_auth = false
-access_key_id = YOURACCESSKEY
-secret_access_key = YOURSECRETACCESSKEY
-endpoint = storage.iran.liara.space
-location_constraint =
-acl =
-server_side_encryption =
-storage_class =
---------------------
-y) Yes this is OK
-e) Edit this remote
-d) Delete this remote
-y/e/d> y
+[test-cache]
+remote = local:/test
+plex_url = http://127.0.0.1:32400
+plex_username = dummyusername
+plex_password = *** ENCRYPTED ***
+chunk_size = 5M
+info_age = 48h
+chunk_total_size = 10G
```
-This will leave the config file looking like this.
+You can then use it like this,
-```
-[Liara]
-type = s3
-provider = Liara
-env_auth = false
-access_key_id = YOURACCESSKEY
-secret_access_key = YOURSECRETACCESSKEY
-region =
-endpoint = storage.iran.liara.space
-location_constraint =
-acl =
-server_side_encryption =
-storage_class =
-```
+List directories in top level of your drive
-### ArvanCloud {#arvan-cloud}
+ rclone lsd test-cache:
-[ArvanCloud](https://www.arvancloud.com/en/products/cloud-storage) ArvanCloud Object Storage goes beyond the limited traditional file storage.
-It gives you access to backup and archived files and allows sharing.
-Files like profile image in the app, images sent by users or scanned documents can be stored securely and easily in our Object Storage service.
+List all the files in your drive
-ArvanCloud provides an S3 interface which can be configured for use with
-rclone like this.
+ rclone ls test-cache:
-```
-No remotes found, make a new one?
-n) New remote
-s) Set configuration password
-n/s> n
-name> ArvanCloud
-Type of storage to configure.
-Choose a number from below, or type in your own value
-[snip]
-XX / Amazon S3 (also Dreamhost, Ceph, ChinaMobile, ArvanCloud, Liara, Minio)
- \ "s3"
-[snip]
-Storage> s3
-Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars). Only applies if access_key_id and secret_access_key is blank.
-Choose a number from below, or type in your own value
- 1 / Enter AWS credentials in the next step
- \ "false"
- 2 / Get AWS credentials from the environment (env vars or IAM)
- \ "true"
-env_auth> 1
-AWS Access Key ID - leave blank for anonymous access or runtime credentials.
-access_key_id> YOURACCESSKEY
-AWS Secret Access Key (password) - leave blank for anonymous access or runtime credentials.
-secret_access_key> YOURSECRETACCESSKEY
-Region to connect to.
-Choose a number from below, or type in your own value
- / The default endpoint - a good choice if you are unsure.
- 1 | US Region, Northern Virginia, or Pacific Northwest.
- | Leave location constraint empty.
- \ "us-east-1"
-[snip]
-region>
-Endpoint for S3 API.
-Leave blank if using ArvanCloud to use the default endpoint for the region.
-Specify if using an S3 clone such as Ceph.
-endpoint> s3.arvanstorage.com
-Location constraint - must be set to match the Region. Used when creating buckets only.
-Choose a number from below, or type in your own value
- 1 / Empty for Iran-Tehran Region.
- \ ""
-[snip]
-location_constraint>
-Canned ACL used when creating buckets and/or storing objects in S3.
-For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
-Choose a number from below, or type in your own value
- 1 / Owner gets FULL_CONTROL. No one else has access rights (default).
- \ "private"
-[snip]
-acl>
-The server-side encryption algorithm used when storing this object in S3.
-Choose a number from below, or type in your own value
- 1 / None
- \ ""
- 2 / AES256
- \ "AES256"
-server_side_encryption>
-The storage class to use when storing objects in S3.
-Choose a number from below, or type in your own value
- 1 / Default
- \ ""
- 2 / Standard storage class
- \ "STANDARD"
-storage_class>
-Remote config
---------------------
-[ArvanCloud]
-env_auth = false
-access_key_id = YOURACCESSKEY
-secret_access_key = YOURSECRETACCESSKEY
-region = ir-thr-at1
-endpoint = s3.arvanstorage.com
-location_constraint =
-acl =
-server_side_encryption =
-storage_class =
---------------------
-y) Yes this is OK
-e) Edit this remote
-d) Delete this remote
-y/e/d> y
-```
+To start a cached mount
-This will leave the config file looking like this.
+ rclone mount --allow-other test-cache: /var/tmp/test-cache
-```
-[ArvanCloud]
-type = s3
-provider = ArvanCloud
-env_auth = false
-access_key_id = YOURACCESSKEY
-secret_access_key = YOURSECRETACCESSKEY
-region =
-endpoint = s3.arvanstorage.com
-location_constraint =
-acl =
-server_side_encryption =
-storage_class =
-```
+### Write Features ###
-### Tencent COS {#tencent-cos}
+### Offline uploading ###
-[Tencent Cloud Object Storage (COS)](https://intl.cloud.tencent.com/product/cos) is a distributed storage service offered by Tencent Cloud for unstructured data. It is secure, stable, massive, convenient, low-delay and low-cost.
+In an effort to make writing through cache more reliable, the backend
+now supports this feature which can be activated by specifying a
+`cache-tmp-upload-path`.
-To configure access to Tencent COS, follow the steps below:
+A files goes through these states when using this feature:
-1. Run `rclone config` and select `n` for a new remote.
+1. An upload is started (usually by copying a file on the cache remote)
+2. When the copy to the temporary location is complete the file is part
+of the cached remote and looks and behaves like any other file (reading included)
+3. After `cache-tmp-wait-time` passes and the file is next in line, `rclone move`
+is used to move the file to the cloud provider
+4. Reading the file still works during the upload but most modifications on it will be prohibited
+5. Once the move is complete the file is unlocked for modifications as it
+becomes as any other regular file
+6. If the file is being read through `cache` when it's actually
+deleted from the temporary path then `cache` will simply swap the source
+to the cloud provider without interrupting the reading (small blip can happen though)
-```
-rclone config
-No remotes found, make a new one?
-n) New remote
-s) Set configuration password
-q) Quit config
-n/s/q> n
-```
+Files are uploaded in sequence and only one file is uploaded at a time.
+Uploads will be stored in a queue and be processed based on the order they were added.
+The queue and the temporary storage is persistent across restarts but
+can be cleared on startup with the `--cache-db-purge` flag.
-2. Give the name of the configuration. For example, name it 'cos'.
+### Write Support ###
-```
-name> cos
-```
+Writes are supported through `cache`.
+One caveat is that a mounted cache remote does not add any retry or fallback
+mechanism to the upload operation. This will depend on the implementation
+of the wrapped remote. Consider using `Offline uploading` for reliable writes.
-3. Select `s3` storage.
+One special case is covered with `cache-writes` which will cache the file
+data at the same time as the upload when it is enabled making it available
+from the cache store immediately once the upload is finished.
-```
-Choose a number from below, or type in your own value
-1 / 1Fichier
- \ "fichier"
- 2 / Alias for an existing remote
- \ "alias"
- 3 / Amazon Drive
- \ "amazon cloud drive"
- 4 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, ChinaMobile, ArvanCloud, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, Liara, Minio, and Tencent COS
- \ "s3"
-[snip]
-Storage> s3
-```
+### Read Features ###
-4. Select `TencentCOS` provider.
-```
-Choose a number from below, or type in your own value
-1 / Amazon Web Services (AWS) S3
- \ "AWS"
-[snip]
-11 / Tencent Cloud Object Storage (COS)
- \ "TencentCOS"
-[snip]
-provider> TencentCOS
-```
+#### Multiple connections ####
-5. Enter your SecretId and SecretKey of Tencent Cloud.
+To counter the high latency between a local PC where rclone is running
+and cloud providers, the cache remote can split multiple requests to the
+cloud provider for smaller file chunks and combines them together locally
+where they can be available almost immediately before the reader usually
+needs them.
-```
-Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
-Only applies if access_key_id and secret_access_key is blank.
-Enter a boolean value (true or false). Press Enter for the default ("false").
-Choose a number from below, or type in your own value
- 1 / Enter AWS credentials in the next step
- \ "false"
- 2 / Get AWS credentials from the environment (env vars or IAM)
- \ "true"
-env_auth> 1
-AWS Access Key ID.
-Leave blank for anonymous access or runtime credentials.
-Enter a string value. Press Enter for the default ("").
-access_key_id> AKIDxxxxxxxxxx
-AWS Secret Access Key (password)
-Leave blank for anonymous access or runtime credentials.
-Enter a string value. Press Enter for the default ("").
-secret_access_key> xxxxxxxxxxx
-```
+This is similar to buffering when media files are played online. Rclone
+will stay around the current marker but always try its best to stay ahead
+and prepare the data before.
-6. Select endpoint for Tencent COS. This is the standard endpoint for different region.
+#### Plex Integration ####
-```
- 1 / Beijing Region.
- \ "cos.ap-beijing.myqcloud.com"
- 2 / Nanjing Region.
- \ "cos.ap-nanjing.myqcloud.com"
- 3 / Shanghai Region.
- \ "cos.ap-shanghai.myqcloud.com"
- 4 / Guangzhou Region.
- \ "cos.ap-guangzhou.myqcloud.com"
-[snip]
-endpoint> 4
-```
+There is a direct integration with Plex which allows cache to detect during reading
+if the file is in playback or not. This helps cache to adapt how it queries
+the cloud provider depending on what is needed for.
-7. Choose acl and storage class.
+Scans will have a minimum amount of workers (1) while in a confirmed playback cache
+will deploy the configured number of workers.
-```
-Note that this ACL is applied when server-side copying objects as S3
-doesn't copy the ACL from the source but rather writes a fresh one.
-Enter a string value. Press Enter for the default ("").
-Choose a number from below, or type in your own value
- 1 / Owner gets Full_CONTROL. No one else has access rights (default).
- \ "default"
-[snip]
-acl> 1
-The storage class to use when storing new objects in Tencent COS.
-Enter a string value. Press Enter for the default ("").
-Choose a number from below, or type in your own value
- 1 / Default
- \ ""
-[snip]
-storage_class> 1
-Edit advanced config? (y/n)
-y) Yes
-n) No (default)
-y/n> n
-Remote config
---------------------
-[cos]
-type = s3
-provider = TencentCOS
-env_auth = false
-access_key_id = xxx
-secret_access_key = xxx
-endpoint = cos.ap-guangzhou.myqcloud.com
-acl = default
---------------------
-y) Yes this is OK (default)
-e) Edit this remote
-d) Delete this remote
-y/e/d> y
-Current remotes:
+This integration opens the doorway to additional performance improvements
+which will be explored in the near future.
-Name Type
-==== ====
-cos s3
-```
+**Note:** If Plex options are not configured, `cache` will function with its
+configured options without adapting any of its settings.
-### Netease NOS
+How to enable? Run `rclone config` and add all the Plex options (endpoint, username
+and password) in your remote and it will be automatically enabled.
-For Netease NOS configure as per the configurator `rclone config`
-setting the provider `Netease`. This will automatically set
-`force_path_style = false` which is necessary for it to run properly.
+Affected settings:
+- `cache-workers`: _Configured value_ during confirmed playback or _1_ all the other times
-### Petabox
+##### Certificate Validation #####
-Here is an example of making a [Petabox](https://petabox.io/)
-configuration. First run:
+When the Plex server is configured to only accept secure connections, it is
+possible to use `.plex.direct` URLs to ensure certificate validation succeeds.
+These URLs are used by Plex internally to connect to the Plex server securely.
-```bash
-rclone config
-```
+The format for these URLs is the following:
-This will guide you through an interactive setup process.
+`https://ip-with-dots-replaced.server-hash.plex.direct:32400/`
-```
-No remotes found, make a new one?
-n) New remote
-s) Set configuration password
-n/s> n
+The `ip-with-dots-replaced` part can be any IPv4 address, where the dots
+have been replaced with dashes, e.g. `127.0.0.1` becomes `127-0-0-1`.
-Enter name for new remote.
-name> My Petabox Storage
+To get the `server-hash` part, the easiest way is to visit
-Option Storage.
-Type of storage to configure.
-Choose a number from below, or type in your own value.
-[snip]
-XX / Amazon S3 Compliant Storage Providers including AWS, ...
- \ "s3"
-[snip]
-Storage> s3
+https://plex.tv/api/resources?includeHttps=1&X-Plex-Token=your-plex-token
-Option provider.
-Choose your S3 provider.
-Choose a number from below, or type in your own value.
-Press Enter to leave empty.
-[snip]
-XX / Petabox Object Storage
- \ (Petabox)
-[snip]
-provider> Petabox
+This page will list all the available Plex servers for your account
+with at least one `.plex.direct` link for each. Copy one URL and replace
+the IP address with the desired address. This can be used as the
+`plex_url` value.
+
+### Known issues ###
+
+#### Mount and --dir-cache-time ####
+
+--dir-cache-time controls the first layer of directory caching which works at the mount layer.
+Being an independent caching mechanism from the `cache` backend, it will manage its own entries
+based on the configured time.
+
+To avoid getting in a scenario where dir cache has obsolete data and cache would have the correct
+one, try to set `--dir-cache-time` to a lower time than `--cache-info-age`. Default values are
+already configured in this way.
+
+#### Windows support - Experimental ####
+
+There are a couple of issues with Windows `mount` functionality that still require some investigations.
+It should be considered as experimental thus far as fixes come in for this OS.
+
+Most of the issues seem to be related to the difference between filesystems
+on Linux flavors and Windows as cache is heavily dependent on them.
+
+Any reports or feedback on how cache behaves on this OS is greatly appreciated.
+
+- https://github.com/rclone/rclone/issues/1935
+- https://github.com/rclone/rclone/issues/1907
+- https://github.com/rclone/rclone/issues/1834
-Option env_auth.
-Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
-Only applies if access_key_id and secret_access_key is blank.
-Choose a number from below, or type in your own boolean value (true or false).
-Press Enter for the default (false).
- 1 / Enter AWS credentials in the next step.
- \ (false)
- 2 / Get AWS credentials from the environment (env vars or IAM).
- \ (true)
-env_auth> 1
+#### Risk of throttling ####
-Option access_key_id.
-AWS Access Key ID.
-Leave blank for anonymous access or runtime credentials.
-Enter a value. Press Enter to leave empty.
-access_key_id> YOUR_ACCESS_KEY_ID
+Future iterations of the cache backend will make use of the pooling functionality
+of the cloud provider to synchronize and at the same time make writing through it
+more tolerant to failures.
-Option secret_access_key.
-AWS Secret Access Key (password).
-Leave blank for anonymous access or runtime credentials.
-Enter a value. Press Enter to leave empty.
-secret_access_key> YOUR_SECRET_ACCESS_KEY
+There are a couple of enhancements in track to add these but in the meantime
+there is a valid concern that the expiring cache listings can lead to cloud provider
+throttles or bans due to repeated queries on it for very large mounts.
-Option region.
-Region where your bucket will be created and your data stored.
-Choose a number from below, or type in your own value.
-Press Enter to leave empty.
- 1 / US East (N. Virginia)
- \ (us-east-1)
- 2 / Europe (Frankfurt)
- \ (eu-central-1)
- 3 / Asia Pacific (Singapore)
- \ (ap-southeast-1)
- 4 / Middle East (Bahrain)
- \ (me-south-1)
- 5 / South America (São Paulo)
- \ (sa-east-1)
-region> 1
+Some recommendations:
+- don't use a very small interval for entry information (`--cache-info-age`)
+- while writes aren't yet optimised, you can still write through `cache` which gives you the advantage
+of adding the file in the cache at the same time if configured to do so.
-Option endpoint.
-Endpoint for Petabox S3 Object Storage.
-Specify the endpoint from the same region.
-Choose a number from below, or type in your own value.
- 1 / US East (N. Virginia)
- \ (s3.petabox.io)
- 2 / US East (N. Virginia)
- \ (s3.us-east-1.petabox.io)
- 3 / Europe (Frankfurt)
- \ (s3.eu-central-1.petabox.io)
- 4 / Asia Pacific (Singapore)
- \ (s3.ap-southeast-1.petabox.io)
- 5 / Middle East (Bahrain)
- \ (s3.me-south-1.petabox.io)
- 6 / South America (São Paulo)
- \ (s3.sa-east-1.petabox.io)
-endpoint> 1
+Future enhancements:
-Option acl.
-Canned ACL used when creating buckets and storing or copying objects.
-This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.
-For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
-Note that this ACL is applied when server-side copying objects as S3
-doesn't copy the ACL from the source but rather writes a fresh one.
-If the acl is an empty string then no X-Amz-Acl: header is added and
-the default (private) will be used.
-Choose a number from below, or type in your own value.
-Press Enter to leave empty.
- / Owner gets FULL_CONTROL.
- 1 | No one else has access rights (default).
- \ (private)
- / Owner gets FULL_CONTROL.
- 2 | The AllUsers group gets READ access.
- \ (public-read)
- / Owner gets FULL_CONTROL.
- 3 | The AllUsers group gets READ and WRITE access.
- | Granting this on a bucket is generally not recommended.
- \ (public-read-write)
- / Owner gets FULL_CONTROL.
- 4 | The AuthenticatedUsers group gets READ access.
- \ (authenticated-read)
- / Object owner gets FULL_CONTROL.
- 5 | Bucket owner gets READ access.
- | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
- \ (bucket-owner-read)
- / Both the object owner and the bucket owner get FULL_CONTROL over the object.
- 6 | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
- \ (bucket-owner-full-control)
-acl> 1
+- https://github.com/rclone/rclone/issues/1937
+- https://github.com/rclone/rclone/issues/1936
-Edit advanced config?
-y) Yes
-n) No (default)
-y/n> No
+#### cache and crypt ####
-Configuration complete.
-Options:
-- type: s3
-- provider: Petabox
-- access_key_id: YOUR_ACCESS_KEY_ID
-- secret_access_key: YOUR_SECRET_ACCESS_KEY
-- region: us-east-1
-- endpoint: s3.petabox.io
-Keep this "My Petabox Storage" remote?
-y) Yes this is OK (default)
-e) Edit this remote
-d) Delete this remote
-y/e/d> y
-```
+One common scenario is to keep your data encrypted in the cloud provider
+using the `crypt` remote. `crypt` uses a similar technique to wrap around
+an existing remote and handles this translation in a seamless way.
-This will leave the config file looking like this.
+There is an issue with wrapping the remotes in this order:
+**cloud remote** -> **crypt** -> **cache**
-```
-[My Petabox Storage]
-type = s3
-provider = Petabox
-access_key_id = YOUR_ACCESS_KEY_ID
-secret_access_key = YOUR_SECRET_ACCESS_KEY
-region = us-east-1
-endpoint = s3.petabox.io
-```
+During testing, I experienced a lot of bans with the remotes in this order.
+I suspect it might be related to how crypt opens files on the cloud provider
+which makes it think we're downloading the full file instead of small chunks.
+Organizing the remotes in this order yields better results:
+**cloud remote** -> **cache** -> **crypt**
-### Storj
+#### absolute remote paths ####
-Storj is a decentralized cloud storage which can be used through its
-native protocol or an S3 compatible gateway.
+`cache` can not differentiate between relative and absolute paths for the wrapped remote.
+Any path given in the `remote` config setting and on the command line will be passed to
+the wrapped remote as is, but for storing the chunks on disk the path will be made
+relative by removing any leading `/` character.
-The S3 compatible gateway is configured using `rclone config` with a
-type of `s3` and with a provider name of `Storj`. Here is an example
-run of the configurator.
+This behavior is irrelevant for most backend types, but there are backends where a leading `/`
+changes the effective directory, e.g. in the `sftp` backend paths starting with a `/` are
+relative to the root of the SSH server and paths without are relative to the user home directory.
+As a result `sftp:bin` and `sftp:/bin` will share the same cache folder, even if they represent
+a different directory on the SSH server.
-```
-Type of storage to configure.
-Storage> s3
-Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
-Only applies if access_key_id and secret_access_key is blank.
-Choose a number from below, or type in your own boolean value (true or false).
-Press Enter for the default (false).
- 1 / Enter AWS credentials in the next step.
- \ (false)
- 2 / Get AWS credentials from the environment (env vars or IAM).
- \ (true)
-env_auth> 1
-Option access_key_id.
-AWS Access Key ID.
-Leave blank for anonymous access or runtime credentials.
-Enter a value. Press Enter to leave empty.
-access_key_id> XXXX (as shown when creating the access grant)
-Option secret_access_key.
-AWS Secret Access Key (password).
-Leave blank for anonymous access or runtime credentials.
-Enter a value. Press Enter to leave empty.
-secret_access_key> XXXX (as shown when creating the access grant)
-Option endpoint.
-Endpoint of the Shared Gateway.
-Choose a number from below, or type in your own value.
-Press Enter to leave empty.
- 1 / EU1 Shared Gateway
- \ (gateway.eu1.storjshare.io)
- 2 / US1 Shared Gateway
- \ (gateway.us1.storjshare.io)
- 3 / Asia-Pacific Shared Gateway
- \ (gateway.ap1.storjshare.io)
-endpoint> 1 (as shown when creating the access grant)
-Edit advanced config?
-y) Yes
-n) No (default)
-y/n> n
-```
+### Cache and Remote Control (--rc) ###
+Cache supports the new `--rc` mode in rclone and can be remote controlled through the following end points:
+By default, the listener is disabled if you do not add the flag.
-Note that s3 credentials are generated when you [create an access
-grant](https://docs.storj.io/dcs/api-reference/s3-compatible-gateway#usage).
+### rc cache/expire
+Purge a remote from the cache backend. Supports either a directory or a file.
+It supports both encrypted and unencrypted file names if cache is wrapped by crypt.
-#### Backend quirks
+Params:
+ - **remote** = path to remote **(required)**
+ - **withData** = true/false to delete cached data (chunks) as well _(optional, false by default)_
-- `--chunk-size` is forced to be 64 MiB or greater. This will use more
- memory than the default of 5 MiB.
-- Server side copy is disabled as it isn't currently supported in the
- gateway.
-- GetTier and SetTier are not supported.
-#### Backend bugs
+### Standard options
-Due to [issue #39](https://github.com/storj/gateway-mt/issues/39)
-uploading multipart files via the S3 gateway causes them to lose their
-metadata. For rclone's purpose this means that the modification time
-is not stored, nor is any MD5SUM (if one is available from the
-source).
+Here are the Standard options specific to cache (Cache a remote).
-This has the following consequences:
+#### --cache-remote
-- Using `rclone rcat` will fail as the medatada doesn't match after upload
-- Uploading files with `rclone mount` will fail for the same reason
- - This can worked around by using `--vfs-cache-mode writes` or `--vfs-cache-mode full` or setting `--s3-upload-cutoff` large
-- Files uploaded via a multipart upload won't have their modtimes
- - This will mean that `rclone sync` will likely keep trying to upload files bigger than `--s3-upload-cutoff`
- - This can be worked around with `--checksum` or `--size-only` or setting `--s3-upload-cutoff` large
- - The maximum value for `--s3-upload-cutoff` is 5GiB though
+Remote to cache.
-One general purpose workaround is to set `--s3-upload-cutoff 5G`. This
-means that rclone will upload files smaller than 5GiB as single parts.
-Note that this can be set in the config file with `upload_cutoff = 5G`
-or configured in the advanced settings. If you regularly transfer
-files larger than 5G then using `--checksum` or `--size-only` in
-`rclone sync` is the recommended workaround.
+Normally should contain a ':' and a path, e.g. "myremote:path/to/dir",
+"myremote:bucket" or maybe "myremote:" (not recommended).
-#### Comparison with the native protocol
+Properties:
-Use the [the native protocol](/storj) to take advantage of
-client-side encryption as well as to achieve the best possible
-download performance. Uploads will be erasure-coded locally, thus a
-1gb upload will result in 2.68gb of data being uploaded to storage
-nodes across the network.
+- Config: remote
+- Env Var: RCLONE_CACHE_REMOTE
+- Type: string
+- Required: true
-Use this backend and the S3 compatible Hosted Gateway to increase
-upload performance and reduce the load on your systems and network.
-Uploads will be encrypted and erasure-coded server-side, thus a 1GB
-upload will result in only in 1GB of data being uploaded to storage
-nodes across the network.
+#### --cache-plex-url
-For more detailed comparison please check the documentation of the
-[storj](/storj) backend.
+The URL of the Plex server.
-## Limitations
+Properties:
-`rclone about` is not supported by the S3 backend. Backends without
-this capability cannot determine free space for an rclone mount or
-use policy `mfs` (most free space) as a member of an rclone union
-remote.
+- Config: plex_url
+- Env Var: RCLONE_CACHE_PLEX_URL
+- Type: string
+- Required: false
-See [List of backends that do not support rclone about](https://rclone.org/overview/#optional-features) and [rclone about](https://rclone.org/commands/rclone_about/)
+#### --cache-plex-username
+The username of the Plex user.
+Properties:
-### Synology C2 Object Storage {#synology-c2}
+- Config: plex_username
+- Env Var: RCLONE_CACHE_PLEX_USERNAME
+- Type: string
+- Required: false
-[Synology C2 Object Storage](https://c2.synology.com/en-global/object-storage/overview) provides a secure, S3-compatible, and cost-effective cloud storage solution without API request, download fees, and deletion penalty.
+#### --cache-plex-password
-The S3 compatible gateway is configured using `rclone config` with a
-type of `s3` and with a provider name of `Synology`. Here is an example
-run of the configurator.
+The password of the Plex user.
-First run:
+**NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/).
-```
-rclone config
-```
+Properties:
-This will guide you through an interactive setup process.
+- Config: plex_password
+- Env Var: RCLONE_CACHE_PLEX_PASSWORD
+- Type: string
+- Required: false
-```
-No remotes found, make a new one?
-n) New remote
-s) Set configuration password
-q) Quit config
+#### --cache-chunk-size
-n/s/q> n
+The size of a chunk (partial file data).
-Enter name for new remote.1
-name> syno
+Use lower numbers for slower connections. If the chunk size is
+changed, any downloaded chunks will be invalid and cache-chunk-path
+will need to be cleared or unexpected EOF errors will occur.
-Type of storage to configure.
-Enter a string value. Press Enter for the default ("").
-Choose a number from below, or type in your own value
+Properties:
- 5 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, China Mobile, Cloudflare, GCS, ArvanCloud, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, IDrive e2, IONOS Cloud, Liara, Lyve Cloud, Minio, Netease, Petabox, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Synology, Tencent COS, Qiniu and Wasabi
- \ "s3"
+- Config: chunk_size
+- Env Var: RCLONE_CACHE_CHUNK_SIZE
+- Type: SizeSuffix
+- Default: 5Mi
+- Examples:
+ - "1M"
+ - 1 MiB
+ - "5M"
+ - 5 MiB
+ - "10M"
+ - 10 MiB
-Storage> s3
+#### --cache-info-age
-Choose your S3 provider.
-Enter a string value. Press Enter for the default ("").
-Choose a number from below, or type in your own value
- 24 / Synology C2 Object Storage
- \ (Synology)
+How long to cache file structure information (directory listings, file size, times, etc.).
+If all write operations are done through the cache then you can safely make
+this value very large as the cache store will also be updated in real time.
-provider> Synology
+Properties:
-Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
-Only applies if access_key_id and secret_access_key is blank.
-Enter a boolean value (true or false). Press Enter for the default ("false").
-Choose a number from below, or type in your own value
- 1 / Enter AWS credentials in the next step
- \ "false"
- 2 / Get AWS credentials from the environment (env vars or IAM)
- \ "true"
+- Config: info_age
+- Env Var: RCLONE_CACHE_INFO_AGE
+- Type: Duration
+- Default: 6h0m0s
+- Examples:
+ - "1h"
+ - 1 hour
+ - "24h"
+ - 24 hours
+ - "48h"
+ - 48 hours
-env_auth> 1
+#### --cache-chunk-total-size
-AWS Access Key ID.
-Leave blank for anonymous access or runtime credentials.
-Enter a string value. Press Enter for the default ("").
+The total size that the chunks can take up on the local disk.
-access_key_id> accesskeyid
+If the cache exceeds this value then it will start to delete the
+oldest chunks until it goes under this value.
-AWS Secret Access Key (password)
-Leave blank for anonymous access or runtime credentials.
-Enter a string value. Press Enter for the default ("").
+Properties:
-secret_access_key> secretaccesskey
+- Config: chunk_total_size
+- Env Var: RCLONE_CACHE_CHUNK_TOTAL_SIZE
+- Type: SizeSuffix
+- Default: 10Gi
+- Examples:
+ - "500M"
+ - 500 MiB
+ - "1G"
+ - 1 GiB
+ - "10G"
+ - 10 GiB
-Region where your data stored.
-Choose a number from below, or type in your own value.
-Press Enter to leave empty.
- 1 / Europe Region 1
- \ (eu-001)
- 2 / Europe Region 2
- \ (eu-002)
- 3 / US Region 1
- \ (us-001)
- 4 / US Region 2
- \ (us-002)
- 5 / Asia (Taiwan)
- \ (tw-001)
+### Advanced options
-region > 1
+Here are the Advanced options specific to cache (Cache a remote).
-Option endpoint.
-Endpoint for Synology C2 Object Storage API.
-Choose a number from below, or type in your own value.
-Press Enter to leave empty.
- 1 / EU Endpoint 1
- \ (eu-001.s3.synologyc2.net)
- 2 / US Endpoint 1
- \ (us-001.s3.synologyc2.net)
- 3 / TW Endpoint 1
- \ (tw-001.s3.synologyc2.net)
+#### --cache-plex-token
-endpoint> 1
+The plex token for authentication - auto set normally.
-Option location_constraint.
-Location constraint - must be set to match the Region.
-Leave blank if not sure. Used when creating buckets only.
-Enter a value. Press Enter to leave empty.
-location_constraint>
+Properties:
-Edit advanced config? (y/n)
-y) Yes
-n) No
-y/n> y
+- Config: plex_token
+- Env Var: RCLONE_CACHE_PLEX_TOKEN
+- Type: string
+- Required: false
-Option no_check_bucket.
-If set, don't attempt to check the bucket exists or create it.
-This can be useful when trying to minimise the number of transactions
-rclone does if you know the bucket exists already.
-It can also be needed if the user you are using does not have bucket
-creation permissions. Before v1.52.0 this would have passed silently
-due to a bug.
-Enter a boolean value (true or false). Press Enter for the default (true).
+#### --cache-plex-insecure
-no_check_bucket> true
+Skip all certificate verification when connecting to the Plex server.
-Configuration complete.
-Options:
-- type: s3
-- provider: Synology
-- region: eu-001
-- endpoint: eu-001.s3.synologyc2.net
-- no_check_bucket: true
-Keep this "syno" remote?
-y) Yes this is OK (default)
-e) Edit this remote
-d) Delete this remote
+Properties:
-y/e/d> y
+- Config: plex_insecure
+- Env Var: RCLONE_CACHE_PLEX_INSECURE
+- Type: string
+- Required: false
-# Backblaze B2
+#### --cache-db-path
-B2 is [Backblaze's cloud storage system](https://www.backblaze.com/b2/).
+Directory to store file structure metadata DB.
-Paths are specified as `remote:bucket` (or `remote:` for the `lsd`
-command.) You may put subdirectories in too, e.g. `remote:bucket/path/to/dir`.
+The remote name is used as the DB file name.
-## Configuration
+Properties:
-Here is an example of making a b2 configuration. First run
+- Config: db_path
+- Env Var: RCLONE_CACHE_DB_PATH
+- Type: string
+- Default: "$HOME/.cache/rclone/cache-backend"
- rclone config
+#### --cache-chunk-path
-This will guide you through an interactive setup process. To authenticate
-you will either need your Account ID (a short hex number) and Master
-Application Key (a long hex number) OR an Application Key, which is the
-recommended method. See below for further details on generating and using
-an Application Key.
+Directory to cache chunk files.
-```
-No remotes found, make a new one?
-n) New remote
-q) Quit config
-n/q> n
-name> remote
-Type of storage to configure.
-Choose a number from below, or type in your own value
-[snip]
-XX / Backblaze B2
- \ "b2"
-[snip]
-Storage> b2
-Account ID or Application Key ID
-account> 123456789abc
-Application Key
-key> 0123456789abcdef0123456789abcdef0123456789
-Endpoint for the service - leave blank normally.
-endpoint>
-Remote config
---------------------
-[remote]
-account = 123456789abc
-key = 0123456789abcdef0123456789abcdef0123456789
-endpoint =
---------------------
-y) Yes this is OK
-e) Edit this remote
-d) Delete this remote
-y/e/d> y
-```
+Path to where partial file data (chunks) are stored locally. The remote
+name is appended to the final path.
-This remote is called `remote` and can now be used like this
+This config follows the "--cache-db-path". If you specify a custom
+location for "--cache-db-path" and don't specify one for "--cache-chunk-path"
+then "--cache-chunk-path" will use the same path as "--cache-db-path".
+
+Properties:
+
+- Config: chunk_path
+- Env Var: RCLONE_CACHE_CHUNK_PATH
+- Type: string
+- Default: "$HOME/.cache/rclone/cache-backend"
-See all buckets
+#### --cache-db-purge
- rclone lsd remote:
+Clear all the cached data for this remote on start.
-Create a new bucket
+Properties:
- rclone mkdir remote:bucket
+- Config: db_purge
+- Env Var: RCLONE_CACHE_DB_PURGE
+- Type: bool
+- Default: false
-List the contents of a bucket
+#### --cache-chunk-clean-interval
- rclone ls remote:bucket
+How often should the cache perform cleanups of the chunk storage.
-Sync `/home/local/directory` to the remote bucket, deleting any
-excess files in the bucket.
+The default value should be ok for most people. If you find that the
+cache goes over "cache-chunk-total-size" too often then try to lower
+this value to force it to perform cleanups more often.
- rclone sync --interactive /home/local/directory remote:bucket
+Properties:
-### Application Keys
+- Config: chunk_clean_interval
+- Env Var: RCLONE_CACHE_CHUNK_CLEAN_INTERVAL
+- Type: Duration
+- Default: 1m0s
-B2 supports multiple [Application Keys for different access permission
-to B2 Buckets](https://www.backblaze.com/b2/docs/application_keys.html).
+#### --cache-read-retries
-You can use these with rclone too; you will need to use rclone version 1.43
-or later.
+How many times to retry a read from a cache storage.
-Follow Backblaze's docs to create an Application Key with the required
-permission and add the `applicationKeyId` as the `account` and the
-`Application Key` itself as the `key`.
+Since reading from a cache stream is independent from downloading file
+data, readers can get to a point where there's no more data in the
+cache. Most of the times this can indicate a connectivity issue if
+cache isn't able to provide file data anymore.
-Note that you must put the _applicationKeyId_ as the `account` – you
-can't use the master Account ID. If you try then B2 will return 401
-errors.
+For really slow connections, increase this to a point where the stream is
+able to provide data but your experience will be very stuttering.
-### --fast-list
+Properties:
-This remote supports `--fast-list` which allows you to use fewer
-transactions in exchange for more memory. See the [rclone
-docs](https://rclone.org/docs/#fast-list) for more details.
+- Config: read_retries
+- Env Var: RCLONE_CACHE_READ_RETRIES
+- Type: int
+- Default: 10
-### Modified time
+#### --cache-workers
-The modified time is stored as metadata on the object as
-`X-Bz-Info-src_last_modified_millis` as milliseconds since 1970-01-01
-in the Backblaze standard. Other tools should be able to use this as
-a modified time.
+How many workers should run in parallel to download chunks.
-Modified times are used in syncing and are fully supported. Note that
-if a modification time needs to be updated on an object then it will
-create a new version of the object.
+Higher values will mean more parallel processing (better CPU needed)
+and more concurrent requests on the cloud provider. This impacts
+several aspects like the cloud provider API limits, more stress on the
+hardware that rclone runs on but it also means that streams will be
+more fluid and data will be available much more faster to readers.
-### Restricted filename characters
+**Note**: If the optional Plex integration is enabled then this
+setting will adapt to the type of reading performed and the value
+specified here will be used as a maximum number of workers to use.
-In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters)
-the following characters are also replaced:
+Properties:
-| Character | Value | Replacement |
-| --------- |:-----:|:-----------:|
-| \ | 0x5C | ï¼¼ |
+- Config: workers
+- Env Var: RCLONE_CACHE_WORKERS
+- Type: int
+- Default: 4
-Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8),
-as they can't be used in JSON strings.
+#### --cache-chunk-no-memory
-Note that in 2020-05 Backblaze started allowing \ characters in file
-names. Rclone hasn't changed its encoding as this could cause syncs to
-re-transfer files. If you want rclone not to replace \ then see the
-`--b2-encoding` flag below and remove the `BackSlash` from the
-string. This can be set in the config.
+Disable the in-memory cache for storing chunks during streaming.
-### SHA1 checksums
+By default, cache will keep file data during streaming in RAM as well
+to provide it to readers as fast as possible.
-The SHA1 checksums of the files are checked on upload and download and
-will be used in the syncing process.
+This transient data is evicted as soon as it is read and the number of
+chunks stored doesn't exceed the number of workers. However, depending
+on other settings like "cache-chunk-size" and "cache-workers" this footprint
+can increase if there are parallel streams too (multiple files being read
+at the same time).
-Large files (bigger than the limit in `--b2-upload-cutoff`) which are
-uploaded in chunks will store their SHA1 on the object as
-`X-Bz-Info-large_file_sha1` as recommended by Backblaze.
+If the hardware permits it, use this feature to provide an overall better
+performance during streaming but it can also be disabled if RAM is not
+available on the local machine.
-For a large file to be uploaded with an SHA1 checksum, the source
-needs to support SHA1 checksums. The local disk supports SHA1
-checksums so large file transfers from local disk will have an SHA1.
-See [the overview](https://rclone.org/overview/#features) for exactly which remotes
-support SHA1.
+Properties:
-Sources which don't support SHA1, in particular `crypt` will upload
-large files without SHA1 checksums. This may be fixed in the future
-(see [#1767](https://github.com/rclone/rclone/issues/1767)).
+- Config: chunk_no_memory
+- Env Var: RCLONE_CACHE_CHUNK_NO_MEMORY
+- Type: bool
+- Default: false
-Files sizes below `--b2-upload-cutoff` will always have an SHA1
-regardless of the source.
+#### --cache-rps
-### Transfers
+Limits the number of requests per second to the source FS (-1 to disable).
-Backblaze recommends that you do lots of transfers simultaneously for
-maximum speed. In tests from my SSD equipped laptop the optimum
-setting is about `--transfers 32` though higher numbers may be used
-for a slight speed improvement. The optimum number for you may vary
-depending on your hardware, how big the files are, how much you want
-to load your computer, etc. The default of `--transfers 4` is
-definitely too low for Backblaze B2 though.
+This setting places a hard limit on the number of requests per second
+that cache will be doing to the cloud provider remote and try to
+respect that value by setting waits between reads.
-Note that uploading big files (bigger than 200 MiB by default) will use
-a 96 MiB RAM buffer by default. There can be at most `--transfers` of
-these in use at any moment, so this sets the upper limit on the memory
-used.
+If you find that you're getting banned or limited on the cloud
+provider through cache and know that a smaller number of requests per
+second will allow you to work with it then you can use this setting
+for that.
-### Versions
+A good balance of all the other settings should make this setting
+useless but it is available to set for more special cases.
-When rclone uploads a new version of a file it creates a [new version
-of it](https://www.backblaze.com/b2/docs/file_versions.html).
-Likewise when you delete a file, the old version will be marked hidden
-and still be available. Conversely, you may opt in to a "hard delete"
-of files with the `--b2-hard-delete` flag which would permanently remove
-the file instead of hiding it.
+**NOTE**: This will limit the number of requests during streams but
+other API calls to the cloud provider like directory listings will
+still pass.
-Old versions of files, where available, are visible using the
-`--b2-versions` flag.
+Properties:
-It is also possible to view a bucket as it was at a certain point in time,
-using the `--b2-version-at` flag. This will show the file versions as they
-were at that time, showing files that have been deleted afterwards, and
-hiding files that were created since.
+- Config: rps
+- Env Var: RCLONE_CACHE_RPS
+- Type: int
+- Default: -1
-If you wish to remove all the old versions then you can use the
-`rclone cleanup remote:bucket` command which will delete all the old
-versions of files, leaving the current ones intact. You can also
-supply a path and only old versions under that path will be deleted,
-e.g. `rclone cleanup remote:bucket/path/to/stuff`.
+#### --cache-writes
-Note that `cleanup` will remove partially uploaded files from the bucket
-if they are more than a day old.
+Cache file data on writes through the FS.
-When you `purge` a bucket, the current and the old versions will be
-deleted then the bucket will be deleted.
+If you need to read files immediately after you upload them through
+cache you can enable this flag to have their data stored in the
+cache store at the same time during upload.
-However `delete` will cause the current versions of the files to
-become hidden old versions.
+Properties:
-Here is a session showing the listing and retrieval of an old
-version followed by a `cleanup` of the old versions.
+- Config: writes
+- Env Var: RCLONE_CACHE_WRITES
+- Type: bool
+- Default: false
-Show current version and all the versions with `--b2-versions` flag.
+#### --cache-tmp-upload-path
-```
-$ rclone -q ls b2:cleanup-test
- 9 one.txt
+Directory to keep temporary files until they are uploaded.
-$ rclone -q --b2-versions ls b2:cleanup-test
- 9 one.txt
- 8 one-v2016-07-04-141032-000.txt
- 16 one-v2016-07-04-141003-000.txt
- 15 one-v2016-07-02-155621-000.txt
-```
+This is the path where cache will use as a temporary storage for new
+files that need to be uploaded to the cloud provider.
-Retrieve an old version
+Specifying a value will enable this feature. Without it, it is
+completely disabled and files will be uploaded directly to the cloud
+provider
-```
-$ rclone -q --b2-versions copy b2:cleanup-test/one-v2016-07-04-141003-000.txt /tmp
+Properties:
-$ ls -l /tmp/one-v2016-07-04-141003-000.txt
--rw-rw-r-- 1 ncw ncw 16 Jul 2 17:46 /tmp/one-v2016-07-04-141003-000.txt
-```
+- Config: tmp_upload_path
+- Env Var: RCLONE_CACHE_TMP_UPLOAD_PATH
+- Type: string
+- Required: false
-Clean up all the old versions and show that they've gone.
+#### --cache-tmp-wait-time
-```
-$ rclone -q cleanup b2:cleanup-test
+How long should files be stored in local cache before being uploaded.
-$ rclone -q ls b2:cleanup-test
- 9 one.txt
+This is the duration that a file must wait in the temporary location
+_cache-tmp-upload-path_ before it is selected for upload.
-$ rclone -q --b2-versions ls b2:cleanup-test
- 9 one.txt
-```
+Note that only one file is uploaded at a time and it can take longer
+to start the upload if a queue formed for this purpose.
-#### Versions naming caveat
+Properties:
-When using `--b2-versions` flag rclone is relying on the file name
-to work out whether the objects are versions or not. Versions' names
-are created by inserting timestamp between file name and its extension.
-```
- 9 file.txt
- 8 file-v2023-07-17-161032-000.txt
- 16 file-v2023-06-15-141003-000.txt
-```
-If there are real files present with the same names as versions, then
-behaviour of `--b2-versions` can be unpredictable.
+- Config: tmp_wait_time
+- Env Var: RCLONE_CACHE_TMP_WAIT_TIME
+- Type: Duration
+- Default: 15s
-### Data usage
+#### --cache-db-wait-time
-It is useful to know how many requests are sent to the server in different scenarios.
+How long to wait for the DB to be available - 0 is unlimited.
-All copy commands send the following 4 requests:
+Only one process can have the DB open at any one time, so rclone waits
+for this duration for the DB to become available before it gives an
+error.
-```
-/b2api/v1/b2_authorize_account
-/b2api/v1/b2_create_bucket
-/b2api/v1/b2_list_buckets
-/b2api/v1/b2_list_file_names
-```
+If you set it to 0 then it will wait forever.
-The `b2_list_file_names` request will be sent once for every 1k files
-in the remote path, providing the checksum and modification time of
-the listed files. As of version 1.33 issue
-[#818](https://github.com/rclone/rclone/issues/818) causes extra requests
-to be sent when using B2 with Crypt. When a copy operation does not
-require any files to be uploaded, no more requests will be sent.
+Properties:
-Uploading files that do not require chunking, will send 2 requests per
-file upload:
+- Config: db_wait_time
+- Env Var: RCLONE_CACHE_DB_WAIT_TIME
+- Type: Duration
+- Default: 1s
-```
-/b2api/v1/b2_get_upload_url
-/b2api/v1/b2_upload_file/
-```
+## Backend commands
-Uploading files requiring chunking, will send 2 requests (one each to
-start and finish the upload) and another 2 requests for each chunk:
+Here are the commands specific to the cache backend.
-```
-/b2api/v1/b2_start_large_file
-/b2api/v1/b2_get_upload_part_url
-/b2api/v1/b2_upload_part/
-/b2api/v1/b2_finish_large_file
-```
+Run them with
-#### Versions
+ rclone backend COMMAND remote:
-Versions can be viewed with the `--b2-versions` flag. When it is set
-rclone will show and act on older versions of files. For example
+The help below will explain what arguments each command takes.
-Listing without `--b2-versions`
+See the [backend](https://rclone.org/commands/rclone_backend/) command for more
+info on how to pass options and arguments.
-```
-$ rclone -q ls b2:cleanup-test
- 9 one.txt
-```
+These can be run on a running backend using the rc command
+[backend/command](https://rclone.org/rc/#backend-command).
-And with
+### stats
-```
-$ rclone -q --b2-versions ls b2:cleanup-test
- 9 one.txt
- 8 one-v2016-07-04-141032-000.txt
- 16 one-v2016-07-04-141003-000.txt
- 15 one-v2016-07-02-155621-000.txt
-```
+Print stats on the cache backend in JSON format.
-Showing that the current version is unchanged but older versions can
-be seen. These have the UTC date that they were uploaded to the
-server to the nearest millisecond appended to them.
+ rclone backend stats remote: [options] [+]
-Note that when using `--b2-versions` no file write operations are
-permitted, so you can't upload files or delete them.
-### B2 and rclone link
-Rclone supports generating file share links for private B2 buckets.
-They can either be for a file for example:
+# Chunker
-```
-./rclone link B2:bucket/path/to/file.txt
-https://f002.backblazeb2.com/file/bucket/path/to/file.txt?Authorization=xxxxxxxx
+The `chunker` overlay transparently splits large files into smaller chunks
+during upload to wrapped remote and transparently assembles them back
+when the file is downloaded. This allows to effectively overcome size limits
+imposed by storage providers.
-```
+## Configuration
-or if run on a directory you will get:
+To use it, first set up the underlying remote following the configuration
+instructions for that remote. You can also use a local pathname instead of
+a remote.
-```
-./rclone link B2:bucket/path
-https://f002.backblazeb2.com/file/bucket/path?Authorization=xxxxxxxx
-```
+First check your chosen remote is working - we'll call it `remote:path` here.
+Note that anything inside `remote:path` will be chunked and anything outside
+won't. This means that if you are using a bucket-based remote (e.g. S3, B2, swift)
+then you should probably put the bucket in the remote `s3:bucket`.
-you can then use the authorization token (the part of the url from the
- `?Authorization=` on) on any file path under that directory. For example:
+Now configure `chunker` using `rclone config`. We will call this one `overlay`
+to separate it from the `remote` itself.
```
-https://f002.backblazeb2.com/file/bucket/path/to/file1?Authorization=xxxxxxxx
-https://f002.backblazeb2.com/file/bucket/path/file2?Authorization=xxxxxxxx
-https://f002.backblazeb2.com/file/bucket/path/folder/file3?Authorization=xxxxxxxx
-
+No remotes found, make a new one?
+n) New remote
+s) Set configuration password
+q) Quit config
+n/s/q> n
+name> overlay
+Type of storage to configure.
+Choose a number from below, or type in your own value
+[snip]
+XX / Transparently chunk/split large files
+ \ "chunker"
+[snip]
+Storage> chunker
+Remote to chunk/unchunk.
+Normally should contain a ':' and a path, e.g. "myremote:path/to/dir",
+"myremote:bucket" or maybe "myremote:" (not recommended).
+Enter a string value. Press Enter for the default ("").
+remote> remote:path
+Files larger than chunk size will be split in chunks.
+Enter a size with suffix K,M,G,T. Press Enter for the default ("2G").
+chunk_size> 100M
+Choose how chunker handles hash sums. All modes but "none" require metadata.
+Enter a string value. Press Enter for the default ("md5").
+Choose a number from below, or type in your own value
+ 1 / Pass any hash supported by wrapped remote for non-chunked files, return nothing otherwise
+ \ "none"
+ 2 / MD5 for composite files
+ \ "md5"
+ 3 / SHA1 for composite files
+ \ "sha1"
+ 4 / MD5 for all files
+ \ "md5all"
+ 5 / SHA1 for all files
+ \ "sha1all"
+ 6 / Copying a file to chunker will request MD5 from the source falling back to SHA1 if unsupported
+ \ "md5quick"
+ 7 / Similar to "md5quick" but prefers SHA1 over MD5
+ \ "sha1quick"
+hash_type> md5
+Edit advanced config? (y/n)
+y) Yes
+n) No
+y/n> n
+Remote config
+--------------------
+[overlay]
+type = chunker
+remote = remote:bucket
+chunk_size = 100M
+hash_type = md5
+--------------------
+y) Yes this is OK
+e) Edit this remote
+d) Delete this remote
+y/e/d> y
```
+### Specifying the remote
-### Standard options
-
-Here are the Standard options specific to b2 (Backblaze B2).
-
-#### --b2-account
-
-Account ID or Application Key ID.
+In normal use, make sure the remote has a `:` in. If you specify the remote
+without a `:` then rclone will use a local directory of that name.
+So if you use a remote of `/path/to/secret/files` then rclone will
+chunk stuff in that directory. If you use a remote of `name` then rclone
+will put files in a directory called `name` in the current directory.
-Properties:
-- Config: account
-- Env Var: RCLONE_B2_ACCOUNT
-- Type: string
-- Required: true
+### Chunking
-#### --b2-key
+When rclone starts a file upload, chunker checks the file size. If it
+doesn't exceed the configured chunk size, chunker will just pass the file
+to the wrapped remote (however, see caveat below). If a file is large, chunker will transparently cut
+data in pieces with temporary names and stream them one by one, on the fly.
+Each data chunk will contain the specified number of bytes, except for the
+last one which may have less data. If file size is unknown in advance
+(this is called a streaming upload), chunker will internally create
+a temporary copy, record its size and repeat the above process.
-Application Key.
+When upload completes, temporary chunk files are finally renamed.
+This scheme guarantees that operations can be run in parallel and look
+from outside as atomic.
+A similar method with hidden temporary chunks is used for other operations
+(copy/move/rename, etc.). If an operation fails, hidden chunks are normally
+destroyed, and the target composite file stays intact.
-Properties:
+When a composite file download is requested, chunker transparently
+assembles it by concatenating data chunks in order. As the split is trivial
+one could even manually concatenate data chunks together to obtain the
+original content.
-- Config: key
-- Env Var: RCLONE_B2_KEY
-- Type: string
-- Required: true
+When the `list` rclone command scans a directory on wrapped remote,
+the potential chunk files are accounted for, grouped and assembled into
+composite directory entries. Any temporary chunks are hidden.
-#### --b2-hard-delete
+List and other commands can sometimes come across composite files with
+missing or invalid chunks, e.g. shadowed by like-named directory or
+another file. This usually means that wrapped file system has been directly
+tampered with or damaged. If chunker detects a missing chunk it will
+by default print warning, skip the whole incomplete group of chunks but
+proceed with current command.
+You can set the `--chunker-fail-hard` flag to have commands abort with
+error message in such cases.
-Permanently delete files on remote removal, otherwise hide files.
+**Caveat**: As it is now, chunker will always create a temporary file in the
+backend and then rename it, even if the file is below the chunk threshold.
+This will result in unnecessary API calls and can severely restrict throughput
+when handling transfers primarily composed of small files on some backends (e.g. Box).
+A workaround to this issue is to use chunker only for files above the chunk threshold
+via `--min-size` and then perform a separate call without chunker on the remaining
+files.
-Properties:
-- Config: hard_delete
-- Env Var: RCLONE_B2_HARD_DELETE
-- Type: bool
-- Default: false
+#### Chunk names
-### Advanced options
+The default chunk name format is `*.rclone_chunk.###`, hence by default
+chunk names are `BIG_FILE_NAME.rclone_chunk.001`,
+`BIG_FILE_NAME.rclone_chunk.002` etc. You can configure another name format
+using the `name_format` configuration file option. The format uses asterisk
+`*` as a placeholder for the base file name and one or more consecutive
+hash characters `#` as a placeholder for sequential chunk number.
+There must be one and only one asterisk. The number of consecutive hash
+characters defines the minimum length of a string representing a chunk number.
+If decimal chunk number has less digits than the number of hashes, it is
+left-padded by zeros. If the decimal string is longer, it is left intact.
+By default numbering starts from 1 but there is another option that allows
+user to start from 0, e.g. for compatibility with legacy software.
-Here are the Advanced options specific to b2 (Backblaze B2).
+For example, if name format is `big_*-##.part` and original file name is
+`data.txt` and numbering starts from 0, then the first chunk will be named
+`big_data.txt-00.part`, the 99th chunk will be `big_data.txt-98.part`
+and the 302nd chunk will become `big_data.txt-301.part`.
-#### --b2-endpoint
+Note that `list` assembles composite directory entries only when chunk names
+match the configured format and treats non-conforming file names as normal
+non-chunked files.
-Endpoint for the service.
+When using `norename` transactions, chunk names will additionally have a unique
+file version suffix. For example, `BIG_FILE_NAME.rclone_chunk.001_bp562k`.
-Leave blank normally.
-Properties:
+### Metadata
-- Config: endpoint
-- Env Var: RCLONE_B2_ENDPOINT
-- Type: string
-- Required: false
+Besides data chunks chunker will by default create metadata object for
+a composite file. The object is named after the original file.
+Chunker allows user to disable metadata completely (the `none` format).
+Note that metadata is normally not created for files smaller than the
+configured chunk size. This may change in future rclone releases.
-#### --b2-test-mode
+#### Simple JSON metadata format
-A flag string for X-Bz-Test-Mode header for debugging.
+This is the default format. It supports hash sums and chunk validation
+for composite files. Meta objects carry the following fields:
-This is for debugging purposes only. Setting it to one of the strings
-below will cause b2 to return specific errors:
+- `ver` - version of format, currently `1`
+- `size` - total size of composite file
+- `nchunks` - number of data chunks in file
+- `md5` - MD5 hashsum of composite file (if present)
+- `sha1` - SHA1 hashsum (if present)
+- `txn` - identifies current version of the file
- * "fail_some_uploads"
- * "expire_some_account_authorization_tokens"
- * "force_cap_exceeded"
+There is no field for composite file name as it's simply equal to the name
+of meta object on the wrapped remote. Please refer to respective sections
+for details on hashsums and modified time handling.
-These will be set in the "X-Bz-Test-Mode" header which is documented
-in the [b2 integrations checklist](https://www.backblaze.com/b2/docs/integration_checklist.html).
+#### No metadata
-Properties:
+You can disable meta objects by setting the meta format option to `none`.
+In this mode chunker will scan directory for all files that follow
+configured chunk name format, group them by detecting chunks with the same
+base name and show group names as virtual composite files.
+This method is more prone to missing chunk errors (especially missing
+last chunk) than format with metadata enabled.
-- Config: test_mode
-- Env Var: RCLONE_B2_TEST_MODE
-- Type: string
-- Required: false
-#### --b2-versions
+### Hashsums
-Include old versions in directory listings.
+Chunker supports hashsums only when a compatible metadata is present.
+Hence, if you choose metadata format of `none`, chunker will report hashsum
+as `UNSUPPORTED`.
-Note that when using this no file write operations are permitted,
-so you can't upload files or delete them.
+Please note that by default metadata is stored only for composite files.
+If a file is smaller than configured chunk size, chunker will transparently
+redirect hash requests to wrapped remote, so support depends on that.
+You will see the empty string as a hashsum of requested type for small
+files if the wrapped remote doesn't support it.
-Properties:
+Many storage backends support MD5 and SHA1 hash types, so does chunker.
+With chunker you can choose one or another but not both.
+MD5 is set by default as the most supported type.
+Since chunker keeps hashes for composite files and falls back to the
+wrapped remote hash for non-chunked ones, we advise you to choose the same
+hash type as supported by wrapped remote so that your file listings
+look coherent.
-- Config: versions
-- Env Var: RCLONE_B2_VERSIONS
-- Type: bool
-- Default: false
+If your storage backend does not support MD5 or SHA1 but you need consistent
+file hashing, configure chunker with `md5all` or `sha1all`. These two modes
+guarantee given hash for all files. If wrapped remote doesn't support it,
+chunker will then add metadata to all files, even small. However, this can
+double the amount of small files in storage and incur additional service charges.
+You can even use chunker to force md5/sha1 support in any other remote
+at expense of sidecar meta objects by setting e.g. `hash_type=sha1all`
+to force hashsums and `chunk_size=1P` to effectively disable chunking.
-#### --b2-version-at
+Normally, when a file is copied to chunker controlled remote, chunker
+will ask the file source for compatible file hash and revert to on-the-fly
+calculation if none is found. This involves some CPU overhead but provides
+a guarantee that given hashsum is available. Also, chunker will reject
+a server-side copy or move operation if source and destination hashsum
+types are different resulting in the extra network bandwidth, too.
+In some rare cases this may be undesired, so chunker provides two optional
+choices: `sha1quick` and `md5quick`. If the source does not support primary
+hash type and the quick mode is enabled, chunker will try to fall back to
+the secondary type. This will save CPU and bandwidth but can result in empty
+hashsums at destination. Beware of consequences: the `sync` command will
+revert (sometimes silently) to time/size comparison if compatible hashsums
+between source and target are not found.
-Show file versions as they were at the specified time.
-Note that when using this no file write operations are permitted,
-so you can't upload files or delete them.
+### Modification times
-Properties:
+Chunker stores modification times using the wrapped remote so support
+depends on that. For a small non-chunked file the chunker overlay simply
+manipulates modification time of the wrapped remote file.
+For a composite file with metadata chunker will get and set
+modification time of the metadata object on the wrapped remote.
+If file is chunked but metadata format is `none` then chunker will
+use modification time of the first data chunk.
-- Config: version_at
-- Env Var: RCLONE_B2_VERSION_AT
-- Type: Time
-- Default: off
-#### --b2-upload-cutoff
+### Migrations
-Cutoff for switching to chunked upload.
+The idiomatic way to migrate to a different chunk size, hash type, transaction
+style or chunk naming scheme is to:
-Files above this size will be uploaded in chunks of "--b2-chunk-size".
+- Collect all your chunked files under a directory and have your
+ chunker remote point to it.
+- Create another directory (most probably on the same cloud storage)
+ and configure a new remote with desired metadata format,
+ hash type, chunk naming etc.
+- Now run `rclone sync --interactive oldchunks: newchunks:` and all your data
+ will be transparently converted in transfer.
+ This may take some time, yet chunker will try server-side
+ copy if possible.
+- After checking data integrity you may remove configuration section
+ of the old remote.
-This value should be set no larger than 4.657 GiB (== 5 GB).
+If rclone gets killed during a long operation on a big composite file,
+hidden temporary chunks may stay in the directory. They will not be
+shown by the `list` command but will eat up your account quota.
+Please note that the `deletefile` command deletes only active
+chunks of a file. As a workaround, you can use remote of the wrapped
+file system to see them.
+An easy way to get rid of hidden garbage is to copy littered directory
+somewhere using the chunker remote and purge the original directory.
+The `copy` command will copy only active chunks while the `purge` will
+remove everything including garbage.
-Properties:
-- Config: upload_cutoff
-- Env Var: RCLONE_B2_UPLOAD_CUTOFF
-- Type: SizeSuffix
-- Default: 200Mi
+### Caveats and Limitations
-#### --b2-copy-cutoff
+Chunker requires wrapped remote to support server-side `move` (or `copy` +
+`delete`) operations, otherwise it will explicitly refuse to start.
+This is because it internally renames temporary chunk files to their final
+names when an operation completes successfully.
-Cutoff for switching to multipart copy.
+Chunker encodes chunk number in file name, so with default `name_format`
+setting it adds 17 characters. Also chunker adds 7 characters of temporary
+suffix during operations. Many file systems limit base file name without path
+by 255 characters. Using rclone's crypt remote as a base file system limits
+file name by 143 characters. Thus, maximum name length is 231 for most files
+and 119 for chunker-over-crypt. A user in need can change name format to
+e.g. `*.rcc##` and save 10 characters (provided at most 99 chunks per file).
-Any files larger than this that need to be server-side copied will be
-copied in chunks of this size.
+Note that a move implemented using the copy-and-delete method may incur
+double charging with some cloud storage providers.
-The minimum is 0 and the maximum is 4.6 GiB.
+Chunker will not automatically rename existing chunks when you run
+`rclone config` on a live remote and change the chunk name format.
+Beware that in result of this some files which have been treated as chunks
+before the change can pop up in directory listings as normal files
+and vice versa. The same warning holds for the chunk size.
+If you desperately need to change critical chunking settings, you should
+run data migration as described above.
-Properties:
+If wrapped remote is case insensitive, the chunker overlay will inherit
+that property (so you can't have a file called "Hello.doc" and "hello.doc"
+in the same directory).
-- Config: copy_cutoff
-- Env Var: RCLONE_B2_COPY_CUTOFF
-- Type: SizeSuffix
-- Default: 4Gi
+Chunker included in rclone releases up to `v1.54` can sometimes fail to
+detect metadata produced by recent versions of rclone. We recommend users
+to keep rclone up-to-date to avoid data corruption.
-#### --b2-chunk-size
+Changing `transactions` is dangerous and requires explicit migration.
-Upload chunk size.
-When uploading large files, chunk the file into this size.
+### Standard options
-Must fit in memory. These chunks are buffered in memory and there
-might a maximum of "--transfers" chunks in progress at once.
+Here are the Standard options specific to chunker (Transparently chunk/split large files).
-5,000,000 Bytes is the minimum size.
+#### --chunker-remote
-Properties:
+Remote to chunk/unchunk.
-- Config: chunk_size
-- Env Var: RCLONE_B2_CHUNK_SIZE
-- Type: SizeSuffix
-- Default: 96Mi
+Normally should contain a ':' and a path, e.g. "myremote:path/to/dir",
+"myremote:bucket" or maybe "myremote:" (not recommended).
-#### --b2-upload-concurrency
+Properties:
-Concurrency for multipart uploads.
+- Config: remote
+- Env Var: RCLONE_CHUNKER_REMOTE
+- Type: string
+- Required: true
-This is the number of chunks of the same file that are uploaded
-concurrently.
+#### --chunker-chunk-size
-Note that chunks are stored in memory and there may be up to
-"--transfers" * "--b2-upload-concurrency" chunks stored at once
-in memory.
+Files larger than chunk size will be split in chunks.
Properties:
-- Config: upload_concurrency
-- Env Var: RCLONE_B2_UPLOAD_CONCURRENCY
-- Type: int
-- Default: 16
+- Config: chunk_size
+- Env Var: RCLONE_CHUNKER_CHUNK_SIZE
+- Type: SizeSuffix
+- Default: 2Gi
-#### --b2-disable-checksum
+#### --chunker-hash-type
-Disable checksums for large (> upload cutoff) files.
+Choose how chunker handles hash sums.
-Normally rclone will calculate the SHA1 checksum of the input before
-uploading it so it can add it to metadata on the object. This is great
-for data integrity checking but can cause long delays for large files
-to start uploading.
+All modes but "none" require metadata.
Properties:
-- Config: disable_checksum
-- Env Var: RCLONE_B2_DISABLE_CHECKSUM
-- Type: bool
-- Default: false
+- Config: hash_type
+- Env Var: RCLONE_CHUNKER_HASH_TYPE
+- Type: string
+- Default: "md5"
+- Examples:
+ - "none"
+ - Pass any hash supported by wrapped remote for non-chunked files.
+ - Return nothing otherwise.
+ - "md5"
+ - MD5 for composite files.
+ - "sha1"
+ - SHA1 for composite files.
+ - "md5all"
+ - MD5 for all files.
+ - "sha1all"
+ - SHA1 for all files.
+ - "md5quick"
+ - Copying a file to chunker will request MD5 from the source.
+ - Falling back to SHA1 if unsupported.
+ - "sha1quick"
+ - Similar to "md5quick" but prefers SHA1 over MD5.
-#### --b2-download-url
+### Advanced options
-Custom endpoint for downloads.
+Here are the Advanced options specific to chunker (Transparently chunk/split large files).
-This is usually set to a Cloudflare CDN URL as Backblaze offers
-free egress for data downloaded through the Cloudflare network.
-Rclone works with private buckets by sending an "Authorization" header.
-If the custom endpoint rewrites the requests for authentication,
-e.g., in Cloudflare Workers, this header needs to be handled properly.
-Leave blank if you want to use the endpoint provided by Backblaze.
+#### --chunker-name-format
-The URL provided here SHOULD have the protocol and SHOULD NOT have
-a trailing slash or specify the /file/bucket subpath as rclone will
-request files with "{download_url}/file/{bucket_name}/{path}".
+String format of chunk file names.
-Example:
-> https://mysubdomain.mydomain.tld
-(No trailing "/", "file" or "bucket")
+The two placeholders are: base file name (*) and chunk number (#...).
+There must be one and only one asterisk and one or more consecutive hash characters.
+If chunk number has less digits than the number of hashes, it is left-padded by zeros.
+If there are more digits in the number, they are left as is.
+Possible chunk files are ignored if their name does not match given format.
Properties:
-- Config: download_url
-- Env Var: RCLONE_B2_DOWNLOAD_URL
+- Config: name_format
+- Env Var: RCLONE_CHUNKER_NAME_FORMAT
- Type: string
-- Required: false
+- Default: "*.rclone_chunk.###"
-#### --b2-download-auth-duration
+#### --chunker-start-from
-Time before the authorization token will expire in s or suffix ms|s|m|h|d.
+Minimum valid chunk number. Usually 0 or 1.
-The duration before the download authorization token will expire.
-The minimum value is 1 second. The maximum value is one week.
+By default chunk numbers start from 1.
Properties:
-- Config: download_auth_duration
-- Env Var: RCLONE_B2_DOWNLOAD_AUTH_DURATION
-- Type: Duration
-- Default: 1w
+- Config: start_from
+- Env Var: RCLONE_CHUNKER_START_FROM
+- Type: int
+- Default: 1
-#### --b2-memory-pool-flush-time
+#### --chunker-meta-format
-How often internal memory buffer pools will be flushed. (no longer used)
+Format of the metadata object or "none".
+
+By default "simplejson".
+Metadata is a small JSON file named after the composite file.
Properties:
-- Config: memory_pool_flush_time
-- Env Var: RCLONE_B2_MEMORY_POOL_FLUSH_TIME
-- Type: Duration
-- Default: 1m0s
+- Config: meta_format
+- Env Var: RCLONE_CHUNKER_META_FORMAT
+- Type: string
+- Default: "simplejson"
+- Examples:
+ - "none"
+ - Do not use metadata files at all.
+ - Requires hash type "none".
+ - "simplejson"
+ - Simple JSON supports hash sums and chunk validation.
+ -
+ - It has the following fields: ver, size, nchunks, md5, sha1.
-#### --b2-memory-pool-use-mmap
+#### --chunker-fail-hard
-Whether to use mmap buffers in internal memory pool. (no longer used)
+Choose how chunker should handle files with missing or invalid chunks.
Properties:
-- Config: memory_pool_use_mmap
-- Env Var: RCLONE_B2_MEMORY_POOL_USE_MMAP
+- Config: fail_hard
+- Env Var: RCLONE_CHUNKER_FAIL_HARD
- Type: bool
- Default: false
+- Examples:
+ - "true"
+ - Report errors and abort current command.
+ - "false"
+ - Warn user, skip incomplete file and proceed.
-#### --b2-encoding
-
-The encoding for the backend.
+#### --chunker-transactions
-See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info.
+Choose how chunker should handle temporary files during transactions.
Properties:
-- Config: encoding
-- Env Var: RCLONE_B2_ENCODING
-- Type: MultiEncoder
-- Default: Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot
-
-
-
-## Limitations
-
-`rclone about` is not supported by the B2 backend. Backends without
-this capability cannot determine free space for an rclone mount or
-use policy `mfs` (most free space) as a member of an rclone union
-remote.
-
-See [List of backends that do not support rclone about](https://rclone.org/overview/#optional-features) and [rclone about](https://rclone.org/commands/rclone_about/)
+- Config: transactions
+- Env Var: RCLONE_CHUNKER_TRANSACTIONS
+- Type: string
+- Default: "rename"
+- Examples:
+ - "rename"
+ - Rename temporary files after a successful transaction.
+ - "norename"
+ - Leave temporary file names and write transaction ID to metadata file.
+ - Metadata is required for no rename transactions (meta format cannot be "none").
+ - If you are using norename transactions you should be careful not to downgrade Rclone
+ - as older versions of Rclone don't support this transaction style and will misinterpret
+ - files manipulated by norename transactions.
+ - This method is EXPERIMENTAL, don't use on production systems.
+ - "auto"
+ - Rename or norename will be used depending on capabilities of the backend.
+ - If meta format is set to "none", rename transactions will always be used.
+ - This method is EXPERIMENTAL, don't use on production systems.
-# Box
-Paths are specified as `remote:path`
-Paths may be as deep as required, e.g. `remote:directory/subdirectory`.
+# Citrix ShareFile
-The initial setup for Box involves getting a token from Box which you
-can do either in your browser, or with a config.json downloaded from Box
-to use JWT authentication. `rclone config` walks you through it.
+[Citrix ShareFile](https://sharefile.com) is a secure file sharing and transfer service aimed as business.
## Configuration
+The initial setup for Citrix ShareFile involves getting a token from
+Citrix ShareFile which you can in your browser. `rclone config` walks you
+through it.
+
Here is an example of how to make a remote called `remote`. First run:
rclone config
@@ -27157,153 +29146,35 @@ q) Quit config
n/s/q> n
name> remote
Type of storage to configure.
-Choose a number from below, or type in your own value
-[snip]
-XX / Box
- \ "box"
-[snip]
-Storage> box
-Box App Client Id - leave blank normally.
-client_id>
-Box App Client Secret - leave blank normally.
-client_secret>
-Box App config.json location
-Leave blank normally.
-Enter a string value. Press Enter for the default ("").
-box_config_file>
-Box App Primary Access Token
-Leave blank normally.
Enter a string value. Press Enter for the default ("").
-access_token>
-
-Enter a string value. Press Enter for the default ("user").
Choose a number from below, or type in your own value
- 1 / Rclone should act on behalf of a user
- \ "user"
- 2 / Rclone should act on behalf of a service account
- \ "enterprise"
-box_sub_type>
-Remote config
-Use web browser to automatically authenticate rclone with remote?
- * Say Y if the machine running rclone has a web browser you can use
- * Say N if running rclone on a (remote) machine without web browser access
-If not sure try Y. If Y failed, try N.
-y) Yes
-n) No
-y/n> y
-If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth
-Log in and authorize rclone for access
-Waiting for code...
-Got code
---------------------
-[remote]
-client_id =
-client_secret =
-token = {"access_token":"XXX","token_type":"bearer","refresh_token":"XXX","expiry":"XXX"}
---------------------
-y) Yes this is OK
-e) Edit this remote
-d) Delete this remote
-y/e/d> y
-```
-
-See the [remote setup docs](https://rclone.org/remote_setup/) for how to set it up on a
-machine with no Internet browser available.
-
-Note that rclone runs a webserver on your local machine to collect the
-token as returned from Box. This only runs from the moment it opens
-your browser to the moment you get back the verification code. This
-is on `http://127.0.0.1:53682/` and this it may require you to unblock
-it temporarily if you are running a host firewall.
-
-Once configured you can then use `rclone` like this,
-
-List directories in top level of your Box
-
- rclone lsd remote:
-
-List all the files in your Box
-
- rclone ls remote:
-
-To copy a local directory to an Box directory called backup
-
- rclone copy /home/source remote:backup
-
-### Using rclone with an Enterprise account with SSO
-
-If you have an "Enterprise" account type with Box with single sign on
-(SSO), you need to create a password to use Box with rclone. This can
-be done at your Enterprise Box account by going to Settings, "Account"
-Tab, and then set the password in the "Authentication" field.
-
-Once you have done this, you can setup your Enterprise Box account
-using the same procedure detailed above in the, using the password you
-have just set.
-
-### Invalid refresh token
-
-According to the [box docs](https://developer.box.com/v2.0/docs/oauth-20#section-6-using-the-access-and-refresh-tokens):
-
-> Each refresh_token is valid for one use in 60 days.
-
-This means that if you
-
- * Don't use the box remote for 60 days
- * Copy the config file with a box refresh token in and use it in two places
- * Get an error on a token refresh
-
-then rclone will return an error which includes the text `Invalid
-refresh token`.
-
-To fix this you will need to use oauth2 again to update the refresh
-token. You can use the methods in [the remote setup
-docs](https://rclone.org/remote_setup/), bearing in mind that if you use the copy the
-config file method, you should not use that remote on the computer you
-did the authentication on.
-
-Here is how to do it.
-
-```
-$ rclone config
-Current remotes:
-
-Name Type
-==== ====
-remote box
-
-e) Edit existing remote
-n) New remote
-d) Delete remote
-r) Rename remote
-c) Copy remote
-s) Set configuration password
-q) Quit config
-e/n/d/r/c/s/q> e
-Choose a number from below, or type in an existing value
- 1 > remote
-remote> remote
---------------------
-[remote]
-type = box
-token = {"access_token":"XXX","token_type":"bearer","refresh_token":"XXX","expiry":"2017-07-08T23:40:08.059167677+01:00"}
---------------------
-Edit remote
-Value "client_id" = ""
-Edit? (y/n)>
-y) Yes
-n) No
-y/n> n
-Value "client_secret" = ""
-Edit? (y/n)>
+XX / Citrix Sharefile
+ \ "sharefile"
+Storage> sharefile
+** See help for sharefile backend at: https://rclone.org/sharefile/ **
+
+ID of the root folder
+
+Leave blank to access "Personal Folders". You can use one of the
+standard values here or any folder ID (long hex number ID).
+Enter a string value. Press Enter for the default ("").
+Choose a number from below, or type in your own value
+ 1 / Access the Personal Folders. (Default)
+ \ ""
+ 2 / Access the Favorites folder.
+ \ "favorites"
+ 3 / Access all the shared folders.
+ \ "allshared"
+ 4 / Access all the individual connectors.
+ \ "connectors"
+ 5 / Access the home, favorites, and shared folders as well as the connectors.
+ \ "top"
+root_folder_id>
+Edit advanced config? (y/n)
y) Yes
n) No
y/n> n
Remote config
-Already have a token - refresh?
-y) Yes
-n) No
-y/n> y
Use web browser to automatically authenticate rclone with remote?
* Say Y if the machine running rclone has a web browser you can use
* Say N if running rclone on a (remote) machine without web browser access
@@ -27311,14 +29182,15 @@ If not sure try Y. If Y failed, try N.
y) Yes
n) No
y/n> y
-If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth
+If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth?state=XXX
Log in and authorize rclone for access
Waiting for code...
Got code
--------------------
[remote]
-type = box
-token = {"access_token":"YYY","token_type":"bearer","refresh_token":"YYY","expiry":"2017-07-23T12:22:29.259137901+01:00"}
+type = sharefile
+endpoint = https://XXX.sharefile.com
+token = {"access_token":"XXX","token_type":"bearer","refresh_token":"XXX","expiry":"2019-09-30T19:41:45.878561877+01:00"}
--------------------
y) Yes this is OK
e) Edit this remote
@@ -27326,15 +29198,47 @@ d) Delete this remote
y/e/d> y
```
-### Modified time and hashes
+See the [remote setup docs](https://rclone.org/remote_setup/) for how to set it up on a
+machine with no Internet browser available.
-Box allows modification times to be set on objects accurate to 1
+Note that rclone runs a webserver on your local machine to collect the
+token as returned from Citrix ShareFile. This only runs from the moment it opens
+your browser to the moment you get back the verification code. This
+is on `http://127.0.0.1:53682/` and this it may require you to unblock
+it temporarily if you are running a host firewall.
+
+Once configured you can then use `rclone` like this,
+
+List directories in top level of your ShareFile
+
+ rclone lsd remote:
+
+List all the files in your ShareFile
+
+ rclone ls remote:
+
+To copy a local directory to an ShareFile directory called backup
+
+ rclone copy /home/source remote:backup
+
+Paths may be as deep as required, e.g. `remote:directory/subdirectory`.
+
+### Modification times and hashes
+
+ShareFile allows modification times to be set on objects accurate to 1
second. These will be used to detect whether objects need syncing or
not.
-Box supports SHA1 type hashes, so you can use the `--checksum`
+ShareFile supports MD5 type hashes, so you can use the `--checksum`
flag.
+### Transfers
+
+For files above 128 MiB rclone will use a chunked transfer. Rclone will
+upload up to `--transfers` chunks at the same time (shared among all
+the multipart uploads). Chunks are buffered in memory and are
+normally 64 MiB so increasing `--transfers` will increase memory use.
+
### Restricted filename characters
In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters)
@@ -27342,64 +29246,33 @@ the following characters are also replaced:
| Character | Value | Replacement |
| --------- |:-----:|:-----------:|
-| \ | 0x5C | ï¼¼ |
+| \\ | 0x5C | ï¼¼ |
+| * | 0x2A | ＊ |
+| < | 0x3C | ＜ |
+| > | 0x3E | ＞ |
+| ? | 0x3F | ？ |
+| : | 0x3A | ： |
+| \| | 0x7C | ｜ |
+| " | 0x22 | ＂ |
-File names can also not end with the following characters.
-These only get replaced if they are the last character in the name:
+File names can also not start or end with the following characters.
+These only get replaced if they are the first or last character in the
+name:
| Character | Value | Replacement |
| --------- |:-----:|:-----------:|
| SP | 0x20 | â |
+| . | 0x2E | ． |
Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8),
as they can't be used in JSON strings.
-### Transfers
-
-For files above 50 MiB rclone will use a chunked transfer. Rclone will
-upload up to `--transfers` chunks at the same time (shared among all
-the multipart uploads). Chunks are buffered in memory and are
-normally 8 MiB so increasing `--transfers` will increase memory use.
-
-### Deleting files
-
-Depending on the enterprise settings for your user, the item will
-either be actually deleted from Box or moved to the trash.
-
-Emptying the trash is supported via the rclone however cleanup command
-however this deletes every trashed file and folder individually so it
-may take a very long time.
-Emptying the trash via the WebUI does not have this limitation
-so it is advised to empty the trash via the WebUI.
-
-### Root folder ID
-
-You can set the `root_folder_id` for rclone. This is the directory
-(identified by its `Folder ID`) that rclone considers to be the root
-of your Box drive.
-
-Normally you will leave this blank and rclone will determine the
-correct root to use itself.
-
-However you can set this to restrict rclone to a specific folder
-hierarchy.
-
-In order to do this you will have to find the `Folder ID` of the
-directory you wish rclone to display. This will be the last segment
-of the URL when you open the relevant folder in the Box web
-interface.
-
-So if the folder you want rclone to use has a URL which looks like
-`https://app.box.com/folder/11xxxxxxxxx8`
-in the browser, then you use `11xxxxxxxxx8` as
-the `root_folder_id` in the config.
-
### Standard options
-Here are the Standard options specific to box (Box).
+Here are the Standard options specific to sharefile (Citrix Sharefile).
-#### --box-client-id
+#### --sharefile-client-id
OAuth Client Id.
@@ -27408,11 +29281,11 @@ Leave blank normally.
Properties:
- Config: client_id
-- Env Var: RCLONE_BOX_CLIENT_ID
+- Env Var: RCLONE_SHAREFILE_CLIENT_ID
- Type: string
- Required: false
-#### --box-client-secret
+#### --sharefile-client-secret
OAuth Client Secret.
@@ -27421,70 +29294,51 @@ Leave blank normally.
Properties:
- Config: client_secret
-- Env Var: RCLONE_BOX_CLIENT_SECRET
-- Type: string
-- Required: false
-
-#### --box-box-config-file
-
-Box App config.json location
-
-Leave blank normally.
-
-Leading `~` will be expanded in the file name as will environment variables such as `${RCLONE_CONFIG_DIR}`.
-
-Properties:
-
-- Config: box_config_file
-- Env Var: RCLONE_BOX_BOX_CONFIG_FILE
+- Env Var: RCLONE_SHAREFILE_CLIENT_SECRET
- Type: string
- Required: false
-#### --box-access-token
+#### --sharefile-root-folder-id
-Box App Primary Access Token
+ID of the root folder.
-Leave blank normally.
+Leave blank to access "Personal Folders". You can use one of the
+standard values here or any folder ID (long hex number ID).
Properties:
-- Config: access_token
-- Env Var: RCLONE_BOX_ACCESS_TOKEN
+- Config: root_folder_id
+- Env Var: RCLONE_SHAREFILE_ROOT_FOLDER_ID
- Type: string
- Required: false
-
-#### --box-box-sub-type
-
-
-
-Properties:
-
-- Config: box_sub_type
-- Env Var: RCLONE_BOX_BOX_SUB_TYPE
-- Type: string
-- Default: "user"
- Examples:
- - "user"
- - Rclone should act on behalf of a user.
- - "enterprise"
- - Rclone should act on behalf of a service account.
+ - ""
+ - Access the Personal Folders (default).
+ - "favorites"
+ - Access the Favorites folder.
+ - "allshared"
+ - Access all the shared folders.
+ - "connectors"
+ - Access all the individual connectors.
+ - "top"
+ - Access the home, favorites, and shared folders as well as the connectors.
### Advanced options
-Here are the Advanced options specific to box (Box).
+Here are the Advanced options specific to sharefile (Citrix Sharefile).
-#### --box-token
+#### --sharefile-token
OAuth Access Token as a JSON blob.
Properties:
- Config: token
-- Env Var: RCLONE_BOX_TOKEN
+- Env Var: RCLONE_SHAREFILE_TOKEN
- Type: string
- Required: false
-#### --box-auth-url
+#### --sharefile-auth-url
Auth server URL.
@@ -27493,11 +29347,11 @@ Leave blank to use the provider defaults.
Properties:
- Config: auth_url
-- Env Var: RCLONE_BOX_AUTH_URL
+- Env Var: RCLONE_SHAREFILE_AUTH_URL
- Type: string
- Required: false
-#### --box-token-url
+#### --sharefile-token-url
Token server url.
@@ -27506,88 +29360,55 @@ Leave blank to use the provider defaults.
Properties:
- Config: token_url
-- Env Var: RCLONE_BOX_TOKEN_URL
+- Env Var: RCLONE_SHAREFILE_TOKEN_URL
- Type: string
- Required: false
-#### --box-root-folder-id
-
-Fill in for rclone to use a non root folder as its starting point.
-
-Properties:
-
-- Config: root_folder_id
-- Env Var: RCLONE_BOX_ROOT_FOLDER_ID
-- Type: string
-- Default: "0"
-
-#### --box-upload-cutoff
+#### --sharefile-upload-cutoff
-Cutoff for switching to multipart upload (>= 50 MiB).
+Cutoff for switching to multipart upload.
Properties:
- Config: upload_cutoff
-- Env Var: RCLONE_BOX_UPLOAD_CUTOFF
+- Env Var: RCLONE_SHAREFILE_UPLOAD_CUTOFF
- Type: SizeSuffix
-- Default: 50Mi
-
-#### --box-commit-retries
-
-Max number of times to try committing a multipart file.
-
-Properties:
-
-- Config: commit_retries
-- Env Var: RCLONE_BOX_COMMIT_RETRIES
-- Type: int
-- Default: 100
-
-#### --box-list-chunk
+- Default: 128Mi
-Size of listing chunk 1-1000.
+#### --sharefile-chunk-size
-Properties:
+Upload chunk size.
-- Config: list_chunk
-- Env Var: RCLONE_BOX_LIST_CHUNK
-- Type: int
-- Default: 1000
+Must a power of 2 >= 256k.
-#### --box-owned-by
+Making this larger will improve performance, but note that each chunk
+is buffered in memory one per transfer.
-Only show items owned by the login (email address) passed in.
+Reducing this will reduce memory usage but decrease performance.
Properties:
-- Config: owned_by
-- Env Var: RCLONE_BOX_OWNED_BY
-- Type: string
-- Required: false
-
-#### --box-impersonate
-
-Impersonate this user ID when using a service account.
+- Config: chunk_size
+- Env Var: RCLONE_SHAREFILE_CHUNK_SIZE
+- Type: SizeSuffix
+- Default: 64Mi
-Settng this flag allows rclone, when using a JWT service account, to
-act on behalf of another user by setting the as-user header.
+#### --sharefile-endpoint
-The user ID is the Box identifier for a user. User IDs can found for
-any user via the GET /users endpoint, which is only available to
-admins, or by calling the GET /users/me endpoint with an authenticated
-user session.
+Endpoint for API calls.
-See: https://developer.box.com/guides/authentication/jwt/as-user/
+This is usually auto discovered as part of the oauth process, but can
+be set manually to something like: https://XXX.sharefile.com
Properties:
-- Config: impersonate
-- Env Var: RCLONE_BOX_IMPERSONATE
+- Config: endpoint
+- Env Var: RCLONE_SHAREFILE_ENDPOINT
- Type: string
- Required: false
-#### --box-encoding
+#### --sharefile-encoding
The encoding for the backend.
@@ -27596,366 +29417,446 @@ See the [encoding section in the overview](https://rclone.org/overview/#encoding
Properties:
- Config: encoding
-- Env Var: RCLONE_BOX_ENCODING
-- Type: MultiEncoder
-- Default: Slash,BackSlash,Del,Ctl,RightSpace,InvalidUtf8,Dot
-
+- Env Var: RCLONE_SHAREFILE_ENCODING
+- Type: Encoding
+- Default: Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,LeftSpace,LeftPeriod,RightSpace,RightPeriod,InvalidUtf8,Dot
## Limitations
-Note that Box is case insensitive so you can't have a file called
+Note that ShareFile is case insensitive so you can't have a file called
"Hello.doc" and one called "hello.doc".
-Box file names can't have the `\` character in. rclone maps this to
-and from an identical looking unicode equivalent `ï¼¼` (U+FF3C Fullwidth
-Reverse Solidus).
-
-Box only supports filenames up to 255 characters in length.
-
-Box has [API rate limits](https://developer.box.com/guides/api-calls/permissions-and-errors/rate-limits/) that sometimes reduce the speed of rclone.
+ShareFile only supports filenames up to 256 characters in length.
-`rclone about` is not supported by the Box backend. Backends without
+`rclone about` is not supported by the Citrix ShareFile backend. Backends without
this capability cannot determine free space for an rclone mount or
use policy `mfs` (most free space) as a member of an rclone union
remote.
See [List of backends that do not support rclone about](https://rclone.org/overview/#optional-features) and [rclone about](https://rclone.org/commands/rclone_about/)
-## Get your own Box App ID
-
-Here is how to create your own Box App ID for rclone:
-
-1. Go to the [Box Developer Console](https://app.box.com/developers/console)
-and login, then click `My Apps` on the sidebar. Click `Create New App`
-and select `Custom App`.
-
-2. In the first screen on the box that pops up, you can pretty much enter
-whatever you want. The `App Name` can be whatever. For `Purpose` choose
-automation to avoid having to fill out anything else. Click `Next`.
-
-3. In the second screen of the creation screen, select
-`User Authentication (OAuth 2.0)`. Then click `Create App`.
-
-4. You should now be on the `Configuration` tab of your new app. If not,
-click on it at the top of the webpage. Copy down `Client ID`
-and `Client Secret`, you'll need those for rclone.
-
-5. Under "OAuth 2.0 Redirect URI", add `http://127.0.0.1:53682/`
-
-6. For `Application Scopes`, select `Read all files and folders stored in Box`
-and `Write all files and folders stored in box` (assuming you want to do both).
-Leave others unchecked. Click `Save Changes` at the top right.
+# Crypt
-# Cache
+Rclone `crypt` remotes encrypt and decrypt other remotes.
-The `cache` remote wraps another existing remote and stores file structure
-and its data for long running tasks like `rclone mount`.
+A remote of type `crypt` does not access a [storage system](https://rclone.org/overview/)
+directly, but instead wraps another remote, which in turn accesses
+the storage system. This is similar to how [alias](https://rclone.org/alias/),
+[union](https://rclone.org/union/), [chunker](https://rclone.org/chunker/)
+and a few others work. It makes the usage very flexible, as you can
+add a layer, in this case an encryption layer, on top of any other
+backend, even in multiple layers. Rclone's functionality
+can be used as with any other remote, for example you can
+[mount](https://rclone.org/commands/rclone_mount/) a crypt remote.
-## Status
+Accessing a storage system through a crypt remote realizes client-side
+encryption, which makes it safe to keep your data in a location you do
+not trust will not get compromised.
+When working against the `crypt` remote, rclone will automatically
+encrypt (before uploading) and decrypt (after downloading) on your local
+system as needed on the fly, leaving the data encrypted at rest in the
+wrapped remote. If you access the storage system using an application
+other than rclone, or access the wrapped remote directly using rclone,
+there will not be any encryption/decryption: Downloading existing content
+will just give you the encrypted (scrambled) format, and anything you
+upload will *not* become encrypted.
-The cache backend code is working but it currently doesn't
-have a maintainer so there are [outstanding bugs](https://github.com/rclone/rclone/issues?q=is%3Aopen+is%3Aissue+label%3Abug+label%3A%22Remote%3A+Cache%22) which aren't getting fixed.
+The encryption is a secret-key encryption (also called symmetric key encryption)
+algorithm, where a password (or pass phrase) is used to generate real encryption key.
+The password can be supplied by user, or you may chose to let rclone
+generate one. It will be stored in the configuration file, in a lightly obscured form.
+If you are in an environment where you are not able to keep your configuration
+secured, you should add
+[configuration encryption](https://rclone.org/docs/#configuration-encryption)
+as protection. As long as you have this configuration file, you will be able to
+decrypt your data. Without the configuration file, as long as you remember
+the password (or keep it in a safe place), you can re-create the configuration
+and gain access to the existing data. You may also configure a corresponding
+remote in a different installation to access the same data.
+See below for guidance to [changing password](#changing-password).
-The cache backend is due to be phased out in favour of the VFS caching
-layer eventually which is more tightly integrated into rclone.
+Encryption uses [cryptographic salt](https://en.wikipedia.org/wiki/Salt_(cryptography)),
+to permute the encryption key so that the same string may be encrypted in
+different ways. When configuring the crypt remote it is optional to enter a salt,
+or to let rclone generate a unique salt. If omitted, rclone uses a built-in unique string.
+Normally in cryptography, the salt is stored together with the encrypted content,
+and do not have to be memorized by the user. This is not the case in rclone,
+because rclone does not store any additional information on the remotes. Use of
+custom salt is effectively a second password that must be memorized.
-Until this happens we recommend only using the cache backend if you
-find you can't work without it. There are many docs online describing
-the use of the cache backend to minimize API hits and by-and-large
-these are out of date and the cache backend isn't needed in those
-scenarios any more.
+[File content](#file-encryption) encryption is performed using
+[NaCl SecretBox](https://godoc.org/golang.org/x/crypto/nacl/secretbox),
+based on XSalsa20 cipher and Poly1305 for integrity.
+[Names](#name-encryption) (file- and directory names) are also encrypted
+by default, but this has some implications and is therefore
+possible to be turned off.
## Configuration
-To get started you just need to have an existing remote which can be configured
-with `cache`.
+Here is an example of how to make a remote called `secret`.
-Here is an example of how to make a remote called `test-cache`. First run:
+To use `crypt`, first set up the underlying remote. Follow the
+`rclone config` instructions for the specific backend.
- rclone config
+Before configuring the crypt remote, check the underlying remote is
+working. In this example the underlying remote is called `remote`.
+We will configure a path `path` within this remote to contain the
+encrypted content. Anything inside `remote:path` will be encrypted
+and anything outside will not.
-This will guide you through an interactive setup process:
+Configure `crypt` using `rclone config`. In this example the `crypt`
+remote is called `secret`, to differentiate it from the underlying
+`remote`.
+
+When you are done you can use the crypt remote named `secret` just
+as you would with any other remote, e.g. `rclone copy D:\docs secret:\docs`,
+and rclone will encrypt and decrypt as needed on the fly.
+If you access the wrapped remote `remote:path` directly you will bypass
+the encryption, and anything you read will be in encrypted form, and
+anything you write will be unencrypted. To avoid issues it is best to
+configure a dedicated path for encrypted content, and access it
+exclusively through a crypt remote.
```
No remotes found, make a new one?
n) New remote
-r) Rename remote
-c) Copy remote
s) Set configuration password
q) Quit config
-n/r/c/s/q> n
-name> test-cache
+n/s/q> n
+name> secret
Type of storage to configure.
+Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
[snip]
-XX / Cache a remote
- \ "cache"
+XX / Encrypt/Decrypt a remote
+ \ "crypt"
[snip]
-Storage> cache
-Remote to cache.
-Normally should contain a ':' and a path, e.g. "myremote:path/to/dir",
+Storage> crypt
+** See help for crypt backend at: https://rclone.org/crypt/ **
+
+Remote to encrypt/decrypt.
+Normally should contain a ':' and a path, eg "myremote:path/to/dir",
"myremote:bucket" or maybe "myremote:" (not recommended).
-remote> local:/test
-Optional: The URL of the Plex server
-plex_url> http://127.0.0.1:32400
-Optional: The username of the Plex user
-plex_username> dummyusername
-Optional: The password of the Plex user
+Enter a string value. Press Enter for the default ("").
+remote> remote:path
+How to encrypt the filenames.
+Enter a string value. Press Enter for the default ("standard").
+Choose a number from below, or type in your own value.
+ / Encrypt the filenames.
+ 1 | See the docs for the details.
+ \ "standard"
+ 2 / Very simple filename obfuscation.
+ \ "obfuscate"
+ / Don't encrypt the file names.
+ 3 | Adds a ".bin" extension only.
+ \ "off"
+filename_encryption>
+Option to either encrypt directory names or leave them intact.
+
+NB If filename_encryption is "off" then this option will do nothing.
+Enter a boolean value (true or false). Press Enter for the default ("true").
+Choose a number from below, or type in your own value
+ 1 / Encrypt directory names.
+ \ "true"
+ 2 / Don't encrypt directory names, leave them intact.
+ \ "false"
+directory_name_encryption>
+Password or pass phrase for encryption.
y) Yes type in my own password
g) Generate random password
-n) No leave this optional password blank
-y/g/n> y
+y/g> y
Enter the password:
password:
Confirm the password:
password:
-The size of a chunk. Lower value good for slow connections but can affect seamless reading.
-Default: 5M
-Choose a number from below, or type in your own value
- 1 / 1 MiB
- \ "1M"
- 2 / 5 MiB
- \ "5M"
- 3 / 10 MiB
- \ "10M"
-chunk_size> 2
-How much time should object info (file size, file hashes, etc.) be stored in cache. Use a very high value if you don't plan on changing the source FS from outside the cache.
-Accepted units are: "s", "m", "h".
-Default: 5m
-Choose a number from below, or type in your own value
- 1 / 1 hour
- \ "1h"
- 2 / 24 hours
- \ "24h"
- 3 / 24 hours
- \ "48h"
-info_age> 2
-The maximum size of stored chunks. When the storage grows beyond this size, the oldest chunks will be deleted.
-Default: 10G
-Choose a number from below, or type in your own value
- 1 / 500 MiB
- \ "500M"
- 2 / 1 GiB
- \ "1G"
- 3 / 10 GiB
- \ "10G"
-chunk_total_size> 3
+Password or pass phrase for salt. Optional but recommended.
+Should be different to the previous password.
+y) Yes type in my own password
+g) Generate random password
+n) No leave this optional password blank (default)
+y/g/n> g
+Password strength in bits.
+64 is just about memorable
+128 is secure
+1024 is the maximum
+Bits> 128
+Your password is: JAsJvRcgR-_veXNfy_sGmQ
+Use this password? Please note that an obscured version of this
+password (and not the password itself) will be stored under your
+configuration file, so keep this generated password in a safe place.
+y) Yes (default)
+n) No
+y/n>
+Edit advanced config? (y/n)
+y) Yes
+n) No (default)
+y/n>
Remote config
--------------------
-[test-cache]
-remote = local:/test
-plex_url = http://127.0.0.1:32400
-plex_username = dummyusername
-plex_password = *** ENCRYPTED ***
-chunk_size = 5M
-info_age = 48h
-chunk_total_size = 10G
+[secret]
+type = crypt
+remote = remote:path
+password = *** ENCRYPTED ***
+password2 = *** ENCRYPTED ***
+--------------------
+y) Yes this is OK (default)
+e) Edit this remote
+d) Delete this remote
+y/e/d>
```
-You can then use it like this,
-
-List directories in top level of your drive
-
- rclone lsd test-cache:
-
-List all the files in your drive
-
- rclone ls test-cache:
-
-To start a cached mount
+**Important** The crypt password stored in `rclone.conf` is lightly
+obscured. That only protects it from cursory inspection. It is not
+secure unless [configuration encryption](https://rclone.org/docs/#configuration-encryption) of `rclone.conf` is specified.
- rclone mount --allow-other test-cache: /var/tmp/test-cache
+A long passphrase is recommended, or `rclone config` can generate a
+random one.
-### Write Features ###
+The obscured password is created using AES-CTR with a static key. The
+salt is stored verbatim at the beginning of the obscured password. This
+static key is shared between all versions of rclone.
-### Offline uploading ###
+If you reconfigure rclone with the same passwords/passphrases
+elsewhere it will be compatible, but the obscured version will be different
+due to the different salt.
-In an effort to make writing through cache more reliable, the backend
-now supports this feature which can be activated by specifying a
-`cache-tmp-upload-path`.
+Rclone does not encrypt
-A files goes through these states when using this feature:
+ * file length - this can be calculated within 16 bytes
+ * modification time - used for syncing
-1. An upload is started (usually by copying a file on the cache remote)
-2. When the copy to the temporary location is complete the file is part
-of the cached remote and looks and behaves like any other file (reading included)
-3. After `cache-tmp-wait-time` passes and the file is next in line, `rclone move`
-is used to move the file to the cloud provider
-4. Reading the file still works during the upload but most modifications on it will be prohibited
-5. Once the move is complete the file is unlocked for modifications as it
-becomes as any other regular file
-6. If the file is being read through `cache` when it's actually
-deleted from the temporary path then `cache` will simply swap the source
-to the cloud provider without interrupting the reading (small blip can happen though)
+### Specifying the remote
-Files are uploaded in sequence and only one file is uploaded at a time.
-Uploads will be stored in a queue and be processed based on the order they were added.
-The queue and the temporary storage is persistent across restarts but
-can be cleared on startup with the `--cache-db-purge` flag.
+When configuring the remote to encrypt/decrypt, you may specify any
+string that rclone accepts as a source/destination of other commands.
-### Write Support ###
+The primary use case is to specify the path into an already configured
+remote (e.g. `remote:path/to/dir` or `remote:bucket`), such that
+data in a remote untrusted location can be stored encrypted.
-Writes are supported through `cache`.
-One caveat is that a mounted cache remote does not add any retry or fallback
-mechanism to the upload operation. This will depend on the implementation
-of the wrapped remote. Consider using `Offline uploading` for reliable writes.
+You may also specify a local filesystem path, such as
+`/path/to/dir` on Linux, `C:\path\to\dir` on Windows. By creating
+a crypt remote pointing to such a local filesystem path, you can
+use rclone as a utility for pure local file encryption, for example
+to keep encrypted files on a removable USB drive.
-One special case is covered with `cache-writes` which will cache the file
-data at the same time as the upload when it is enabled making it available
-from the cache store immediately once the upload is finished.
+**Note**: A string which do not contain a `:` will by rclone be treated
+as a relative path in the local filesystem. For example, if you enter
+the name `remote` without the trailing `:`, it will be treated as
+a subdirectory of the current directory with name "remote".
-### Read Features ###
+If a path `remote:path/to/dir` is specified, rclone stores encrypted
+files in `path/to/dir` on the remote. With file name encryption, files
+saved to `secret:subdir/subfile` are stored in the unencrypted path
+`path/to/dir` but the `subdir/subpath` element is encrypted.
-#### Multiple connections ####
+The path you specify does not have to exist, rclone will create
+it when needed.
-To counter the high latency between a local PC where rclone is running
-and cloud providers, the cache remote can split multiple requests to the
-cloud provider for smaller file chunks and combines them together locally
-where they can be available almost immediately before the reader usually
-needs them.
+If you intend to use the wrapped remote both directly for keeping
+unencrypted content, as well as through a crypt remote for encrypted
+content, it is recommended to point the crypt remote to a separate
+directory within the wrapped remote. If you use a bucket-based storage
+system (e.g. Swift, S3, Google Compute Storage, B2) it is generally
+advisable to wrap the crypt remote around a specific bucket (`s3:bucket`).
+If wrapping around the entire root of the storage (`s3:`), and use the
+optional file name encryption, rclone will encrypt the bucket name.
-This is similar to buffering when media files are played online. Rclone
-will stay around the current marker but always try its best to stay ahead
-and prepare the data before.
+### Changing password
-#### Plex Integration ####
+Should the password, or the configuration file containing a lightly obscured
+form of the password, be compromised, you need to re-encrypt your data with
+a new password. Since rclone uses secret-key encryption, where the encryption
+key is generated directly from the password kept on the client, it is not
+possible to change the password/key of already encrypted content. Just changing
+the password configured for an existing crypt remote means you will no longer
+able to decrypt any of the previously encrypted content. The only possibility
+is to re-upload everything via a crypt remote configured with your new password.
-There is a direct integration with Plex which allows cache to detect during reading
-if the file is in playback or not. This helps cache to adapt how it queries
-the cloud provider depending on what is needed for.
+Depending on the size of your data, your bandwidth, storage quota etc, there are
+different approaches you can take:
+- If you have everything in a different location, for example on your local system,
+you could remove all of the prior encrypted files, change the password for your
+configured crypt remote (or delete and re-create the crypt configuration),
+and then re-upload everything from the alternative location.
+- If you have enough space on the storage system you can create a new crypt
+remote pointing to a separate directory on the same backend, and then use
+rclone to copy everything from the original crypt remote to the new,
+effectively decrypting everything on the fly using the old password and
+re-encrypting using the new password. When done, delete the original crypt
+remote directory and finally the rclone crypt configuration with the old password.
+All data will be streamed from the storage system and back, so you will
+get half the bandwidth and be charged twice if you have upload and download quota
+on the storage system.
-Scans will have a minimum amount of workers (1) while in a confirmed playback cache
-will deploy the configured number of workers.
+**Note**: A security problem related to the random password generator
+was fixed in rclone version 1.53.3 (released 2020-11-19). Passwords generated
+by rclone config in version 1.49.0 (released 2019-08-26) to 1.53.2
+(released 2020-10-26) are not considered secure and should be changed.
+If you made up your own password, or used rclone version older than 1.49.0 or
+newer than 1.53.2 to generate it, you are *not* affected by this issue.
+See [issue #4783](https://github.com/rclone/rclone/issues/4783) for more
+details, and a tool you can use to check if you are affected.
-This integration opens the doorway to additional performance improvements
-which will be explored in the near future.
+### Example
-**Note:** If Plex options are not configured, `cache` will function with its
-configured options without adapting any of its settings.
+Create the following file structure using "standard" file name
+encryption.
-How to enable? Run `rclone config` and add all the Plex options (endpoint, username
-and password) in your remote and it will be automatically enabled.
+```
+plaintext/
+├── file0.txt
+├── file1.txt
+└── subdir
+ ├── file2.txt
+ ├── file3.txt
+ └── subsubdir
+ └── file4.txt
+```
-Affected settings:
-- `cache-workers`: _Configured value_ during confirmed playback or _1_ all the other times
+Copy these to the remote, and list them
-##### Certificate Validation #####
+```
+$ rclone -q copy plaintext secret:
+$ rclone -q ls secret:
+ 7 file1.txt
+ 6 file0.txt
+ 8 subdir/file2.txt
+ 10 subdir/subsubdir/file4.txt
+ 9 subdir/file3.txt
+```
-When the Plex server is configured to only accept secure connections, it is
-possible to use `.plex.direct` URLs to ensure certificate validation succeeds.
-These URLs are used by Plex internally to connect to the Plex server securely.
+The crypt remote looks like
-The format for these URLs is the following:
+```
+$ rclone -q ls remote:path
+ 55 hagjclgavj2mbiqm6u6cnjjqcg
+ 54 v05749mltvv1tf4onltun46gls
+ 57 86vhrsv86mpbtd3a0akjuqslj8/dlj7fkq4kdq72emafg7a7s41uo
+ 58 86vhrsv86mpbtd3a0akjuqslj8/7uu829995du6o42n32otfhjqp4/b9pausrfansjth5ob3jkdqd4lc
+ 56 86vhrsv86mpbtd3a0akjuqslj8/8njh1sk437gttmep3p70g81aps
+```
-`https://ip-with-dots-replaced.server-hash.plex.direct:32400/`
+The directory structure is preserved
-The `ip-with-dots-replaced` part can be any IPv4 address, where the dots
-have been replaced with dashes, e.g. `127.0.0.1` becomes `127-0-0-1`.
+```
+$ rclone -q ls secret:subdir
+ 8 file2.txt
+ 9 file3.txt
+ 10 subsubdir/file4.txt
+```
-To get the `server-hash` part, the easiest way is to visit
+Without file name encryption `.bin` extensions are added to underlying
+names. This prevents the cloud provider attempting to interpret file
+content.
-https://plex.tv/api/resources?includeHttps=1&X-Plex-Token=your-plex-token
+```
+$ rclone -q ls remote:path
+ 54 file0.txt.bin
+ 57 subdir/file3.txt.bin
+ 56 subdir/file2.txt.bin
+ 58 subdir/subsubdir/file4.txt.bin
+ 55 file1.txt.bin
+```
-This page will list all the available Plex servers for your account
-with at least one `.plex.direct` link for each. Copy one URL and replace
-the IP address with the desired address. This can be used as the
-`plex_url` value.
+### File name encryption modes
-### Known issues ###
+Off
-#### Mount and --dir-cache-time ####
+ * doesn't hide file names or directory structure
+ * allows for longer file names (~246 characters)
+ * can use sub paths and copy single files
---dir-cache-time controls the first layer of directory caching which works at the mount layer.
-Being an independent caching mechanism from the `cache` backend, it will manage its own entries
-based on the configured time.
+Standard
-To avoid getting in a scenario where dir cache has obsolete data and cache would have the correct
-one, try to set `--dir-cache-time` to a lower time than `--cache-info-age`. Default values are
-already configured in this way.
+ * file names encrypted
+ * file names can't be as long (~143 characters)
+ * can use sub paths and copy single files
+ * directory structure visible
+ * identical files names will have identical uploaded names
+ * can use shortcuts to shorten the directory recursion
-#### Windows support - Experimental ####
+Obfuscation
-There are a couple of issues with Windows `mount` functionality that still require some investigations.
-It should be considered as experimental thus far as fixes come in for this OS.
+This is a simple "rotate" of the filename, with each file having a rot
+distance based on the filename. Rclone stores the distance at the
+beginning of the filename. A file called "hello" may become "53.jgnnq".
-Most of the issues seem to be related to the difference between filesystems
-on Linux flavors and Windows as cache is heavily dependent on them.
+Obfuscation is not a strong encryption of filenames, but hinders
+automated scanning tools picking up on filename patterns. It is an
+intermediate between "off" and "standard" which allows for longer path
+segment names.
-Any reports or feedback on how cache behaves on this OS is greatly appreciated.
-
-- https://github.com/rclone/rclone/issues/1935
-- https://github.com/rclone/rclone/issues/1907
-- https://github.com/rclone/rclone/issues/1834
+There is a possibility with some unicode based filenames that the
+obfuscation is weak and may map lower case characters to upper case
+equivalents.
-#### Risk of throttling ####
+Obfuscation cannot be relied upon for strong protection.
-Future iterations of the cache backend will make use of the pooling functionality
-of the cloud provider to synchronize and at the same time make writing through it
-more tolerant to failures.
+ * file names very lightly obfuscated
+ * file names can be longer than standard encryption
+ * can use sub paths and copy single files
+ * directory structure visible
+ * identical files names will have identical uploaded names
-There are a couple of enhancements in track to add these but in the meantime
-there is a valid concern that the expiring cache listings can lead to cloud provider
-throttles or bans due to repeated queries on it for very large mounts.
+Cloud storage systems have limits on file name length and
+total path length which rclone is more likely to breach using
+"Standard" file name encryption. Where file names are less than 156
+characters in length issues should not be encountered, irrespective of
+cloud storage provider.
-Some recommendations:
-- don't use a very small interval for entry information (`--cache-info-age`)
-- while writes aren't yet optimised, you can still write through `cache` which gives you the advantage
-of adding the file in the cache at the same time if configured to do so.
+An experimental advanced option `filename_encoding` is now provided to
+address this problem to a certain degree.
+For cloud storage systems with case sensitive file names (e.g. Google Drive),
+`base64` can be used to reduce file name length.
+For cloud storage systems using UTF-16 to store file names internally
+(e.g. OneDrive, Dropbox, Box), `base32768` can be used to drastically reduce
+file name length.
-Future enhancements:
+An alternative, future rclone file name encryption mode may tolerate
+backend provider path length limits.
-- https://github.com/rclone/rclone/issues/1937
-- https://github.com/rclone/rclone/issues/1936
+### Directory name encryption
-#### cache and crypt ####
+Crypt offers the option of encrypting dir names or leaving them intact.
+There are two options:
-One common scenario is to keep your data encrypted in the cloud provider
-using the `crypt` remote. `crypt` uses a similar technique to wrap around
-an existing remote and handles this translation in a seamless way.
+True
-There is an issue with wrapping the remotes in this order:
-**cloud remote** -> **crypt** -> **cache**
+Encrypts the whole file path including directory names
+Example:
+`1/12/123.txt` is encrypted to
+`p0e52nreeaj0a5ea7s64m4j72s/l42g6771hnv3an9cgc8cr2n1ng/qgm4avr35m5loi1th53ato71v0`
-During testing, I experienced a lot of bans with the remotes in this order.
-I suspect it might be related to how crypt opens files on the cloud provider
-which makes it think we're downloading the full file instead of small chunks.
-Organizing the remotes in this order yields better results:
-**cloud remote** -> **cache** -> **crypt**
+False
-#### absolute remote paths ####
+Only encrypts file names, skips directory names
+Example:
+`1/12/123.txt` is encrypted to
+`1/12/qgm4avr35m5loi1th53ato71v0`
-`cache` can not differentiate between relative and absolute paths for the wrapped remote.
-Any path given in the `remote` config setting and on the command line will be passed to
-the wrapped remote as is, but for storing the chunks on disk the path will be made
-relative by removing any leading `/` character.
-This behavior is irrelevant for most backend types, but there are backends where a leading `/`
-changes the effective directory, e.g. in the `sftp` backend paths starting with a `/` are
-relative to the root of the SSH server and paths without are relative to the user home directory.
-As a result `sftp:bin` and `sftp:/bin` will share the same cache folder, even if they represent
-a different directory on the SSH server.
+### Modification times and hashes
-### Cache and Remote Control (--rc) ###
-Cache supports the new `--rc` mode in rclone and can be remote controlled through the following end points:
-By default, the listener is disabled if you do not add the flag.
+Crypt stores modification times using the underlying remote so support
+depends on that.
-### rc cache/expire
-Purge a remote from the cache backend. Supports either a directory or a file.
-It supports both encrypted and unencrypted file names if cache is wrapped by crypt.
+Hashes are not stored for crypt. However the data integrity is
+protected by an extremely strong crypto authenticator.
-Params:
- - **remote** = path to remote **(required)**
- - **withData** = true/false to delete cached data (chunks) as well _(optional, false by default)_
+Use the `rclone cryptcheck` command to check the
+integrity of an encrypted remote instead of `rclone check` which can't
+check the checksums properly.
### Standard options
-Here are the Standard options specific to cache (Cache a remote).
+Here are the Standard options specific to crypt (Encrypt/Decrypt a remote).
-#### --cache-remote
+#### --crypt-remote
-Remote to cache.
+Remote to encrypt/decrypt.
Normally should contain a ':' and a path, e.g. "myremote:path/to/dir",
"myremote:bucket" or maybe "myremote:" (not recommended).
@@ -27963,855 +29864,705 @@ Normally should contain a ':' and a path, e.g. "myremote:path/to/dir",
Properties:
- Config: remote
-- Env Var: RCLONE_CACHE_REMOTE
+- Env Var: RCLONE_CRYPT_REMOTE
- Type: string
- Required: true
-#### --cache-plex-url
+#### --crypt-filename-encryption
-The URL of the Plex server.
+How to encrypt the filenames.
Properties:
-- Config: plex_url
-- Env Var: RCLONE_CACHE_PLEX_URL
+- Config: filename_encryption
+- Env Var: RCLONE_CRYPT_FILENAME_ENCRYPTION
- Type: string
-- Required: false
+- Default: "standard"
+- Examples:
+ - "standard"
+ - Encrypt the filenames.
+ - See the docs for the details.
+ - "obfuscate"
+ - Very simple filename obfuscation.
+ - "off"
+ - Don't encrypt the file names.
+ - Adds a ".bin", or "suffix" extension only.
-#### --cache-plex-username
+#### --crypt-directory-name-encryption
-The username of the Plex user.
+Option to either encrypt directory names or leave them intact.
+
+NB If filename_encryption is "off" then this option will do nothing.
Properties:
-- Config: plex_username
-- Env Var: RCLONE_CACHE_PLEX_USERNAME
-- Type: string
-- Required: false
+- Config: directory_name_encryption
+- Env Var: RCLONE_CRYPT_DIRECTORY_NAME_ENCRYPTION
+- Type: bool
+- Default: true
+- Examples:
+ - "true"
+ - Encrypt directory names.
+ - "false"
+ - Don't encrypt directory names, leave them intact.
-#### --cache-plex-password
+#### --crypt-password
-The password of the Plex user.
+Password or pass phrase for encryption.
**NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/).
Properties:
-- Config: plex_password
-- Env Var: RCLONE_CACHE_PLEX_PASSWORD
+- Config: password
+- Env Var: RCLONE_CRYPT_PASSWORD
- Type: string
-- Required: false
+- Required: true
-#### --cache-chunk-size
+#### --crypt-password2
-The size of a chunk (partial file data).
+Password or pass phrase for salt.
-Use lower numbers for slower connections. If the chunk size is
-changed, any downloaded chunks will be invalid and cache-chunk-path
-will need to be cleared or unexpected EOF errors will occur.
+Optional but recommended.
+Should be different to the previous password.
+
+**NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/).
Properties:
-- Config: chunk_size
-- Env Var: RCLONE_CACHE_CHUNK_SIZE
-- Type: SizeSuffix
-- Default: 5Mi
-- Examples:
- - "1M"
- - 1 MiB
- - "5M"
- - 5 MiB
- - "10M"
- - 10 MiB
+- Config: password2
+- Env Var: RCLONE_CRYPT_PASSWORD2
+- Type: string
+- Required: false
-#### --cache-info-age
+### Advanced options
-How long to cache file structure information (directory listings, file size, times, etc.).
-If all write operations are done through the cache then you can safely make
-this value very large as the cache store will also be updated in real time.
+Here are the Advanced options specific to crypt (Encrypt/Decrypt a remote).
-Properties:
+#### --crypt-server-side-across-configs
-- Config: info_age
-- Env Var: RCLONE_CACHE_INFO_AGE
-- Type: Duration
-- Default: 6h0m0s
-- Examples:
- - "1h"
- - 1 hour
- - "24h"
- - 24 hours
- - "48h"
- - 48 hours
+Deprecated: use --server-side-across-configs instead.
-#### --cache-chunk-total-size
+Allow server-side operations (e.g. copy) to work across different crypt configs.
-The total size that the chunks can take up on the local disk.
+Normally this option is not what you want, but if you have two crypts
+pointing to the same backend you can use it.
-If the cache exceeds this value then it will start to delete the
-oldest chunks until it goes under this value.
+This can be used, for example, to change file name encryption type
+without re-uploading all the data. Just make two crypt backends
+pointing to two different directories with the single changed
+parameter and use rclone move to move the files between the crypt
+remotes.
Properties:
-- Config: chunk_total_size
-- Env Var: RCLONE_CACHE_CHUNK_TOTAL_SIZE
-- Type: SizeSuffix
-- Default: 10Gi
-- Examples:
- - "500M"
- - 500 MiB
- - "1G"
- - 1 GiB
- - "10G"
- - 10 GiB
+- Config: server_side_across_configs
+- Env Var: RCLONE_CRYPT_SERVER_SIDE_ACROSS_CONFIGS
+- Type: bool
+- Default: false
-### Advanced options
+#### --crypt-show-mapping
-Here are the Advanced options specific to cache (Cache a remote).
+For all files listed show how the names encrypt.
-#### --cache-plex-token
+If this flag is set then for each file that the remote is asked to
+list, it will log (at level INFO) a line stating the decrypted file
+name and the encrypted file name.
-The plex token for authentication - auto set normally.
+This is so you can work out which encrypted names are which decrypted
+names just in case you need to do something with the encrypted file
+names, or for debugging purposes.
Properties:
-- Config: plex_token
-- Env Var: RCLONE_CACHE_PLEX_TOKEN
-- Type: string
-- Required: false
+- Config: show_mapping
+- Env Var: RCLONE_CRYPT_SHOW_MAPPING
+- Type: bool
+- Default: false
-#### --cache-plex-insecure
+#### --crypt-no-data-encryption
-Skip all certificate verification when connecting to the Plex server.
+Option to either encrypt file data or leave it unencrypted.
+
+Properties:
+
+- Config: no_data_encryption
+- Env Var: RCLONE_CRYPT_NO_DATA_ENCRYPTION
+- Type: bool
+- Default: false
+- Examples:
+ - "true"
+ - Don't encrypt file data, leave it unencrypted.
+ - "false"
+ - Encrypt file data.
+
+#### --crypt-pass-bad-blocks
+
+If set this will pass bad blocks through as all 0.
+
+This should not be set in normal operation, it should only be set if
+trying to recover an encrypted file with errors and it is desired to
+recover as much of the file as possible.
Properties:
-- Config: plex_insecure
-- Env Var: RCLONE_CACHE_PLEX_INSECURE
-- Type: string
-- Required: false
+- Config: pass_bad_blocks
+- Env Var: RCLONE_CRYPT_PASS_BAD_BLOCKS
+- Type: bool
+- Default: false
-#### --cache-db-path
+#### --crypt-filename-encoding
-Directory to store file structure metadata DB.
+How to encode the encrypted filename to text string.
-The remote name is used as the DB file name.
+This option could help with shortening the encrypted filename. The
+suitable option would depend on the way your remote count the filename
+length and if it's case sensitive.
Properties:
-- Config: db_path
-- Env Var: RCLONE_CACHE_DB_PATH
+- Config: filename_encoding
+- Env Var: RCLONE_CRYPT_FILENAME_ENCODING
- Type: string
-- Default: "$HOME/.cache/rclone/cache-backend"
-
-#### --cache-chunk-path
+- Default: "base32"
+- Examples:
+ - "base32"
+ - Encode using base32. Suitable for all remote.
+ - "base64"
+ - Encode using base64. Suitable for case sensitive remote.
+ - "base32768"
+ - Encode using base32768. Suitable if your remote counts UTF-16 or
+ - Unicode codepoint instead of UTF-8 byte length. (Eg. Onedrive, Dropbox)
-Directory to cache chunk files.
+#### --crypt-suffix
-Path to where partial file data (chunks) are stored locally. The remote
-name is appended to the final path.
+If this is set it will override the default suffix of ".bin".
-This config follows the "--cache-db-path". If you specify a custom
-location for "--cache-db-path" and don't specify one for "--cache-chunk-path"
-then "--cache-chunk-path" will use the same path as "--cache-db-path".
+Setting suffix to "none" will result in an empty suffix. This may be useful
+when the path length is critical.
Properties:
-- Config: chunk_path
-- Env Var: RCLONE_CACHE_CHUNK_PATH
+- Config: suffix
+- Env Var: RCLONE_CRYPT_SUFFIX
- Type: string
-- Default: "$HOME/.cache/rclone/cache-backend"
+- Default: ".bin"
-#### --cache-db-purge
+### Metadata
-Clear all the cached data for this remote on start.
+Any metadata supported by the underlying remote is read and written.
-Properties:
+See the [metadata](https://rclone.org/docs/#metadata) docs for more info.
-- Config: db_purge
-- Env Var: RCLONE_CACHE_DB_PURGE
-- Type: bool
-- Default: false
+## Backend commands
-#### --cache-chunk-clean-interval
+Here are the commands specific to the crypt backend.
-How often should the cache perform cleanups of the chunk storage.
+Run them with
-The default value should be ok for most people. If you find that the
-cache goes over "cache-chunk-total-size" too often then try to lower
-this value to force it to perform cleanups more often.
+ rclone backend COMMAND remote:
-Properties:
+The help below will explain what arguments each command takes.
-- Config: chunk_clean_interval
-- Env Var: RCLONE_CACHE_CHUNK_CLEAN_INTERVAL
-- Type: Duration
-- Default: 1m0s
+See the [backend](https://rclone.org/commands/rclone_backend/) command for more
+info on how to pass options and arguments.
-#### --cache-read-retries
+These can be run on a running backend using the rc command
+[backend/command](https://rclone.org/rc/#backend-command).
-How many times to retry a read from a cache storage.
+### encode
-Since reading from a cache stream is independent from downloading file
-data, readers can get to a point where there's no more data in the
-cache. Most of the times this can indicate a connectivity issue if
-cache isn't able to provide file data anymore.
+Encode the given filename(s)
-For really slow connections, increase this to a point where the stream is
-able to provide data but your experience will be very stuttering.
+ rclone backend encode remote: [options] [+]
-Properties:
+This encodes the filenames given as arguments returning a list of
+strings of the encoded results.
-- Config: read_retries
-- Env Var: RCLONE_CACHE_READ_RETRIES
-- Type: int
-- Default: 10
+Usage Example:
-#### --cache-workers
+ rclone backend encode crypt: file1 [file2...]
+ rclone rc backend/command command=encode fs=crypt: file1 [file2...]
-How many workers should run in parallel to download chunks.
-Higher values will mean more parallel processing (better CPU needed)
-and more concurrent requests on the cloud provider. This impacts
-several aspects like the cloud provider API limits, more stress on the
-hardware that rclone runs on but it also means that streams will be
-more fluid and data will be available much more faster to readers.
+### decode
-**Note**: If the optional Plex integration is enabled then this
-setting will adapt to the type of reading performed and the value
-specified here will be used as a maximum number of workers to use.
+Decode the given filename(s)
-Properties:
+ rclone backend decode remote: [options] [+]
-- Config: workers
-- Env Var: RCLONE_CACHE_WORKERS
-- Type: int
-- Default: 4
+This decodes the filenames given as arguments returning a list of
+strings of the decoded results. It will return an error if any of the
+inputs are invalid.
-#### --cache-chunk-no-memory
+Usage Example:
-Disable the in-memory cache for storing chunks during streaming.
+ rclone backend decode crypt: encryptedfile1 [encryptedfile2...]
+ rclone rc backend/command command=decode fs=crypt: encryptedfile1 [encryptedfile2...]
-By default, cache will keep file data during streaming in RAM as well
-to provide it to readers as fast as possible.
-This transient data is evicted as soon as it is read and the number of
-chunks stored doesn't exceed the number of workers. However, depending
-on other settings like "cache-chunk-size" and "cache-workers" this footprint
-can increase if there are parallel streams too (multiple files being read
-at the same time).
-If the hardware permits it, use this feature to provide an overall better
-performance during streaming but it can also be disabled if RAM is not
-available on the local machine.
-Properties:
+## Backing up an encrypted remote
-- Config: chunk_no_memory
-- Env Var: RCLONE_CACHE_CHUNK_NO_MEMORY
-- Type: bool
-- Default: false
+If you wish to backup an encrypted remote, it is recommended that you use
+`rclone sync` on the encrypted files, and make sure the passwords are
+the same in the new encrypted remote.
-#### --cache-rps
+This will have the following advantages
-Limits the number of requests per second to the source FS (-1 to disable).
+ * `rclone sync` will check the checksums while copying
+ * you can use `rclone check` between the encrypted remotes
+ * you don't decrypt and encrypt unnecessarily
-This setting places a hard limit on the number of requests per second
-that cache will be doing to the cloud provider remote and try to
-respect that value by setting waits between reads.
+For example, let's say you have your original remote at `remote:` with
+the encrypted version at `eremote:` with path `remote:crypt`. You
+would then set up the new remote `remote2:` and then the encrypted
+version `eremote2:` with path `remote2:crypt` using the same passwords
+as `eremote:`.
-If you find that you're getting banned or limited on the cloud
-provider through cache and know that a smaller number of requests per
-second will allow you to work with it then you can use this setting
-for that.
+To sync the two remotes you would do
-A good balance of all the other settings should make this setting
-useless but it is available to set for more special cases.
+ rclone sync --interactive remote:crypt remote2:crypt
-**NOTE**: This will limit the number of requests during streams but
-other API calls to the cloud provider like directory listings will
-still pass.
+And to check the integrity you would do
-Properties:
+ rclone check remote:crypt remote2:crypt
-- Config: rps
-- Env Var: RCLONE_CACHE_RPS
-- Type: int
-- Default: -1
+## File formats
-#### --cache-writes
+### File encryption
-Cache file data on writes through the FS.
+Files are encrypted 1:1 source file to destination object. The file
+has a header and is divided into chunks.
-If you need to read files immediately after you upload them through
-cache you can enable this flag to have their data stored in the
-cache store at the same time during upload.
+#### Header
-Properties:
+ * 8 bytes magic string `RCLONE\x00\x00`
+ * 24 bytes Nonce (IV)
-- Config: writes
-- Env Var: RCLONE_CACHE_WRITES
-- Type: bool
-- Default: false
+The initial nonce is generated from the operating systems crypto
+strong random number generator. The nonce is incremented for each
+chunk read making sure each nonce is unique for each block written.
+The chance of a nonce being reused is minuscule. If you wrote an
+exabyte of data (10¹⸠bytes) you would have a probability of
+approximately 2×10â»Â³Â² of re-using a nonce.
-#### --cache-tmp-upload-path
+#### Chunk
-Directory to keep temporary files until they are uploaded.
+Each chunk will contain 64 KiB of data, except for the last one which
+may have less data. The data chunk is in standard NaCl SecretBox
+format. SecretBox uses XSalsa20 and Poly1305 to encrypt and
+authenticate messages.
-This is the path where cache will use as a temporary storage for new
-files that need to be uploaded to the cloud provider.
+Each chunk contains:
-Specifying a value will enable this feature. Without it, it is
-completely disabled and files will be uploaded directly to the cloud
-provider
+ * 16 Bytes of Poly1305 authenticator
+ * 1 - 65536 bytes XSalsa20 encrypted data
-Properties:
+64k chunk size was chosen as the best performing chunk size (the
+authenticator takes too much time below this and the performance drops
+off due to cache effects above this). Note that these chunks are
+buffered in memory so they can't be too big.
-- Config: tmp_upload_path
-- Env Var: RCLONE_CACHE_TMP_UPLOAD_PATH
-- Type: string
-- Required: false
+This uses a 32 byte (256 bit key) key derived from the user password.
-#### --cache-tmp-wait-time
+#### Examples
-How long should files be stored in local cache before being uploaded.
+1 byte file will encrypt to
-This is the duration that a file must wait in the temporary location
-_cache-tmp-upload-path_ before it is selected for upload.
+ * 32 bytes header
+ * 17 bytes data chunk
-Note that only one file is uploaded at a time and it can take longer
-to start the upload if a queue formed for this purpose.
+49 bytes total
-Properties:
+1 MiB (1048576 bytes) file will encrypt to
-- Config: tmp_wait_time
-- Env Var: RCLONE_CACHE_TMP_WAIT_TIME
-- Type: Duration
-- Default: 15s
+ * 32 bytes header
+ * 16 chunks of 65568 bytes
-#### --cache-db-wait-time
+1049120 bytes total (a 0.05% overhead). This is the overhead for big
+files.
-How long to wait for the DB to be available - 0 is unlimited.
+### Name encryption
-Only one process can have the DB open at any one time, so rclone waits
-for this duration for the DB to become available before it gives an
-error.
+File names are encrypted segment by segment - the path is broken up
+into `/` separated strings and these are encrypted individually.
-If you set it to 0 then it will wait forever.
+File segments are padded using PKCS#7 to a multiple of 16 bytes
+before encryption.
-Properties:
+They are then encrypted with EME using AES with 256 bit key. EME
+(ECB-Mix-ECB) is a wide-block encryption mode presented in the 2003
+paper "A Parallelizable Enciphering Mode" by Halevi and Rogaway.
-- Config: db_wait_time
-- Env Var: RCLONE_CACHE_DB_WAIT_TIME
-- Type: Duration
-- Default: 1s
+This makes for deterministic encryption which is what we want - the
+same filename must encrypt to the same thing otherwise we can't find
+it on the cloud storage system.
-## Backend commands
+This means that
-Here are the commands specific to the cache backend.
+ * filenames with the same name will encrypt the same
+ * filenames which start the same won't have a common prefix
-Run them with
+This uses a 32 byte key (256 bits) and a 16 byte (128 bits) IV both of
+which are derived from the user password.
- rclone backend COMMAND remote:
+After encryption they are written out using a modified version of
+standard `base32` encoding as described in RFC4648. The standard
+encoding is modified in two ways:
-The help below will explain what arguments each command takes.
+ * it becomes lower case (no-one likes upper case filenames!)
+ * we strip the padding character `=`
-See the [backend](https://rclone.org/commands/rclone_backend/) command for more
-info on how to pass options and arguments.
+`base32` is used rather than the more efficient `base64` so rclone can be
+used on case insensitive remotes (e.g. Windows, Amazon Drive).
-These can be run on a running backend using the rc command
-[backend/command](https://rclone.org/rc/#backend-command).
+### Key derivation
-### stats
+Rclone uses `scrypt` with parameters `N=16384, r=8, p=1` with an
+optional user supplied salt (password2) to derive the 32+32+16 = 80
+bytes of key material required. If the user doesn't supply a salt
+then rclone uses an internal one.
-Print stats on the cache backend in JSON format.
+`scrypt` makes it impractical to mount a dictionary attack on rclone
+encrypted data. For full protection against this you should always use
+a salt.
- rclone backend stats remote: [options] [+]
+## SEE ALSO
+* [rclone cryptdecode](https://rclone.org/commands/rclone_cryptdecode/) - Show forward/reverse mapping of encrypted filenames
+# Compress
-# Chunker
+## Warning
-The `chunker` overlay transparently splits large files into smaller chunks
-during upload to wrapped remote and transparently assembles them back
-when the file is downloaded. This allows to effectively overcome size limits
-imposed by storage providers.
+This remote is currently **experimental**. Things may break and data may be lost. Anything you do with this remote is
+at your own risk. Please understand the risks associated with using experimental code and don't use this remote in
+critical applications.
+
+The `Compress` remote adds compression to another remote. It is best used with remotes containing
+many large compressible files.
## Configuration
-To use it, first set up the underlying remote following the configuration
-instructions for that remote. You can also use a local pathname instead of
-a remote.
+To use this remote, all you need to do is specify another remote and a compression mode to use:
-First check your chosen remote is working - we'll call it `remote:path` here.
-Note that anything inside `remote:path` will be chunked and anything outside
-won't. This means that if you are using a bucket-based remote (e.g. S3, B2, swift)
-then you should probably put the bucket in the remote `s3:bucket`.
+```
+Current remotes:
-Now configure `chunker` using `rclone config`. We will call this one `overlay`
-to separate it from the `remote` itself.
+Name Type
+==== ====
+remote_to_press sometype
-```
-No remotes found, make a new one?
+e) Edit existing remote
+$ rclone config
n) New remote
+d) Delete remote
+r) Rename remote
+c) Copy remote
s) Set configuration password
q) Quit config
-n/s/q> n
-name> overlay
-Type of storage to configure.
-Choose a number from below, or type in your own value
-[snip]
-XX / Transparently chunk/split large files
- \ "chunker"
-[snip]
-Storage> chunker
-Remote to chunk/unchunk.
-Normally should contain a ':' and a path, e.g. "myremote:path/to/dir",
-"myremote:bucket" or maybe "myremote:" (not recommended).
+e/n/d/r/c/s/q> n
+name> compress
+...
+ 8 / Compress a remote
+ \ "compress"
+...
+Storage> compress
+** See help for compress backend at: https://rclone.org/compress/ **
+
+Remote to compress.
Enter a string value. Press Enter for the default ("").
-remote> remote:path
-Files larger than chunk size will be split in chunks.
-Enter a size with suffix K,M,G,T. Press Enter for the default ("2G").
-chunk_size> 100M
-Choose how chunker handles hash sums. All modes but "none" require metadata.
-Enter a string value. Press Enter for the default ("md5").
+remote> remote_to_press:subdir
+Compression mode.
+Enter a string value. Press Enter for the default ("gzip").
Choose a number from below, or type in your own value
- 1 / Pass any hash supported by wrapped remote for non-chunked files, return nothing otherwise
- \ "none"
- 2 / MD5 for composite files
- \ "md5"
- 3 / SHA1 for composite files
- \ "sha1"
- 4 / MD5 for all files
- \ "md5all"
- 5 / SHA1 for all files
- \ "sha1all"
- 6 / Copying a file to chunker will request MD5 from the source falling back to SHA1 if unsupported
- \ "md5quick"
- 7 / Similar to "md5quick" but prefers SHA1 over MD5
- \ "sha1quick"
-hash_type> md5
+ 1 / Gzip compression balanced for speed and compression strength.
+ \ "gzip"
+compression_mode> gzip
Edit advanced config? (y/n)
y) Yes
-n) No
+n) No (default)
y/n> n
Remote config
--------------------
-[overlay]
-type = chunker
-remote = remote:bucket
-chunk_size = 100M
-hash_type = md5
+[compress]
+type = compress
+remote = remote_to_press:subdir
+compression_mode = gzip
--------------------
-y) Yes this is OK
+y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y
```
-### Specifying the remote
-
-In normal use, make sure the remote has a `:` in. If you specify the remote
-without a `:` then rclone will use a local directory of that name.
-So if you use a remote of `/path/to/secret/files` then rclone will
-chunk stuff in that directory. If you use a remote of `name` then rclone
-will put files in a directory called `name` in the current directory.
-
-
-### Chunking
-
-When rclone starts a file upload, chunker checks the file size. If it
-doesn't exceed the configured chunk size, chunker will just pass the file
-to the wrapped remote (however, see caveat below). If a file is large, chunker will transparently cut
-data in pieces with temporary names and stream them one by one, on the fly.
-Each data chunk will contain the specified number of bytes, except for the
-last one which may have less data. If file size is unknown in advance
-(this is called a streaming upload), chunker will internally create
-a temporary copy, record its size and repeat the above process.
-
-When upload completes, temporary chunk files are finally renamed.
-This scheme guarantees that operations can be run in parallel and look
-from outside as atomic.
-A similar method with hidden temporary chunks is used for other operations
-(copy/move/rename, etc.). If an operation fails, hidden chunks are normally
-destroyed, and the target composite file stays intact.
-
-When a composite file download is requested, chunker transparently
-assembles it by concatenating data chunks in order. As the split is trivial
-one could even manually concatenate data chunks together to obtain the
-original content.
-
-When the `list` rclone command scans a directory on wrapped remote,
-the potential chunk files are accounted for, grouped and assembled into
-composite directory entries. Any temporary chunks are hidden.
-
-List and other commands can sometimes come across composite files with
-missing or invalid chunks, e.g. shadowed by like-named directory or
-another file. This usually means that wrapped file system has been directly
-tampered with or damaged. If chunker detects a missing chunk it will
-by default print warning, skip the whole incomplete group of chunks but
-proceed with current command.
-You can set the `--chunker-fail-hard` flag to have commands abort with
-error message in such cases.
-
-**Caveat**: As it is now, chunker will always create a temporary file in the
-backend and then rename it, even if the file is below the chunk threshold.
-This will result in unnecessary API calls and can severely restrict throughput
-when handling transfers primarily composed of small files on some backends (e.g. Box).
-A workaround to this issue is to use chunker only for files above the chunk threshold
-via `--min-size` and then perform a separate call without chunker on the remaining
-files.
-
-
-#### Chunk names
-
-The default chunk name format is `*.rclone_chunk.###`, hence by default
-chunk names are `BIG_FILE_NAME.rclone_chunk.001`,
-`BIG_FILE_NAME.rclone_chunk.002` etc. You can configure another name format
-using the `name_format` configuration file option. The format uses asterisk
-`*` as a placeholder for the base file name and one or more consecutive
-hash characters `#` as a placeholder for sequential chunk number.
-There must be one and only one asterisk. The number of consecutive hash
-characters defines the minimum length of a string representing a chunk number.
-If decimal chunk number has less digits than the number of hashes, it is
-left-padded by zeros. If the decimal string is longer, it is left intact.
-By default numbering starts from 1 but there is another option that allows
-user to start from 0, e.g. for compatibility with legacy software.
+### Compression Modes
-For example, if name format is `big_*-##.part` and original file name is
-`data.txt` and numbering starts from 0, then the first chunk will be named
-`big_data.txt-00.part`, the 99th chunk will be `big_data.txt-98.part`
-and the 302nd chunk will become `big_data.txt-301.part`.
+Currently only gzip compression is supported. It provides a decent balance between speed and size and is well
+supported by other applications. Compression strength can further be configured via an advanced setting where 0 is no
+compression and 9 is strongest compression.
-Note that `list` assembles composite directory entries only when chunk names
-match the configured format and treats non-conforming file names as normal
-non-chunked files.
+### File types
-When using `norename` transactions, chunk names will additionally have a unique
-file version suffix. For example, `BIG_FILE_NAME.rclone_chunk.001_bp562k`.
+If you open a remote wrapped by compress, you will see that there are many files with an extension corresponding to
+the compression algorithm you chose. These files are standard files that can be opened by various archive programs,
+but they have some hidden metadata that allows them to be used by rclone.
+While you may download and decompress these files at will, do **not** manually delete or rename files. Files without
+correct metadata files will not be recognized by rclone.
+### File names
-### Metadata
+The compressed files will be named `*.###########.gz` where `*` is the base file and the `#` part is base64 encoded
+size of the uncompressed file. The file names should not be changed by anything other than the rclone compression backend.
-Besides data chunks chunker will by default create metadata object for
-a composite file. The object is named after the original file.
-Chunker allows user to disable metadata completely (the `none` format).
-Note that metadata is normally not created for files smaller than the
-configured chunk size. This may change in future rclone releases.
-#### Simple JSON metadata format
+### Standard options
-This is the default format. It supports hash sums and chunk validation
-for composite files. Meta objects carry the following fields:
+Here are the Standard options specific to compress (Compress a remote).
-- `ver` - version of format, currently `1`
-- `size` - total size of composite file
-- `nchunks` - number of data chunks in file
-- `md5` - MD5 hashsum of composite file (if present)
-- `sha1` - SHA1 hashsum (if present)
-- `txn` - identifies current version of the file
+#### --compress-remote
-There is no field for composite file name as it's simply equal to the name
-of meta object on the wrapped remote. Please refer to respective sections
-for details on hashsums and modified time handling.
+Remote to compress.
-#### No metadata
+Properties:
-You can disable meta objects by setting the meta format option to `none`.
-In this mode chunker will scan directory for all files that follow
-configured chunk name format, group them by detecting chunks with the same
-base name and show group names as virtual composite files.
-This method is more prone to missing chunk errors (especially missing
-last chunk) than format with metadata enabled.
+- Config: remote
+- Env Var: RCLONE_COMPRESS_REMOTE
+- Type: string
+- Required: true
+#### --compress-mode
-### Hashsums
+Compression mode.
-Chunker supports hashsums only when a compatible metadata is present.
-Hence, if you choose metadata format of `none`, chunker will report hashsum
-as `UNSUPPORTED`.
+Properties:
-Please note that by default metadata is stored only for composite files.
-If a file is smaller than configured chunk size, chunker will transparently
-redirect hash requests to wrapped remote, so support depends on that.
-You will see the empty string as a hashsum of requested type for small
-files if the wrapped remote doesn't support it.
+- Config: mode
+- Env Var: RCLONE_COMPRESS_MODE
+- Type: string
+- Default: "gzip"
+- Examples:
+ - "gzip"
+ - Standard gzip compression with fastest parameters.
-Many storage backends support MD5 and SHA1 hash types, so does chunker.
-With chunker you can choose one or another but not both.
-MD5 is set by default as the most supported type.
-Since chunker keeps hashes for composite files and falls back to the
-wrapped remote hash for non-chunked ones, we advise you to choose the same
-hash type as supported by wrapped remote so that your file listings
-look coherent.
+### Advanced options
-If your storage backend does not support MD5 or SHA1 but you need consistent
-file hashing, configure chunker with `md5all` or `sha1all`. These two modes
-guarantee given hash for all files. If wrapped remote doesn't support it,
-chunker will then add metadata to all files, even small. However, this can
-double the amount of small files in storage and incur additional service charges.
-You can even use chunker to force md5/sha1 support in any other remote
-at expense of sidecar meta objects by setting e.g. `hash_type=sha1all`
-to force hashsums and `chunk_size=1P` to effectively disable chunking.
+Here are the Advanced options specific to compress (Compress a remote).
-Normally, when a file is copied to chunker controlled remote, chunker
-will ask the file source for compatible file hash and revert to on-the-fly
-calculation if none is found. This involves some CPU overhead but provides
-a guarantee that given hashsum is available. Also, chunker will reject
-a server-side copy or move operation if source and destination hashsum
-types are different resulting in the extra network bandwidth, too.
-In some rare cases this may be undesired, so chunker provides two optional
-choices: `sha1quick` and `md5quick`. If the source does not support primary
-hash type and the quick mode is enabled, chunker will try to fall back to
-the secondary type. This will save CPU and bandwidth but can result in empty
-hashsums at destination. Beware of consequences: the `sync` command will
-revert (sometimes silently) to time/size comparison if compatible hashsums
-between source and target are not found.
+#### --compress-level
+GZIP compression level (-2 to 9).
-### Modified time
+Generally -1 (default, equivalent to 5) is recommended.
+Levels 1 to 9 increase compression at the cost of speed. Going past 6
+generally offers very little return.
-Chunker stores modification times using the wrapped remote so support
-depends on that. For a small non-chunked file the chunker overlay simply
-manipulates modification time of the wrapped remote file.
-For a composite file with metadata chunker will get and set
-modification time of the metadata object on the wrapped remote.
-If file is chunked but metadata format is `none` then chunker will
-use modification time of the first data chunk.
+Level -2 uses Huffman encoding only. Only use if you know what you
+are doing.
+Level 0 turns off compression.
+Properties:
-### Migrations
+- Config: level
+- Env Var: RCLONE_COMPRESS_LEVEL
+- Type: int
+- Default: -1
-The idiomatic way to migrate to a different chunk size, hash type, transaction
-style or chunk naming scheme is to:
+#### --compress-ram-cache-limit
-- Collect all your chunked files under a directory and have your
- chunker remote point to it.
-- Create another directory (most probably on the same cloud storage)
- and configure a new remote with desired metadata format,
- hash type, chunk naming etc.
-- Now run `rclone sync --interactive oldchunks: newchunks:` and all your data
- will be transparently converted in transfer.
- This may take some time, yet chunker will try server-side
- copy if possible.
-- After checking data integrity you may remove configuration section
- of the old remote.
+Some remotes don't allow the upload of files with unknown size.
+In this case the compressed file will need to be cached to determine
+it's size.
-If rclone gets killed during a long operation on a big composite file,
-hidden temporary chunks may stay in the directory. They will not be
-shown by the `list` command but will eat up your account quota.
-Please note that the `deletefile` command deletes only active
-chunks of a file. As a workaround, you can use remote of the wrapped
-file system to see them.
-An easy way to get rid of hidden garbage is to copy littered directory
-somewhere using the chunker remote and purge the original directory.
-The `copy` command will copy only active chunks while the `purge` will
-remove everything including garbage.
+Files smaller than this limit will be cached in RAM, files larger than
+this limit will be cached on disk.
+Properties:
-### Caveats and Limitations
+- Config: ram_cache_limit
+- Env Var: RCLONE_COMPRESS_RAM_CACHE_LIMIT
+- Type: SizeSuffix
+- Default: 20Mi
-Chunker requires wrapped remote to support server-side `move` (or `copy` +
-`delete`) operations, otherwise it will explicitly refuse to start.
-This is because it internally renames temporary chunk files to their final
-names when an operation completes successfully.
+### Metadata
-Chunker encodes chunk number in file name, so with default `name_format`
-setting it adds 17 characters. Also chunker adds 7 characters of temporary
-suffix during operations. Many file systems limit base file name without path
-by 255 characters. Using rclone's crypt remote as a base file system limits
-file name by 143 characters. Thus, maximum name length is 231 for most files
-and 119 for chunker-over-crypt. A user in need can change name format to
-e.g. `*.rcc##` and save 10 characters (provided at most 99 chunks per file).
+Any metadata supported by the underlying remote is read and written.
-Note that a move implemented using the copy-and-delete method may incur
-double charging with some cloud storage providers.
+See the [metadata](https://rclone.org/docs/#metadata) docs for more info.
-Chunker will not automatically rename existing chunks when you run
-`rclone config` on a live remote and change the chunk name format.
-Beware that in result of this some files which have been treated as chunks
-before the change can pop up in directory listings as normal files
-and vice versa. The same warning holds for the chunk size.
-If you desperately need to change critical chunking settings, you should
-run data migration as described above.
-If wrapped remote is case insensitive, the chunker overlay will inherit
-that property (so you can't have a file called "Hello.doc" and "hello.doc"
-in the same directory).
-Chunker included in rclone releases up to `v1.54` can sometimes fail to
-detect metadata produced by recent versions of rclone. We recommend users
-to keep rclone up-to-date to avoid data corruption.
+# Combine
-Changing `transactions` is dangerous and requires explicit migration.
+The `combine` backend joins remotes together into a single directory
+tree.
+For example you might have a remote for images on one provider:
-### Standard options
+```
+$ rclone tree s3:imagesbucket
+/
+├── image1.jpg
+└── image2.jpg
+```
-Here are the Standard options specific to chunker (Transparently chunk/split large files).
+And a remote for files on another:
-#### --chunker-remote
+```
+$ rclone tree drive:important/files
+/
+├── file1.txt
+└── file2.txt
+```
-Remote to chunk/unchunk.
+The `combine` backend can join these together into a synthetic
+directory structure like this:
-Normally should contain a ':' and a path, e.g. "myremote:path/to/dir",
-"myremote:bucket" or maybe "myremote:" (not recommended).
+```
+$ rclone tree combined:
+/
+├── files
+│ ├── file1.txt
+│ └── file2.txt
+└── images
+ ├── image1.jpg
+ └── image2.jpg
+```
-Properties:
+You'd do this by specifying an `upstreams` parameter in the config
+like this
-- Config: remote
-- Env Var: RCLONE_CHUNKER_REMOTE
-- Type: string
-- Required: true
+ upstreams = images=s3:imagesbucket files=drive:important/files
-#### --chunker-chunk-size
+During the initial setup with `rclone config` you will specify the
+upstreams remotes as a space separated list. The upstream remotes can
+either be a local paths or other remotes.
-Files larger than chunk size will be split in chunks.
+## Configuration
-Properties:
+Here is an example of how to make a combine called `remote` for the
+example above. First run:
-- Config: chunk_size
-- Env Var: RCLONE_CHUNKER_CHUNK_SIZE
-- Type: SizeSuffix
-- Default: 2Gi
+ rclone config
-#### --chunker-hash-type
+This will guide you through an interactive setup process:
-Choose how chunker handles hash sums.
+```
+No remotes found, make a new one?
+n) New remote
+s) Set configuration password
+q) Quit config
+n/s/q> n
+name> remote
+Option Storage.
+Type of storage to configure.
+Choose a number from below, or type in your own value.
+...
+XX / Combine several remotes into one
+ \ (combine)
+...
+Storage> combine
+Option upstreams.
+Upstreams for combining
+These should be in the form
+ dir=remote:path dir2=remote2:path
+Where before the = is specified the root directory and after is the remote to
+put there.
+Embedded spaces can be added using quotes
+ "dir=remote:path with space" "dir2=remote2:path with space"
+Enter a fs.SpaceSepList value.
+upstreams> images=s3:imagesbucket files=drive:important/files
+--------------------
+[remote]
+type = combine
+upstreams = images=s3:imagesbucket files=drive:important/files
+--------------------
+y) Yes this is OK (default)
+e) Edit this remote
+d) Delete this remote
+y/e/d> y
+```
-All modes but "none" require metadata.
+### Configuring for Google Drive Shared Drives
-Properties:
+Rclone has a convenience feature for making a combine backend for all
+the shared drives you have access to.
-- Config: hash_type
-- Env Var: RCLONE_CHUNKER_HASH_TYPE
-- Type: string
-- Default: "md5"
-- Examples:
- - "none"
- - Pass any hash supported by wrapped remote for non-chunked files.
- - Return nothing otherwise.
- - "md5"
- - MD5 for composite files.
- - "sha1"
- - SHA1 for composite files.
- - "md5all"
- - MD5 for all files.
- - "sha1all"
- - SHA1 for all files.
- - "md5quick"
- - Copying a file to chunker will request MD5 from the source.
- - Falling back to SHA1 if unsupported.
- - "sha1quick"
- - Similar to "md5quick" but prefers SHA1 over MD5.
+Assuming your main (non shared drive) Google drive remote is called
+`drive:` you would run
-### Advanced options
+ rclone backend -o config drives drive:
-Here are the Advanced options specific to chunker (Transparently chunk/split large files).
+This would produce something like this:
-#### --chunker-name-format
+ [My Drive]
+ type = alias
+ remote = drive,team_drive=0ABCDEF-01234567890,root_folder_id=:
-String format of chunk file names.
+ [Test Drive]
+ type = alias
+ remote = drive,team_drive=0ABCDEFabcdefghijkl,root_folder_id=:
-The two placeholders are: base file name (*) and chunk number (#...).
-There must be one and only one asterisk and one or more consecutive hash characters.
-If chunk number has less digits than the number of hashes, it is left-padded by zeros.
-If there are more digits in the number, they are left as is.
-Possible chunk files are ignored if their name does not match given format.
+ [AllDrives]
+ type = combine
+ upstreams = "My Drive=My Drive:" "Test Drive=Test Drive:"
-Properties:
+If you then add that config to your config file (find it with `rclone
+config file`) then you can access all the shared drives in one place
+with the `AllDrives:` remote.
-- Config: name_format
-- Env Var: RCLONE_CHUNKER_NAME_FORMAT
-- Type: string
-- Default: "*.rclone_chunk.###"
+See [the Google Drive docs](https://rclone.org/drive/#drives) for full info.
-#### --chunker-start-from
-Minimum valid chunk number. Usually 0 or 1.
+### Standard options
-By default chunk numbers start from 1.
+Here are the Standard options specific to combine (Combine several remotes into one).
-Properties:
+#### --combine-upstreams
-- Config: start_from
-- Env Var: RCLONE_CHUNKER_START_FROM
-- Type: int
-- Default: 1
+Upstreams for combining
-#### --chunker-meta-format
+These should be in the form
-Format of the metadata object or "none".
+ dir=remote:path dir2=remote2:path
-By default "simplejson".
-Metadata is a small JSON file named after the composite file.
+Where before the = is specified the root directory and after is the remote to
+put there.
-Properties:
+Embedded spaces can be added using quotes
-- Config: meta_format
-- Env Var: RCLONE_CHUNKER_META_FORMAT
-- Type: string
-- Default: "simplejson"
-- Examples:
- - "none"
- - Do not use metadata files at all.
- - Requires hash type "none".
- - "simplejson"
- - Simple JSON supports hash sums and chunk validation.
- -
- - It has the following fields: ver, size, nchunks, md5, sha1.
+ "dir=remote:path with space" "dir2=remote2:path with space"
-#### --chunker-fail-hard
-Choose how chunker should handle files with missing or invalid chunks.
Properties:
-- Config: fail_hard
-- Env Var: RCLONE_CHUNKER_FAIL_HARD
-- Type: bool
-- Default: false
-- Examples:
- - "true"
- - Report errors and abort current command.
- - "false"
- - Warn user, skip incomplete file and proceed.
+- Config: upstreams
+- Env Var: RCLONE_COMBINE_UPSTREAMS
+- Type: SpaceSepList
+- Default:
-#### --chunker-transactions
+### Metadata
-Choose how chunker should handle temporary files during transactions.
+Any metadata supported by the underlying remote is read and written.
-Properties:
+See the [metadata](https://rclone.org/docs/#metadata) docs for more info.
-- Config: transactions
-- Env Var: RCLONE_CHUNKER_TRANSACTIONS
-- Type: string
-- Default: "rename"
-- Examples:
- - "rename"
- - Rename temporary files after a successful transaction.
- - "norename"
- - Leave temporary file names and write transaction ID to metadata file.
- - Metadata is required for no rename transactions (meta format cannot be "none").
- - If you are using norename transactions you should be careful not to downgrade Rclone
- - as older versions of Rclone don't support this transaction style and will misinterpret
- - files manipulated by norename transactions.
- - This method is EXPERIMENTAL, don't use on production systems.
- - "auto"
- - Rename or norename will be used depending on capabilities of the backend.
- - If meta format is set to "none", rename transactions will always be used.
- - This method is EXPERIMENTAL, don't use on production systems.
+# Dropbox
-# Citrix ShareFile
+Paths are specified as `remote:path`
-[Citrix ShareFile](https://sharefile.com) is a secure file sharing and transfer service aimed as business.
+Dropbox paths may be as deep as required, e.g.
+`remote:directory/subdirectory`.
## Configuration
-The initial setup for Citrix ShareFile involves getting a token from
-Citrix ShareFile which you can in your browser. `rclone config` walks you
+The initial setup for dropbox involves getting a token from Dropbox
+which you need to do in your browser. `rclone config` walks you
through it.
Here is an example of how to make a remote called `remote`. First run:
@@ -28821,58 +30572,31 @@ Here is an example of how to make a remote called `remote`. First run:
This will guide you through an interactive setup process:
```
-No remotes found, make a new one?
n) New remote
-s) Set configuration password
+d) Delete remote
q) Quit config
-n/s/q> n
+e/n/d/q> n
name> remote
Type of storage to configure.
-Enter a string value. Press Enter for the default ("").
-Choose a number from below, or type in your own value
-XX / Citrix Sharefile
- \ "sharefile"
-Storage> sharefile
-** See help for sharefile backend at: https://rclone.org/sharefile/ **
-
-ID of the root folder
-
-Leave blank to access "Personal Folders". You can use one of the
-standard values here or any folder ID (long hex number ID).
-Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
- 1 / Access the Personal Folders. (Default)
- \ ""
- 2 / Access the Favorites folder.
- \ "favorites"
- 3 / Access all the shared folders.
- \ "allshared"
- 4 / Access all the individual connectors.
- \ "connectors"
- 5 / Access the home, favorites, and shared folders as well as the connectors.
- \ "top"
-root_folder_id>
-Edit advanced config? (y/n)
-y) Yes
-n) No
-y/n> n
+[snip]
+XX / Dropbox
+ \ "dropbox"
+[snip]
+Storage> dropbox
+Dropbox App Key - leave blank normally.
+app_key>
+Dropbox App Secret - leave blank normally.
+app_secret>
Remote config
-Use web browser to automatically authenticate rclone with remote?
- * Say Y if the machine running rclone has a web browser you can use
- * Say N if running rclone on a (remote) machine without web browser access
-If not sure try Y. If Y failed, try N.
-y) Yes
-n) No
-y/n> y
-If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth?state=XXX
-Log in and authorize rclone for access
-Waiting for code...
-Got code
+Please visit:
+https://www.dropbox.com/1/oauth2/authorize?client_id=XXXXXXXXXXXXXXX&response_type=code
+Enter the code: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX_XXXXXXXXXX
--------------------
[remote]
-type = sharefile
-endpoint = https://XXX.sharefile.com
-token = {"access_token":"XXX","token_type":"bearer","refresh_token":"XXX","expiry":"2019-09-30T19:41:45.878561877+01:00"}
+app_key =
+app_secret =
+token = XXXXXXXXXXXXXXXXXXXXXXXXXXXXX_XXXX_XXXXXXXXXXXXXXXXXXXXXXXXXXXXX
--------------------
y) Yes this is OK
e) Edit this remote
@@ -28880,81 +30604,146 @@ d) Delete this remote
y/e/d> y
```
-See the [remote setup docs](https://rclone.org/remote_setup/) for how to set it up on a
-machine with no Internet browser available.
+See the [remote setup docs](https://rclone.org/remote_setup/) for how to set it up on a
+machine with no Internet browser available.
+
+Note that rclone runs a webserver on your local machine to collect the
+token as returned from Dropbox. This only
+runs from the moment it opens your browser to the moment you get back
+the verification code. This is on `http://127.0.0.1:53682/` and it
+may require you to unblock it temporarily if you are running a host
+firewall, or use manual mode.
+
+You can then use it like this,
+
+List directories in top level of your dropbox
+
+ rclone lsd remote:
+
+List all the files in your dropbox
+
+ rclone ls remote:
+
+To copy a local directory to a dropbox directory called backup
+
+ rclone copy /home/source remote:backup
+
+### Dropbox for business
+
+Rclone supports Dropbox for business and Team Folders.
+
+When using Dropbox for business `remote:` and `remote:path/to/file`
+will refer to your personal folder.
+
+If you wish to see Team Folders you must use a leading `/` in the
+path, so `rclone lsd remote:/` will refer to the root and show you all
+Team Folders and your User Folder.
+
+You can then use team folders like this `remote:/TeamFolder` and
+`remote:/TeamFolder/path/to/file`.
+
+A leading `/` for a Dropbox personal account will do nothing, but it
+will take an extra HTTP transaction so it should be avoided.
+
+### Modification times and hashes
+
+Dropbox supports modified times, but the only way to set a
+modification time is to re-upload the file.
+
+This means that if you uploaded your data with an older version of
+rclone which didn't support the v2 API and modified times, rclone will
+decide to upload all your old data to fix the modification times. If
+you don't want this to happen use `--size-only` or `--checksum` flag
+to stop it.
+
+Dropbox supports [its own hash
+type](https://www.dropbox.com/developers/reference/content-hash) which
+is checked for all transfers.
+
+### Restricted filename characters
+
+| Character | Value | Replacement |
+| --------- |:-----:|:-----------:|
+| NUL | 0x00 | †|
+| / | 0x2F | ï¼ |
+| DEL | 0x7F | â¡ |
+| \ | 0x5C | ï¼¼ |
+
+File names can also not end with the following characters.
+These only get replaced if they are the last character in the name:
+
+| Character | Value | Replacement |
+| --------- |:-----:|:-----------:|
+| SP | 0x20 | â |
+
+Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8),
+as they can't be used in JSON strings.
-Note that rclone runs a webserver on your local machine to collect the
-token as returned from Citrix ShareFile. This only runs from the moment it opens
-your browser to the moment you get back the verification code. This
-is on `http://127.0.0.1:53682/` and this it may require you to unblock
-it temporarily if you are running a host firewall.
+### Batch mode uploads {#batch-mode}
-Once configured you can then use `rclone` like this,
+Using batch mode uploads is very important for performance when using
+the Dropbox API. See [the dropbox performance guide](https://developers.dropbox.com/dbx-performance-guide)
+for more info.
-List directories in top level of your ShareFile
+There are 3 modes rclone can use for uploads.
- rclone lsd remote:
+#### --dropbox-batch-mode off
-List all the files in your ShareFile
+In this mode rclone will not use upload batching. This was the default
+before rclone v1.55. It has the disadvantage that it is very likely to
+encounter `too_many_requests` errors like this
- rclone ls remote:
+ NOTICE: too_many_requests/.: Too many requests or write operations. Trying again in 15 seconds.
-To copy a local directory to an ShareFile directory called backup
+When rclone receives these it has to wait for 15s or sometimes 300s
+before continuing which really slows down transfers.
- rclone copy /home/source remote:backup
+This will happen especially if `--transfers` is large, so this mode
+isn't recommended except for compatibility or investigating problems.
-Paths may be as deep as required, e.g. `remote:directory/subdirectory`.
+#### --dropbox-batch-mode sync
-### Modified time and hashes
+In this mode rclone will batch up uploads to the size specified by
+`--dropbox-batch-size` and commit them together.
-ShareFile allows modification times to be set on objects accurate to 1
-second. These will be used to detect whether objects need syncing or
-not.
+Using this mode means you can use a much higher `--transfers`
+parameter (32 or 64 works fine) without receiving `too_many_requests`
+errors.
-ShareFile supports MD5 type hashes, so you can use the `--checksum`
-flag.
+This mode ensures full data integrity.
-### Transfers
+Note that there may be a pause when quitting rclone while rclone
+finishes up the last batch using this mode.
-For files above 128 MiB rclone will use a chunked transfer. Rclone will
-upload up to `--transfers` chunks at the same time (shared among all
-the multipart uploads). Chunks are buffered in memory and are
-normally 64 MiB so increasing `--transfers` will increase memory use.
+#### --dropbox-batch-mode async
-### Restricted filename characters
+In this mode rclone will batch up uploads to the size specified by
+`--dropbox-batch-size` and commit them together.
-In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters)
-the following characters are also replaced:
+However it will not wait for the status of the batch to be returned to
+the caller. This means rclone can use a much bigger batch size (much
+bigger than `--transfers`), at the cost of not being able to check the
+status of the upload.
-| Character | Value | Replacement |
-| --------- |:-----:|:-----------:|
-| \\ | 0x5C | ï¼¼ |
-| * | 0x2A | ＊ |
-| < | 0x3C | ＜ |
-| > | 0x3E | ＞ |
-| ? | 0x3F | ？ |
-| : | 0x3A | ： |
-| \| | 0x7C | ｜ |
-| " | 0x22 | ＂ |
+This provides the maximum possible upload speed especially with lots
+of small files, however rclone can't check the file got uploaded
+properly using this mode.
-File names can also not start or end with the following characters.
-These only get replaced if they are the first or last character in the
-name:
+If you are using this mode then using "rclone check" after the
+transfer completes is recommended. Or you could do an initial transfer
+with `--dropbox-batch-mode async` then do a final transfer with
+`--dropbox-batch-mode sync` (the default).
-| Character | Value | Replacement |
-| --------- |:-----:|:-----------:|
-| SP | 0x20 | â |
-| . | 0x2E | ． |
+Note that there may be a pause when quitting rclone while rclone
+finishes up the last batch using this mode.
-Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8),
-as they can't be used in JSON strings.
### Standard options
-Here are the Standard options specific to sharefile (Citrix Sharefile).
+Here are the Standard options specific to dropbox (Dropbox).
-#### --sharefile-client-id
+#### --dropbox-client-id
OAuth Client Id.
@@ -28963,11 +30752,11 @@ Leave blank normally.
Properties:
- Config: client_id
-- Env Var: RCLONE_SHAREFILE_CLIENT_ID
+- Env Var: RCLONE_DROPBOX_CLIENT_ID
- Type: string
- Required: false
-#### --sharefile-client-secret
+#### --dropbox-client-secret
OAuth Client Secret.
@@ -28976,51 +30765,26 @@ Leave blank normally.
Properties:
- Config: client_secret
-- Env Var: RCLONE_SHAREFILE_CLIENT_SECRET
-- Type: string
-- Required: false
-
-#### --sharefile-root-folder-id
-
-ID of the root folder.
-
-Leave blank to access "Personal Folders". You can use one of the
-standard values here or any folder ID (long hex number ID).
-
-Properties:
-
-- Config: root_folder_id
-- Env Var: RCLONE_SHAREFILE_ROOT_FOLDER_ID
+- Env Var: RCLONE_DROPBOX_CLIENT_SECRET
- Type: string
- Required: false
-- Examples:
- - ""
- - Access the Personal Folders (default).
- - "favorites"
- - Access the Favorites folder.
- - "allshared"
- - Access all the shared folders.
- - "connectors"
- - Access all the individual connectors.
- - "top"
- - Access the home, favorites, and shared folders as well as the connectors.
### Advanced options
-Here are the Advanced options specific to sharefile (Citrix Sharefile).
+Here are the Advanced options specific to dropbox (Dropbox).
-#### --sharefile-token
+#### --dropbox-token
OAuth Access Token as a JSON blob.
Properties:
- Config: token
-- Env Var: RCLONE_SHAREFILE_TOKEN
+- Env Var: RCLONE_DROPBOX_TOKEN
- Type: string
- Required: false
-#### --sharefile-auth-url
+#### --dropbox-auth-url
Auth server URL.
@@ -29029,11 +30793,11 @@ Leave blank to use the provider defaults.
Properties:
- Config: auth_url
-- Env Var: RCLONE_SHAREFILE_AUTH_URL
+- Env Var: RCLONE_DROPBOX_AUTH_URL
- Type: string
- Required: false
-#### --sharefile-token-url
+#### --dropbox-token-url
Token server url.
@@ -29042,1958 +30806,1711 @@ Leave blank to use the provider defaults.
Properties:
- Config: token_url
-- Env Var: RCLONE_SHAREFILE_TOKEN_URL
+- Env Var: RCLONE_DROPBOX_TOKEN_URL
- Type: string
- Required: false
-#### --sharefile-upload-cutoff
-
-Cutoff for switching to multipart upload.
-
-Properties:
-
-- Config: upload_cutoff
-- Env Var: RCLONE_SHAREFILE_UPLOAD_CUTOFF
-- Type: SizeSuffix
-- Default: 128Mi
-
-#### --sharefile-chunk-size
-
-Upload chunk size.
+#### --dropbox-chunk-size
-Must a power of 2 >= 256k.
+Upload chunk size (< 150Mi).
-Making this larger will improve performance, but note that each chunk
-is buffered in memory one per transfer.
+Any files larger than this will be uploaded in chunks of this size.
-Reducing this will reduce memory usage but decrease performance.
+Note that chunks are buffered in memory (one at a time) so rclone can
+deal with retries. Setting this larger will increase the speed
+slightly (at most 10% for 128 MiB in tests) at the cost of using more
+memory. It can be set smaller if you are tight on memory.
Properties:
- Config: chunk_size
-- Env Var: RCLONE_SHAREFILE_CHUNK_SIZE
+- Env Var: RCLONE_DROPBOX_CHUNK_SIZE
- Type: SizeSuffix
-- Default: 64Mi
-
-#### --sharefile-endpoint
-
-Endpoint for API calls.
-
-This is usually auto discovered as part of the oauth process, but can
-be set manually to something like: https://XXX.sharefile.com
-
-
-Properties:
-
-- Config: endpoint
-- Env Var: RCLONE_SHAREFILE_ENDPOINT
-- Type: string
-- Required: false
-
-#### --sharefile-encoding
-
-The encoding for the backend.
-
-See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info.
-
-Properties:
-
-- Config: encoding
-- Env Var: RCLONE_SHAREFILE_ENCODING
-- Type: MultiEncoder
-- Default: Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,LeftSpace,LeftPeriod,RightSpace,RightPeriod,InvalidUtf8,Dot
-
-
-## Limitations
-
-Note that ShareFile is case insensitive so you can't have a file called
-"Hello.doc" and one called "hello.doc".
-
-ShareFile only supports filenames up to 256 characters in length.
-
-`rclone about` is not supported by the Citrix ShareFile backend. Backends without
-this capability cannot determine free space for an rclone mount or
-use policy `mfs` (most free space) as a member of an rclone union
-remote.
-
-See [List of backends that do not support rclone about](https://rclone.org/overview/#optional-features) and [rclone about](https://rclone.org/commands/rclone_about/)
-
-# Crypt
-
-Rclone `crypt` remotes encrypt and decrypt other remotes.
-
-A remote of type `crypt` does not access a [storage system](https://rclone.org/overview/)
-directly, but instead wraps another remote, which in turn accesses
-the storage system. This is similar to how [alias](https://rclone.org/alias/),
-[union](https://rclone.org/union/), [chunker](https://rclone.org/chunker/)
-and a few others work. It makes the usage very flexible, as you can
-add a layer, in this case an encryption layer, on top of any other
-backend, even in multiple layers. Rclone's functionality
-can be used as with any other remote, for example you can
-[mount](https://rclone.org/commands/rclone_mount/) a crypt remote.
-
-Accessing a storage system through a crypt remote realizes client-side
-encryption, which makes it safe to keep your data in a location you do
-not trust will not get compromised.
-When working against the `crypt` remote, rclone will automatically
-encrypt (before uploading) and decrypt (after downloading) on your local
-system as needed on the fly, leaving the data encrypted at rest in the
-wrapped remote. If you access the storage system using an application
-other than rclone, or access the wrapped remote directly using rclone,
-there will not be any encryption/decryption: Downloading existing content
-will just give you the encrypted (scrambled) format, and anything you
-upload will *not* become encrypted.
-
-The encryption is a secret-key encryption (also called symmetric key encryption)
-algorithm, where a password (or pass phrase) is used to generate real encryption key.
-The password can be supplied by user, or you may chose to let rclone
-generate one. It will be stored in the configuration file, in a lightly obscured form.
-If you are in an environment where you are not able to keep your configuration
-secured, you should add
-[configuration encryption](https://rclone.org/docs/#configuration-encryption)
-as protection. As long as you have this configuration file, you will be able to
-decrypt your data. Without the configuration file, as long as you remember
-the password (or keep it in a safe place), you can re-create the configuration
-and gain access to the existing data. You may also configure a corresponding
-remote in a different installation to access the same data.
-See below for guidance to [changing password](#changing-password).
-
-Encryption uses [cryptographic salt](https://en.wikipedia.org/wiki/Salt_(cryptography)),
-to permute the encryption key so that the same string may be encrypted in
-different ways. When configuring the crypt remote it is optional to enter a salt,
-or to let rclone generate a unique salt. If omitted, rclone uses a built-in unique string.
-Normally in cryptography, the salt is stored together with the encrypted content,
-and do not have to be memorized by the user. This is not the case in rclone,
-because rclone does not store any additional information on the remotes. Use of
-custom salt is effectively a second password that must be memorized.
-
-[File content](#file-encryption) encryption is performed using
-[NaCl SecretBox](https://godoc.org/golang.org/x/crypto/nacl/secretbox),
-based on XSalsa20 cipher and Poly1305 for integrity.
-[Names](#name-encryption) (file- and directory names) are also encrypted
-by default, but this has some implications and is therefore
-possible to be turned off.
-
-## Configuration
-
-Here is an example of how to make a remote called `secret`.
-
-To use `crypt`, first set up the underlying remote. Follow the
-`rclone config` instructions for the specific backend.
-
-Before configuring the crypt remote, check the underlying remote is
-working. In this example the underlying remote is called `remote`.
-We will configure a path `path` within this remote to contain the
-encrypted content. Anything inside `remote:path` will be encrypted
-and anything outside will not.
-
-Configure `crypt` using `rclone config`. In this example the `crypt`
-remote is called `secret`, to differentiate it from the underlying
-`remote`.
-
-When you are done you can use the crypt remote named `secret` just
-as you would with any other remote, e.g. `rclone copy D:\docs secret:\docs`,
-and rclone will encrypt and decrypt as needed on the fly.
-If you access the wrapped remote `remote:path` directly you will bypass
-the encryption, and anything you read will be in encrypted form, and
-anything you write will be unencrypted. To avoid issues it is best to
-configure a dedicated path for encrypted content, and access it
-exclusively through a crypt remote.
-
-```
-No remotes found, make a new one?
-n) New remote
-s) Set configuration password
-q) Quit config
-n/s/q> n
-name> secret
-Type of storage to configure.
-Enter a string value. Press Enter for the default ("").
-Choose a number from below, or type in your own value
-[snip]
-XX / Encrypt/Decrypt a remote
- \ "crypt"
-[snip]
-Storage> crypt
-** See help for crypt backend at: https://rclone.org/crypt/ **
-
-Remote to encrypt/decrypt.
-Normally should contain a ':' and a path, eg "myremote:path/to/dir",
-"myremote:bucket" or maybe "myremote:" (not recommended).
-Enter a string value. Press Enter for the default ("").
-remote> remote:path
-How to encrypt the filenames.
-Enter a string value. Press Enter for the default ("standard").
-Choose a number from below, or type in your own value.
- / Encrypt the filenames.
- 1 | See the docs for the details.
- \ "standard"
- 2 / Very simple filename obfuscation.
- \ "obfuscate"
- / Don't encrypt the file names.
- 3 | Adds a ".bin" extension only.
- \ "off"
-filename_encryption>
-Option to either encrypt directory names or leave them intact.
-
-NB If filename_encryption is "off" then this option will do nothing.
-Enter a boolean value (true or false). Press Enter for the default ("true").
-Choose a number from below, or type in your own value
- 1 / Encrypt directory names.
- \ "true"
- 2 / Don't encrypt directory names, leave them intact.
- \ "false"
-directory_name_encryption>
-Password or pass phrase for encryption.
-y) Yes type in my own password
-g) Generate random password
-y/g> y
-Enter the password:
-password:
-Confirm the password:
-password:
-Password or pass phrase for salt. Optional but recommended.
-Should be different to the previous password.
-y) Yes type in my own password
-g) Generate random password
-n) No leave this optional password blank (default)
-y/g/n> g
-Password strength in bits.
-64 is just about memorable
-128 is secure
-1024 is the maximum
-Bits> 128
-Your password is: JAsJvRcgR-_veXNfy_sGmQ
-Use this password? Please note that an obscured version of this
-password (and not the password itself) will be stored under your
-configuration file, so keep this generated password in a safe place.
-y) Yes (default)
-n) No
-y/n>
-Edit advanced config? (y/n)
-y) Yes
-n) No (default)
-y/n>
-Remote config
---------------------
-[secret]
-type = crypt
-remote = remote:path
-password = *** ENCRYPTED ***
-password2 = *** ENCRYPTED ***
---------------------
-y) Yes this is OK (default)
-e) Edit this remote
-d) Delete this remote
-y/e/d>
-```
-
-**Important** The crypt password stored in `rclone.conf` is lightly
-obscured. That only protects it from cursory inspection. It is not
-secure unless [configuration encryption](https://rclone.org/docs/#configuration-encryption) of `rclone.conf` is specified.
-
-A long passphrase is recommended, or `rclone config` can generate a
-random one.
-
-The obscured password is created using AES-CTR with a static key. The
-salt is stored verbatim at the beginning of the obscured password. This
-static key is shared between all versions of rclone.
-
-If you reconfigure rclone with the same passwords/passphrases
-elsewhere it will be compatible, but the obscured version will be different
-due to the different salt.
-
-Rclone does not encrypt
-
- * file length - this can be calculated within 16 bytes
- * modification time - used for syncing
-
-### Specifying the remote
-
-When configuring the remote to encrypt/decrypt, you may specify any
-string that rclone accepts as a source/destination of other commands.
-
-The primary use case is to specify the path into an already configured
-remote (e.g. `remote:path/to/dir` or `remote:bucket`), such that
-data in a remote untrusted location can be stored encrypted.
-
-You may also specify a local filesystem path, such as
-`/path/to/dir` on Linux, `C:\path\to\dir` on Windows. By creating
-a crypt remote pointing to such a local filesystem path, you can
-use rclone as a utility for pure local file encryption, for example
-to keep encrypted files on a removable USB drive.
-
-**Note**: A string which do not contain a `:` will by rclone be treated
-as a relative path in the local filesystem. For example, if you enter
-the name `remote` without the trailing `:`, it will be treated as
-a subdirectory of the current directory with name "remote".
-
-If a path `remote:path/to/dir` is specified, rclone stores encrypted
-files in `path/to/dir` on the remote. With file name encryption, files
-saved to `secret:subdir/subfile` are stored in the unencrypted path
-`path/to/dir` but the `subdir/subpath` element is encrypted.
-
-The path you specify does not have to exist, rclone will create
-it when needed.
-
-If you intend to use the wrapped remote both directly for keeping
-unencrypted content, as well as through a crypt remote for encrypted
-content, it is recommended to point the crypt remote to a separate
-directory within the wrapped remote. If you use a bucket-based storage
-system (e.g. Swift, S3, Google Compute Storage, B2) it is generally
-advisable to wrap the crypt remote around a specific bucket (`s3:bucket`).
-If wrapping around the entire root of the storage (`s3:`), and use the
-optional file name encryption, rclone will encrypt the bucket name.
-
-### Changing password
-
-Should the password, or the configuration file containing a lightly obscured
-form of the password, be compromised, you need to re-encrypt your data with
-a new password. Since rclone uses secret-key encryption, where the encryption
-key is generated directly from the password kept on the client, it is not
-possible to change the password/key of already encrypted content. Just changing
-the password configured for an existing crypt remote means you will no longer
-able to decrypt any of the previously encrypted content. The only possibility
-is to re-upload everything via a crypt remote configured with your new password.
-
-Depending on the size of your data, your bandwidth, storage quota etc, there are
-different approaches you can take:
-- If you have everything in a different location, for example on your local system,
-you could remove all of the prior encrypted files, change the password for your
-configured crypt remote (or delete and re-create the crypt configuration),
-and then re-upload everything from the alternative location.
-- If you have enough space on the storage system you can create a new crypt
-remote pointing to a separate directory on the same backend, and then use
-rclone to copy everything from the original crypt remote to the new,
-effectively decrypting everything on the fly using the old password and
-re-encrypting using the new password. When done, delete the original crypt
-remote directory and finally the rclone crypt configuration with the old password.
-All data will be streamed from the storage system and back, so you will
-get half the bandwidth and be charged twice if you have upload and download quota
-on the storage system.
-
-**Note**: A security problem related to the random password generator
-was fixed in rclone version 1.53.3 (released 2020-11-19). Passwords generated
-by rclone config in version 1.49.0 (released 2019-08-26) to 1.53.2
-(released 2020-10-26) are not considered secure and should be changed.
-If you made up your own password, or used rclone version older than 1.49.0 or
-newer than 1.53.2 to generate it, you are *not* affected by this issue.
-See [issue #4783](https://github.com/rclone/rclone/issues/4783) for more
-details, and a tool you can use to check if you are affected.
-
-### Example
-
-Create the following file structure using "standard" file name
-encryption.
-
-```
-plaintext/
-├── file0.txt
-├── file1.txt
-└── subdir
- ├── file2.txt
- ├── file3.txt
- └── subsubdir
- └── file4.txt
-```
-
-Copy these to the remote, and list them
-
-```
-$ rclone -q copy plaintext secret:
-$ rclone -q ls secret:
- 7 file1.txt
- 6 file0.txt
- 8 subdir/file2.txt
- 10 subdir/subsubdir/file4.txt
- 9 subdir/file3.txt
-```
-
-The crypt remote looks like
-
-```
-$ rclone -q ls remote:path
- 55 hagjclgavj2mbiqm6u6cnjjqcg
- 54 v05749mltvv1tf4onltun46gls
- 57 86vhrsv86mpbtd3a0akjuqslj8/dlj7fkq4kdq72emafg7a7s41uo
- 58 86vhrsv86mpbtd3a0akjuqslj8/7uu829995du6o42n32otfhjqp4/b9pausrfansjth5ob3jkdqd4lc
- 56 86vhrsv86mpbtd3a0akjuqslj8/8njh1sk437gttmep3p70g81aps
-```
+- Default: 48Mi
-The directory structure is preserved
+#### --dropbox-impersonate
-```
-$ rclone -q ls secret:subdir
- 8 file2.txt
- 9 file3.txt
- 10 subsubdir/file4.txt
-```
+Impersonate this user when using a business account.
-Without file name encryption `.bin` extensions are added to underlying
-names. This prevents the cloud provider attempting to interpret file
-content.
+Note that if you want to use impersonate, you should make sure this
+flag is set when running "rclone config" as this will cause rclone to
+request the "members.read" scope which it won't normally. This is
+needed to lookup a members email address into the internal ID that
+dropbox uses in the API.
-```
-$ rclone -q ls remote:path
- 54 file0.txt.bin
- 57 subdir/file3.txt.bin
- 56 subdir/file2.txt.bin
- 58 subdir/subsubdir/file4.txt.bin
- 55 file1.txt.bin
-```
+Using the "members.read" scope will require a Dropbox Team Admin
+to approve during the OAuth flow.
-### File name encryption modes
+You will have to use your own App (setting your own client_id and
+client_secret) to use this option as currently rclone's default set of
+permissions doesn't include "members.read". This can be added once
+v1.55 or later is in use everywhere.
-Off
- * doesn't hide file names or directory structure
- * allows for longer file names (~246 characters)
- * can use sub paths and copy single files
+Properties:
-Standard
+- Config: impersonate
+- Env Var: RCLONE_DROPBOX_IMPERSONATE
+- Type: string
+- Required: false
- * file names encrypted
- * file names can't be as long (~143 characters)
- * can use sub paths and copy single files
- * directory structure visible
- * identical files names will have identical uploaded names
- * can use shortcuts to shorten the directory recursion
+#### --dropbox-shared-files
-Obfuscation
+Instructs rclone to work on individual shared files.
-This is a simple "rotate" of the filename, with each file having a rot
-distance based on the filename. Rclone stores the distance at the
-beginning of the filename. A file called "hello" may become "53.jgnnq".
+In this mode rclone's features are extremely limited - only list (ls, lsl, etc.)
+operations and read operations (e.g. downloading) are supported in this mode.
+All other operations will be disabled.
-Obfuscation is not a strong encryption of filenames, but hinders
-automated scanning tools picking up on filename patterns. It is an
-intermediate between "off" and "standard" which allows for longer path
-segment names.
+Properties:
-There is a possibility with some unicode based filenames that the
-obfuscation is weak and may map lower case characters to upper case
-equivalents.
+- Config: shared_files
+- Env Var: RCLONE_DROPBOX_SHARED_FILES
+- Type: bool
+- Default: false
-Obfuscation cannot be relied upon for strong protection.
+#### --dropbox-shared-folders
- * file names very lightly obfuscated
- * file names can be longer than standard encryption
- * can use sub paths and copy single files
- * directory structure visible
- * identical files names will have identical uploaded names
+Instructs rclone to work on shared folders.
+
+When this flag is used with no path only the List operation is supported and
+all available shared folders will be listed. If you specify a path the first part
+will be interpreted as the name of shared folder. Rclone will then try to mount this
+shared to the root namespace. On success shared folder rclone proceeds normally.
+The shared folder is now pretty much a normal folder and all normal operations
+are supported.
-Cloud storage systems have limits on file name length and
-total path length which rclone is more likely to breach using
-"Standard" file name encryption. Where file names are less than 156
-characters in length issues should not be encountered, irrespective of
-cloud storage provider.
+Note that we don't unmount the shared folder afterwards so the
+--dropbox-shared-folders can be omitted after the first use of a particular
+shared folder.
-An experimental advanced option `filename_encoding` is now provided to
-address this problem to a certain degree.
-For cloud storage systems with case sensitive file names (e.g. Google Drive),
-`base64` can be used to reduce file name length.
-For cloud storage systems using UTF-16 to store file names internally
-(e.g. OneDrive, Dropbox, Box), `base32768` can be used to drastically reduce
-file name length.
+Properties:
-An alternative, future rclone file name encryption mode may tolerate
-backend provider path length limits.
+- Config: shared_folders
+- Env Var: RCLONE_DROPBOX_SHARED_FOLDERS
+- Type: bool
+- Default: false
-### Directory name encryption
+#### --dropbox-pacer-min-sleep
-Crypt offers the option of encrypting dir names or leaving them intact.
-There are two options:
+Minimum time to sleep between API calls.
-True
+Properties:
-Encrypts the whole file path including directory names
-Example:
-`1/12/123.txt` is encrypted to
-`p0e52nreeaj0a5ea7s64m4j72s/l42g6771hnv3an9cgc8cr2n1ng/qgm4avr35m5loi1th53ato71v0`
+- Config: pacer_min_sleep
+- Env Var: RCLONE_DROPBOX_PACER_MIN_SLEEP
+- Type: Duration
+- Default: 10ms
-False
+#### --dropbox-encoding
-Only encrypts file names, skips directory names
-Example:
-`1/12/123.txt` is encrypted to
-`1/12/qgm4avr35m5loi1th53ato71v0`
+The encoding for the backend.
+See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info.
-### Modified time and hashes
+Properties:
-Crypt stores modification times using the underlying remote so support
-depends on that.
+- Config: encoding
+- Env Var: RCLONE_DROPBOX_ENCODING
+- Type: Encoding
+- Default: Slash,BackSlash,Del,RightSpace,InvalidUtf8,Dot
-Hashes are not stored for crypt. However the data integrity is
-protected by an extremely strong crypto authenticator.
+#### --dropbox-batch-mode
-Use the `rclone cryptcheck` command to check the
-integrity of an encrypted remote instead of `rclone check` which can't
-check the checksums properly.
+Upload file batching sync|async|off.
+This sets the batch mode used by rclone.
-### Standard options
+For full info see [the main docs](https://rclone.org/dropbox/#batch-mode)
-Here are the Standard options specific to crypt (Encrypt/Decrypt a remote).
+This has 3 possible values
-#### --crypt-remote
+- off - no batching
+- sync - batch uploads and check completion (default)
+- async - batch upload and don't check completion
-Remote to encrypt/decrypt.
+Rclone will close any outstanding batches when it exits which may make
+a delay on quit.
-Normally should contain a ':' and a path, e.g. "myremote:path/to/dir",
-"myremote:bucket" or maybe "myremote:" (not recommended).
Properties:
-- Config: remote
-- Env Var: RCLONE_CRYPT_REMOTE
+- Config: batch_mode
+- Env Var: RCLONE_DROPBOX_BATCH_MODE
- Type: string
-- Required: true
+- Default: "sync"
-#### --crypt-filename-encryption
+#### --dropbox-batch-size
-How to encrypt the filenames.
+Max number of files in upload batch.
-Properties:
+This sets the batch size of files to upload. It has to be less than 1000.
-- Config: filename_encryption
-- Env Var: RCLONE_CRYPT_FILENAME_ENCRYPTION
-- Type: string
-- Default: "standard"
-- Examples:
- - "standard"
- - Encrypt the filenames.
- - See the docs for the details.
- - "obfuscate"
- - Very simple filename obfuscation.
- - "off"
- - Don't encrypt the file names.
- - Adds a ".bin", or "suffix" extension only.
+By default this is 0 which means rclone which calculate the batch size
+depending on the setting of batch_mode.
-#### --crypt-directory-name-encryption
+- batch_mode: async - default batch_size is 100
+- batch_mode: sync - default batch_size is the same as --transfers
+- batch_mode: off - not in use
-Option to either encrypt directory names or leave them intact.
+Rclone will close any outstanding batches when it exits which may make
+a delay on quit.
+
+Setting this is a great idea if you are uploading lots of small files
+as it will make them a lot quicker. You can use --transfers 32 to
+maximise throughput.
-NB If filename_encryption is "off" then this option will do nothing.
Properties:
-- Config: directory_name_encryption
-- Env Var: RCLONE_CRYPT_DIRECTORY_NAME_ENCRYPTION
-- Type: bool
-- Default: true
-- Examples:
- - "true"
- - Encrypt directory names.
- - "false"
- - Don't encrypt directory names, leave them intact.
+- Config: batch_size
+- Env Var: RCLONE_DROPBOX_BATCH_SIZE
+- Type: int
+- Default: 0
-#### --crypt-password
+#### --dropbox-batch-timeout
-Password or pass phrase for encryption.
+Max time to allow an idle upload batch before uploading.
-**NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/).
+If an upload batch is idle for more than this long then it will be
+uploaded.
-Properties:
+The default for this is 0 which means rclone will choose a sensible
+default based on the batch_mode in use.
-- Config: password
-- Env Var: RCLONE_CRYPT_PASSWORD
-- Type: string
-- Required: true
+- batch_mode: async - default batch_timeout is 10s
+- batch_mode: sync - default batch_timeout is 500ms
+- batch_mode: off - not in use
-#### --crypt-password2
-Password or pass phrase for salt.
+Properties:
-Optional but recommended.
-Should be different to the previous password.
+- Config: batch_timeout
+- Env Var: RCLONE_DROPBOX_BATCH_TIMEOUT
+- Type: Duration
+- Default: 0s
-**NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/).
+#### --dropbox-batch-commit-timeout
+
+Max time to wait for a batch to finish committing
Properties:
-- Config: password2
-- Env Var: RCLONE_CRYPT_PASSWORD2
-- Type: string
-- Required: false
+- Config: batch_commit_timeout
+- Env Var: RCLONE_DROPBOX_BATCH_COMMIT_TIMEOUT
+- Type: Duration
+- Default: 10m0s
-### Advanced options
-Here are the Advanced options specific to crypt (Encrypt/Decrypt a remote).
-#### --crypt-server-side-across-configs
+## Limitations
-Deprecated: use --server-side-across-configs instead.
+Note that Dropbox is case insensitive so you can't have a file called
+"Hello.doc" and one called "hello.doc".
-Allow server-side operations (e.g. copy) to work across different crypt configs.
+There are some file names such as `thumbs.db` which Dropbox can't
+store. There is a full list of them in the ["Ignored Files" section
+of this document](https://www.dropbox.com/en/help/145). Rclone will
+issue an error message `File name disallowed - not uploading` if it
+attempts to upload one of those file names, but the sync won't fail.
-Normally this option is not what you want, but if you have two crypts
-pointing to the same backend you can use it.
+Some errors may occur if you try to sync copyright-protected files
+because Dropbox has its own [copyright detector](https://techcrunch.com/2014/03/30/how-dropbox-knows-when-youre-sharing-copyrighted-stuff-without-actually-looking-at-your-stuff/) that
+prevents this sort of file being downloaded. This will return the error `ERROR :
+/path/to/your/file: Failed to copy: failed to open source object:
+path/restricted_content/.`
-This can be used, for example, to change file name encryption type
-without re-uploading all the data. Just make two crypt backends
-pointing to two different directories with the single changed
-parameter and use rclone move to move the files between the crypt
-remotes.
+If you have more than 10,000 files in a directory then `rclone purge
+dropbox:dir` will return the error `Failed to purge: There are too
+many files involved in this operation`. As a work-around do an
+`rclone delete dropbox:dir` followed by an `rclone rmdir dropbox:dir`.
-Properties:
+When using `rclone link` you'll need to set `--expire` if using a
+non-personal account otherwise the visibility may not be correct.
+(Note that `--expire` isn't supported on personal accounts). See the
+[forum discussion](https://forum.rclone.org/t/rclone-link-dropbox-permissions/23211) and the
+[dropbox SDK issue](https://github.com/dropbox/dropbox-sdk-go-unofficial/issues/75).
-- Config: server_side_across_configs
-- Env Var: RCLONE_CRYPT_SERVER_SIDE_ACROSS_CONFIGS
-- Type: bool
-- Default: false
+## Get your own Dropbox App ID
-#### --crypt-show-mapping
+When you use rclone with Dropbox in its default configuration you are using rclone's App ID. This is shared between all the rclone users.
-For all files listed show how the names encrypt.
+Here is how to create your own Dropbox App ID for rclone:
-If this flag is set then for each file that the remote is asked to
-list, it will log (at level INFO) a line stating the decrypted file
-name and the encrypted file name.
+1. Log into the [Dropbox App console](https://www.dropbox.com/developers/apps/create) with your Dropbox Account (It need not
+to be the same account as the Dropbox you want to access)
-This is so you can work out which encrypted names are which decrypted
-names just in case you need to do something with the encrypted file
-names, or for debugging purposes.
+2. Choose an API => Usually this should be `Dropbox API`
-Properties:
+3. Choose the type of access you want to use => `Full Dropbox` or `App Folder`. If you want to use Team Folders, `Full Dropbox` is required ([see here](https://www.dropboxforum.com/t5/Dropbox-API-Support-Feedback/How-to-create-team-folder-inside-my-app-s-folder/m-p/601005/highlight/true#M27911)).
-- Config: show_mapping
-- Env Var: RCLONE_CRYPT_SHOW_MAPPING
-- Type: bool
-- Default: false
+4. Name your App. The app name is global, so you can't use `rclone` for example
-#### --crypt-no-data-encryption
+5. Click the button `Create App`
-Option to either encrypt file data or leave it unencrypted.
+6. Switch to the `Permissions` tab. Enable at least the following permissions: `account_info.read`, `files.metadata.write`, `files.content.write`, `files.content.read`, `sharing.write`. The `files.metadata.read` and `sharing.read` checkboxes will be marked too. Click `Submit`
-Properties:
+7. Switch to the `Settings` tab. Fill `OAuth2 - Redirect URIs` as `http://localhost:53682/` and click on `Add`
-- Config: no_data_encryption
-- Env Var: RCLONE_CRYPT_NO_DATA_ENCRYPTION
-- Type: bool
-- Default: false
-- Examples:
- - "true"
- - Don't encrypt file data, leave it unencrypted.
- - "false"
- - Encrypt file data.
+8. Find the `App key` and `App secret` values on the `Settings` tab. Use these values in rclone config to add a new remote or edit an existing remote. The `App key` setting corresponds to `client_id` in rclone config, the `App secret` corresponds to `client_secret`
-#### --crypt-pass-bad-blocks
+# Enterprise File Fabric
-If set this will pass bad blocks through as all 0.
+This backend supports [Storage Made Easy's Enterprise File
+Fabricâ„¢](https://storagemadeeasy.com/about/) which provides a software
+solution to integrate and unify File and Object Storage accessible
+through a global file system.
-This should not be set in normal operation, it should only be set if
-trying to recover an encrypted file with errors and it is desired to
-recover as much of the file as possible.
+## Configuration
-Properties:
+The initial setup for the Enterprise File Fabric backend involves
+getting a token from the Enterprise File Fabric which you need to
+do in your browser. `rclone config` walks you through it.
-- Config: pass_bad_blocks
-- Env Var: RCLONE_CRYPT_PASS_BAD_BLOCKS
-- Type: bool
-- Default: false
+Here is an example of how to make a remote called `remote`. First run:
-#### --crypt-filename-encoding
+ rclone config
-How to encode the encrypted filename to text string.
+This will guide you through an interactive setup process:
-This option could help with shortening the encrypted filename. The
-suitable option would depend on the way your remote count the filename
-length and if it's case sensitive.
+```
+No remotes found, make a new one?
+n) New remote
+s) Set configuration password
+q) Quit config
+n/s/q> n
+name> remote
+Type of storage to configure.
+Enter a string value. Press Enter for the default ("").
+Choose a number from below, or type in your own value
+[snip]
+XX / Enterprise File Fabric
+ \ "filefabric"
+[snip]
+Storage> filefabric
+** See help for filefabric backend at: https://rclone.org/filefabric/ **
-Properties:
+URL of the Enterprise File Fabric to connect to
+Enter a string value. Press Enter for the default ("").
+Choose a number from below, or type in your own value
+ 1 / Storage Made Easy US
+ \ "https://storagemadeeasy.com"
+ 2 / Storage Made Easy EU
+ \ "https://eu.storagemadeeasy.com"
+ 3 / Connect to your Enterprise File Fabric
+ \ "https://yourfabric.smestorage.com"
+url> https://yourfabric.smestorage.com/
+ID of the root folder
+Leave blank normally.
-- Config: filename_encoding
-- Env Var: RCLONE_CRYPT_FILENAME_ENCODING
-- Type: string
-- Default: "base32"
-- Examples:
- - "base32"
- - Encode using base32. Suitable for all remote.
- - "base64"
- - Encode using base64. Suitable for case sensitive remote.
- - "base32768"
- - Encode using base32768. Suitable if your remote counts UTF-16 or
- - Unicode codepoint instead of UTF-8 byte length. (Eg. Onedrive, Dropbox)
+Fill in to make rclone start with directory of a given ID.
-#### --crypt-suffix
+Enter a string value. Press Enter for the default ("").
+root_folder_id>
+Permanent Authentication Token
-If this is set it will override the default suffix of ".bin".
+A Permanent Authentication Token can be created in the Enterprise File
+Fabric, on the users Dashboard under Security, there is an entry
+you'll see called "My Authentication Tokens". Click the Manage button
+to create one.
-Setting suffix to "none" will result in an empty suffix. This may be useful
-when the path length is critical.
+These tokens are normally valid for several years.
-Properties:
+For more info see: https://docs.storagemadeeasy.com/organisationcloud/api-tokens
-- Config: suffix
-- Env Var: RCLONE_CRYPT_SUFFIX
-- Type: string
-- Default: ".bin"
+Enter a string value. Press Enter for the default ("").
+permanent_token> xxxxxxxxxxxxxxx-xxxxxxxxxxxxxxxx
+Edit advanced config? (y/n)
+y) Yes
+n) No (default)
+y/n> n
+Remote config
+--------------------
+[remote]
+type = filefabric
+url = https://yourfabric.smestorage.com/
+permanent_token = xxxxxxxxxxxxxxx-xxxxxxxxxxxxxxxx
+--------------------
+y) Yes this is OK (default)
+e) Edit this remote
+d) Delete this remote
+y/e/d> y
+```
-### Metadata
+Once configured you can then use `rclone` like this,
-Any metadata supported by the underlying remote is read and written.
+List directories in top level of your Enterprise File Fabric
-See the [metadata](https://rclone.org/docs/#metadata) docs for more info.
+ rclone lsd remote:
-## Backend commands
+List all the files in your Enterprise File Fabric
-Here are the commands specific to the crypt backend.
+ rclone ls remote:
-Run them with
+To copy a local directory to an Enterprise File Fabric directory called backup
- rclone backend COMMAND remote:
+ rclone copy /home/source remote:backup
-The help below will explain what arguments each command takes.
+### Modification times and hashes
-See the [backend](https://rclone.org/commands/rclone_backend/) command for more
-info on how to pass options and arguments.
+The Enterprise File Fabric allows modification times to be set on
+files accurate to 1 second. These will be used to detect whether
+objects need syncing or not.
-These can be run on a running backend using the rc command
-[backend/command](https://rclone.org/rc/#backend-command).
+The Enterprise File Fabric does not support any data hashes at this time.
-### encode
+### Restricted filename characters
-Encode the given filename(s)
+The [default restricted characters set](https://rclone.org/overview/#restricted-characters)
+will be replaced.
- rclone backend encode remote: [options] [+]
+Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8),
+as they can't be used in JSON strings.
-This encodes the filenames given as arguments returning a list of
-strings of the encoded results.
+### Empty files
-Usage Example:
+Empty files aren't supported by the Enterprise File Fabric. Rclone will therefore
+upload an empty file as a single space with a mime type of
+`application/vnd.rclone.empty.file` and files with that mime type are
+treated as empty.
- rclone backend encode crypt: file1 [file2...]
- rclone rc backend/command command=encode fs=crypt: file1 [file2...]
+### Root folder ID ###
+You can set the `root_folder_id` for rclone. This is the directory
+(identified by its `Folder ID`) that rclone considers to be the root
+of your Enterprise File Fabric.
-### decode
+Normally you will leave this blank and rclone will determine the
+correct root to use itself.
-Decode the given filename(s)
+However you can set this to restrict rclone to a specific folder
+hierarchy.
- rclone backend decode remote: [options] [+]
+In order to do this you will have to find the `Folder ID` of the
+directory you wish rclone to display. These aren't displayed in the
+web interface, but you can use `rclone lsf` to find them, for example
-This decodes the filenames given as arguments returning a list of
-strings of the decoded results. It will return an error if any of the
-inputs are invalid.
+```
+$ rclone lsf --dirs-only -Fip --csv filefabric:
+120673758,Burnt PDFs/
+120673759,My Quick Uploads/
+120673755,My Syncs/
+120673756,My backups/
+120673757,My contacts/
+120673761,S3 Storage/
+```
-Usage Example:
+The ID for "S3 Storage" would be `120673761`.
- rclone backend decode crypt: encryptedfile1 [encryptedfile2...]
- rclone rc backend/command command=decode fs=crypt: encryptedfile1 [encryptedfile2...]
+### Standard options
+
+Here are the Standard options specific to filefabric (Enterprise File Fabric).
+#### --filefabric-url
+URL of the Enterprise File Fabric to connect to.
-## Backing up an encrypted remote
+Properties:
+
+- Config: url
+- Env Var: RCLONE_FILEFABRIC_URL
+- Type: string
+- Required: true
+- Examples:
+ - "https://storagemadeeasy.com"
+ - Storage Made Easy US
+ - "https://eu.storagemadeeasy.com"
+ - Storage Made Easy EU
+ - "https://yourfabric.smestorage.com"
+ - Connect to your Enterprise File Fabric
-If you wish to backup an encrypted remote, it is recommended that you use
-`rclone sync` on the encrypted files, and make sure the passwords are
-the same in the new encrypted remote.
+#### --filefabric-root-folder-id
-This will have the following advantages
+ID of the root folder.
- * `rclone sync` will check the checksums while copying
- * you can use `rclone check` between the encrypted remotes
- * you don't decrypt and encrypt unnecessarily
+Leave blank normally.
-For example, let's say you have your original remote at `remote:` with
-the encrypted version at `eremote:` with path `remote:crypt`. You
-would then set up the new remote `remote2:` and then the encrypted
-version `eremote2:` with path `remote2:crypt` using the same passwords
-as `eremote:`.
+Fill in to make rclone start with directory of a given ID.
-To sync the two remotes you would do
- rclone sync --interactive remote:crypt remote2:crypt
+Properties:
-And to check the integrity you would do
+- Config: root_folder_id
+- Env Var: RCLONE_FILEFABRIC_ROOT_FOLDER_ID
+- Type: string
+- Required: false
- rclone check remote:crypt remote2:crypt
+#### --filefabric-permanent-token
-## File formats
+Permanent Authentication Token.
-### File encryption
+A Permanent Authentication Token can be created in the Enterprise File
+Fabric, on the users Dashboard under Security, there is an entry
+you'll see called "My Authentication Tokens". Click the Manage button
+to create one.
-Files are encrypted 1:1 source file to destination object. The file
-has a header and is divided into chunks.
+These tokens are normally valid for several years.
-#### Header
+For more info see: https://docs.storagemadeeasy.com/organisationcloud/api-tokens
- * 8 bytes magic string `RCLONE\x00\x00`
- * 24 bytes Nonce (IV)
-The initial nonce is generated from the operating systems crypto
-strong random number generator. The nonce is incremented for each
-chunk read making sure each nonce is unique for each block written.
-The chance of a nonce being re-used is minuscule. If you wrote an
-exabyte of data (10¹⸠bytes) you would have a probability of
-approximately 2×10â»Â³Â² of re-using a nonce.
+Properties:
-#### Chunk
+- Config: permanent_token
+- Env Var: RCLONE_FILEFABRIC_PERMANENT_TOKEN
+- Type: string
+- Required: false
-Each chunk will contain 64 KiB of data, except for the last one which
-may have less data. The data chunk is in standard NaCl SecretBox
-format. SecretBox uses XSalsa20 and Poly1305 to encrypt and
-authenticate messages.
+### Advanced options
-Each chunk contains:
+Here are the Advanced options specific to filefabric (Enterprise File Fabric).
- * 16 Bytes of Poly1305 authenticator
- * 1 - 65536 bytes XSalsa20 encrypted data
+#### --filefabric-token
-64k chunk size was chosen as the best performing chunk size (the
-authenticator takes too much time below this and the performance drops
-off due to cache effects above this). Note that these chunks are
-buffered in memory so they can't be too big.
+Session Token.
-This uses a 32 byte (256 bit key) key derived from the user password.
+This is a session token which rclone caches in the config file. It is
+usually valid for 1 hour.
-#### Examples
+Don't set this value - rclone will set it automatically.
-1 byte file will encrypt to
- * 32 bytes header
- * 17 bytes data chunk
+Properties:
-49 bytes total
+- Config: token
+- Env Var: RCLONE_FILEFABRIC_TOKEN
+- Type: string
+- Required: false
-1 MiB (1048576 bytes) file will encrypt to
+#### --filefabric-token-expiry
- * 32 bytes header
- * 16 chunks of 65568 bytes
+Token expiry time.
-1049120 bytes total (a 0.05% overhead). This is the overhead for big
-files.
+Don't set this value - rclone will set it automatically.
-### Name encryption
-File names are encrypted segment by segment - the path is broken up
-into `/` separated strings and these are encrypted individually.
+Properties:
-File segments are padded using PKCS#7 to a multiple of 16 bytes
-before encryption.
+- Config: token_expiry
+- Env Var: RCLONE_FILEFABRIC_TOKEN_EXPIRY
+- Type: string
+- Required: false
-They are then encrypted with EME using AES with 256 bit key. EME
-(ECB-Mix-ECB) is a wide-block encryption mode presented in the 2003
-paper "A Parallelizable Enciphering Mode" by Halevi and Rogaway.
+#### --filefabric-version
-This makes for deterministic encryption which is what we want - the
-same filename must encrypt to the same thing otherwise we can't find
-it on the cloud storage system.
+Version read from the file fabric.
-This means that
+Don't set this value - rclone will set it automatically.
- * filenames with the same name will encrypt the same
- * filenames which start the same won't have a common prefix
-This uses a 32 byte key (256 bits) and a 16 byte (128 bits) IV both of
-which are derived from the user password.
+Properties:
-After encryption they are written out using a modified version of
-standard `base32` encoding as described in RFC4648. The standard
-encoding is modified in two ways:
+- Config: version
+- Env Var: RCLONE_FILEFABRIC_VERSION
+- Type: string
+- Required: false
- * it becomes lower case (no-one likes upper case filenames!)
- * we strip the padding character `=`
+#### --filefabric-encoding
-`base32` is used rather than the more efficient `base64` so rclone can be
-used on case insensitive remotes (e.g. Windows, Amazon Drive).
+The encoding for the backend.
-### Key derivation
+See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info.
-Rclone uses `scrypt` with parameters `N=16384, r=8, p=1` with an
-optional user supplied salt (password2) to derive the 32+32+16 = 80
-bytes of key material required. If the user doesn't supply a salt
-then rclone uses an internal one.
+Properties:
-`scrypt` makes it impractical to mount a dictionary attack on rclone
-encrypted data. For full protection against this you should always use
-a salt.
+- Config: encoding
+- Env Var: RCLONE_FILEFABRIC_ENCODING
+- Type: Encoding
+- Default: Slash,Del,Ctl,InvalidUtf8,Dot
-## SEE ALSO
-* [rclone cryptdecode](https://rclone.org/commands/rclone_cryptdecode/) - Show forward/reverse mapping of encrypted filenames
-# Compress
+# FTP
-## Warning
+FTP is the File Transfer Protocol. Rclone FTP support is provided using the
+[github.com/jlaffaye/ftp](https://godoc.org/github.com/jlaffaye/ftp)
+package.
-This remote is currently **experimental**. Things may break and data may be lost. Anything you do with this remote is
-at your own risk. Please understand the risks associated with using experimental code and don't use this remote in
-critical applications.
+[Limitations of Rclone's FTP backend](#limitations)
-The `Compress` remote adds compression to another remote. It is best used with remotes containing
-many large compressible files.
+Paths are specified as `remote:path`. If the path does not begin with
+a `/` it is relative to the home directory of the user. An empty path
+`remote:` refers to the user's home directory.
## Configuration
-To use this remote, all you need to do is specify another remote and a compression mode to use:
+To create an FTP configuration named `remote`, run
-```
-Current remotes:
+ rclone config
-Name Type
-==== ====
-remote_to_press sometype
+Rclone config guides you through an interactive setup process. A minimal
+rclone FTP remote definition only requires host, username and password.
+For an anonymous FTP server, see [below](#anonymous-ftp).
-e) Edit existing remote
-$ rclone config
+```
+No remotes found, make a new one?
n) New remote
-d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
-e/n/d/r/c/s/q> n
-name> compress
-...
- 8 / Compress a remote
- \ "compress"
-...
-Storage> compress
-** See help for compress backend at: https://rclone.org/compress/ **
+n/r/c/s/q> n
+name> remote
+Type of storage to configure.
+Enter a string value. Press Enter for the default ("").
+Choose a number from below, or type in your own value
+[snip]
+XX / FTP
+ \ "ftp"
+[snip]
+Storage> ftp
+** See help for ftp backend at: https://rclone.org/ftp/ **
-Remote to compress.
+FTP host to connect to
Enter a string value. Press Enter for the default ("").
-remote> remote_to_press:subdir
-Compression mode.
-Enter a string value. Press Enter for the default ("gzip").
Choose a number from below, or type in your own value
- 1 / Gzip compression balanced for speed and compression strength.
- \ "gzip"
-compression_mode> gzip
-Edit advanced config? (y/n)
-y) Yes
-n) No (default)
-y/n> n
+ 1 / Connect to ftp.example.com
+ \ "ftp.example.com"
+host> ftp.example.com
+FTP username
+Enter a string value. Press Enter for the default ("$USER").
+user>
+FTP port number
+Enter a signed integer. Press Enter for the default (21).
+port>
+FTP password
+y) Yes type in my own password
+g) Generate random password
+y/g> y
+Enter the password:
+password:
+Confirm the password:
+password:
+Use FTP over TLS (Implicit)
+Enter a boolean value (true or false). Press Enter for the default ("false").
+tls>
+Use FTP over TLS (Explicit)
+Enter a boolean value (true or false). Press Enter for the default ("false").
+explicit_tls>
Remote config
--------------------
-[compress]
-type = compress
-remote = remote_to_press:subdir
-compression_mode = gzip
+[remote]
+type = ftp
+host = ftp.example.com
+pass = *** ENCRYPTED ***
--------------------
-y) Yes this is OK (default)
+y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> y
```
-### Compression Modes
-
-Currently only gzip compression is supported. It provides a decent balance between speed and size and is well
-supported by other applications. Compression strength can further be configured via an advanced setting where 0 is no
-compression and 9 is strongest compression.
-
-### File types
-
-If you open a remote wrapped by compress, you will see that there are many files with an extension corresponding to
-the compression algorithm you chose. These files are standard files that can be opened by various archive programs,
-but they have some hidden metadata that allows them to be used by rclone.
-While you may download and decompress these files at will, do **not** manually delete or rename files. Files without
-correct metadata files will not be recognized by rclone.
-
-### File names
-
-The compressed files will be named `*.###########.gz` where `*` is the base file and the `#` part is base64 encoded
-size of the uncompressed file. The file names should not be changed by anything other than the rclone compression backend.
-
-
-### Standard options
-
-Here are the Standard options specific to compress (Compress a remote).
-
-#### --compress-remote
-
-Remote to compress.
-
-Properties:
-
-- Config: remote
-- Env Var: RCLONE_COMPRESS_REMOTE
-- Type: string
-- Required: true
-
-#### --compress-mode
-
-Compression mode.
-
-Properties:
-
-- Config: mode
-- Env Var: RCLONE_COMPRESS_MODE
-- Type: string
-- Default: "gzip"
-- Examples:
- - "gzip"
- - Standard gzip compression with fastest parameters.
-
-### Advanced options
+To see all directories in the home directory of `remote`
-Here are the Advanced options specific to compress (Compress a remote).
+ rclone lsd remote:
-#### --compress-level
+Make a new directory
-GZIP compression level (-2 to 9).
+ rclone mkdir remote:path/to/directory
-Generally -1 (default, equivalent to 5) is recommended.
-Levels 1 to 9 increase compression at the cost of speed. Going past 6
-generally offers very little return.
+List the contents of a directory
-Level -2 uses Huffman encoding only. Only use if you know what you
-are doing.
-Level 0 turns off compression.
+ rclone ls remote:path/to/directory
-Properties:
+Sync `/home/local/directory` to the remote directory, deleting any
+excess files in the directory.
-- Config: level
-- Env Var: RCLONE_COMPRESS_LEVEL
-- Type: int
-- Default: -1
+ rclone sync --interactive /home/local/directory remote:directory
-#### --compress-ram-cache-limit
+### Anonymous FTP
-Some remotes don't allow the upload of files with unknown size.
-In this case the compressed file will need to be cached to determine
-it's size.
+When connecting to a FTP server that allows anonymous login, you can use the
+special "anonymous" username. Traditionally, this user account accepts any
+string as a password, although it is common to use either the password
+"anonymous" or "guest". Some servers require the use of a valid e-mail
+address as password.
-Files smaller than this limit will be cached in RAM, files larger than
-this limit will be cached on disk.
+Using [on-the-fly](#backend-path-to-dir) or
+[connection string](https://rclone.org/docs/#connection-strings) remotes makes it easy to access
+such servers, without requiring any configuration in advance. The following
+are examples of that:
-Properties:
+ rclone lsf :ftp: --ftp-host=speedtest.tele2.net --ftp-user=anonymous --ftp-pass=$(rclone obscure dummy)
+ rclone lsf :ftp,host=speedtest.tele2.net,user=anonymous,pass=$(rclone obscure dummy):
-- Config: ram_cache_limit
-- Env Var: RCLONE_COMPRESS_RAM_CACHE_LIMIT
-- Type: SizeSuffix
-- Default: 20Mi
+The above examples work in Linux shells and in PowerShell, but not Windows
+Command Prompt. They execute the [rclone obscure](https://rclone.org/commands/rclone_obscure/)
+command to create a password string in the format required by the
+[pass](#ftp-pass) option. The following examples are exactly the same, except use
+an already obscured string representation of the same password "dummy", and
+therefore works even in Windows Command Prompt:
-### Metadata
+ rclone lsf :ftp: --ftp-host=speedtest.tele2.net --ftp-user=anonymous --ftp-pass=IXs2wc8OJOz7SYLBk47Ji1rHTmxM
+ rclone lsf :ftp,host=speedtest.tele2.net,user=anonymous,pass=IXs2wc8OJOz7SYLBk47Ji1rHTmxM:
-Any metadata supported by the underlying remote is read and written.
+### Implicit TLS
-See the [metadata](https://rclone.org/docs/#metadata) docs for more info.
+Rlone FTP supports implicit FTP over TLS servers (FTPS). This has to
+be enabled in the FTP backend config for the remote, or with
+[`--ftp-tls`](#ftp-tls). The default FTPS port is `990`, not `21` and
+can be set with [`--ftp-port`](#ftp-port).
+### Restricted filename characters
+In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters)
+the following characters are also replaced:
-# Combine
+File names cannot end with the following characters. Replacement is
+limited to the last character in a file name:
-The `combine` backend joins remotes together into a single directory
-tree.
+| Character | Value | Replacement |
+| --------- |:-----:|:-----------:|
+| SP | 0x20 | â |
-For example you might have a remote for images on one provider:
+Not all FTP servers can have all characters in file names, for example:
-```
-$ rclone tree s3:imagesbucket
-/
-├── image1.jpg
-└── image2.jpg
-```
+| FTP Server| Forbidden characters |
+| --------- |:--------------------:|
+| proftpd | `*` |
+| pureftpd | `\ [ ]` |
-And a remote for files on another:
+This backend's interactive configuration wizard provides a selection of
+sensible encoding settings for major FTP servers: ProFTPd, PureFTPd, VsFTPd.
+Just hit a selection number when prompted.
-```
-$ rclone tree drive:important/files
-/
-├── file1.txt
-└── file2.txt
-```
-The `combine` backend can join these together into a synthetic
-directory structure like this:
+### Standard options
-```
-$ rclone tree combined:
-/
-├── files
-│ ├── file1.txt
-│ └── file2.txt
-└── images
- ├── image1.jpg
- └── image2.jpg
-```
+Here are the Standard options specific to ftp (FTP).
-You'd do this by specifying an `upstreams` parameter in the config
-like this
+#### --ftp-host
- upstreams = images=s3:imagesbucket files=drive:important/files
+FTP host to connect to.
-During the initial setup with `rclone config` you will specify the
-upstreams remotes as a space separated list. The upstream remotes can
-either be a local paths or other remotes.
+E.g. "ftp.example.com".
-## Configuration
+Properties:
-Here is an example of how to make a combine called `remote` for the
-example above. First run:
+- Config: host
+- Env Var: RCLONE_FTP_HOST
+- Type: string
+- Required: true
- rclone config
+#### --ftp-user
-This will guide you through an interactive setup process:
+FTP username.
-```
-No remotes found, make a new one?
-n) New remote
-s) Set configuration password
-q) Quit config
-n/s/q> n
-name> remote
-Option Storage.
-Type of storage to configure.
-Choose a number from below, or type in your own value.
-...
-XX / Combine several remotes into one
- \ (combine)
-...
-Storage> combine
-Option upstreams.
-Upstreams for combining
-These should be in the form
- dir=remote:path dir2=remote2:path
-Where before the = is specified the root directory and after is the remote to
-put there.
-Embedded spaces can be added using quotes
- "dir=remote:path with space" "dir2=remote2:path with space"
-Enter a fs.SpaceSepList value.
-upstreams> images=s3:imagesbucket files=drive:important/files
---------------------
-[remote]
-type = combine
-upstreams = images=s3:imagesbucket files=drive:important/files
---------------------
-y) Yes this is OK (default)
-e) Edit this remote
-d) Delete this remote
-y/e/d> y
-```
+Properties:
-### Configuring for Google Drive Shared Drives
+- Config: user
+- Env Var: RCLONE_FTP_USER
+- Type: string
+- Default: "$USER"
-Rclone has a convenience feature for making a combine backend for all
-the shared drives you have access to.
+#### --ftp-port
-Assuming your main (non shared drive) Google drive remote is called
-`drive:` you would run
+FTP port number.
- rclone backend -o config drives drive:
+Properties:
-This would produce something like this:
+- Config: port
+- Env Var: RCLONE_FTP_PORT
+- Type: int
+- Default: 21
- [My Drive]
- type = alias
- remote = drive,team_drive=0ABCDEF-01234567890,root_folder_id=:
+#### --ftp-pass
- [Test Drive]
- type = alias
- remote = drive,team_drive=0ABCDEFabcdefghijkl,root_folder_id=:
+FTP password.
- [AllDrives]
- type = combine
- upstreams = "My Drive=My Drive:" "Test Drive=Test Drive:"
+**NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/).
-If you then add that config to your config file (find it with `rclone
-config file`) then you can access all the shared drives in one place
-with the `AllDrives:` remote.
+Properties:
-See [the Google Drive docs](https://rclone.org/drive/#drives) for full info.
+- Config: pass
+- Env Var: RCLONE_FTP_PASS
+- Type: string
+- Required: false
+#### --ftp-tls
-### Standard options
+Use Implicit FTPS (FTP over TLS).
-Here are the Standard options specific to combine (Combine several remotes into one).
+When using implicit FTP over TLS the client connects using TLS
+right from the start which breaks compatibility with
+non-TLS-aware servers. This is usually served over port 990 rather
+than port 21. Cannot be used in combination with explicit FTPS.
-#### --combine-upstreams
+Properties:
-Upstreams for combining
+- Config: tls
+- Env Var: RCLONE_FTP_TLS
+- Type: bool
+- Default: false
-These should be in the form
+#### --ftp-explicit-tls
- dir=remote:path dir2=remote2:path
+Use Explicit FTPS (FTP over TLS).
-Where before the = is specified the root directory and after is the remote to
-put there.
+When using explicit FTP over TLS the client explicitly requests
+security from the server in order to upgrade a plain text connection
+to an encrypted one. Cannot be used in combination with implicit FTPS.
-Embedded spaces can be added using quotes
+Properties:
- "dir=remote:path with space" "dir2=remote2:path with space"
+- Config: explicit_tls
+- Env Var: RCLONE_FTP_EXPLICIT_TLS
+- Type: bool
+- Default: false
+### Advanced options
+Here are the Advanced options specific to ftp (FTP).
-Properties:
+#### --ftp-concurrency
-- Config: upstreams
-- Env Var: RCLONE_COMBINE_UPSTREAMS
-- Type: SpaceSepList
-- Default:
+Maximum number of FTP simultaneous connections, 0 for unlimited.
-### Metadata
+Note that setting this is very likely to cause deadlocks so it should
+be used with care.
-Any metadata supported by the underlying remote is read and written.
+If you are doing a sync or copy then make sure concurrency is one more
+than the sum of `--transfers` and `--checkers`.
-See the [metadata](https://rclone.org/docs/#metadata) docs for more info.
+If you use `--check-first` then it just needs to be one more than the
+maximum of `--checkers` and `--transfers`.
+So for `concurrency 3` you'd use `--checkers 2 --transfers 2
+--check-first` or `--checkers 1 --transfers 1`.
-# Dropbox
-Paths are specified as `remote:path`
+Properties:
-Dropbox paths may be as deep as required, e.g.
-`remote:directory/subdirectory`.
+- Config: concurrency
+- Env Var: RCLONE_FTP_CONCURRENCY
+- Type: int
+- Default: 0
-## Configuration
+#### --ftp-no-check-certificate
-The initial setup for dropbox involves getting a token from Dropbox
-which you need to do in your browser. `rclone config` walks you
-through it.
+Do not verify the TLS certificate of the server.
-Here is an example of how to make a remote called `remote`. First run:
+Properties:
- rclone config
+- Config: no_check_certificate
+- Env Var: RCLONE_FTP_NO_CHECK_CERTIFICATE
+- Type: bool
+- Default: false
-This will guide you through an interactive setup process:
+#### --ftp-disable-epsv
-```
-n) New remote
-d) Delete remote
-q) Quit config
-e/n/d/q> n
-name> remote
-Type of storage to configure.
-Choose a number from below, or type in your own value
-[snip]
-XX / Dropbox
- \ "dropbox"
-[snip]
-Storage> dropbox
-Dropbox App Key - leave blank normally.
-app_key>
-Dropbox App Secret - leave blank normally.
-app_secret>
-Remote config
-Please visit:
-https://www.dropbox.com/1/oauth2/authorize?client_id=XXXXXXXXXXXXXXX&response_type=code
-Enter the code: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX_XXXXXXXXXX
---------------------
-[remote]
-app_key =
-app_secret =
-token = XXXXXXXXXXXXXXXXXXXXXXXXXXXXX_XXXX_XXXXXXXXXXXXXXXXXXXXXXXXXXXXX
---------------------
-y) Yes this is OK
-e) Edit this remote
-d) Delete this remote
-y/e/d> y
-```
+Disable using EPSV even if server advertises support.
-See the [remote setup docs](https://rclone.org/remote_setup/) for how to set it up on a
-machine with no Internet browser available.
+Properties:
-Note that rclone runs a webserver on your local machine to collect the
-token as returned from Dropbox. This only
-runs from the moment it opens your browser to the moment you get back
-the verification code. This is on `http://127.0.0.1:53682/` and it
-may require you to unblock it temporarily if you are running a host
-firewall, or use manual mode.
+- Config: disable_epsv
+- Env Var: RCLONE_FTP_DISABLE_EPSV
+- Type: bool
+- Default: false
-You can then use it like this,
+#### --ftp-disable-mlsd
-List directories in top level of your dropbox
+Disable using MLSD even if server advertises support.
- rclone lsd remote:
+Properties:
-List all the files in your dropbox
+- Config: disable_mlsd
+- Env Var: RCLONE_FTP_DISABLE_MLSD
+- Type: bool
+- Default: false
- rclone ls remote:
+#### --ftp-disable-utf8
-To copy a local directory to a dropbox directory called backup
+Disable using UTF-8 even if server advertises support.
- rclone copy /home/source remote:backup
+Properties:
-### Dropbox for business
+- Config: disable_utf8
+- Env Var: RCLONE_FTP_DISABLE_UTF8
+- Type: bool
+- Default: false
-Rclone supports Dropbox for business and Team Folders.
+#### --ftp-writing-mdtm
-When using Dropbox for business `remote:` and `remote:path/to/file`
-will refer to your personal folder.
+Use MDTM to set modification time (VsFtpd quirk)
-If you wish to see Team Folders you must use a leading `/` in the
-path, so `rclone lsd remote:/` will refer to the root and show you all
-Team Folders and your User Folder.
+Properties:
-You can then use team folders like this `remote:/TeamFolder` and
-`remote:/TeamFolder/path/to/file`.
+- Config: writing_mdtm
+- Env Var: RCLONE_FTP_WRITING_MDTM
+- Type: bool
+- Default: false
-A leading `/` for a Dropbox personal account will do nothing, but it
-will take an extra HTTP transaction so it should be avoided.
+#### --ftp-force-list-hidden
-### Modified time and Hashes
+Use LIST -a to force listing of hidden files and folders. This will disable the use of MLSD.
-Dropbox supports modified times, but the only way to set a
-modification time is to re-upload the file.
+Properties:
-This means that if you uploaded your data with an older version of
-rclone which didn't support the v2 API and modified times, rclone will
-decide to upload all your old data to fix the modification times. If
-you don't want this to happen use `--size-only` or `--checksum` flag
-to stop it.
+- Config: force_list_hidden
+- Env Var: RCLONE_FTP_FORCE_LIST_HIDDEN
+- Type: bool
+- Default: false
-Dropbox supports [its own hash
-type](https://www.dropbox.com/developers/reference/content-hash) which
-is checked for all transfers.
+#### --ftp-idle-timeout
-### Restricted filename characters
+Max time before closing idle connections.
-| Character | Value | Replacement |
-| --------- |:-----:|:-----------:|
-| NUL | 0x00 | †|
-| / | 0x2F | ï¼ |
-| DEL | 0x7F | â¡ |
-| \ | 0x5C | ï¼¼ |
+If no connections have been returned to the connection pool in the time
+given, rclone will empty the connection pool.
-File names can also not end with the following characters.
-These only get replaced if they are the last character in the name:
+Set to 0 to keep connections indefinitely.
-| Character | Value | Replacement |
-| --------- |:-----:|:-----------:|
-| SP | 0x20 | â |
-Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8),
-as they can't be used in JSON strings.
+Properties:
-### Batch mode uploads {#batch-mode}
+- Config: idle_timeout
+- Env Var: RCLONE_FTP_IDLE_TIMEOUT
+- Type: Duration
+- Default: 1m0s
-Using batch mode uploads is very important for performance when using
-the Dropbox API. See [the dropbox performance guide](https://developers.dropbox.com/dbx-performance-guide)
-for more info.
+#### --ftp-close-timeout
-There are 3 modes rclone can use for uploads.
+Maximum time to wait for a response to close.
-#### --dropbox-batch-mode off
+Properties:
-In this mode rclone will not use upload batching. This was the default
-before rclone v1.55. It has the disadvantage that it is very likely to
-encounter `too_many_requests` errors like this
+- Config: close_timeout
+- Env Var: RCLONE_FTP_CLOSE_TIMEOUT
+- Type: Duration
+- Default: 1m0s
- NOTICE: too_many_requests/.: Too many requests or write operations. Trying again in 15 seconds.
+#### --ftp-tls-cache-size
-When rclone receives these it has to wait for 15s or sometimes 300s
-before continuing which really slows down transfers.
+Size of TLS session cache for all control and data connections.
-This will happen especially if `--transfers` is large, so this mode
-isn't recommended except for compatibility or investigating problems.
+TLS cache allows to resume TLS sessions and reuse PSK between connections.
+Increase if default size is not enough resulting in TLS resumption errors.
+Enabled by default. Use 0 to disable.
-#### --dropbox-batch-mode sync
+Properties:
-In this mode rclone will batch up uploads to the size specified by
-`--dropbox-batch-size` and commit them together.
+- Config: tls_cache_size
+- Env Var: RCLONE_FTP_TLS_CACHE_SIZE
+- Type: int
+- Default: 32
-Using this mode means you can use a much higher `--transfers`
-parameter (32 or 64 works fine) without receiving `too_many_requests`
-errors.
+#### --ftp-disable-tls13
-This mode ensures full data integrity.
+Disable TLS 1.3 (workaround for FTP servers with buggy TLS)
-Note that there may be a pause when quitting rclone while rclone
-finishes up the last batch using this mode.
+Properties:
-#### --dropbox-batch-mode async
+- Config: disable_tls13
+- Env Var: RCLONE_FTP_DISABLE_TLS13
+- Type: bool
+- Default: false
-In this mode rclone will batch up uploads to the size specified by
-`--dropbox-batch-size` and commit them together.
+#### --ftp-shut-timeout
-However it will not wait for the status of the batch to be returned to
-the caller. This means rclone can use a much bigger batch size (much
-bigger than `--transfers`), at the cost of not being able to check the
-status of the upload.
+Maximum time to wait for data connection closing status.
-This provides the maximum possible upload speed especially with lots
-of small files, however rclone can't check the file got uploaded
-properly using this mode.
+Properties:
-If you are using this mode then using "rclone check" after the
-transfer completes is recommended. Or you could do an initial transfer
-with `--dropbox-batch-mode async` then do a final transfer with
-`--dropbox-batch-mode sync` (the default).
+- Config: shut_timeout
+- Env Var: RCLONE_FTP_SHUT_TIMEOUT
+- Type: Duration
+- Default: 1m0s
-Note that there may be a pause when quitting rclone while rclone
-finishes up the last batch using this mode.
+#### --ftp-ask-password
+Allow asking for FTP password when needed.
+If this is set and no password is supplied then rclone will ask for a password
-### Standard options
-Here are the Standard options specific to dropbox (Dropbox).
+Properties:
-#### --dropbox-client-id
+- Config: ask_password
+- Env Var: RCLONE_FTP_ASK_PASSWORD
+- Type: bool
+- Default: false
-OAuth Client Id.
+#### --ftp-socks-proxy
-Leave blank normally.
+Socks 5 proxy host.
+
+ Supports the format user:pass@host:port, user@host:port, host:port.
+
+ Example:
+
+ myUser:myPass@localhost:9005
+
Properties:
-- Config: client_id
-- Env Var: RCLONE_DROPBOX_CLIENT_ID
+- Config: socks_proxy
+- Env Var: RCLONE_FTP_SOCKS_PROXY
- Type: string
- Required: false
-#### --dropbox-client-secret
+#### --ftp-encoding
-OAuth Client Secret.
+The encoding for the backend.
-Leave blank normally.
+See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info.
Properties:
-- Config: client_secret
-- Env Var: RCLONE_DROPBOX_CLIENT_SECRET
-- Type: string
-- Required: false
-
-### Advanced options
-
-Here are the Advanced options specific to dropbox (Dropbox).
+- Config: encoding
+- Env Var: RCLONE_FTP_ENCODING
+- Type: Encoding
+- Default: Slash,Del,Ctl,RightSpace,Dot
+- Examples:
+ - "Asterisk,Ctl,Dot,Slash"
+ - ProFTPd can't handle '*' in file names
+ - "BackSlash,Ctl,Del,Dot,RightSpace,Slash,SquareBracket"
+ - PureFTPd can't handle '[]' or '*' in file names
+ - "Ctl,LeftPeriod,Slash"
+ - VsFTPd can't handle file names starting with dot
-#### --dropbox-token
-OAuth Access Token as a JSON blob.
-Properties:
+## Limitations
-- Config: token
-- Env Var: RCLONE_DROPBOX_TOKEN
-- Type: string
-- Required: false
+FTP servers acting as rclone remotes must support `passive` mode.
+The mode cannot be configured as `passive` is the only supported one.
+Rclone's FTP implementation is not compatible with `active` mode
+as [the library it uses doesn't support it](https://github.com/jlaffaye/ftp/issues/29).
+This will likely never be supported due to security concerns.
-#### --dropbox-auth-url
+Rclone's FTP backend does not support any checksums but can compare
+file sizes.
-Auth server URL.
+`rclone about` is not supported by the FTP backend. Backends without
+this capability cannot determine free space for an rclone mount or
+use policy `mfs` (most free space) as a member of an rclone union
+remote.
-Leave blank to use the provider defaults.
+See [List of backends that do not support rclone about](https://rclone.org/overview/#optional-features) and [rclone about](https://rclone.org/commands/rclone_about/)
-Properties:
+The implementation of : `--dump headers`,
+`--dump bodies`, `--dump auth` for debugging isn't the same as
+for rclone HTTP based backends - it has less fine grained control.
-- Config: auth_url
-- Env Var: RCLONE_DROPBOX_AUTH_URL
-- Type: string
-- Required: false
+`--timeout` isn't supported (but `--contimeout` is).
-#### --dropbox-token-url
+`--bind` isn't supported.
-Token server url.
+Rclone's FTP backend could support server-side move but does not
+at present.
-Leave blank to use the provider defaults.
+The `ftp_proxy` environment variable is not currently supported.
-Properties:
+### Modification times
-- Config: token_url
-- Env Var: RCLONE_DROPBOX_TOKEN_URL
-- Type: string
-- Required: false
+File modification time (timestamps) is supported to 1 second resolution
+for major FTP servers: ProFTPd, PureFTPd, VsFTPd, and FileZilla FTP server.
+The `VsFTPd` server has non-standard implementation of time related protocol
+commands and needs a special configuration setting: `writing_mdtm = true`.
-#### --dropbox-chunk-size
+Support for precise file time with other FTP servers varies depending on what
+protocol extensions they advertise. If all the `MLSD`, `MDTM` and `MFTM`
+extensions are present, rclone will use them together to provide precise time.
+Otherwise the times you see on the FTP server through rclone are those of the
+last file upload.
-Upload chunk size (< 150Mi).
+You can use the following command to check whether rclone can use precise time
+with your FTP server: `rclone backend features your_ftp_remote:` (the trailing
+colon is important). Look for the number in the line tagged by `Precision`
+designating the remote time precision expressed as nanoseconds. A value of
+`1000000000` means that file time precision of 1 second is available.
+A value of `3153600000000000000` (or another large number) means "unsupported".
-Any files larger than this will be uploaded in chunks of this size.
+# Google Cloud Storage
-Note that chunks are buffered in memory (one at a time) so rclone can
-deal with retries. Setting this larger will increase the speed
-slightly (at most 10% for 128 MiB in tests) at the cost of using more
-memory. It can be set smaller if you are tight on memory.
+Paths are specified as `remote:bucket` (or `remote:` for the `lsd`
+command.) You may put subdirectories in too, e.g. `remote:bucket/path/to/dir`.
-Properties:
+## Configuration
-- Config: chunk_size
-- Env Var: RCLONE_DROPBOX_CHUNK_SIZE
-- Type: SizeSuffix
-- Default: 48Mi
+The initial setup for google cloud storage involves getting a token from Google Cloud Storage
+which you need to do in your browser. `rclone config` walks you
+through it.
-#### --dropbox-impersonate
+Here is an example of how to make a remote called `remote`. First run:
-Impersonate this user when using a business account.
+ rclone config
-Note that if you want to use impersonate, you should make sure this
-flag is set when running "rclone config" as this will cause rclone to
-request the "members.read" scope which it won't normally. This is
-needed to lookup a members email address into the internal ID that
-dropbox uses in the API.
+This will guide you through an interactive setup process:
-Using the "members.read" scope will require a Dropbox Team Admin
-to approve during the OAuth flow.
+```
+n) New remote
+d) Delete remote
+q) Quit config
+e/n/d/q> n
+name> remote
+Type of storage to configure.
+Choose a number from below, or type in your own value
+[snip]
+XX / Google Cloud Storage (this is not Google Drive)
+ \ "google cloud storage"
+[snip]
+Storage> google cloud storage
+Google Application Client Id - leave blank normally.
+client_id>
+Google Application Client Secret - leave blank normally.
+client_secret>
+Project number optional - needed only for list/create/delete buckets - see your developer console.
+project_number> 12345678
+Service Account Credentials JSON file path - needed only if you want use SA instead of interactive login.
+service_account_file>
+Access Control List for new objects.
+Choose a number from below, or type in your own value
+ 1 / Object owner gets OWNER access, and all Authenticated Users get READER access.
+ \ "authenticatedRead"
+ 2 / Object owner gets OWNER access, and project team owners get OWNER access.
+ \ "bucketOwnerFullControl"
+ 3 / Object owner gets OWNER access, and project team owners get READER access.
+ \ "bucketOwnerRead"
+ 4 / Object owner gets OWNER access [default if left blank].
+ \ "private"
+ 5 / Object owner gets OWNER access, and project team members get access according to their roles.
+ \ "projectPrivate"
+ 6 / Object owner gets OWNER access, and all Users get READER access.
+ \ "publicRead"
+object_acl> 4
+Access Control List for new buckets.
+Choose a number from below, or type in your own value
+ 1 / Project team owners get OWNER access, and all Authenticated Users get READER access.
+ \ "authenticatedRead"
+ 2 / Project team owners get OWNER access [default if left blank].
+ \ "private"
+ 3 / Project team members get access according to their roles.
+ \ "projectPrivate"
+ 4 / Project team owners get OWNER access, and all Users get READER access.
+ \ "publicRead"
+ 5 / Project team owners get OWNER access, and all Users get WRITER access.
+ \ "publicReadWrite"
+bucket_acl> 2
+Location for the newly created buckets.
+Choose a number from below, or type in your own value
+ 1 / Empty for default location (US).
+ \ ""
+ 2 / Multi-regional location for Asia.
+ \ "asia"
+ 3 / Multi-regional location for Europe.
+ \ "eu"
+ 4 / Multi-regional location for United States.
+ \ "us"
+ 5 / Taiwan.
+ \ "asia-east1"
+ 6 / Tokyo.
+ \ "asia-northeast1"
+ 7 / Singapore.
+ \ "asia-southeast1"
+ 8 / Sydney.
+ \ "australia-southeast1"
+ 9 / Belgium.
+ \ "europe-west1"
+10 / London.
+ \ "europe-west2"
+11 / Iowa.
+ \ "us-central1"
+12 / South Carolina.
+ \ "us-east1"
+13 / Northern Virginia.
+ \ "us-east4"
+14 / Oregon.
+ \ "us-west1"
+location> 12
+The storage class to use when storing objects in Google Cloud Storage.
+Choose a number from below, or type in your own value
+ 1 / Default
+ \ ""
+ 2 / Multi-regional storage class
+ \ "MULTI_REGIONAL"
+ 3 / Regional storage class
+ \ "REGIONAL"
+ 4 / Nearline storage class
+ \ "NEARLINE"
+ 5 / Coldline storage class
+ \ "COLDLINE"
+ 6 / Durable reduced availability storage class
+ \ "DURABLE_REDUCED_AVAILABILITY"
+storage_class> 5
+Remote config
+Use web browser to automatically authenticate rclone with remote?
+ * Say Y if the machine running rclone has a web browser you can use
+ * Say N if running rclone on a (remote) machine without web browser access
+If not sure try Y. If Y failed, try N.
+y) Yes
+n) No
+y/n> y
+If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth
+Log in and authorize rclone for access
+Waiting for code...
+Got code
+--------------------
+[remote]
+type = google cloud storage
+client_id =
+client_secret =
+token = {"AccessToken":"xxxx.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","RefreshToken":"x/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx_xxxxxxxxx","Expiry":"2014-07-17T20:49:14.929208288+01:00","Extra":null}
+project_number = 12345678
+object_acl = private
+bucket_acl = private
+--------------------
+y) Yes this is OK
+e) Edit this remote
+d) Delete this remote
+y/e/d> y
+```
-You will have to use your own App (setting your own client_id and
-client_secret) to use this option as currently rclone's default set of
-permissions doesn't include "members.read". This can be added once
-v1.55 or later is in use everywhere.
+See the [remote setup docs](https://rclone.org/remote_setup/) for how to set it up on a
+machine with no Internet browser available.
+Note that rclone runs a webserver on your local machine to collect the
+token as returned from Google if using web browser to automatically
+authenticate. This only
+runs from the moment it opens your browser to the moment you get back
+the verification code. This is on `http://127.0.0.1:53682/` and this
+it may require you to unblock it temporarily if you are running a host
+firewall, or use manual mode.
-Properties:
+This remote is called `remote` and can now be used like this
-- Config: impersonate
-- Env Var: RCLONE_DROPBOX_IMPERSONATE
-- Type: string
-- Required: false
+See all the buckets in your project
-#### --dropbox-shared-files
+ rclone lsd remote:
-Instructs rclone to work on individual shared files.
+Make a new bucket
-In this mode rclone's features are extremely limited - only list (ls, lsl, etc.)
-operations and read operations (e.g. downloading) are supported in this mode.
-All other operations will be disabled.
+ rclone mkdir remote:bucket
-Properties:
+List the contents of a bucket
-- Config: shared_files
-- Env Var: RCLONE_DROPBOX_SHARED_FILES
-- Type: bool
-- Default: false
+ rclone ls remote:bucket
-#### --dropbox-shared-folders
+Sync `/home/local/directory` to the remote bucket, deleting any excess
+files in the bucket.
-Instructs rclone to work on shared folders.
-
-When this flag is used with no path only the List operation is supported and
-all available shared folders will be listed. If you specify a path the first part
-will be interpreted as the name of shared folder. Rclone will then try to mount this
-shared to the root namespace. On success shared folder rclone proceeds normally.
-The shared folder is now pretty much a normal folder and all normal operations
-are supported.
+ rclone sync --interactive /home/local/directory remote:bucket
-Note that we don't unmount the shared folder afterwards so the
---dropbox-shared-folders can be omitted after the first use of a particular
-shared folder.
+### Service Account support
-Properties:
+You can set up rclone with Google Cloud Storage in an unattended mode,
+i.e. not tied to a specific end-user Google account. This is useful
+when you want to synchronise files onto machines that don't have
+actively logged-in users, for example build machines.
-- Config: shared_folders
-- Env Var: RCLONE_DROPBOX_SHARED_FOLDERS
-- Type: bool
-- Default: false
+To get credentials for Google Cloud Platform
+[IAM Service Accounts](https://cloud.google.com/iam/docs/service-accounts),
+please head to the
+[Service Account](https://console.cloud.google.com/permissions/serviceaccounts)
+section of the Google Developer Console. Service Accounts behave just
+like normal `User` permissions in
+[Google Cloud Storage ACLs](https://cloud.google.com/storage/docs/access-control),
+so you can limit their access (e.g. make them read only). After
+creating an account, a JSON file containing the Service Account's
+credentials will be downloaded onto your machines. These credentials
+are what rclone will use for authentication.
-#### --dropbox-batch-mode
+To use a Service Account instead of OAuth2 token flow, enter the path
+to your Service Account credentials at the `service_account_file`
+prompt and rclone won't use the browser based authentication
+flow. If you'd rather stuff the contents of the credentials file into
+the rclone config file, you can set `service_account_credentials` with
+the actual contents of the file instead, or set the equivalent
+environment variable.
-Upload file batching sync|async|off.
+### Anonymous Access
-This sets the batch mode used by rclone.
+For downloads of objects that permit public access you can configure rclone
+to use anonymous access by setting `anonymous` to `true`.
+With unauthorized access you can't write or create files but only read or list
+those buckets and objects that have public read access.
-For full info see [the main docs](https://rclone.org/dropbox/#batch-mode)
+### Application Default Credentials
-This has 3 possible values
+If no other source of credentials is provided, rclone will fall back
+to
+[Application Default Credentials](https://cloud.google.com/video-intelligence/docs/common/auth#authenticating_with_application_default_credentials)
+this is useful both when you already have configured authentication
+for your developer account, or in production when running on a google
+compute host. Note that if running in docker, you may need to run
+additional commands on your google compute machine -
+[see this page](https://cloud.google.com/container-registry/docs/advanced-authentication#gcloud_as_a_docker_credential_helper).
-- off - no batching
-- sync - batch uploads and check completion (default)
-- async - batch upload and don't check completion
+Note that in the case application default credentials are used, there
+is no need to explicitly configure a project number.
-Rclone will close any outstanding batches when it exits which may make
-a delay on quit.
+### --fast-list
+This remote supports `--fast-list` which allows you to use fewer
+transactions in exchange for more memory. See the [rclone
+docs](https://rclone.org/docs/#fast-list) for more details.
-Properties:
+### Custom upload headers
-- Config: batch_mode
-- Env Var: RCLONE_DROPBOX_BATCH_MODE
-- Type: string
-- Default: "sync"
+You can set custom upload headers with the `--header-upload`
+flag. Google Cloud Storage supports the headers as described in the
+[working with metadata documentation](https://cloud.google.com/storage/docs/gsutil/addlhelp/WorkingWithObjectMetadata)
-#### --dropbox-batch-size
+- Cache-Control
+- Content-Disposition
+- Content-Encoding
+- Content-Language
+- Content-Type
+- X-Goog-Storage-Class
+- X-Goog-Meta-
-Max number of files in upload batch.
+Eg `--header-upload "Content-Type text/potato"`
-This sets the batch size of files to upload. It has to be less than 1000.
+Note that the last of these is for setting custom metadata in the form
+`--header-upload "x-goog-meta-key: value"`
-By default this is 0 which means rclone which calculate the batch size
-depending on the setting of batch_mode.
+### Modification times
-- batch_mode: async - default batch_size is 100
-- batch_mode: sync - default batch_size is the same as --transfers
-- batch_mode: off - not in use
+Google Cloud Storage stores md5sum natively.
+Google's [gsutil](https://cloud.google.com/storage/docs/gsutil) tool stores modification time
+with one-second precision as `goog-reserved-file-mtime` in file metadata.
-Rclone will close any outstanding batches when it exits which may make
-a delay on quit.
+To ensure compatibility with gsutil, rclone stores modification time in 2 separate metadata entries.
+`mtime` uses RFC3339 format with one-nanosecond precision.
+`goog-reserved-file-mtime` uses Unix timestamp format with one-second precision.
+To get modification time from object metadata, rclone reads the metadata in the following order: `mtime`, `goog-reserved-file-mtime`, object updated time.
-Setting this is a great idea if you are uploading lots of small files
-as it will make them a lot quicker. You can use --transfers 32 to
-maximise throughput.
+Note that rclone's default modify window is 1ns.
+Files uploaded by gsutil only contain timestamps with one-second precision.
+If you use rclone to sync files previously uploaded by gsutil,
+rclone will attempt to update modification time for all these files.
+To avoid these possibly unnecessary updates, use `--modify-window 1s`.
+### Restricted filename characters
-Properties:
+| Character | Value | Replacement |
+| --------- |:-----:|:-----------:|
+| NUL | 0x00 | †|
+| LF | 0x0A | ⊠|
+| CR | 0x0D | â |
+| / | 0x2F | ï¼ |
-- Config: batch_size
-- Env Var: RCLONE_DROPBOX_BATCH_SIZE
-- Type: int
-- Default: 0
+Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8),
+as they can't be used in JSON strings.
-#### --dropbox-batch-timeout
-Max time to allow an idle upload batch before uploading.
+### Standard options
-If an upload batch is idle for more than this long then it will be
-uploaded.
+Here are the Standard options specific to google cloud storage (Google Cloud Storage (this is not Google Drive)).
-The default for this is 0 which means rclone will choose a sensible
-default based on the batch_mode in use.
+#### --gcs-client-id
-- batch_mode: async - default batch_timeout is 10s
-- batch_mode: sync - default batch_timeout is 500ms
-- batch_mode: off - not in use
+OAuth Client Id.
+Leave blank normally.
Properties:
-- Config: batch_timeout
-- Env Var: RCLONE_DROPBOX_BATCH_TIMEOUT
-- Type: Duration
-- Default: 0s
-
-#### --dropbox-batch-commit-timeout
-
-Max time to wait for a batch to finish committing
-
-Properties:
+- Config: client_id
+- Env Var: RCLONE_GCS_CLIENT_ID
+- Type: string
+- Required: false
-- Config: batch_commit_timeout
-- Env Var: RCLONE_DROPBOX_BATCH_COMMIT_TIMEOUT
-- Type: Duration
-- Default: 10m0s
+#### --gcs-client-secret
-#### --dropbox-pacer-min-sleep
+OAuth Client Secret.
-Minimum time to sleep between API calls.
+Leave blank normally.
Properties:
-- Config: pacer_min_sleep
-- Env Var: RCLONE_DROPBOX_PACER_MIN_SLEEP
-- Type: Duration
-- Default: 10ms
+- Config: client_secret
+- Env Var: RCLONE_GCS_CLIENT_SECRET
+- Type: string
+- Required: false
-#### --dropbox-encoding
+#### --gcs-project-number
-The encoding for the backend.
+Project number.
-See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info.
+Optional - needed only for list/create/delete buckets - see your developer console.
Properties:
-- Config: encoding
-- Env Var: RCLONE_DROPBOX_ENCODING
-- Type: MultiEncoder
-- Default: Slash,BackSlash,Del,RightSpace,InvalidUtf8,Dot
-
+- Config: project_number
+- Env Var: RCLONE_GCS_PROJECT_NUMBER
+- Type: string
+- Required: false
+#### --gcs-user-project
-## Limitations
+User project.
-Note that Dropbox is case insensitive so you can't have a file called
-"Hello.doc" and one called "hello.doc".
+Optional - needed only for requester pays.
-There are some file names such as `thumbs.db` which Dropbox can't
-store. There is a full list of them in the ["Ignored Files" section
-of this document](https://www.dropbox.com/en/help/145). Rclone will
-issue an error message `File name disallowed - not uploading` if it
-attempts to upload one of those file names, but the sync won't fail.
+Properties:
-Some errors may occur if you try to sync copyright-protected files
-because Dropbox has its own [copyright detector](https://techcrunch.com/2014/03/30/how-dropbox-knows-when-youre-sharing-copyrighted-stuff-without-actually-looking-at-your-stuff/) that
-prevents this sort of file being downloaded. This will return the error `ERROR :
-/path/to/your/file: Failed to copy: failed to open source object:
-path/restricted_content/.`
+- Config: user_project
+- Env Var: RCLONE_GCS_USER_PROJECT
+- Type: string
+- Required: false
-If you have more than 10,000 files in a directory then `rclone purge
-dropbox:dir` will return the error `Failed to purge: There are too
-many files involved in this operation`. As a work-around do an
-`rclone delete dropbox:dir` followed by an `rclone rmdir dropbox:dir`.
+#### --gcs-service-account-file
-When using `rclone link` you'll need to set `--expire` if using a
-non-personal account otherwise the visibility may not be correct.
-(Note that `--expire` isn't supported on personal accounts). See the
-[forum discussion](https://forum.rclone.org/t/rclone-link-dropbox-permissions/23211) and the
-[dropbox SDK issue](https://github.com/dropbox/dropbox-sdk-go-unofficial/issues/75).
+Service Account Credentials JSON file path.
-## Get your own Dropbox App ID
+Leave blank normally.
+Needed only if you want use SA instead of interactive login.
-When you use rclone with Dropbox in its default configuration you are using rclone's App ID. This is shared between all the rclone users.
+Leading `~` will be expanded in the file name as will environment variables such as `${RCLONE_CONFIG_DIR}`.
-Here is how to create your own Dropbox App ID for rclone:
+Properties:
-1. Log into the [Dropbox App console](https://www.dropbox.com/developers/apps/create) with your Dropbox Account (It need not
-to be the same account as the Dropbox you want to access)
+- Config: service_account_file
+- Env Var: RCLONE_GCS_SERVICE_ACCOUNT_FILE
+- Type: string
+- Required: false
-2. Choose an API => Usually this should be `Dropbox API`
+#### --gcs-service-account-credentials
-3. Choose the type of access you want to use => `Full Dropbox` or `App Folder`. If you want to use Team Folders, `Full Dropbox` is required ([see here](https://www.dropboxforum.com/t5/Dropbox-API-Support-Feedback/How-to-create-team-folder-inside-my-app-s-folder/m-p/601005/highlight/true#M27911)).
+Service Account Credentials JSON blob.
-4. Name your App. The app name is global, so you can't use `rclone` for example
+Leave blank normally.
+Needed only if you want use SA instead of interactive login.
-5. Click the button `Create App`
+Properties:
-6. Switch to the `Permissions` tab. Enable at least the following permissions: `account_info.read`, `files.metadata.write`, `files.content.write`, `files.content.read`, `sharing.write`. The `files.metadata.read` and `sharing.read` checkboxes will be marked too. Click `Submit`
+- Config: service_account_credentials
+- Env Var: RCLONE_GCS_SERVICE_ACCOUNT_CREDENTIALS
+- Type: string
+- Required: false
-7. Switch to the `Settings` tab. Fill `OAuth2 - Redirect URIs` as `http://localhost:53682/` and click on `Add`
+#### --gcs-anonymous
-8. Find the `App key` and `App secret` values on the `Settings` tab. Use these values in rclone config to add a new remote or edit an existing remote. The `App key` setting corresponds to `client_id` in rclone config, the `App secret` corresponds to `client_secret`
+Access public buckets and objects without credentials.
-# Enterprise File Fabric
+Set to 'true' if you just want to download files and don't configure credentials.
-This backend supports [Storage Made Easy's Enterprise File
-Fabricâ„¢](https://storagemadeeasy.com/about/) which provides a software
-solution to integrate and unify File and Object Storage accessible
-through a global file system.
+Properties:
-## Configuration
+- Config: anonymous
+- Env Var: RCLONE_GCS_ANONYMOUS
+- Type: bool
+- Default: false
-The initial setup for the Enterprise File Fabric backend involves
-getting a token from the Enterprise File Fabric which you need to
-do in your browser. `rclone config` walks you through it.
+#### --gcs-object-acl
-Here is an example of how to make a remote called `remote`. First run:
+Access Control List for new objects.
- rclone config
+Properties:
-This will guide you through an interactive setup process:
+- Config: object_acl
+- Env Var: RCLONE_GCS_OBJECT_ACL
+- Type: string
+- Required: false
+- Examples:
+ - "authenticatedRead"
+ - Object owner gets OWNER access.
+ - All Authenticated Users get READER access.
+ - "bucketOwnerFullControl"
+ - Object owner gets OWNER access.
+ - Project team owners get OWNER access.
+ - "bucketOwnerRead"
+ - Object owner gets OWNER access.
+ - Project team owners get READER access.
+ - "private"
+ - Object owner gets OWNER access.
+ - Default if left blank.
+ - "projectPrivate"
+ - Object owner gets OWNER access.
+ - Project team members get access according to their roles.
+ - "publicRead"
+ - Object owner gets OWNER access.
+ - All Users get READER access.
-```
-No remotes found, make a new one?
-n) New remote
-s) Set configuration password
-q) Quit config
-n/s/q> n
-name> remote
-Type of storage to configure.
-Enter a string value. Press Enter for the default ("").
-Choose a number from below, or type in your own value
-[snip]
-XX / Enterprise File Fabric
- \ "filefabric"
-[snip]
-Storage> filefabric
-** See help for filefabric backend at: https://rclone.org/filefabric/ **
+#### --gcs-bucket-acl
-URL of the Enterprise File Fabric to connect to
-Enter a string value. Press Enter for the default ("").
-Choose a number from below, or type in your own value
- 1 / Storage Made Easy US
- \ "https://storagemadeeasy.com"
- 2 / Storage Made Easy EU
- \ "https://eu.storagemadeeasy.com"
- 3 / Connect to your Enterprise File Fabric
- \ "https://yourfabric.smestorage.com"
-url> https://yourfabric.smestorage.com/
-ID of the root folder
-Leave blank normally.
+Access Control List for new buckets.
-Fill in to make rclone start with directory of a given ID.
+Properties:
-Enter a string value. Press Enter for the default ("").
-root_folder_id>
-Permanent Authentication Token
+- Config: bucket_acl
+- Env Var: RCLONE_GCS_BUCKET_ACL
+- Type: string
+- Required: false
+- Examples:
+ - "authenticatedRead"
+ - Project team owners get OWNER access.
+ - All Authenticated Users get READER access.
+ - "private"
+ - Project team owners get OWNER access.
+ - Default if left blank.
+ - "projectPrivate"
+ - Project team members get access according to their roles.
+ - "publicRead"
+ - Project team owners get OWNER access.
+ - All Users get READER access.
+ - "publicReadWrite"
+ - Project team owners get OWNER access.
+ - All Users get WRITER access.
-A Permanent Authentication Token can be created in the Enterprise File
-Fabric, on the users Dashboard under Security, there is an entry
-you'll see called "My Authentication Tokens". Click the Manage button
-to create one.
+#### --gcs-bucket-policy-only
-These tokens are normally valid for several years.
+Access checks should use bucket-level IAM policies.
-For more info see: https://docs.storagemadeeasy.com/organisationcloud/api-tokens
+If you want to upload objects to a bucket with Bucket Policy Only set
+then you will need to set this.
-Enter a string value. Press Enter for the default ("").
-permanent_token> xxxxxxxxxxxxxxx-xxxxxxxxxxxxxxxx
-Edit advanced config? (y/n)
-y) Yes
-n) No (default)
-y/n> n
-Remote config
---------------------
-[remote]
-type = filefabric
-url = https://yourfabric.smestorage.com/
-permanent_token = xxxxxxxxxxxxxxx-xxxxxxxxxxxxxxxx
---------------------
-y) Yes this is OK (default)
-e) Edit this remote
-d) Delete this remote
-y/e/d> y
-```
+When it is set, rclone:
-Once configured you can then use `rclone` like this,
+- ignores ACLs set on buckets
+- ignores ACLs set on objects
+- creates buckets with Bucket Policy Only set
-List directories in top level of your Enterprise File Fabric
+Docs: https://cloud.google.com/storage/docs/bucket-policy-only
- rclone lsd remote:
-List all the files in your Enterprise File Fabric
+Properties:
- rclone ls remote:
+- Config: bucket_policy_only
+- Env Var: RCLONE_GCS_BUCKET_POLICY_ONLY
+- Type: bool
+- Default: false
-To copy a local directory to an Enterprise File Fabric directory called backup
+#### --gcs-location
- rclone copy /home/source remote:backup
+Location for the newly created buckets.
-### Modified time and hashes
+Properties:
-The Enterprise File Fabric allows modification times to be set on
-files accurate to 1 second. These will be used to detect whether
-objects need syncing or not.
+- Config: location
+- Env Var: RCLONE_GCS_LOCATION
+- Type: string
+- Required: false
+- Examples:
+ - ""
+ - Empty for default location (US)
+ - "asia"
+ - Multi-regional location for Asia
+ - "eu"
+ - Multi-regional location for Europe
+ - "us"
+ - Multi-regional location for United States
+ - "asia-east1"
+ - Taiwan
+ - "asia-east2"
+ - Hong Kong
+ - "asia-northeast1"
+ - Tokyo
+ - "asia-northeast2"
+ - Osaka
+ - "asia-northeast3"
+ - Seoul
+ - "asia-south1"
+ - Mumbai
+ - "asia-south2"
+ - Delhi
+ - "asia-southeast1"
+ - Singapore
+ - "asia-southeast2"
+ - Jakarta
+ - "australia-southeast1"
+ - Sydney
+ - "australia-southeast2"
+ - Melbourne
+ - "europe-north1"
+ - Finland
+ - "europe-west1"
+ - Belgium
+ - "europe-west2"
+ - London
+ - "europe-west3"
+ - Frankfurt
+ - "europe-west4"
+ - Netherlands
+ - "europe-west6"
+ - Zürich
+ - "europe-central2"
+ - Warsaw
+ - "us-central1"
+ - Iowa
+ - "us-east1"
+ - South Carolina
+ - "us-east4"
+ - Northern Virginia
+ - "us-west1"
+ - Oregon
+ - "us-west2"
+ - California
+ - "us-west3"
+ - Salt Lake City
+ - "us-west4"
+ - Las Vegas
+ - "northamerica-northeast1"
+ - Montréal
+ - "northamerica-northeast2"
+ - Toronto
+ - "southamerica-east1"
+ - São Paulo
+ - "southamerica-west1"
+ - Santiago
+ - "asia1"
+ - Dual region: asia-northeast1 and asia-northeast2.
+ - "eur4"
+ - Dual region: europe-north1 and europe-west4.
+ - "nam4"
+ - Dual region: us-central1 and us-east1.
-The Enterprise File Fabric does not support any data hashes at this time.
+#### --gcs-storage-class
-### Restricted filename characters
+The storage class to use when storing objects in Google Cloud Storage.
-The [default restricted characters set](https://rclone.org/overview/#restricted-characters)
-will be replaced.
+Properties:
-Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8),
-as they can't be used in JSON strings.
+- Config: storage_class
+- Env Var: RCLONE_GCS_STORAGE_CLASS
+- Type: string
+- Required: false
+- Examples:
+ - ""
+ - Default
+ - "MULTI_REGIONAL"
+ - Multi-regional storage class
+ - "REGIONAL"
+ - Regional storage class
+ - "NEARLINE"
+ - Nearline storage class
+ - "COLDLINE"
+ - Coldline storage class
+ - "ARCHIVE"
+ - Archive storage class
+ - "DURABLE_REDUCED_AVAILABILITY"
+ - Durable reduced availability storage class
-### Empty files
+#### --gcs-env-auth
-Empty files aren't supported by the Enterprise File Fabric. Rclone will therefore
-upload an empty file as a single space with a mime type of
-`application/vnd.rclone.empty.file` and files with that mime type are
-treated as empty.
+Get GCP IAM credentials from runtime (environment variables or instance meta data if no env vars).
-### Root folder ID ###
+Only applies if service_account_file and service_account_credentials is blank.
-You can set the `root_folder_id` for rclone. This is the directory
-(identified by its `Folder ID`) that rclone considers to be the root
-of your Enterprise File Fabric.
+Properties:
-Normally you will leave this blank and rclone will determine the
-correct root to use itself.
+- Config: env_auth
+- Env Var: RCLONE_GCS_ENV_AUTH
+- Type: bool
+- Default: false
+- Examples:
+ - "false"
+ - Enter credentials in the next step.
+ - "true"
+ - Get GCP IAM credentials from the environment (env vars or IAM).
-However you can set this to restrict rclone to a specific folder
-hierarchy.
+### Advanced options
-In order to do this you will have to find the `Folder ID` of the
-directory you wish rclone to display. These aren't displayed in the
-web interface, but you can use `rclone lsf` to find them, for example
+Here are the Advanced options specific to google cloud storage (Google Cloud Storage (this is not Google Drive)).
-```
-$ rclone lsf --dirs-only -Fip --csv filefabric:
-120673758,Burnt PDFs/
-120673759,My Quick Uploads/
-120673755,My Syncs/
-120673756,My backups/
-120673757,My contacts/
-120673761,S3 Storage/
-```
+#### --gcs-token
-The ID for "S3 Storage" would be `120673761`.
+OAuth Access Token as a JSON blob.
+Properties:
-### Standard options
+- Config: token
+- Env Var: RCLONE_GCS_TOKEN
+- Type: string
+- Required: false
-Here are the Standard options specific to filefabric (Enterprise File Fabric).
+#### --gcs-auth-url
-#### --filefabric-url
+Auth server URL.
-URL of the Enterprise File Fabric to connect to.
+Leave blank to use the provider defaults.
Properties:
-- Config: url
-- Env Var: RCLONE_FILEFABRIC_URL
+- Config: auth_url
+- Env Var: RCLONE_GCS_AUTH_URL
- Type: string
-- Required: true
-- Examples:
- - "https://storagemadeeasy.com"
- - Storage Made Easy US
- - "https://eu.storagemadeeasy.com"
- - Storage Made Easy EU
- - "https://yourfabric.smestorage.com"
- - Connect to your Enterprise File Fabric
-
-#### --filefabric-root-folder-id
-
-ID of the root folder.
+- Required: false
-Leave blank normally.
+#### --gcs-token-url
-Fill in to make rclone start with directory of a given ID.
+Token server url.
+Leave blank to use the provider defaults.
Properties:
-- Config: root_folder_id
-- Env Var: RCLONE_FILEFABRIC_ROOT_FOLDER_ID
+- Config: token_url
+- Env Var: RCLONE_GCS_TOKEN_URL
- Type: string
- Required: false
-#### --filefabric-permanent-token
-
-Permanent Authentication Token.
-
-A Permanent Authentication Token can be created in the Enterprise File
-Fabric, on the users Dashboard under Security, there is an entry
-you'll see called "My Authentication Tokens". Click the Manage button
-to create one.
+#### --gcs-directory-markers
-These tokens are normally valid for several years.
+Upload an empty object with a trailing slash when a new directory is created
-For more info see: https://docs.storagemadeeasy.com/organisationcloud/api-tokens
+Empty folders are unsupported for bucket based remotes, this option creates an empty
+object ending with "/", to persist the folder.
Properties:
-- Config: permanent_token
-- Env Var: RCLONE_FILEFABRIC_PERMANENT_TOKEN
-- Type: string
-- Required: false
-
-### Advanced options
-
-Here are the Advanced options specific to filefabric (Enterprise File Fabric).
-
-#### --filefabric-token
+- Config: directory_markers
+- Env Var: RCLONE_GCS_DIRECTORY_MARKERS
+- Type: bool
+- Default: false
-Session Token.
+#### --gcs-no-check-bucket
-This is a session token which rclone caches in the config file. It is
-usually valid for 1 hour.
+If set, don't attempt to check the bucket exists or create it.
-Don't set this value - rclone will set it automatically.
+This can be useful when trying to minimise the number of transactions
+rclone does if you know the bucket exists already.
Properties:
-- Config: token
-- Env Var: RCLONE_FILEFABRIC_TOKEN
-- Type: string
-- Required: false
+- Config: no_check_bucket
+- Env Var: RCLONE_GCS_NO_CHECK_BUCKET
+- Type: bool
+- Default: false
-#### --filefabric-token-expiry
+#### --gcs-decompress
-Token expiry time.
+If set this will decompress gzip encoded objects.
-Don't set this value - rclone will set it automatically.
+It is possible to upload objects to GCS with "Content-Encoding: gzip"
+set. Normally rclone will download these files as compressed objects.
+
+If this flag is set then rclone will decompress these files with
+"Content-Encoding: gzip" as they are received. This means that rclone
+can't check the size and hash but the file contents will be decompressed.
Properties:
-- Config: token_expiry
-- Env Var: RCLONE_FILEFABRIC_TOKEN_EXPIRY
-- Type: string
-- Required: false
-
-#### --filefabric-version
+- Config: decompress
+- Env Var: RCLONE_GCS_DECOMPRESS
+- Type: bool
+- Default: false
-Version read from the file fabric.
+#### --gcs-endpoint
-Don't set this value - rclone will set it automatically.
+Endpoint for the service.
+Leave blank normally.
Properties:
-- Config: version
-- Env Var: RCLONE_FILEFABRIC_VERSION
+- Config: endpoint
+- Env Var: RCLONE_GCS_ENDPOINT
- Type: string
- Required: false
-#### --filefabric-encoding
+#### --gcs-encoding
The encoding for the backend.
@@ -31002,33 +32519,38 @@ See the [encoding section in the overview](https://rclone.org/overview/#encoding
Properties:
- Config: encoding
-- Env Var: RCLONE_FILEFABRIC_ENCODING
-- Type: MultiEncoder
-- Default: Slash,Del,Ctl,InvalidUtf8,Dot
+- Env Var: RCLONE_GCS_ENCODING
+- Type: Encoding
+- Default: Slash,CrLf,InvalidUtf8,Dot
-# FTP
+## Limitations
-FTP is the File Transfer Protocol. Rclone FTP support is provided using the
-[github.com/jlaffaye/ftp](https://godoc.org/github.com/jlaffaye/ftp)
-package.
+`rclone about` is not supported by the Google Cloud Storage backend. Backends without
+this capability cannot determine free space for an rclone mount or
+use policy `mfs` (most free space) as a member of an rclone union
+remote.
-[Limitations of Rclone's FTP backend](#limitations)
+See [List of backends that do not support rclone about](https://rclone.org/overview/#optional-features) and [rclone about](https://rclone.org/commands/rclone_about/)
-Paths are specified as `remote:path`. If the path does not begin with
-a `/` it is relative to the home directory of the user. An empty path
-`remote:` refers to the user's home directory.
+# Google Drive
+
+Paths are specified as `drive:path`
+
+Drive paths may be as deep as required, e.g. `drive:directory/subdirectory`.
## Configuration
-To create an FTP configuration named `remote`, run
+The initial setup for drive involves getting a token from Google drive
+which you need to do in your browser. `rclone config` walks you
+through it.
- rclone config
+Here is an example of how to make a remote called `remote`. First run:
-Rclone config guides you through an interactive setup process. A minimal
-rclone FTP remote definition only requires host, username and password.
-For an anonymous FTP server, see [below](#anonymous-ftp).
+ rclone config
+
+This will guide you through an interactive setup process:
```
No remotes found, make a new one?
@@ -31040,47 +32562,59 @@ q) Quit config
n/r/c/s/q> n
name> remote
Type of storage to configure.
-Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
[snip]
-XX / FTP
- \ "ftp"
+XX / Google Drive
+ \ "drive"
[snip]
-Storage> ftp
-** See help for ftp backend at: https://rclone.org/ftp/ **
-
-FTP host to connect to
-Enter a string value. Press Enter for the default ("").
+Storage> drive
+Google Application Client Id - leave blank normally.
+client_id>
+Google Application Client Secret - leave blank normally.
+client_secret>
+Scope that rclone should use when requesting access from drive.
Choose a number from below, or type in your own value
- 1 / Connect to ftp.example.com
- \ "ftp.example.com"
-host> ftp.example.com
-FTP username
-Enter a string value. Press Enter for the default ("$USER").
-user>
-FTP port number
-Enter a signed integer. Press Enter for the default (21).
-port>
-FTP password
-y) Yes type in my own password
-g) Generate random password
-y/g> y
-Enter the password:
-password:
-Confirm the password:
-password:
-Use FTP over TLS (Implicit)
-Enter a boolean value (true or false). Press Enter for the default ("false").
-tls>
-Use FTP over TLS (Explicit)
-Enter a boolean value (true or false). Press Enter for the default ("false").
-explicit_tls>
+ 1 / Full access all files, excluding Application Data Folder.
+ \ "drive"
+ 2 / Read-only access to file metadata and file contents.
+ \ "drive.readonly"
+ / Access to files created by rclone only.
+ 3 | These are visible in the drive website.
+ | File authorization is revoked when the user deauthorizes the app.
+ \ "drive.file"
+ / Allows read and write access to the Application Data folder.
+ 4 | This is not visible in the drive website.
+ \ "drive.appfolder"
+ / Allows read-only access to file metadata but
+ 5 | does not allow any access to read or download file content.
+ \ "drive.metadata.readonly"
+scope> 1
+Service Account Credentials JSON file path - needed only if you want use SA instead of interactive login.
+service_account_file>
Remote config
+Use web browser to automatically authenticate rclone with remote?
+ * Say Y if the machine running rclone has a web browser you can use
+ * Say N if running rclone on a (remote) machine without web browser access
+If not sure try Y. If Y failed, try N.
+y) Yes
+n) No
+y/n> y
+If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth
+Log in and authorize rclone for access
+Waiting for code...
+Got code
+Configure this as a Shared Drive (Team Drive)?
+y) Yes
+n) No
+y/n> n
--------------------
[remote]
-type = ftp
-host = ftp.example.com
-pass = *** ENCRYPTED ***
+client_id =
+client_secret =
+scope = drive
+root_folder_id =
+service_account_file =
+token = {"access_token":"XXX","token_type":"Bearer","refresh_token":"XXX","expiry":"2014-03-16T13:57:58.955387075Z"}
--------------------
y) Yes this is OK
e) Edit this remote
@@ -31088,774 +32622,611 @@ d) Delete this remote
y/e/d> y
```
-To see all directories in the home directory of `remote`
-
- rclone lsd remote:
-
-Make a new directory
-
- rclone mkdir remote:path/to/directory
-
-List the contents of a directory
-
- rclone ls remote:path/to/directory
-
-Sync `/home/local/directory` to the remote directory, deleting any
-excess files in the directory.
-
- rclone sync --interactive /home/local/directory remote:directory
-
-### Anonymous FTP
-
-When connecting to a FTP server that allows anonymous login, you can use the
-special "anonymous" username. Traditionally, this user account accepts any
-string as a password, although it is common to use either the password
-"anonymous" or "guest". Some servers require the use of a valid e-mail
-address as password.
-
-Using [on-the-fly](#backend-path-to-dir) or
-[connection string](https://rclone.org/docs/#connection-strings) remotes makes it easy to access
-such servers, without requiring any configuration in advance. The following
-are examples of that:
-
- rclone lsf :ftp: --ftp-host=speedtest.tele2.net --ftp-user=anonymous --ftp-pass=$(rclone obscure dummy)
- rclone lsf :ftp,host=speedtest.tele2.net,user=anonymous,pass=$(rclone obscure dummy):
-
-The above examples work in Linux shells and in PowerShell, but not Windows
-Command Prompt. They execute the [rclone obscure](https://rclone.org/commands/rclone_obscure/)
-command to create a password string in the format required by the
-[pass](#ftp-pass) option. The following examples are exactly the same, except use
-an already obscured string representation of the same password "dummy", and
-therefore works even in Windows Command Prompt:
-
- rclone lsf :ftp: --ftp-host=speedtest.tele2.net --ftp-user=anonymous --ftp-pass=IXs2wc8OJOz7SYLBk47Ji1rHTmxM
- rclone lsf :ftp,host=speedtest.tele2.net,user=anonymous,pass=IXs2wc8OJOz7SYLBk47Ji1rHTmxM:
-
-### Implicit TLS
-
-Rlone FTP supports implicit FTP over TLS servers (FTPS). This has to
-be enabled in the FTP backend config for the remote, or with
-[`--ftp-tls`](#ftp-tls). The default FTPS port is `990`, not `21` and
-can be set with [`--ftp-port`](#ftp-port).
-
-### Restricted filename characters
-
-In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters)
-the following characters are also replaced:
-
-File names cannot end with the following characters. Replacement is
-limited to the last character in a file name:
-
-| Character | Value | Replacement |
-| --------- |:-----:|:-----------:|
-| SP | 0x20 | â |
-
-Not all FTP servers can have all characters in file names, for example:
-
-| FTP Server| Forbidden characters |
-| --------- |:--------------------:|
-| proftpd | `*` |
-| pureftpd | `\ [ ]` |
-
-This backend's interactive configuration wizard provides a selection of
-sensible encoding settings for major FTP servers: ProFTPd, PureFTPd, VsFTPd.
-Just hit a selection number when prompted.
-
-
-### Standard options
-
-Here are the Standard options specific to ftp (FTP).
-
-#### --ftp-host
-
-FTP host to connect to.
-
-E.g. "ftp.example.com".
-
-Properties:
-
-- Config: host
-- Env Var: RCLONE_FTP_HOST
-- Type: string
-- Required: true
-
-#### --ftp-user
-
-FTP username.
-
-Properties:
-
-- Config: user
-- Env Var: RCLONE_FTP_USER
-- Type: string
-- Default: "$USER"
-
-#### --ftp-port
-
-FTP port number.
-
-Properties:
-
-- Config: port
-- Env Var: RCLONE_FTP_PORT
-- Type: int
-- Default: 21
-
-#### --ftp-pass
-
-FTP password.
-
-**NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/).
-
-Properties:
-
-- Config: pass
-- Env Var: RCLONE_FTP_PASS
-- Type: string
-- Required: false
-
-#### --ftp-tls
-
-Use Implicit FTPS (FTP over TLS).
-
-When using implicit FTP over TLS the client connects using TLS
-right from the start which breaks compatibility with
-non-TLS-aware servers. This is usually served over port 990 rather
-than port 21. Cannot be used in combination with explicit FTPS.
-
-Properties:
-
-- Config: tls
-- Env Var: RCLONE_FTP_TLS
-- Type: bool
-- Default: false
-
-#### --ftp-explicit-tls
-
-Use Explicit FTPS (FTP over TLS).
-
-When using explicit FTP over TLS the client explicitly requests
-security from the server in order to upgrade a plain text connection
-to an encrypted one. Cannot be used in combination with implicit FTPS.
-
-Properties:
-
-- Config: explicit_tls
-- Env Var: RCLONE_FTP_EXPLICIT_TLS
-- Type: bool
-- Default: false
-
-### Advanced options
-
-Here are the Advanced options specific to ftp (FTP).
-
-#### --ftp-concurrency
-
-Maximum number of FTP simultaneous connections, 0 for unlimited.
-
-Note that setting this is very likely to cause deadlocks so it should
-be used with care.
-
-If you are doing a sync or copy then make sure concurrency is one more
-than the sum of `--transfers` and `--checkers`.
-
-If you use `--check-first` then it just needs to be one more than the
-maximum of `--checkers` and `--transfers`.
-
-So for `concurrency 3` you'd use `--checkers 2 --transfers 2
---check-first` or `--checkers 1 --transfers 1`.
-
-
-
-Properties:
-
-- Config: concurrency
-- Env Var: RCLONE_FTP_CONCURRENCY
-- Type: int
-- Default: 0
-
-#### --ftp-no-check-certificate
-
-Do not verify the TLS certificate of the server.
+See the [remote setup docs](https://rclone.org/remote_setup/) for how to set it up on a
+machine with no Internet browser available.
-Properties:
+Note that rclone runs a webserver on your local machine to collect the
+token as returned from Google if using web browser to automatically
+authenticate. This only
+runs from the moment it opens your browser to the moment you get back
+the verification code. This is on `http://127.0.0.1:53682/` and it
+may require you to unblock it temporarily if you are running a host
+firewall, or use manual mode.
-- Config: no_check_certificate
-- Env Var: RCLONE_FTP_NO_CHECK_CERTIFICATE
-- Type: bool
-- Default: false
+You can then use it like this,
-#### --ftp-disable-epsv
+List directories in top level of your drive
-Disable using EPSV even if server advertises support.
+ rclone lsd remote:
-Properties:
+List all the files in your drive
-- Config: disable_epsv
-- Env Var: RCLONE_FTP_DISABLE_EPSV
-- Type: bool
-- Default: false
+ rclone ls remote:
-#### --ftp-disable-mlsd
+To copy a local directory to a drive directory called backup
-Disable using MLSD even if server advertises support.
+ rclone copy /home/source remote:backup
-Properties:
+### Scopes
-- Config: disable_mlsd
-- Env Var: RCLONE_FTP_DISABLE_MLSD
-- Type: bool
-- Default: false
+Rclone allows you to select which scope you would like for rclone to
+use. This changes what type of token is granted to rclone. [The
+scopes are defined
+here](https://developers.google.com/drive/v3/web/about-auth).
-#### --ftp-disable-utf8
+A comma-separated list is allowed e.g. `drive.readonly,drive.file`.
-Disable using UTF-8 even if server advertises support.
+The scope are
-Properties:
+#### drive
-- Config: disable_utf8
-- Env Var: RCLONE_FTP_DISABLE_UTF8
-- Type: bool
-- Default: false
+This is the default scope and allows full access to all files, except
+for the Application Data Folder (see below).
-#### --ftp-writing-mdtm
+Choose this one if you aren't sure.
-Use MDTM to set modification time (VsFtpd quirk)
+#### drive.readonly
-Properties:
+This allows read only access to all files. Files may be listed and
+downloaded but not uploaded, renamed or deleted.
-- Config: writing_mdtm
-- Env Var: RCLONE_FTP_WRITING_MDTM
-- Type: bool
-- Default: false
+#### drive.file
-#### --ftp-force-list-hidden
+With this scope rclone can read/view/modify only those files and
+folders it creates.
-Use LIST -a to force listing of hidden files and folders. This will disable the use of MLSD.
+So if you uploaded files to drive via the web interface (or any other
+means) they will not be visible to rclone.
-Properties:
+This can be useful if you are using rclone to backup data and you want
+to be sure confidential data on your drive is not visible to rclone.
-- Config: force_list_hidden
-- Env Var: RCLONE_FTP_FORCE_LIST_HIDDEN
-- Type: bool
-- Default: false
+Files created with this scope are visible in the web interface.
-#### --ftp-idle-timeout
+#### drive.appfolder
-Max time before closing idle connections.
+This gives rclone its own private area to store files. Rclone will
+not be able to see any other files on your drive and you won't be able
+to see rclone's files from the web interface either.
-If no connections have been returned to the connection pool in the time
-given, rclone will empty the connection pool.
+#### drive.metadata.readonly
-Set to 0 to keep connections indefinitely.
+This allows read only access to file names only. It does not allow
+rclone to download or upload data, or rename or delete files or
+directories.
+### Root folder ID
-Properties:
+This option has been moved to the advanced section. You can set the `root_folder_id` for rclone. This is the directory
+(identified by its `Folder ID`) that rclone considers to be the root
+of your drive.
-- Config: idle_timeout
-- Env Var: RCLONE_FTP_IDLE_TIMEOUT
-- Type: Duration
-- Default: 1m0s
+Normally you will leave this blank and rclone will determine the
+correct root to use itself.
-#### --ftp-close-timeout
+However you can set this to restrict rclone to a specific folder
+hierarchy or to access data within the "Computers" tab on the drive
+web interface (where files from Google's Backup and Sync desktop
+program go).
-Maximum time to wait for a response to close.
+In order to do this you will have to find the `Folder ID` of the
+directory you wish rclone to display. This will be the last segment
+of the URL when you open the relevant folder in the drive web
+interface.
-Properties:
+So if the folder you want rclone to use has a URL which looks like
+`https://drive.google.com/drive/folders/1XyfxxxxxxxxxxxxxxxxxxxxxxxxxKHCh`
+in the browser, then you use `1XyfxxxxxxxxxxxxxxxxxxxxxxxxxKHCh` as
+the `root_folder_id` in the config.
-- Config: close_timeout
-- Env Var: RCLONE_FTP_CLOSE_TIMEOUT
-- Type: Duration
-- Default: 1m0s
+**NB** folders under the "Computers" tab seem to be read only (drive
+gives a 500 error) when using rclone.
-#### --ftp-tls-cache-size
+There doesn't appear to be an API to discover the folder IDs of the
+"Computers" tab - please contact us if you know otherwise!
-Size of TLS session cache for all control and data connections.
+Note also that rclone can't access any data under the "Backups" tab on
+the google drive web interface yet.
-TLS cache allows to resume TLS sessions and reuse PSK between connections.
-Increase if default size is not enough resulting in TLS resumption errors.
-Enabled by default. Use 0 to disable.
+### Service Account support
-Properties:
+You can set up rclone with Google Drive in an unattended mode,
+i.e. not tied to a specific end-user Google account. This is useful
+when you want to synchronise files onto machines that don't have
+actively logged-in users, for example build machines.
-- Config: tls_cache_size
-- Env Var: RCLONE_FTP_TLS_CACHE_SIZE
-- Type: int
-- Default: 32
+To use a Service Account instead of OAuth2 token flow, enter the path
+to your Service Account credentials at the `service_account_file`
+prompt during `rclone config` and rclone won't use the browser based
+authentication flow. If you'd rather stuff the contents of the
+credentials file into the rclone config file, you can set
+`service_account_credentials` with the actual contents of the file
+instead, or set the equivalent environment variable.
-#### --ftp-disable-tls13
+#### Use case - Google Apps/G-suite account and individual Drive
-Disable TLS 1.3 (workaround for FTP servers with buggy TLS)
+Let's say that you are the administrator of a Google Apps (old) or
+G-suite account.
+The goal is to store data on an individual's Drive account, who IS
+a member of the domain.
+We'll call the domain **example.com**, and the user
+**foo@example.com**.
-Properties:
+There's a few steps we need to go through to accomplish this:
-- Config: disable_tls13
-- Env Var: RCLONE_FTP_DISABLE_TLS13
-- Type: bool
-- Default: false
+##### 1. Create a service account for example.com
+ - To create a service account and obtain its credentials, go to the
+[Google Developer Console](https://console.developers.google.com).
+ - You must have a project - create one if you don't.
+ - Then go to "IAM & admin" -> "Service Accounts".
+ - Use the "Create Service Account" button. Fill in "Service account name"
+and "Service account ID" with something that identifies your client.
+ - Select "Create And Continue". Step 2 and 3 are optional.
+ - These credentials are what rclone will use for authentication.
+If you ever need to remove access, press the "Delete service
+account key" button.
-#### --ftp-shut-timeout
+##### 2. Allowing API access to example.com Google Drive
+ - Go to example.com's admin console
+ - Go into "Security" (or use the search bar)
+ - Select "Show more" and then "Advanced settings"
+ - Select "Manage API client access" in the "Authentication" section
+ - In the "Client Name" field enter the service account's
+"Client ID" - this can be found in the Developer Console under
+"IAM & Admin" -> "Service Accounts", then "View Client ID" for
+the newly created service account.
+It is a ~21 character numerical string.
+ - In the next field, "One or More API Scopes", enter
+`https://www.googleapis.com/auth/drive`
+to grant access to Google Drive specifically.
-Maximum time to wait for data connection closing status.
+##### 3. Configure rclone, assuming a new install
-Properties:
+```
+rclone config
-- Config: shut_timeout
-- Env Var: RCLONE_FTP_SHUT_TIMEOUT
-- Type: Duration
-- Default: 1m0s
+n/s/q> n # New
+name>gdrive # Gdrive is an example name
+Storage> # Select the number shown for Google Drive
+client_id> # Can be left blank
+client_secret> # Can be left blank
+scope> # Select your scope, 1 for example
+root_folder_id> # Can be left blank
+service_account_file> /home/foo/myJSONfile.json # This is where the JSON file goes!
+y/n> # Auto config, n
-#### --ftp-ask-password
+```
-Allow asking for FTP password when needed.
+##### 4. Verify that it's working
+ - `rclone -v --drive-impersonate foo@example.com lsf gdrive:backup`
+ - The arguments do:
+ - `-v` - verbose logging
+ - `--drive-impersonate foo@example.com` - this is what does
+the magic, pretending to be user foo.
+ - `lsf` - list files in a parsing friendly way
+ - `gdrive:backup` - use the remote called gdrive, work in
+the folder named backup.
-If this is set and no password is supplied then rclone will ask for a password
+Note: in case you configured a specific root folder on gdrive and rclone is unable to access the contents of that folder when using `--drive-impersonate`, do this instead:
+ - in the gdrive web interface, share your root folder with the user/email of the new Service Account you created/selected at step #1
+ - use rclone without specifying the `--drive-impersonate` option, like this:
+ `rclone -v lsf gdrive:backup`
-Properties:
+### Shared drives (team drives)
-- Config: ask_password
-- Env Var: RCLONE_FTP_ASK_PASSWORD
-- Type: bool
-- Default: false
+If you want to configure the remote to point to a Google Shared Drive
+(previously known as Team Drives) then answer `y` to the question
+`Configure this as a Shared Drive (Team Drive)?`.
-#### --ftp-socks-proxy
+This will fetch the list of Shared Drives from google and allow you to
+configure which one you want to use. You can also type in a Shared
+Drive ID if you prefer.
-Socks 5 proxy host.
-
- Supports the format user:pass@host:port, user@host:port, host:port.
-
- Example:
-
- myUser:myPass@localhost:9005
-
+For example:
-Properties:
+```
+Configure this as a Shared Drive (Team Drive)?
+y) Yes
+n) No
+y/n> y
+Fetching Shared Drive list...
+Choose a number from below, or type in your own value
+ 1 / Rclone Test
+ \ "xxxxxxxxxxxxxxxxxxxx"
+ 2 / Rclone Test 2
+ \ "yyyyyyyyyyyyyyyyyyyy"
+ 3 / Rclone Test 3
+ \ "zzzzzzzzzzzzzzzzzzzz"
+Enter a Shared Drive ID> 1
+--------------------
+[remote]
+client_id =
+client_secret =
+token = {"AccessToken":"xxxx.x.xxxxx_xxxxxxxxxxx_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","RefreshToken":"1/xxxxxxxxxxxxxxxx_xxxxxxxxxxxxxxxxxxxxxxxxxx","Expiry":"2014-03-16T13:57:58.955387075Z","Extra":null}
+team_drive = xxxxxxxxxxxxxxxxxxxx
+--------------------
+y) Yes this is OK
+e) Edit this remote
+d) Delete this remote
+y/e/d> y
+```
-- Config: socks_proxy
-- Env Var: RCLONE_FTP_SOCKS_PROXY
-- Type: string
-- Required: false
+### --fast-list
-#### --ftp-encoding
+This remote supports `--fast-list` which allows you to use fewer
+transactions in exchange for more memory. See the [rclone
+docs](https://rclone.org/docs/#fast-list) for more details.
-The encoding for the backend.
+It does this by combining multiple `list` calls into a single API request.
-See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info.
+This works by combining many `'%s' in parents` filters into one expression.
+To list the contents of directories a, b and c, the following requests will be send by the regular `List` function:
+```
+trashed=false and 'a' in parents
+trashed=false and 'b' in parents
+trashed=false and 'c' in parents
+```
+These can now be combined into a single request:
+```
+trashed=false and ('a' in parents or 'b' in parents or 'c' in parents)
+```
-Properties:
+The implementation of `ListR` will put up to 50 `parents` filters into one request.
+It will use the `--checkers` value to specify the number of requests to run in parallel.
-- Config: encoding
-- Env Var: RCLONE_FTP_ENCODING
-- Type: MultiEncoder
-- Default: Slash,Del,Ctl,RightSpace,Dot
-- Examples:
- - "Asterisk,Ctl,Dot,Slash"
- - ProFTPd can't handle '*' in file names
- - "BackSlash,Ctl,Del,Dot,RightSpace,Slash,SquareBracket"
- - PureFTPd can't handle '[]' or '*' in file names
- - "Ctl,LeftPeriod,Slash"
- - VsFTPd can't handle file names starting with dot
+In tests, these batch requests were up to 20x faster than the regular method.
+Running the following command against different sized folders gives:
+```
+rclone lsjson -vv -R --checkers=6 gdrive:folder
+```
+small folder (220 directories, 700 files):
+- without `--fast-list`: 38s
+- with `--fast-list`: 10s
-## Limitations
+large folder (10600 directories, 39000 files):
-FTP servers acting as rclone remotes must support `passive` mode.
-The mode cannot be configured as `passive` is the only supported one.
-Rclone's FTP implementation is not compatible with `active` mode
-as [the library it uses doesn't support it](https://github.com/jlaffaye/ftp/issues/29).
-This will likely never be supported due to security concerns.
+- without `--fast-list`: 22:05 min
+- with `--fast-list`: 58s
-Rclone's FTP backend does not support any checksums but can compare
-file sizes.
+### Modification times and hashes
-`rclone about` is not supported by the FTP backend. Backends without
-this capability cannot determine free space for an rclone mount or
-use policy `mfs` (most free space) as a member of an rclone union
-remote.
+Google drive stores modification times accurate to 1 ms.
-See [List of backends that do not support rclone about](https://rclone.org/overview/#optional-features) and [rclone about](https://rclone.org/commands/rclone_about/)
+Hash algorithms MD5, SHA1 and SHA256 are supported. Note, however,
+that a small fraction of files uploaded may not have SHA1 or SHA256
+hashes especially if they were uploaded before 2018.
-The implementation of : `--dump headers`,
-`--dump bodies`, `--dump auth` for debugging isn't the same as
-for rclone HTTP based backends - it has less fine grained control.
+### Restricted filename characters
-`--timeout` isn't supported (but `--contimeout` is).
+Only Invalid UTF-8 bytes will be [replaced](https://rclone.org/overview/#invalid-utf8),
+as they can't be used in JSON strings.
-`--bind` isn't supported.
+In contrast to other backends, `/` can also be used in names and `.`
+or `..` are valid names.
-Rclone's FTP backend could support server-side move but does not
-at present.
+### Revisions
-The `ftp_proxy` environment variable is not currently supported.
+Google drive stores revisions of files. When you upload a change to
+an existing file to google drive using rclone it will create a new
+revision of that file.
-#### Modified time
+Revisions follow the standard google policy which at time of writing
+was
-File modification time (timestamps) is supported to 1 second resolution
-for major FTP servers: ProFTPd, PureFTPd, VsFTPd, and FileZilla FTP server.
-The `VsFTPd` server has non-standard implementation of time related protocol
-commands and needs a special configuration setting: `writing_mdtm = true`.
+ * They are deleted after 30 days or 100 revisions (whatever comes first).
+ * They do not count towards a user storage quota.
-Support for precise file time with other FTP servers varies depending on what
-protocol extensions they advertise. If all the `MLSD`, `MDTM` and `MFTM`
-extensions are present, rclone will use them together to provide precise time.
-Otherwise the times you see on the FTP server through rclone are those of the
-last file upload.
+### Deleting files
-You can use the following command to check whether rclone can use precise time
-with your FTP server: `rclone backend features your_ftp_remote:` (the trailing
-colon is important). Look for the number in the line tagged by `Precision`
-designating the remote time precision expressed as nanoseconds. A value of
-`1000000000` means that file time precision of 1 second is available.
-A value of `3153600000000000000` (or another large number) means "unsupported".
+By default rclone will send all files to the trash when deleting
+files. If deleting them permanently is required then use the
+`--drive-use-trash=false` flag, or set the equivalent environment
+variable.
-# Google Cloud Storage
+### Shortcuts
-Paths are specified as `remote:bucket` (or `remote:` for the `lsd`
-command.) You may put subdirectories in too, e.g. `remote:bucket/path/to/dir`.
+In March 2020 Google introduced a new feature in Google Drive called
+[drive shortcuts](https://support.google.com/drive/answer/9700156)
+([API](https://developers.google.com/drive/api/v3/shortcuts)). These
+will (by September 2020) [replace the ability for files or folders to
+be in multiple folders at once](https://cloud.google.com/blog/products/g-suite/simplifying-google-drives-folder-structure-and-sharing-models).
-## Configuration
+Shortcuts are files that link to other files on Google Drive somewhat
+like a symlink in unix, except they point to the underlying file data
+(e.g. the inode in unix terms) so they don't break if the source is
+renamed or moved about.
-The initial setup for google cloud storage involves getting a token from Google Cloud Storage
-which you need to do in your browser. `rclone config` walks you
-through it.
+By default rclone treats these as follows.
-Here is an example of how to make a remote called `remote`. First run:
+For shortcuts pointing to files:
- rclone config
+- When listing a file shortcut appears as the destination file.
+- When downloading the contents of the destination file is downloaded.
+- When updating shortcut file with a non shortcut file, the shortcut is removed then a new file is uploaded in place of the shortcut.
+- When server-side moving (renaming) the shortcut is renamed, not the destination file.
+- When server-side copying the shortcut is copied, not the contents of the shortcut. (unless `--drive-copy-shortcut-content` is in use in which case the contents of the shortcut gets copied).
+- When deleting the shortcut is deleted not the linked file.
+- When setting the modification time, the modification time of the linked file will be set.
-This will guide you through an interactive setup process:
+For shortcuts pointing to folders:
-```
-n) New remote
-d) Delete remote
-q) Quit config
-e/n/d/q> n
-name> remote
-Type of storage to configure.
-Choose a number from below, or type in your own value
-[snip]
-XX / Google Cloud Storage (this is not Google Drive)
- \ "google cloud storage"
-[snip]
-Storage> google cloud storage
-Google Application Client Id - leave blank normally.
-client_id>
-Google Application Client Secret - leave blank normally.
-client_secret>
-Project number optional - needed only for list/create/delete buckets - see your developer console.
-project_number> 12345678
-Service Account Credentials JSON file path - needed only if you want use SA instead of interactive login.
-service_account_file>
-Access Control List for new objects.
-Choose a number from below, or type in your own value
- 1 / Object owner gets OWNER access, and all Authenticated Users get READER access.
- \ "authenticatedRead"
- 2 / Object owner gets OWNER access, and project team owners get OWNER access.
- \ "bucketOwnerFullControl"
- 3 / Object owner gets OWNER access, and project team owners get READER access.
- \ "bucketOwnerRead"
- 4 / Object owner gets OWNER access [default if left blank].
- \ "private"
- 5 / Object owner gets OWNER access, and project team members get access according to their roles.
- \ "projectPrivate"
- 6 / Object owner gets OWNER access, and all Users get READER access.
- \ "publicRead"
-object_acl> 4
-Access Control List for new buckets.
-Choose a number from below, or type in your own value
- 1 / Project team owners get OWNER access, and all Authenticated Users get READER access.
- \ "authenticatedRead"
- 2 / Project team owners get OWNER access [default if left blank].
- \ "private"
- 3 / Project team members get access according to their roles.
- \ "projectPrivate"
- 4 / Project team owners get OWNER access, and all Users get READER access.
- \ "publicRead"
- 5 / Project team owners get OWNER access, and all Users get WRITER access.
- \ "publicReadWrite"
-bucket_acl> 2
-Location for the newly created buckets.
-Choose a number from below, or type in your own value
- 1 / Empty for default location (US).
- \ ""
- 2 / Multi-regional location for Asia.
- \ "asia"
- 3 / Multi-regional location for Europe.
- \ "eu"
- 4 / Multi-regional location for United States.
- \ "us"
- 5 / Taiwan.
- \ "asia-east1"
- 6 / Tokyo.
- \ "asia-northeast1"
- 7 / Singapore.
- \ "asia-southeast1"
- 8 / Sydney.
- \ "australia-southeast1"
- 9 / Belgium.
- \ "europe-west1"
-10 / London.
- \ "europe-west2"
-11 / Iowa.
- \ "us-central1"
-12 / South Carolina.
- \ "us-east1"
-13 / Northern Virginia.
- \ "us-east4"
-14 / Oregon.
- \ "us-west1"
-location> 12
-The storage class to use when storing objects in Google Cloud Storage.
-Choose a number from below, or type in your own value
- 1 / Default
- \ ""
- 2 / Multi-regional storage class
- \ "MULTI_REGIONAL"
- 3 / Regional storage class
- \ "REGIONAL"
- 4 / Nearline storage class
- \ "NEARLINE"
- 5 / Coldline storage class
- \ "COLDLINE"
- 6 / Durable reduced availability storage class
- \ "DURABLE_REDUCED_AVAILABILITY"
-storage_class> 5
-Remote config
-Use web browser to automatically authenticate rclone with remote?
- * Say Y if the machine running rclone has a web browser you can use
- * Say N if running rclone on a (remote) machine without web browser access
-If not sure try Y. If Y failed, try N.
-y) Yes
-n) No
-y/n> y
-If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth
-Log in and authorize rclone for access
-Waiting for code...
-Got code
---------------------
-[remote]
-type = google cloud storage
-client_id =
-client_secret =
-token = {"AccessToken":"xxxx.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","RefreshToken":"x/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx_xxxxxxxxx","Expiry":"2014-07-17T20:49:14.929208288+01:00","Extra":null}
-project_number = 12345678
-object_acl = private
-bucket_acl = private
---------------------
-y) Yes this is OK
-e) Edit this remote
-d) Delete this remote
-y/e/d> y
-```
+- When listing the shortcut appears as a folder and that folder will contain the contents of the linked folder appear (including any sub folders)
+- When downloading the contents of the linked folder and sub contents are downloaded
+- When uploading to a shortcut folder the file will be placed in the linked folder
+- When server-side moving (renaming) the shortcut is renamed, not the destination folder
+- When server-side copying the contents of the linked folder is copied, not the shortcut.
+- When deleting with `rclone rmdir` or `rclone purge` the shortcut is deleted not the linked folder.
+- **NB** When deleting with `rclone remove` or `rclone mount` the contents of the linked folder will be deleted.
-See the [remote setup docs](https://rclone.org/remote_setup/) for how to set it up on a
-machine with no Internet browser available.
+The [rclone backend](https://rclone.org/commands/rclone_backend/) command can be used to create shortcuts.
-Note that rclone runs a webserver on your local machine to collect the
-token as returned from Google if using web browser to automatically
-authenticate. This only
-runs from the moment it opens your browser to the moment you get back
-the verification code. This is on `http://127.0.0.1:53682/` and this
-it may require you to unblock it temporarily if you are running a host
-firewall, or use manual mode.
+Shortcuts can be completely ignored with the `--drive-skip-shortcuts` flag
+or the corresponding `skip_shortcuts` configuration setting.
-This remote is called `remote` and can now be used like this
+### Emptying trash
-See all the buckets in your project
+If you wish to empty your trash you can use the `rclone cleanup remote:`
+command which will permanently delete all your trashed files. This command
+does not take any path arguments.
- rclone lsd remote:
+Note that Google Drive takes some time (minutes to days) to empty the
+trash even though the command returns within a few seconds. No output
+is echoed, so there will be no confirmation even using -v or -vv.
-Make a new bucket
+### Quota information
- rclone mkdir remote:bucket
+To view your current quota you can use the `rclone about remote:`
+command which will display your usage limit (quota), the usage in Google
+Drive, the size of all files in the Trash and the space used by other
+Google services such as Gmail. This command does not take any path
+arguments.
-List the contents of a bucket
+#### Import/Export of google documents
- rclone ls remote:bucket
+Google documents can be exported from and uploaded to Google Drive.
-Sync `/home/local/directory` to the remote bucket, deleting any excess
-files in the bucket.
+When rclone downloads a Google doc it chooses a format to download
+depending upon the `--drive-export-formats` setting.
+By default the export formats are `docx,xlsx,pptx,svg` which are a
+sensible default for an editable document.
- rclone sync --interactive /home/local/directory remote:bucket
+When choosing a format, rclone runs down the list provided in order
+and chooses the first file format the doc can be exported as from the
+list. If the file can't be exported to a format on the formats list,
+then rclone will choose a format from the default list.
-### Service Account support
+If you prefer an archive copy then you might use `--drive-export-formats
+pdf`, or if you prefer openoffice/libreoffice formats you might use
+`--drive-export-formats ods,odt,odp`.
-You can set up rclone with Google Cloud Storage in an unattended mode,
-i.e. not tied to a specific end-user Google account. This is useful
-when you want to synchronise files onto machines that don't have
-actively logged-in users, for example build machines.
+Note that rclone adds the extension to the google doc, so if it is
+called `My Spreadsheet` on google docs, it will be exported as `My
+Spreadsheet.xlsx` or `My Spreadsheet.pdf` etc.
-To get credentials for Google Cloud Platform
-[IAM Service Accounts](https://cloud.google.com/iam/docs/service-accounts),
-please head to the
-[Service Account](https://console.cloud.google.com/permissions/serviceaccounts)
-section of the Google Developer Console. Service Accounts behave just
-like normal `User` permissions in
-[Google Cloud Storage ACLs](https://cloud.google.com/storage/docs/access-control),
-so you can limit their access (e.g. make them read only). After
-creating an account, a JSON file containing the Service Account's
-credentials will be downloaded onto your machines. These credentials
-are what rclone will use for authentication.
+When importing files into Google Drive, rclone will convert all
+files with an extension in `--drive-import-formats` to their
+associated document type.
+rclone will not convert any files by default, since the conversion
+is lossy process.
-To use a Service Account instead of OAuth2 token flow, enter the path
-to your Service Account credentials at the `service_account_file`
-prompt and rclone won't use the browser based authentication
-flow. If you'd rather stuff the contents of the credentials file into
-the rclone config file, you can set `service_account_credentials` with
-the actual contents of the file instead, or set the equivalent
-environment variable.
+The conversion must result in a file with the same extension when
+the `--drive-export-formats` rules are applied to the uploaded document.
-### Anonymous Access
+Here are some examples for allowed and prohibited conversions.
-For downloads of objects that permit public access you can configure rclone
-to use anonymous access by setting `anonymous` to `true`.
-With unauthorized access you can't write or create files but only read or list
-those buckets and objects that have public read access.
+| export-formats | import-formats | Upload Ext | Document Ext | Allowed |
+| -------------- | -------------- | ---------- | ------------ | ------- |
+| odt | odt | odt | odt | Yes |
+| odt | docx,odt | odt | odt | Yes |
+| | docx | docx | docx | Yes |
+| | odt | odt | docx | No |
+| odt,docx | docx,odt | docx | odt | No |
+| docx,odt | docx,odt | docx | docx | Yes |
+| docx,odt | docx,odt | odt | docx | No |
-### Application Default Credentials
+This limitation can be disabled by specifying `--drive-allow-import-name-change`.
+When using this flag, rclone can convert multiple files types resulting
+in the same document type at once, e.g. with `--drive-import-formats docx,odt,txt`,
+all files having these extension would result in a document represented as a docx file.
+This brings the additional risk of overwriting a document, if multiple files
+have the same stem. Many rclone operations will not handle this name change
+in any way. They assume an equal name when copying files and might copy the
+file again or delete them when the name changes.
-If no other source of credentials is provided, rclone will fall back
-to
-[Application Default Credentials](https://cloud.google.com/video-intelligence/docs/common/auth#authenticating_with_application_default_credentials)
-this is useful both when you already have configured authentication
-for your developer account, or in production when running on a google
-compute host. Note that if running in docker, you may need to run
-additional commands on your google compute machine -
-[see this page](https://cloud.google.com/container-registry/docs/advanced-authentication#gcloud_as_a_docker_credential_helper).
+Here are the possible export extensions with their corresponding mime types.
+Most of these can also be used for importing, but there more that are not
+listed here. Some of these additional ones might only be available when
+the operating system provides the correct MIME type entries.
-Note that in the case application default credentials are used, there
-is no need to explicitly configure a project number.
+This list can be changed by Google Drive at any time and might not
+represent the currently available conversions.
-### --fast-list
+| Extension | Mime Type | Description |
+| --------- |-----------| ------------|
+| bmp | image/bmp | Windows Bitmap format |
+| csv | text/csv | Standard CSV format for Spreadsheets |
+| doc | application/msword | Classic Word file |
+| docx | application/vnd.openxmlformats-officedocument.wordprocessingml.document | Microsoft Office Document |
+| epub | application/epub+zip | E-book format |
+| html | text/html | An HTML Document |
+| jpg | image/jpeg | A JPEG Image File |
+| json | application/vnd.google-apps.script+json | JSON Text Format for Google Apps scripts |
+| odp | application/vnd.oasis.opendocument.presentation | Openoffice Presentation |
+| ods | application/vnd.oasis.opendocument.spreadsheet | Openoffice Spreadsheet |
+| ods | application/x-vnd.oasis.opendocument.spreadsheet | Openoffice Spreadsheet |
+| odt | application/vnd.oasis.opendocument.text | Openoffice Document |
+| pdf | application/pdf | Adobe PDF Format |
+| pjpeg | image/pjpeg | Progressive JPEG Image |
+| png | image/png | PNG Image Format|
+| pptx | application/vnd.openxmlformats-officedocument.presentationml.presentation | Microsoft Office Powerpoint |
+| rtf | application/rtf | Rich Text Format |
+| svg | image/svg+xml | Scalable Vector Graphics Format |
+| tsv | text/tab-separated-values | Standard TSV format for spreadsheets |
+| txt | text/plain | Plain Text |
+| wmf | application/x-msmetafile | Windows Meta File |
+| xls | application/vnd.ms-excel | Classic Excel file |
+| xlsx | application/vnd.openxmlformats-officedocument.spreadsheetml.sheet | Microsoft Office Spreadsheet |
+| zip | application/zip | A ZIP file of HTML, Images CSS |
-This remote supports `--fast-list` which allows you to use fewer
-transactions in exchange for more memory. See the [rclone
-docs](https://rclone.org/docs/#fast-list) for more details.
+Google documents can also be exported as link files. These files will
+open a browser window for the Google Docs website of that document
+when opened. The link file extension has to be specified as a
+`--drive-export-formats` parameter. They will match all available
+Google Documents.
-### Custom upload headers
+| Extension | Description | OS Support |
+| --------- | ----------- | ---------- |
+| desktop | freedesktop.org specified desktop entry | Linux |
+| link.html | An HTML Document with a redirect | All |
+| url | INI style link file | macOS, Windows |
+| webloc | macOS specific XML format | macOS |
-You can set custom upload headers with the `--header-upload`
-flag. Google Cloud Storage supports the headers as described in the
-[working with metadata documentation](https://cloud.google.com/storage/docs/gsutil/addlhelp/WorkingWithObjectMetadata)
-- Cache-Control
-- Content-Disposition
-- Content-Encoding
-- Content-Language
-- Content-Type
-- X-Goog-Storage-Class
-- X-Goog-Meta-
+### Standard options
-Eg `--header-upload "Content-Type text/potato"`
+Here are the Standard options specific to drive (Google Drive).
-Note that the last of these is for setting custom metadata in the form
-`--header-upload "x-goog-meta-key: value"`
+#### --drive-client-id
-### Modification time
+Google Application Client Id
+Setting your own is recommended.
+See https://rclone.org/drive/#making-your-own-client-id for how to create your own.
+If you leave this blank, it will use an internal key which is low performance.
-Google Cloud Storage stores md5sum natively.
-Google's [gsutil](https://cloud.google.com/storage/docs/gsutil) tool stores modification time
-with one-second precision as `goog-reserved-file-mtime` in file metadata.
+Properties:
-To ensure compatibility with gsutil, rclone stores modification time in 2 separate metadata entries.
-`mtime` uses RFC3339 format with one-nanosecond precision.
-`goog-reserved-file-mtime` uses Unix timestamp format with one-second precision.
-To get modification time from object metadata, rclone reads the metadata in the following order: `mtime`, `goog-reserved-file-mtime`, object updated time.
+- Config: client_id
+- Env Var: RCLONE_DRIVE_CLIENT_ID
+- Type: string
+- Required: false
-Note that rclone's default modify window is 1ns.
-Files uploaded by gsutil only contain timestamps with one-second precision.
-If you use rclone to sync files previously uploaded by gsutil,
-rclone will attempt to update modification time for all these files.
-To avoid these possibly unnecessary updates, use `--modify-window 1s`.
+#### --drive-client-secret
-### Restricted filename characters
+OAuth Client Secret.
-| Character | Value | Replacement |
-| --------- |:-----:|:-----------:|
-| NUL | 0x00 | †|
-| LF | 0x0A | ⊠|
-| CR | 0x0D | â |
-| / | 0x2F | ï¼ |
+Leave blank normally.
-Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8),
-as they can't be used in JSON strings.
+Properties:
+- Config: client_secret
+- Env Var: RCLONE_DRIVE_CLIENT_SECRET
+- Type: string
+- Required: false
-### Standard options
+#### --drive-scope
-Here are the Standard options specific to google cloud storage (Google Cloud Storage (this is not Google Drive)).
+Comma separated list of scopes that rclone should use when requesting access from drive.
-#### --gcs-client-id
+Properties:
-OAuth Client Id.
+- Config: scope
+- Env Var: RCLONE_DRIVE_SCOPE
+- Type: string
+- Required: false
+- Examples:
+ - "drive"
+ - Full access all files, excluding Application Data Folder.
+ - "drive.readonly"
+ - Read-only access to file metadata and file contents.
+ - "drive.file"
+ - Access to files created by rclone only.
+ - These are visible in the drive website.
+ - File authorization is revoked when the user deauthorizes the app.
+ - "drive.appfolder"
+ - Allows read and write access to the Application Data folder.
+ - This is not visible in the drive website.
+ - "drive.metadata.readonly"
+ - Allows read-only access to file metadata but
+ - does not allow any access to read or download file content.
+
+#### --drive-service-account-file
+
+Service Account Credentials JSON file path.
Leave blank normally.
+Needed only if you want use SA instead of interactive login.
+
+Leading `~` will be expanded in the file name as will environment variables such as `${RCLONE_CONFIG_DIR}`.
Properties:
-- Config: client_id
-- Env Var: RCLONE_GCS_CLIENT_ID
+- Config: service_account_file
+- Env Var: RCLONE_DRIVE_SERVICE_ACCOUNT_FILE
- Type: string
- Required: false
-#### --gcs-client-secret
+#### --drive-alternate-export
-OAuth Client Secret.
+Deprecated: No longer needed.
-Leave blank normally.
+Properties:
+
+- Config: alternate_export
+- Env Var: RCLONE_DRIVE_ALTERNATE_EXPORT
+- Type: bool
+- Default: false
+
+### Advanced options
+
+Here are the Advanced options specific to drive (Google Drive).
+
+#### --drive-token
+
+OAuth Access Token as a JSON blob.
Properties:
-- Config: client_secret
-- Env Var: RCLONE_GCS_CLIENT_SECRET
+- Config: token
+- Env Var: RCLONE_DRIVE_TOKEN
- Type: string
- Required: false
-#### --gcs-project-number
+#### --drive-auth-url
-Project number.
+Auth server URL.
-Optional - needed only for list/create/delete buckets - see your developer console.
+Leave blank to use the provider defaults.
Properties:
-- Config: project_number
-- Env Var: RCLONE_GCS_PROJECT_NUMBER
+- Config: auth_url
+- Env Var: RCLONE_DRIVE_AUTH_URL
- Type: string
- Required: false
-#### --gcs-user-project
+#### --drive-token-url
-User project.
+Token server url.
-Optional - needed only for requester pays.
+Leave blank to use the provider defaults.
Properties:
-- Config: user_project
-- Env Var: RCLONE_GCS_USER_PROJECT
+- Config: token_url
+- Env Var: RCLONE_DRIVE_TOKEN_URL
- Type: string
- Required: false
-#### --gcs-service-account-file
-
-Service Account Credentials JSON file path.
+#### --drive-root-folder-id
+ID of the root folder.
Leave blank normally.
-Needed only if you want use SA instead of interactive login.
-Leading `~` will be expanded in the file name as will environment variables such as `${RCLONE_CONFIG_DIR}`.
+Fill in to access "Computers" folders (see docs), or for rclone to use
+a non root folder as its starting point.
+
Properties:
-- Config: service_account_file
-- Env Var: RCLONE_GCS_SERVICE_ACCOUNT_FILE
+- Config: root_folder_id
+- Env Var: RCLONE_DRIVE_ROOT_FOLDER_ID
- Type: string
- Required: false
-#### --gcs-service-account-credentials
+#### --drive-service-account-credentials
Service Account Credentials JSON blob.
@@ -31865,1978 +33236,1962 @@ Needed only if you want use SA instead of interactive login.
Properties:
- Config: service_account_credentials
-- Env Var: RCLONE_GCS_SERVICE_ACCOUNT_CREDENTIALS
+- Env Var: RCLONE_DRIVE_SERVICE_ACCOUNT_CREDENTIALS
- Type: string
- Required: false
-#### --gcs-anonymous
+#### --drive-team-drive
-Access public buckets and objects without credentials.
+ID of the Shared Drive (Team Drive).
-Set to 'true' if you just want to download files and don't configure credentials.
+Properties:
+
+- Config: team_drive
+- Env Var: RCLONE_DRIVE_TEAM_DRIVE
+- Type: string
+- Required: false
+
+#### --drive-auth-owner-only
+
+Only consider files owned by the authenticated user.
Properties:
-- Config: anonymous
-- Env Var: RCLONE_GCS_ANONYMOUS
+- Config: auth_owner_only
+- Env Var: RCLONE_DRIVE_AUTH_OWNER_ONLY
- Type: bool
- Default: false
-#### --gcs-object-acl
+#### --drive-use-trash
-Access Control List for new objects.
+Send files to the trash instead of deleting permanently.
+
+Defaults to true, namely sending files to the trash.
+Use `--drive-use-trash=false` to delete files permanently instead.
Properties:
-- Config: object_acl
-- Env Var: RCLONE_GCS_OBJECT_ACL
-- Type: string
-- Required: false
-- Examples:
- - "authenticatedRead"
- - Object owner gets OWNER access.
- - All Authenticated Users get READER access.
- - "bucketOwnerFullControl"
- - Object owner gets OWNER access.
- - Project team owners get OWNER access.
- - "bucketOwnerRead"
- - Object owner gets OWNER access.
- - Project team owners get READER access.
- - "private"
- - Object owner gets OWNER access.
- - Default if left blank.
- - "projectPrivate"
- - Object owner gets OWNER access.
- - Project team members get access according to their roles.
- - "publicRead"
- - Object owner gets OWNER access.
- - All Users get READER access.
+- Config: use_trash
+- Env Var: RCLONE_DRIVE_USE_TRASH
+- Type: bool
+- Default: true
-#### --gcs-bucket-acl
+#### --drive-copy-shortcut-content
-Access Control List for new buckets.
+Server side copy contents of shortcuts instead of the shortcut.
+
+When doing server side copies, normally rclone will copy shortcuts as
+shortcuts.
+
+If this flag is used then rclone will copy the contents of shortcuts
+rather than shortcuts themselves when doing server side copies.
Properties:
-- Config: bucket_acl
-- Env Var: RCLONE_GCS_BUCKET_ACL
-- Type: string
-- Required: false
-- Examples:
- - "authenticatedRead"
- - Project team owners get OWNER access.
- - All Authenticated Users get READER access.
- - "private"
- - Project team owners get OWNER access.
- - Default if left blank.
- - "projectPrivate"
- - Project team members get access according to their roles.
- - "publicRead"
- - Project team owners get OWNER access.
- - All Users get READER access.
- - "publicReadWrite"
- - Project team owners get OWNER access.
- - All Users get WRITER access.
+- Config: copy_shortcut_content
+- Env Var: RCLONE_DRIVE_COPY_SHORTCUT_CONTENT
+- Type: bool
+- Default: false
-#### --gcs-bucket-policy-only
+#### --drive-skip-gdocs
-Access checks should use bucket-level IAM policies.
+Skip google documents in all listings.
-If you want to upload objects to a bucket with Bucket Policy Only set
-then you will need to set this.
+If given, gdocs practically become invisible to rclone.
-When it is set, rclone:
+Properties:
-- ignores ACLs set on buckets
-- ignores ACLs set on objects
-- creates buckets with Bucket Policy Only set
+- Config: skip_gdocs
+- Env Var: RCLONE_DRIVE_SKIP_GDOCS
+- Type: bool
+- Default: false
-Docs: https://cloud.google.com/storage/docs/bucket-policy-only
+#### --drive-show-all-gdocs
+
+Show all Google Docs including non-exportable ones in listings.
+
+If you try a server side copy on a Google Form without this flag, you
+will get this error:
+
+ No export formats found for "application/vnd.google-apps.form"
+
+However adding this flag will allow the form to be server side copied.
+
+Note that rclone doesn't add extensions to the Google Docs file names
+in this mode.
+
+Do **not** use this flag when trying to download Google Docs - rclone
+will fail to download them.
Properties:
-- Config: bucket_policy_only
-- Env Var: RCLONE_GCS_BUCKET_POLICY_ONLY
+- Config: show_all_gdocs
+- Env Var: RCLONE_DRIVE_SHOW_ALL_GDOCS
- Type: bool
- Default: false
-#### --gcs-location
-
-Location for the newly created buckets.
+#### --drive-skip-checksum-gphotos
-Properties:
+Skip checksums on Google photos and videos only.
-- Config: location
-- Env Var: RCLONE_GCS_LOCATION
-- Type: string
-- Required: false
-- Examples:
- - ""
- - Empty for default location (US)
- - "asia"
- - Multi-regional location for Asia
- - "eu"
- - Multi-regional location for Europe
- - "us"
- - Multi-regional location for United States
- - "asia-east1"
- - Taiwan
- - "asia-east2"
- - Hong Kong
- - "asia-northeast1"
- - Tokyo
- - "asia-northeast2"
- - Osaka
- - "asia-northeast3"
- - Seoul
- - "asia-south1"
- - Mumbai
- - "asia-south2"
- - Delhi
- - "asia-southeast1"
- - Singapore
- - "asia-southeast2"
- - Jakarta
- - "australia-southeast1"
- - Sydney
- - "australia-southeast2"
- - Melbourne
- - "europe-north1"
- - Finland
- - "europe-west1"
- - Belgium
- - "europe-west2"
- - London
- - "europe-west3"
- - Frankfurt
- - "europe-west4"
- - Netherlands
- - "europe-west6"
- - Zürich
- - "europe-central2"
- - Warsaw
- - "us-central1"
- - Iowa
- - "us-east1"
- - South Carolina
- - "us-east4"
- - Northern Virginia
- - "us-west1"
- - Oregon
- - "us-west2"
- - California
- - "us-west3"
- - Salt Lake City
- - "us-west4"
- - Las Vegas
- - "northamerica-northeast1"
- - Montréal
- - "northamerica-northeast2"
- - Toronto
- - "southamerica-east1"
- - São Paulo
- - "southamerica-west1"
- - Santiago
- - "asia1"
- - Dual region: asia-northeast1 and asia-northeast2.
- - "eur4"
- - Dual region: europe-north1 and europe-west4.
- - "nam4"
- - Dual region: us-central1 and us-east1.
+Use this if you get checksum errors when transferring Google photos or
+videos.
-#### --gcs-storage-class
+Setting this flag will cause Google photos and videos to return a
+blank checksums.
-The storage class to use when storing objects in Google Cloud Storage.
+Google photos are identified by being in the "photos" space.
+
+Corrupted checksums are caused by Google modifying the image/video but
+not updating the checksum.
Properties:
-- Config: storage_class
-- Env Var: RCLONE_GCS_STORAGE_CLASS
-- Type: string
-- Required: false
-- Examples:
- - ""
- - Default
- - "MULTI_REGIONAL"
- - Multi-regional storage class
- - "REGIONAL"
- - Regional storage class
- - "NEARLINE"
- - Nearline storage class
- - "COLDLINE"
- - Coldline storage class
- - "ARCHIVE"
- - Archive storage class
- - "DURABLE_REDUCED_AVAILABILITY"
- - Durable reduced availability storage class
+- Config: skip_checksum_gphotos
+- Env Var: RCLONE_DRIVE_SKIP_CHECKSUM_GPHOTOS
+- Type: bool
+- Default: false
-#### --gcs-env-auth
+#### --drive-shared-with-me
-Get GCP IAM credentials from runtime (environment variables or instance meta data if no env vars).
+Only show files that are shared with me.
-Only applies if service_account_file and service_account_credentials is blank.
+Instructs rclone to operate on your "Shared with me" folder (where
+Google Drive lets you access the files and folders others have shared
+with you).
+
+This works both with the "list" (lsd, lsl, etc.) and the "copy"
+commands (copy, sync, etc.), and with all other commands too.
Properties:
-- Config: env_auth
-- Env Var: RCLONE_GCS_ENV_AUTH
+- Config: shared_with_me
+- Env Var: RCLONE_DRIVE_SHARED_WITH_ME
- Type: bool
- Default: false
-- Examples:
- - "false"
- - Enter credentials in the next step.
- - "true"
- - Get GCP IAM credentials from the environment (env vars or IAM).
-### Advanced options
-
-Here are the Advanced options specific to google cloud storage (Google Cloud Storage (this is not Google Drive)).
+#### --drive-trashed-only
-#### --gcs-token
+Only show files that are in the trash.
-OAuth Access Token as a JSON blob.
+This will show trashed files in their original directory structure.
Properties:
-- Config: token
-- Env Var: RCLONE_GCS_TOKEN
-- Type: string
-- Required: false
+- Config: trashed_only
+- Env Var: RCLONE_DRIVE_TRASHED_ONLY
+- Type: bool
+- Default: false
-#### --gcs-auth-url
+#### --drive-starred-only
-Auth server URL.
+Only show files that are starred.
-Leave blank to use the provider defaults.
+Properties:
+
+- Config: starred_only
+- Env Var: RCLONE_DRIVE_STARRED_ONLY
+- Type: bool
+- Default: false
+
+#### --drive-formats
+
+Deprecated: See export_formats.
Properties:
-- Config: auth_url
-- Env Var: RCLONE_GCS_AUTH_URL
+- Config: formats
+- Env Var: RCLONE_DRIVE_FORMATS
- Type: string
- Required: false
-#### --gcs-token-url
+#### --drive-export-formats
-Token server url.
+Comma separated list of preferred formats for downloading Google docs.
-Leave blank to use the provider defaults.
+Properties:
+
+- Config: export_formats
+- Env Var: RCLONE_DRIVE_EXPORT_FORMATS
+- Type: string
+- Default: "docx,xlsx,pptx,svg"
+
+#### --drive-import-formats
+
+Comma separated list of preferred formats for uploading Google docs.
Properties:
-- Config: token_url
-- Env Var: RCLONE_GCS_TOKEN_URL
+- Config: import_formats
+- Env Var: RCLONE_DRIVE_IMPORT_FORMATS
- Type: string
- Required: false
-#### --gcs-directory-markers
-
-Upload an empty object with a trailing slash when a new directory is created
+#### --drive-allow-import-name-change
-Empty folders are unsupported for bucket based remotes, this option creates an empty
-object ending with "/", to persist the folder.
+Allow the filetype to change when uploading Google docs.
+E.g. file.doc to file.docx. This will confuse sync and reupload every time.
Properties:
-- Config: directory_markers
-- Env Var: RCLONE_GCS_DIRECTORY_MARKERS
+- Config: allow_import_name_change
+- Env Var: RCLONE_DRIVE_ALLOW_IMPORT_NAME_CHANGE
- Type: bool
- Default: false
-#### --gcs-no-check-bucket
+#### --drive-use-created-date
-If set, don't attempt to check the bucket exists or create it.
+Use file created date instead of modified date.
-This can be useful when trying to minimise the number of transactions
-rclone does if you know the bucket exists already.
+Useful when downloading data and you want the creation date used in
+place of the last modified date.
+
+**WARNING**: This flag may have some unexpected consequences.
+When uploading to your drive all files will be overwritten unless they
+haven't been modified since their creation. And the inverse will occur
+while downloading. This side effect can be avoided by using the
+"--checksum" flag.
+
+This feature was implemented to retain photos capture date as recorded
+by google photos. You will first need to check the "Create a Google
+Photos folder" option in your google drive settings. You can then copy
+or move the photos locally and use the date the image was taken
+(created) set as the modification date.
Properties:
-- Config: no_check_bucket
-- Env Var: RCLONE_GCS_NO_CHECK_BUCKET
+- Config: use_created_date
+- Env Var: RCLONE_DRIVE_USE_CREATED_DATE
- Type: bool
- Default: false
-#### --gcs-decompress
-
-If set this will decompress gzip encoded objects.
+#### --drive-use-shared-date
-It is possible to upload objects to GCS with "Content-Encoding: gzip"
-set. Normally rclone will download these files as compressed objects.
+Use date file was shared instead of modified date.
-If this flag is set then rclone will decompress these files with
-"Content-Encoding: gzip" as they are received. This means that rclone
-can't check the size and hash but the file contents will be decompressed.
+Note that, as with "--drive-use-created-date", this flag may have
+unexpected consequences when uploading/downloading files.
+If both this flag and "--drive-use-created-date" are set, the created
+date is used.
Properties:
-- Config: decompress
-- Env Var: RCLONE_GCS_DECOMPRESS
+- Config: use_shared_date
+- Env Var: RCLONE_DRIVE_USE_SHARED_DATE
- Type: bool
- Default: false
-#### --gcs-endpoint
+#### --drive-list-chunk
-Endpoint for the service.
+Size of listing chunk 100-1000, 0 to disable.
-Leave blank normally.
+Properties:
+
+- Config: list_chunk
+- Env Var: RCLONE_DRIVE_LIST_CHUNK
+- Type: int
+- Default: 1000
+
+#### --drive-impersonate
+
+Impersonate this user when using a service account.
Properties:
-- Config: endpoint
-- Env Var: RCLONE_GCS_ENDPOINT
+- Config: impersonate
+- Env Var: RCLONE_DRIVE_IMPERSONATE
- Type: string
- Required: false
-#### --gcs-encoding
+#### --drive-upload-cutoff
-The encoding for the backend.
+Cutoff for switching to chunked upload.
-See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info.
+Properties:
+
+- Config: upload_cutoff
+- Env Var: RCLONE_DRIVE_UPLOAD_CUTOFF
+- Type: SizeSuffix
+- Default: 8Mi
+
+#### --drive-chunk-size
+
+Upload chunk size.
+
+Must a power of 2 >= 256k.
+
+Making this larger will improve performance, but note that each chunk
+is buffered in memory one per transfer.
+
+Reducing this will reduce memory usage but decrease performance.
Properties:
-- Config: encoding
-- Env Var: RCLONE_GCS_ENCODING
-- Type: MultiEncoder
-- Default: Slash,CrLf,InvalidUtf8,Dot
+- Config: chunk_size
+- Env Var: RCLONE_DRIVE_CHUNK_SIZE
+- Type: SizeSuffix
+- Default: 8Mi
+#### --drive-acknowledge-abuse
+Set to allow files which return cannotDownloadAbusiveFile to be downloaded.
-## Limitations
+If downloading a file returns the error "This file has been identified
+as malware or spam and cannot be downloaded" with the error code
+"cannotDownloadAbusiveFile" then supply this flag to rclone to
+indicate you acknowledge the risks of downloading the file and rclone
+will download it anyway.
-`rclone about` is not supported by the Google Cloud Storage backend. Backends without
-this capability cannot determine free space for an rclone mount or
-use policy `mfs` (most free space) as a member of an rclone union
-remote.
+Note that if you are using service account it will need Manager
+permission (not Content Manager) to for this flag to work. If the SA
+does not have the right permission, Google will just ignore the flag.
-See [List of backends that do not support rclone about](https://rclone.org/overview/#optional-features) and [rclone about](https://rclone.org/commands/rclone_about/)
+Properties:
-# Google Drive
+- Config: acknowledge_abuse
+- Env Var: RCLONE_DRIVE_ACKNOWLEDGE_ABUSE
+- Type: bool
+- Default: false
-Paths are specified as `drive:path`
+#### --drive-keep-revision-forever
-Drive paths may be as deep as required, e.g. `drive:directory/subdirectory`.
+Keep new head revision of each file forever.
-## Configuration
+Properties:
-The initial setup for drive involves getting a token from Google drive
-which you need to do in your browser. `rclone config` walks you
-through it.
+- Config: keep_revision_forever
+- Env Var: RCLONE_DRIVE_KEEP_REVISION_FOREVER
+- Type: bool
+- Default: false
-Here is an example of how to make a remote called `remote`. First run:
+#### --drive-size-as-quota
- rclone config
+Show sizes as storage quota usage, not actual size.
-This will guide you through an interactive setup process:
+Show the size of a file as the storage quota used. This is the
+current version plus any older versions that have been set to keep
+forever.
-```
-No remotes found, make a new one?
-n) New remote
-r) Rename remote
-c) Copy remote
-s) Set configuration password
-q) Quit config
-n/r/c/s/q> n
-name> remote
-Type of storage to configure.
-Choose a number from below, or type in your own value
-[snip]
-XX / Google Drive
- \ "drive"
-[snip]
-Storage> drive
-Google Application Client Id - leave blank normally.
-client_id>
-Google Application Client Secret - leave blank normally.
-client_secret>
-Scope that rclone should use when requesting access from drive.
-Choose a number from below, or type in your own value
- 1 / Full access all files, excluding Application Data Folder.
- \ "drive"
- 2 / Read-only access to file metadata and file contents.
- \ "drive.readonly"
- / Access to files created by rclone only.
- 3 | These are visible in the drive website.
- | File authorization is revoked when the user deauthorizes the app.
- \ "drive.file"
- / Allows read and write access to the Application Data folder.
- 4 | This is not visible in the drive website.
- \ "drive.appfolder"
- / Allows read-only access to file metadata but
- 5 | does not allow any access to read or download file content.
- \ "drive.metadata.readonly"
-scope> 1
-Service Account Credentials JSON file path - needed only if you want use SA instead of interactive login.
-service_account_file>
-Remote config
-Use web browser to automatically authenticate rclone with remote?
- * Say Y if the machine running rclone has a web browser you can use
- * Say N if running rclone on a (remote) machine without web browser access
-If not sure try Y. If Y failed, try N.
-y) Yes
-n) No
-y/n> y
-If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth
-Log in and authorize rclone for access
-Waiting for code...
-Got code
-Configure this as a Shared Drive (Team Drive)?
-y) Yes
-n) No
-y/n> n
---------------------
-[remote]
-client_id =
-client_secret =
-scope = drive
-root_folder_id =
-service_account_file =
-token = {"access_token":"XXX","token_type":"Bearer","refresh_token":"XXX","expiry":"2014-03-16T13:57:58.955387075Z"}
---------------------
-y) Yes this is OK
-e) Edit this remote
-d) Delete this remote
-y/e/d> y
-```
+**WARNING**: This flag may have some unexpected consequences.
-See the [remote setup docs](https://rclone.org/remote_setup/) for how to set it up on a
-machine with no Internet browser available.
+It is not recommended to set this flag in your config - the
+recommended usage is using the flag form --drive-size-as-quota when
+doing rclone ls/lsl/lsf/lsjson/etc only.
-Note that rclone runs a webserver on your local machine to collect the
-token as returned from Google if using web browser to automatically
-authenticate. This only
-runs from the moment it opens your browser to the moment you get back
-the verification code. This is on `http://127.0.0.1:53682/` and it
-may require you to unblock it temporarily if you are running a host
-firewall, or use manual mode.
+If you do use this flag for syncing (not recommended) then you will
+need to use --ignore size also.
-You can then use it like this,
+Properties:
-List directories in top level of your drive
+- Config: size_as_quota
+- Env Var: RCLONE_DRIVE_SIZE_AS_QUOTA
+- Type: bool
+- Default: false
- rclone lsd remote:
+#### --drive-v2-download-min-size
-List all the files in your drive
+If Object's are greater, use drive v2 API to download.
- rclone ls remote:
+Properties:
-To copy a local directory to a drive directory called backup
+- Config: v2_download_min_size
+- Env Var: RCLONE_DRIVE_V2_DOWNLOAD_MIN_SIZE
+- Type: SizeSuffix
+- Default: off
- rclone copy /home/source remote:backup
+#### --drive-pacer-min-sleep
-### Scopes
+Minimum time to sleep between API calls.
-Rclone allows you to select which scope you would like for rclone to
-use. This changes what type of token is granted to rclone. [The
-scopes are defined
-here](https://developers.google.com/drive/v3/web/about-auth).
+Properties:
-The scope are
+- Config: pacer_min_sleep
+- Env Var: RCLONE_DRIVE_PACER_MIN_SLEEP
+- Type: Duration
+- Default: 100ms
-#### drive
+#### --drive-pacer-burst
-This is the default scope and allows full access to all files, except
-for the Application Data Folder (see below).
+Number of API calls to allow without sleeping.
-Choose this one if you aren't sure.
+Properties:
-#### drive.readonly
+- Config: pacer_burst
+- Env Var: RCLONE_DRIVE_PACER_BURST
+- Type: int
+- Default: 100
-This allows read only access to all files. Files may be listed and
-downloaded but not uploaded, renamed or deleted.
+#### --drive-server-side-across-configs
-#### drive.file
+Deprecated: use --server-side-across-configs instead.
-With this scope rclone can read/view/modify only those files and
-folders it creates.
+Allow server-side operations (e.g. copy) to work across different drive configs.
-So if you uploaded files to drive via the web interface (or any other
-means) they will not be visible to rclone.
+This can be useful if you wish to do a server-side copy between two
+different Google drives. Note that this isn't enabled by default
+because it isn't easy to tell if it will work between any two
+configurations.
-This can be useful if you are using rclone to backup data and you want
-to be sure confidential data on your drive is not visible to rclone.
+Properties:
-Files created with this scope are visible in the web interface.
+- Config: server_side_across_configs
+- Env Var: RCLONE_DRIVE_SERVER_SIDE_ACROSS_CONFIGS
+- Type: bool
+- Default: false
-#### drive.appfolder
+#### --drive-disable-http2
-This gives rclone its own private area to store files. Rclone will
-not be able to see any other files on your drive and you won't be able
-to see rclone's files from the web interface either.
+Disable drive using http2.
-#### drive.metadata.readonly
+There is currently an unsolved issue with the google drive backend and
+HTTP/2. HTTP/2 is therefore disabled by default for the drive backend
+but can be re-enabled here. When the issue is solved this flag will
+be removed.
-This allows read only access to file names only. It does not allow
-rclone to download or upload data, or rename or delete files or
-directories.
+See: https://github.com/rclone/rclone/issues/3631
-### Root folder ID
-This option has been moved to the advanced section. You can set the `root_folder_id` for rclone. This is the directory
-(identified by its `Folder ID`) that rclone considers to be the root
-of your drive.
-Normally you will leave this blank and rclone will determine the
-correct root to use itself.
+Properties:
-However you can set this to restrict rclone to a specific folder
-hierarchy or to access data within the "Computers" tab on the drive
-web interface (where files from Google's Backup and Sync desktop
-program go).
+- Config: disable_http2
+- Env Var: RCLONE_DRIVE_DISABLE_HTTP2
+- Type: bool
+- Default: true
-In order to do this you will have to find the `Folder ID` of the
-directory you wish rclone to display. This will be the last segment
-of the URL when you open the relevant folder in the drive web
-interface.
+#### --drive-stop-on-upload-limit
-So if the folder you want rclone to use has a URL which looks like
-`https://drive.google.com/drive/folders/1XyfxxxxxxxxxxxxxxxxxxxxxxxxxKHCh`
-in the browser, then you use `1XyfxxxxxxxxxxxxxxxxxxxxxxxxxKHCh` as
-the `root_folder_id` in the config.
+Make upload limit errors be fatal.
-**NB** folders under the "Computers" tab seem to be read only (drive
-gives a 500 error) when using rclone.
+At the time of writing it is only possible to upload 750 GiB of data to
+Google Drive a day (this is an undocumented limit). When this limit is
+reached Google Drive produces a slightly different error message. When
+this flag is set it causes these errors to be fatal. These will stop
+the in-progress sync.
-There doesn't appear to be an API to discover the folder IDs of the
-"Computers" tab - please contact us if you know otherwise!
+Note that this detection is relying on error message strings which
+Google don't document so it may break in the future.
-Note also that rclone can't access any data under the "Backups" tab on
-the google drive web interface yet.
+See: https://github.com/rclone/rclone/issues/3857
-### Service Account support
-You can set up rclone with Google Drive in an unattended mode,
-i.e. not tied to a specific end-user Google account. This is useful
-when you want to synchronise files onto machines that don't have
-actively logged-in users, for example build machines.
+Properties:
-To use a Service Account instead of OAuth2 token flow, enter the path
-to your Service Account credentials at the `service_account_file`
-prompt during `rclone config` and rclone won't use the browser based
-authentication flow. If you'd rather stuff the contents of the
-credentials file into the rclone config file, you can set
-`service_account_credentials` with the actual contents of the file
-instead, or set the equivalent environment variable.
+- Config: stop_on_upload_limit
+- Env Var: RCLONE_DRIVE_STOP_ON_UPLOAD_LIMIT
+- Type: bool
+- Default: false
-#### Use case - Google Apps/G-suite account and individual Drive
+#### --drive-stop-on-download-limit
-Let's say that you are the administrator of a Google Apps (old) or
-G-suite account.
-The goal is to store data on an individual's Drive account, who IS
-a member of the domain.
-We'll call the domain **example.com**, and the user
-**foo@example.com**.
+Make download limit errors be fatal.
+
+At the time of writing it is only possible to download 10 TiB of data from
+Google Drive a day (this is an undocumented limit). When this limit is
+reached Google Drive produces a slightly different error message. When
+this flag is set it causes these errors to be fatal. These will stop
+the in-progress sync.
+
+Note that this detection is relying on error message strings which
+Google don't document so it may break in the future.
+
+
+Properties:
+
+- Config: stop_on_download_limit
+- Env Var: RCLONE_DRIVE_STOP_ON_DOWNLOAD_LIMIT
+- Type: bool
+- Default: false
+
+#### --drive-skip-shortcuts
+
+If set skip shortcut files.
+
+Normally rclone dereferences shortcut files making them appear as if
+they are the original file (see [the shortcuts section](#shortcuts)).
+If this flag is set then rclone will ignore shortcut files completely.
+
+
+Properties:
+
+- Config: skip_shortcuts
+- Env Var: RCLONE_DRIVE_SKIP_SHORTCUTS
+- Type: bool
+- Default: false
-There's a few steps we need to go through to accomplish this:
+#### --drive-skip-dangling-shortcuts
-##### 1. Create a service account for example.com
- - To create a service account and obtain its credentials, go to the
-[Google Developer Console](https://console.developers.google.com).
- - You must have a project - create one if you don't.
- - Then go to "IAM & admin" -> "Service Accounts".
- - Use the "Create Service Account" button. Fill in "Service account name"
-and "Service account ID" with something that identifies your client.
- - Select "Create And Continue". Step 2 and 3 are optional.
- - These credentials are what rclone will use for authentication.
-If you ever need to remove access, press the "Delete service
-account key" button.
+If set skip dangling shortcut files.
-##### 2. Allowing API access to example.com Google Drive
- - Go to example.com's admin console
- - Go into "Security" (or use the search bar)
- - Select "Show more" and then "Advanced settings"
- - Select "Manage API client access" in the "Authentication" section
- - In the "Client Name" field enter the service account's
-"Client ID" - this can be found in the Developer Console under
-"IAM & Admin" -> "Service Accounts", then "View Client ID" for
-the newly created service account.
-It is a ~21 character numerical string.
- - In the next field, "One or More API Scopes", enter
-`https://www.googleapis.com/auth/drive`
-to grant access to Google Drive specifically.
+If this is set then rclone will not show any dangling shortcuts in listings.
-##### 3. Configure rclone, assuming a new install
-```
-rclone config
+Properties:
-n/s/q> n # New
-name>gdrive # Gdrive is an example name
-Storage> # Select the number shown for Google Drive
-client_id> # Can be left blank
-client_secret> # Can be left blank
-scope> # Select your scope, 1 for example
-root_folder_id> # Can be left blank
-service_account_file> /home/foo/myJSONfile.json # This is where the JSON file goes!
-y/n> # Auto config, n
+- Config: skip_dangling_shortcuts
+- Env Var: RCLONE_DRIVE_SKIP_DANGLING_SHORTCUTS
+- Type: bool
+- Default: false
-```
+#### --drive-resource-key
-##### 4. Verify that it's working
- - `rclone -v --drive-impersonate foo@example.com lsf gdrive:backup`
- - The arguments do:
- - `-v` - verbose logging
- - `--drive-impersonate foo@example.com` - this is what does
-the magic, pretending to be user foo.
- - `lsf` - list files in a parsing friendly way
- - `gdrive:backup` - use the remote called gdrive, work in
-the folder named backup.
+Resource key for accessing a link-shared file.
-Note: in case you configured a specific root folder on gdrive and rclone is unable to access the contents of that folder when using `--drive-impersonate`, do this instead:
- - in the gdrive web interface, share your root folder with the user/email of the new Service Account you created/selected at step #1
- - use rclone without specifying the `--drive-impersonate` option, like this:
- `rclone -v lsf gdrive:backup`
+If you need to access files shared with a link like this
+ https://drive.google.com/drive/folders/XXX?resourcekey=YYY&usp=sharing
-### Shared drives (team drives)
+Then you will need to use the first part "XXX" as the "root_folder_id"
+and the second part "YYY" as the "resource_key" otherwise you will get
+404 not found errors when trying to access the directory.
-If you want to configure the remote to point to a Google Shared Drive
-(previously known as Team Drives) then answer `y` to the question
-`Configure this as a Shared Drive (Team Drive)?`.
+See: https://developers.google.com/drive/api/guides/resource-keys
-This will fetch the list of Shared Drives from google and allow you to
-configure which one you want to use. You can also type in a Shared
-Drive ID if you prefer.
+This resource key requirement only applies to a subset of old files.
-For example:
+Note also that opening the folder once in the web interface (with the
+user you've authenticated rclone with) seems to be enough so that the
+resource key is not needed.
-```
-Configure this as a Shared Drive (Team Drive)?
-y) Yes
-n) No
-y/n> y
-Fetching Shared Drive list...
-Choose a number from below, or type in your own value
- 1 / Rclone Test
- \ "xxxxxxxxxxxxxxxxxxxx"
- 2 / Rclone Test 2
- \ "yyyyyyyyyyyyyyyyyyyy"
- 3 / Rclone Test 3
- \ "zzzzzzzzzzzzzzzzzzzz"
-Enter a Shared Drive ID> 1
---------------------
-[remote]
-client_id =
-client_secret =
-token = {"AccessToken":"xxxx.x.xxxxx_xxxxxxxxxxx_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","RefreshToken":"1/xxxxxxxxxxxxxxxx_xxxxxxxxxxxxxxxxxxxxxxxxxx","Expiry":"2014-03-16T13:57:58.955387075Z","Extra":null}
-team_drive = xxxxxxxxxxxxxxxxxxxx
---------------------
-y) Yes this is OK
-e) Edit this remote
-d) Delete this remote
-y/e/d> y
-```
-### --fast-list
+Properties:
-This remote supports `--fast-list` which allows you to use fewer
-transactions in exchange for more memory. See the [rclone
-docs](https://rclone.org/docs/#fast-list) for more details.
+- Config: resource_key
+- Env Var: RCLONE_DRIVE_RESOURCE_KEY
+- Type: string
+- Required: false
-It does this by combining multiple `list` calls into a single API request.
+#### --drive-fast-list-bug-fix
-This works by combining many `'%s' in parents` filters into one expression.
-To list the contents of directories a, b and c, the following requests will be send by the regular `List` function:
-```
-trashed=false and 'a' in parents
-trashed=false and 'b' in parents
-trashed=false and 'c' in parents
-```
-These can now be combined into a single request:
-```
-trashed=false and ('a' in parents or 'b' in parents or 'c' in parents)
-```
+Work around a bug in Google Drive listing.
-The implementation of `ListR` will put up to 50 `parents` filters into one request.
-It will use the `--checkers` value to specify the number of requests to run in parallel.
+Normally rclone will work around a bug in Google Drive when using
+--fast-list (ListR) where the search "(A in parents) or (B in
+parents)" returns nothing sometimes. See #3114, #4289 and
+https://issuetracker.google.com/issues/149522397
-In tests, these batch requests were up to 20x faster than the regular method.
-Running the following command against different sized folders gives:
-```
-rclone lsjson -vv -R --checkers=6 gdrive:folder
-```
+Rclone detects this by finding no items in more than one directory
+when listing and retries them as lists of individual directories.
-small folder (220 directories, 700 files):
+This means that if you have a lot of empty directories rclone will end
+up listing them all individually and this can take many more API
+calls.
-- without `--fast-list`: 38s
-- with `--fast-list`: 10s
+This flag allows the work-around to be disabled. This is **not**
+recommended in normal use - only if you have a particular case you are
+having trouble with like many empty directories.
-large folder (10600 directories, 39000 files):
-- without `--fast-list`: 22:05 min
-- with `--fast-list`: 58s
+Properties:
-### Modified time
+- Config: fast_list_bug_fix
+- Env Var: RCLONE_DRIVE_FAST_LIST_BUG_FIX
+- Type: bool
+- Default: true
-Google drive stores modification times accurate to 1 ms.
+#### --drive-metadata-owner
-### Restricted filename characters
+Control whether owner should be read or written in metadata.
-Only Invalid UTF-8 bytes will be [replaced](https://rclone.org/overview/#invalid-utf8),
-as they can't be used in JSON strings.
+Owner is a standard part of the file metadata so is easy to read. But it
+isn't always desirable to set the owner from the metadata.
-In contrast to other backends, `/` can also be used in names and `.`
-or `..` are valid names.
+Note that you can't set the owner on Shared Drives, and that setting
+ownership will generate an email to the new owner (this can't be
+disabled), and you can't transfer ownership to someone outside your
+organization.
-### Revisions
-Google drive stores revisions of files. When you upload a change to
-an existing file to google drive using rclone it will create a new
-revision of that file.
+Properties:
-Revisions follow the standard google policy which at time of writing
-was
+- Config: metadata_owner
+- Env Var: RCLONE_DRIVE_METADATA_OWNER
+- Type: Bits
+- Default: read
+- Examples:
+ - "off"
+ - Do not read or write the value
+ - "read"
+ - Read the value only
+ - "write"
+ - Write the value only
+ - "read,write"
+ - Read and Write the value.
- * They are deleted after 30 days or 100 revisions (whatever comes first).
- * They do not count towards a user storage quota.
+#### --drive-metadata-permissions
-### Deleting files
+Control whether permissions should be read or written in metadata.
-By default rclone will send all files to the trash when deleting
-files. If deleting them permanently is required then use the
-`--drive-use-trash=false` flag, or set the equivalent environment
-variable.
+Reading permissions metadata from files can be done quickly, but it
+isn't always desirable to set the permissions from the metadata.
-### Shortcuts
+Note that rclone drops any inherited permissions on Shared Drives and
+any owner permission on My Drives as these are duplicated in the owner
+metadata.
-In March 2020 Google introduced a new feature in Google Drive called
-[drive shortcuts](https://support.google.com/drive/answer/9700156)
-([API](https://developers.google.com/drive/api/v3/shortcuts)). These
-will (by September 2020) [replace the ability for files or folders to
-be in multiple folders at once](https://cloud.google.com/blog/products/g-suite/simplifying-google-drives-folder-structure-and-sharing-models).
-Shortcuts are files that link to other files on Google Drive somewhat
-like a symlink in unix, except they point to the underlying file data
-(e.g. the inode in unix terms) so they don't break if the source is
-renamed or moved about.
+Properties:
-By default rclone treats these as follows.
+- Config: metadata_permissions
+- Env Var: RCLONE_DRIVE_METADATA_PERMISSIONS
+- Type: Bits
+- Default: off
+- Examples:
+ - "off"
+ - Do not read or write the value
+ - "read"
+ - Read the value only
+ - "write"
+ - Write the value only
+ - "read,write"
+ - Read and Write the value.
-For shortcuts pointing to files:
+#### --drive-metadata-labels
-- When listing a file shortcut appears as the destination file.
-- When downloading the contents of the destination file is downloaded.
-- When updating shortcut file with a non shortcut file, the shortcut is removed then a new file is uploaded in place of the shortcut.
-- When server-side moving (renaming) the shortcut is renamed, not the destination file.
-- When server-side copying the shortcut is copied, not the contents of the shortcut. (unless `--drive-copy-shortcut-content` is in use in which case the contents of the shortcut gets copied).
-- When deleting the shortcut is deleted not the linked file.
-- When setting the modification time, the modification time of the linked file will be set.
+Control whether labels should be read or written in metadata.
-For shortcuts pointing to folders:
+Reading labels metadata from files takes an extra API transaction and
+will slow down listings. It isn't always desirable to set the labels
+from the metadata.
-- When listing the shortcut appears as a folder and that folder will contain the contents of the linked folder appear (including any sub folders)
-- When downloading the contents of the linked folder and sub contents are downloaded
-- When uploading to a shortcut folder the file will be placed in the linked folder
-- When server-side moving (renaming) the shortcut is renamed, not the destination folder
-- When server-side copying the contents of the linked folder is copied, not the shortcut.
-- When deleting with `rclone rmdir` or `rclone purge` the shortcut is deleted not the linked folder.
-- **NB** When deleting with `rclone remove` or `rclone mount` the contents of the linked folder will be deleted.
+The format of labels is documented in the drive API documentation at
+https://developers.google.com/drive/api/reference/rest/v3/Label -
+rclone just provides a JSON dump of this format.
-The [rclone backend](https://rclone.org/commands/rclone_backend/) command can be used to create shortcuts.
+When setting labels, the label and fields must already exist - rclone
+will not create them. This means that if you are transferring labels
+from two different accounts you will have to create the labels in
+advance and use the metadata mapper to translate the IDs between the
+two accounts.
-Shortcuts can be completely ignored with the `--drive-skip-shortcuts` flag
-or the corresponding `skip_shortcuts` configuration setting.
-### Emptying trash
+Properties:
-If you wish to empty your trash you can use the `rclone cleanup remote:`
-command which will permanently delete all your trashed files. This command
-does not take any path arguments.
+- Config: metadata_labels
+- Env Var: RCLONE_DRIVE_METADATA_LABELS
+- Type: Bits
+- Default: off
+- Examples:
+ - "off"
+ - Do not read or write the value
+ - "read"
+ - Read the value only
+ - "write"
+ - Write the value only
+ - "read,write"
+ - Read and Write the value.
-Note that Google Drive takes some time (minutes to days) to empty the
-trash even though the command returns within a few seconds. No output
-is echoed, so there will be no confirmation even using -v or -vv.
+#### --drive-encoding
-### Quota information
+The encoding for the backend.
-To view your current quota you can use the `rclone about remote:`
-command which will display your usage limit (quota), the usage in Google
-Drive, the size of all files in the Trash and the space used by other
-Google services such as Gmail. This command does not take any path
-arguments.
+See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info.
-#### Import/Export of google documents
+Properties:
-Google documents can be exported from and uploaded to Google Drive.
+- Config: encoding
+- Env Var: RCLONE_DRIVE_ENCODING
+- Type: Encoding
+- Default: InvalidUtf8
-When rclone downloads a Google doc it chooses a format to download
-depending upon the `--drive-export-formats` setting.
-By default the export formats are `docx,xlsx,pptx,svg` which are a
-sensible default for an editable document.
+#### --drive-env-auth
-When choosing a format, rclone runs down the list provided in order
-and chooses the first file format the doc can be exported as from the
-list. If the file can't be exported to a format on the formats list,
-then rclone will choose a format from the default list.
+Get IAM credentials from runtime (environment variables or instance meta data if no env vars).
-If you prefer an archive copy then you might use `--drive-export-formats
-pdf`, or if you prefer openoffice/libreoffice formats you might use
-`--drive-export-formats ods,odt,odp`.
+Only applies if service_account_file and service_account_credentials is blank.
-Note that rclone adds the extension to the google doc, so if it is
-called `My Spreadsheet` on google docs, it will be exported as `My
-Spreadsheet.xlsx` or `My Spreadsheet.pdf` etc.
+Properties:
-When importing files into Google Drive, rclone will convert all
-files with an extension in `--drive-import-formats` to their
-associated document type.
-rclone will not convert any files by default, since the conversion
-is lossy process.
+- Config: env_auth
+- Env Var: RCLONE_DRIVE_ENV_AUTH
+- Type: bool
+- Default: false
+- Examples:
+ - "false"
+ - Enter credentials in the next step.
+ - "true"
+ - Get GCP IAM credentials from the environment (env vars or IAM).
-The conversion must result in a file with the same extension when
-the `--drive-export-formats` rules are applied to the uploaded document.
+### Metadata
-Here are some examples for allowed and prohibited conversions.
+User metadata is stored in the properties field of the drive object.
-| export-formats | import-formats | Upload Ext | Document Ext | Allowed |
-| -------------- | -------------- | ---------- | ------------ | ------- |
-| odt | odt | odt | odt | Yes |
-| odt | docx,odt | odt | odt | Yes |
-| | docx | docx | docx | Yes |
-| | odt | odt | docx | No |
-| odt,docx | docx,odt | docx | odt | No |
-| docx,odt | docx,odt | docx | docx | Yes |
-| docx,odt | docx,odt | odt | docx | No |
+Here are the possible system metadata items for the drive backend.
-This limitation can be disabled by specifying `--drive-allow-import-name-change`.
-When using this flag, rclone can convert multiple files types resulting
-in the same document type at once, e.g. with `--drive-import-formats docx,odt,txt`,
-all files having these extension would result in a document represented as a docx file.
-This brings the additional risk of overwriting a document, if multiple files
-have the same stem. Many rclone operations will not handle this name change
-in any way. They assume an equal name when copying files and might copy the
-file again or delete them when the name changes.
+| Name | Help | Type | Example | Read Only |
+|------|------|------|---------|-----------|
+| btime | Time of file birth (creation) with mS accuracy. Note that this is only writable on fresh uploads - it can't be written for updates. | RFC 3339 | 2006-01-02T15:04:05.999Z07:00 | N |
+| content-type | The MIME type of the file. | string | text/plain | N |
+| copy-requires-writer-permission | Whether the options to copy, print, or download this file, should be disabled for readers and commenters. | boolean | true | N |
+| description | A short description of the file. | string | Contract for signing | N |
+| folder-color-rgb | The color for a folder or a shortcut to a folder as an RGB hex string. | string | 881133 | N |
+| labels | Labels attached to this file in a JSON dump of Googled drive format. Enable with --drive-metadata-labels. | JSON | [] | N |
+| mtime | Time of last modification with mS accuracy. | RFC 3339 | 2006-01-02T15:04:05.999Z07:00 | N |
+| owner | The owner of the file. Usually an email address. Enable with --drive-metadata-owner. | string | user@example.com | N |
+| permissions | Permissions in a JSON dump of Google drive format. On shared drives these will only be present if they aren't inherited. Enable with --drive-metadata-permissions. | JSON | {} | N |
+| starred | Whether the user has starred the file. | boolean | false | N |
+| viewed-by-me | Whether the file has been viewed by this user. | boolean | true | **Y** |
+| writers-can-share | Whether users with only writer permission can modify the file's permissions. Not populated for items in shared drives. | boolean | false | N |
-Here are the possible export extensions with their corresponding mime types.
-Most of these can also be used for importing, but there more that are not
-listed here. Some of these additional ones might only be available when
-the operating system provides the correct MIME type entries.
+See the [metadata](https://rclone.org/docs/#metadata) docs for more info.
-This list can be changed by Google Drive at any time and might not
-represent the currently available conversions.
+## Backend commands
-| Extension | Mime Type | Description |
-| --------- |-----------| ------------|
-| bmp | image/bmp | Windows Bitmap format |
-| csv | text/csv | Standard CSV format for Spreadsheets |
-| doc | application/msword | Classic Word file |
-| docx | application/vnd.openxmlformats-officedocument.wordprocessingml.document | Microsoft Office Document |
-| epub | application/epub+zip | E-book format |
-| html | text/html | An HTML Document |
-| jpg | image/jpeg | A JPEG Image File |
-| json | application/vnd.google-apps.script+json | JSON Text Format for Google Apps scripts |
-| odp | application/vnd.oasis.opendocument.presentation | Openoffice Presentation |
-| ods | application/vnd.oasis.opendocument.spreadsheet | Openoffice Spreadsheet |
-| ods | application/x-vnd.oasis.opendocument.spreadsheet | Openoffice Spreadsheet |
-| odt | application/vnd.oasis.opendocument.text | Openoffice Document |
-| pdf | application/pdf | Adobe PDF Format |
-| pjpeg | image/pjpeg | Progressive JPEG Image |
-| png | image/png | PNG Image Format|
-| pptx | application/vnd.openxmlformats-officedocument.presentationml.presentation | Microsoft Office Powerpoint |
-| rtf | application/rtf | Rich Text Format |
-| svg | image/svg+xml | Scalable Vector Graphics Format |
-| tsv | text/tab-separated-values | Standard TSV format for spreadsheets |
-| txt | text/plain | Plain Text |
-| wmf | application/x-msmetafile | Windows Meta File |
-| xls | application/vnd.ms-excel | Classic Excel file |
-| xlsx | application/vnd.openxmlformats-officedocument.spreadsheetml.sheet | Microsoft Office Spreadsheet |
-| zip | application/zip | A ZIP file of HTML, Images CSS |
+Here are the commands specific to the drive backend.
-Google documents can also be exported as link files. These files will
-open a browser window for the Google Docs website of that document
-when opened. The link file extension has to be specified as a
-`--drive-export-formats` parameter. They will match all available
-Google Documents.
+Run them with
-| Extension | Description | OS Support |
-| --------- | ----------- | ---------- |
-| desktop | freedesktop.org specified desktop entry | Linux |
-| link.html | An HTML Document with a redirect | All |
-| url | INI style link file | macOS, Windows |
-| webloc | macOS specific XML format | macOS |
+ rclone backend COMMAND remote:
+The help below will explain what arguments each command takes.
-### Standard options
+See the [backend](https://rclone.org/commands/rclone_backend/) command for more
+info on how to pass options and arguments.
-Here are the Standard options specific to drive (Google Drive).
+These can be run on a running backend using the rc command
+[backend/command](https://rclone.org/rc/#backend-command).
-#### --drive-client-id
+### get
-Google Application Client Id
-Setting your own is recommended.
-See https://rclone.org/drive/#making-your-own-client-id for how to create your own.
-If you leave this blank, it will use an internal key which is low performance.
+Get command for fetching the drive config parameters
-Properties:
+ rclone backend get remote: [options] [+]
-- Config: client_id
-- Env Var: RCLONE_DRIVE_CLIENT_ID
-- Type: string
-- Required: false
+This is a get command which will be used to fetch the various drive config parameters
-#### --drive-client-secret
+Usage Examples:
-OAuth Client Secret.
+ rclone backend get drive: [-o service_account_file] [-o chunk_size]
+ rclone rc backend/command command=get fs=drive: [-o service_account_file] [-o chunk_size]
-Leave blank normally.
-Properties:
+Options:
-- Config: client_secret
-- Env Var: RCLONE_DRIVE_CLIENT_SECRET
-- Type: string
-- Required: false
+- "chunk_size": show the current upload chunk size
+- "service_account_file": show the current service account file
-#### --drive-scope
+### set
-Scope that rclone should use when requesting access from drive.
+Set command for updating the drive config parameters
-Properties:
+ rclone backend set remote: [options] [+]
-- Config: scope
-- Env Var: RCLONE_DRIVE_SCOPE
-- Type: string
-- Required: false
-- Examples:
- - "drive"
- - Full access all files, excluding Application Data Folder.
- - "drive.readonly"
- - Read-only access to file metadata and file contents.
- - "drive.file"
- - Access to files created by rclone only.
- - These are visible in the drive website.
- - File authorization is revoked when the user deauthorizes the app.
- - "drive.appfolder"
- - Allows read and write access to the Application Data folder.
- - This is not visible in the drive website.
- - "drive.metadata.readonly"
- - Allows read-only access to file metadata but
- - does not allow any access to read or download file content.
+This is a set command which will be used to update the various drive config parameters
-#### --drive-service-account-file
+Usage Examples:
-Service Account Credentials JSON file path.
+ rclone backend set drive: [-o service_account_file=sa.json] [-o chunk_size=67108864]
+ rclone rc backend/command command=set fs=drive: [-o service_account_file=sa.json] [-o chunk_size=67108864]
-Leave blank normally.
-Needed only if you want use SA instead of interactive login.
-Leading `~` will be expanded in the file name as will environment variables such as `${RCLONE_CONFIG_DIR}`.
+Options:
-Properties:
+- "chunk_size": update the current upload chunk size
+- "service_account_file": update the current service account file
-- Config: service_account_file
-- Env Var: RCLONE_DRIVE_SERVICE_ACCOUNT_FILE
-- Type: string
-- Required: false
+### shortcut
-#### --drive-alternate-export
+Create shortcuts from files or directories
-Deprecated: No longer needed.
+ rclone backend shortcut remote: [options] [+]
-Properties:
+This command creates shortcuts from files or directories.
-- Config: alternate_export
-- Env Var: RCLONE_DRIVE_ALTERNATE_EXPORT
-- Type: bool
-- Default: false
+Usage:
-### Advanced options
+ rclone backend shortcut drive: source_item destination_shortcut
+ rclone backend shortcut drive: source_item -o target=drive2: destination_shortcut
-Here are the Advanced options specific to drive (Google Drive).
+In the first example this creates a shortcut from the "source_item"
+which can be a file or a directory to the "destination_shortcut". The
+"source_item" and the "destination_shortcut" should be relative paths
+from "drive:"
-#### --drive-token
+In the second example this creates a shortcut from the "source_item"
+relative to "drive:" to the "destination_shortcut" relative to
+"drive2:". This may fail with a permission error if the user
+authenticated with "drive2:" can't read files from "drive:".
-OAuth Access Token as a JSON blob.
-Properties:
+Options:
-- Config: token
-- Env Var: RCLONE_DRIVE_TOKEN
-- Type: string
-- Required: false
+- "target": optional target remote for the shortcut destination
-#### --drive-auth-url
+### drives
-Auth server URL.
+List the Shared Drives available to this account
-Leave blank to use the provider defaults.
+ rclone backend drives remote: [options] [+]
-Properties:
+This command lists the Shared Drives (Team Drives) available to this
+account.
-- Config: auth_url
-- Env Var: RCLONE_DRIVE_AUTH_URL
-- Type: string
-- Required: false
+Usage:
-#### --drive-token-url
+ rclone backend [-o config] drives drive:
-Token server url.
+This will return a JSON list of objects like this
-Leave blank to use the provider defaults.
+ [
+ {
+ "id": "0ABCDEF-01234567890",
+ "kind": "drive#teamDrive",
+ "name": "My Drive"
+ },
+ {
+ "id": "0ABCDEFabcdefghijkl",
+ "kind": "drive#teamDrive",
+ "name": "Test Drive"
+ }
+ ]
-Properties:
+With the -o config parameter it will output the list in a format
+suitable for adding to a config file to make aliases for all the
+drives found and a combined drive.
-- Config: token_url
-- Env Var: RCLONE_DRIVE_TOKEN_URL
-- Type: string
-- Required: false
+ [My Drive]
+ type = alias
+ remote = drive,team_drive=0ABCDEF-01234567890,root_folder_id=:
-#### --drive-root-folder-id
+ [Test Drive]
+ type = alias
+ remote = drive,team_drive=0ABCDEFabcdefghijkl,root_folder_id=:
+
+ [AllDrives]
+ type = combine
+ upstreams = "My Drive=My Drive:" "Test Drive=Test Drive:"
+
+Adding this to the rclone config file will cause those team drives to
+be accessible with the aliases shown. Any illegal characters will be
+substituted with "_" and duplicate names will have numbers suffixed.
+It will also add a remote called AllDrives which shows all the shared
+drives combined into one directory tree.
-ID of the root folder.
-Leave blank normally.
-Fill in to access "Computers" folders (see docs), or for rclone to use
-a non root folder as its starting point.
+### untrash
+Untrash files and directories
-Properties:
+ rclone backend untrash remote: [options] [+]
-- Config: root_folder_id
-- Env Var: RCLONE_DRIVE_ROOT_FOLDER_ID
-- Type: string
-- Required: false
+This command untrashes all the files and directories in the directory
+passed in recursively.
-#### --drive-service-account-credentials
+Usage:
-Service Account Credentials JSON blob.
+This takes an optional directory to trash which make this easier to
+use via the API.
-Leave blank normally.
-Needed only if you want use SA instead of interactive login.
+ rclone backend untrash drive:directory
+ rclone backend --interactive untrash drive:directory subdir
-Properties:
+Use the --interactive/-i or --dry-run flag to see what would be restored before restoring it.
-- Config: service_account_credentials
-- Env Var: RCLONE_DRIVE_SERVICE_ACCOUNT_CREDENTIALS
-- Type: string
-- Required: false
+Result:
-#### --drive-team-drive
+ {
+ "Untrashed": 17,
+ "Errors": 0
+ }
-ID of the Shared Drive (Team Drive).
-Properties:
+### copyid
-- Config: team_drive
-- Env Var: RCLONE_DRIVE_TEAM_DRIVE
-- Type: string
-- Required: false
+Copy files by ID
-#### --drive-auth-owner-only
+ rclone backend copyid remote: [options] [+]
-Only consider files owned by the authenticated user.
+This command copies files by ID
-Properties:
+Usage:
-- Config: auth_owner_only
-- Env Var: RCLONE_DRIVE_AUTH_OWNER_ONLY
-- Type: bool
-- Default: false
+ rclone backend copyid drive: ID path
+ rclone backend copyid drive: ID1 path1 ID2 path2
-#### --drive-use-trash
+It copies the drive file with ID given to the path (an rclone path which
+will be passed internally to rclone copyto). The ID and path pairs can be
+repeated.
-Send files to the trash instead of deleting permanently.
+The path should end with a / to indicate copy the file as named to
+this directory. If it doesn't end with a / then the last path
+component will be used as the file name.
-Defaults to true, namely sending files to the trash.
-Use `--drive-use-trash=false` to delete files permanently instead.
+If the destination is a drive backend then server-side copying will be
+attempted if possible.
-Properties:
+Use the --interactive/-i or --dry-run flag to see what would be copied before copying.
-- Config: use_trash
-- Env Var: RCLONE_DRIVE_USE_TRASH
-- Type: bool
-- Default: true
-#### --drive-copy-shortcut-content
+### exportformats
-Server side copy contents of shortcuts instead of the shortcut.
+Dump the export formats for debug purposes
-When doing server side copies, normally rclone will copy shortcuts as
-shortcuts.
+ rclone backend exportformats remote: [options] [+]
-If this flag is used then rclone will copy the contents of shortcuts
-rather than shortcuts themselves when doing server side copies.
+### importformats
-Properties:
+Dump the import formats for debug purposes
-- Config: copy_shortcut_content
-- Env Var: RCLONE_DRIVE_COPY_SHORTCUT_CONTENT
-- Type: bool
-- Default: false
+ rclone backend importformats remote: [options] [+]
-#### --drive-skip-gdocs
-Skip google documents in all listings.
-If given, gdocs practically become invisible to rclone.
+## Limitations
-Properties:
+Drive has quite a lot of rate limiting. This causes rclone to be
+limited to transferring about 2 files per second only. Individual
+files may be transferred much faster at 100s of MiB/s but lots of
+small files can take a long time.
-- Config: skip_gdocs
-- Env Var: RCLONE_DRIVE_SKIP_GDOCS
-- Type: bool
-- Default: false
+Server side copies are also subject to a separate rate limit. If you
+see User rate limit exceeded errors, wait at least 24 hours and retry.
+You can disable server-side copies with `--disable copy` to download
+and upload the files if you prefer.
-#### --drive-skip-checksum-gphotos
+### Limitations of Google Docs
-Skip MD5 checksum on Google photos and videos only.
+Google docs will appear as size -1 in `rclone ls`, `rclone ncdu` etc,
+and as size 0 in anything which uses the VFS layer, e.g. `rclone mount`
+and `rclone serve`. When calculating directory totals, e.g. in
+`rclone size` and `rclone ncdu`, they will be counted in as empty
+files.
-Use this if you get checksum errors when transferring Google photos or
-videos.
+This is because rclone can't find out the size of the Google docs
+without downloading them.
-Setting this flag will cause Google photos and videos to return a
-blank MD5 checksum.
+Google docs will transfer correctly with `rclone sync`, `rclone copy`
+etc as rclone knows to ignore the size when doing the transfer.
-Google photos are identified by being in the "photos" space.
+However an unfortunate consequence of this is that you may not be able
+to download Google docs using `rclone mount`. If it doesn't work you
+will get a 0 sized file. If you try again the doc may gain its
+correct size and be downloadable. Whether it will work on not depends
+on the application accessing the mount and the OS you are running -
+experiment to find out if it does work for you!
-Corrupted checksums are caused by Google modifying the image/video but
-not updating the checksum.
+### Duplicated files
-Properties:
+Sometimes, for no reason I've been able to track down, drive will
+duplicate a file that rclone uploads. Drive unlike all the other
+remotes can have duplicated files.
-- Config: skip_checksum_gphotos
-- Env Var: RCLONE_DRIVE_SKIP_CHECKSUM_GPHOTOS
-- Type: bool
-- Default: false
+Duplicated files cause problems with the syncing and you will see
+messages in the log about duplicates.
-#### --drive-shared-with-me
+Use `rclone dedupe` to fix duplicated files.
-Only show files that are shared with me.
+Note that this isn't just a problem with rclone, even Google Photos on
+Android duplicates files on drive sometimes.
-Instructs rclone to operate on your "Shared with me" folder (where
-Google Drive lets you access the files and folders others have shared
-with you).
+### Rclone appears to be re-copying files it shouldn't
-This works both with the "list" (lsd, lsl, etc.) and the "copy"
-commands (copy, sync, etc.), and with all other commands too.
+The most likely cause of this is the duplicated file issue above - run
+`rclone dedupe` and check your logs for duplicate object or directory
+messages.
-Properties:
+This can also be caused by a delay/caching on google drive's end when
+comparing directory listings. Specifically with team drives used in
+combination with --fast-list. Files that were uploaded recently may
+not appear on the directory list sent to rclone when using --fast-list.
-- Config: shared_with_me
-- Env Var: RCLONE_DRIVE_SHARED_WITH_ME
-- Type: bool
-- Default: false
+Waiting a moderate period of time between attempts (estimated to be
+approximately 1 hour) and/or not using --fast-list both seem to be
+effective in preventing the problem.
-#### --drive-trashed-only
+### SHA1 or SHA256 hashes may be missing
-Only show files that are in the trash.
+All files have MD5 hashes, but a small fraction of files uploaded may
+not have SHA1 or SHA256 hashes especially if they were uploaded before 2018.
-This will show trashed files in their original directory structure.
+## Making your own client_id
-Properties:
+When you use rclone with Google drive in its default configuration you
+are using rclone's client_id. This is shared between all the rclone
+users. There is a global rate limit on the number of queries per
+second that each client_id can do set by Google. rclone already has a
+high quota and I will continue to make sure it is high enough by
+contacting Google.
-- Config: trashed_only
-- Env Var: RCLONE_DRIVE_TRASHED_ONLY
-- Type: bool
-- Default: false
+It is strongly recommended to use your own client ID as the default rclone ID is heavily used. If you have multiple services running, it is recommended to use an API key for each service. The default Google quota is 10 transactions per second so it is recommended to stay under that number as if you use more than that, it will cause rclone to rate limit and make things slower.
-#### --drive-starred-only
+Here is how to create your own Google Drive client ID for rclone:
-Only show files that are starred.
+1. Log into the [Google API
+Console](https://console.developers.google.com/) with your Google
+account. It doesn't matter what Google account you use. (It need not
+be the same account as the Google Drive you want to access)
-Properties:
+2. Select a project or create a new project.
-- Config: starred_only
-- Env Var: RCLONE_DRIVE_STARRED_ONLY
-- Type: bool
-- Default: false
+3. Under "ENABLE APIS AND SERVICES" search for "Drive", and enable the
+"Google Drive API".
-#### --drive-formats
+4. Click "Credentials" in the left-side panel (not "Create
+credentials", which opens the wizard).
-Deprecated: See export_formats.
+5. If you already configured an "Oauth Consent Screen", then skip
+to the next step; if not, click on "CONFIGURE CONSENT SCREEN" button
+(near the top right corner of the right panel), then select "External"
+and click on "CREATE"; on the next screen, enter an "Application name"
+("rclone" is OK); enter "User Support Email" (your own email is OK);
+enter "Developer Contact Email" (your own email is OK); then click on
+"Save" (all other data is optional). You will also have to add some scopes,
+including `.../auth/docs` and `.../auth/drive` in order to be able to edit,
+create and delete files with RClone. You may also want to include the
+`../auth/drive.metadata.readonly` scope. After adding scopes, click
+"Save and continue" to add test users. Be sure to add your own account to
+the test users. Once you've added yourself as a test user and saved the
+changes, click again on "Credentials" on the left panel to go back to
+the "Credentials" screen.
-Properties:
+ (PS: if you are a GSuite user, you could also select "Internal" instead
+of "External" above, but this will restrict API use to Google Workspace
+users in your organisation).
-- Config: formats
-- Env Var: RCLONE_DRIVE_FORMATS
-- Type: string
-- Required: false
+6. Click on the "+ CREATE CREDENTIALS" button at the top of the screen,
+then select "OAuth client ID".
-#### --drive-export-formats
+7. Choose an application type of "Desktop app" and click "Create". (the default name is fine)
-Comma separated list of preferred formats for downloading Google docs.
+8. It will show you a client ID and client secret. Make a note of these.
+
+ (If you selected "External" at Step 5 continue to Step 9.
+ If you chose "Internal" you don't need to publish and can skip straight to
+ Step 10 but your destination drive must be part of the same Google Workspace.)
-Properties:
+9. Go to "Oauth consent screen" and then click "PUBLISH APP" button and confirm.
+ You will also want to add yourself as a test user.
-- Config: export_formats
-- Env Var: RCLONE_DRIVE_EXPORT_FORMATS
-- Type: string
-- Default: "docx,xlsx,pptx,svg"
+10. Provide the noted client ID and client secret to rclone.
-#### --drive-import-formats
+Be aware that, due to the "enhanced security" recently introduced by
+Google, you are theoretically expected to "submit your app for verification"
+and then wait a few weeks(!) for their response; in practice, you can go right
+ahead and use the client ID and client secret with rclone, the only issue will
+be a very scary confirmation screen shown when you connect via your browser
+for rclone to be able to get its token-id (but as this only happens during
+the remote configuration, it's not such a big deal). Keeping the application in
+"Testing" will work as well, but the limitation is that any grants will expire
+after a week, which can be annoying to refresh constantly. If, for whatever
+reason, a short grant time is not a problem, then keeping the application in
+testing mode would also be sufficient.
-Comma separated list of preferred formats for uploading Google docs.
+(Thanks to @balazer on github for these instructions.)
-Properties:
+Sometimes, creation of an OAuth consent in Google API Console fails due to an error message
+“The request failed because changes to one of the field of the resource is not supportedâ€.
+As a convenient workaround, the necessary Google Drive API key can be created on the
+[Python Quickstart](https://developers.google.com/drive/api/v3/quickstart/python) page.
+Just push the Enable the Drive API button to receive the Client ID and Secret.
+Note that it will automatically create a new project in the API Console.
-- Config: import_formats
-- Env Var: RCLONE_DRIVE_IMPORT_FORMATS
-- Type: string
-- Required: false
+# Google Photos
-#### --drive-allow-import-name-change
+The rclone backend for [Google Photos](https://www.google.com/photos/about/) is
+a specialized backend for transferring photos and videos to and from
+Google Photos.
-Allow the filetype to change when uploading Google docs.
+**NB** The Google Photos API which rclone uses has quite a few
+limitations, so please read the [limitations section](#limitations)
+carefully to make sure it is suitable for your use.
-E.g. file.doc to file.docx. This will confuse sync and reupload every time.
+## Configuration
-Properties:
+The initial setup for google cloud storage involves getting a token from Google Photos
+which you need to do in your browser. `rclone config` walks you
+through it.
-- Config: allow_import_name_change
-- Env Var: RCLONE_DRIVE_ALLOW_IMPORT_NAME_CHANGE
-- Type: bool
-- Default: false
+Here is an example of how to make a remote called `remote`. First run:
-#### --drive-use-created-date
+ rclone config
-Use file created date instead of modified date.
+This will guide you through an interactive setup process:
-Useful when downloading data and you want the creation date used in
-place of the last modified date.
+```
+No remotes found, make a new one?
+n) New remote
+s) Set configuration password
+q) Quit config
+n/s/q> n
+name> remote
+Type of storage to configure.
+Enter a string value. Press Enter for the default ("").
+Choose a number from below, or type in your own value
+[snip]
+XX / Google Photos
+ \ "google photos"
+[snip]
+Storage> google photos
+** See help for google photos backend at: https://rclone.org/googlephotos/ **
-**WARNING**: This flag may have some unexpected consequences.
+Google Application Client Id
+Leave blank normally.
+Enter a string value. Press Enter for the default ("").
+client_id>
+Google Application Client Secret
+Leave blank normally.
+Enter a string value. Press Enter for the default ("").
+client_secret>
+Set to make the Google Photos backend read only.
-When uploading to your drive all files will be overwritten unless they
-haven't been modified since their creation. And the inverse will occur
-while downloading. This side effect can be avoided by using the
-"--checksum" flag.
+If you choose read only then rclone will only request read only access
+to your photos, otherwise rclone will request full access.
+Enter a boolean value (true or false). Press Enter for the default ("false").
+read_only>
+Edit advanced config? (y/n)
+y) Yes
+n) No
+y/n> n
+Remote config
+Use web browser to automatically authenticate rclone with remote?
+ * Say Y if the machine running rclone has a web browser you can use
+ * Say N if running rclone on a (remote) machine without web browser access
+If not sure try Y. If Y failed, try N.
+y) Yes
+n) No
+y/n> y
+If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth
+Log in and authorize rclone for access
+Waiting for code...
+Got code
-This feature was implemented to retain photos capture date as recorded
-by google photos. You will first need to check the "Create a Google
-Photos folder" option in your google drive settings. You can then copy
-or move the photos locally and use the date the image was taken
-(created) set as the modification date.
+*** IMPORTANT: All media items uploaded to Google Photos with rclone
+*** are stored in full resolution at original quality. These uploads
+*** will count towards storage in your Google Account.
-Properties:
+--------------------
+[remote]
+type = google photos
+token = {"access_token":"XXX","token_type":"Bearer","refresh_token":"XXX","expiry":"2019-06-28T17:38:04.644930156+01:00"}
+--------------------
+y) Yes this is OK
+e) Edit this remote
+d) Delete this remote
+y/e/d> y
+```
-- Config: use_created_date
-- Env Var: RCLONE_DRIVE_USE_CREATED_DATE
-- Type: bool
-- Default: false
+See the [remote setup docs](https://rclone.org/remote_setup/) for how to set it up on a
+machine with no Internet browser available.
-#### --drive-use-shared-date
+Note that rclone runs a webserver on your local machine to collect the
+token as returned from Google if using web browser to automatically
+authenticate. This only
+runs from the moment it opens your browser to the moment you get back
+the verification code. This is on `http://127.0.0.1:53682/` and this
+may require you to unblock it temporarily if you are running a host
+firewall, or use manual mode.
-Use date file was shared instead of modified date.
+This remote is called `remote` and can now be used like this
-Note that, as with "--drive-use-created-date", this flag may have
-unexpected consequences when uploading/downloading files.
+See all the albums in your photos
-If both this flag and "--drive-use-created-date" are set, the created
-date is used.
+ rclone lsd remote:album
-Properties:
+Make a new album
-- Config: use_shared_date
-- Env Var: RCLONE_DRIVE_USE_SHARED_DATE
-- Type: bool
-- Default: false
+ rclone mkdir remote:album/newAlbum
-#### --drive-list-chunk
+List the contents of an album
-Size of listing chunk 100-1000, 0 to disable.
+ rclone ls remote:album/newAlbum
-Properties:
+Sync `/home/local/images` to the Google Photos, removing any excess
+files in the album.
-- Config: list_chunk
-- Env Var: RCLONE_DRIVE_LIST_CHUNK
-- Type: int
-- Default: 1000
+ rclone sync --interactive /home/local/image remote:album/newAlbum
-#### --drive-impersonate
+### Layout
-Impersonate this user when using a service account.
+As Google Photos is not a general purpose cloud storage system, the
+backend is laid out to help you navigate it.
-Properties:
+The directories under `media` show different ways of categorizing the
+media. Each file will appear multiple times. So if you want to make
+a backup of your google photos you might choose to backup
+`remote:media/by-month`. (**NB** `remote:media/by-day` is rather slow
+at the moment so avoid for syncing.)
-- Config: impersonate
-- Env Var: RCLONE_DRIVE_IMPERSONATE
-- Type: string
-- Required: false
+Note that all your photos and videos will appear somewhere under
+`media`, but they may not appear under `album` unless you've put them
+into albums.
-#### --drive-upload-cutoff
+```
+/
+- upload
+ - file1.jpg
+ - file2.jpg
+ - ...
+- media
+ - all
+ - file1.jpg
+ - file2.jpg
+ - ...
+ - by-year
+ - 2000
+ - file1.jpg
+ - ...
+ - 2001
+ - file2.jpg
+ - ...
+ - ...
+ - by-month
+ - 2000
+ - 2000-01
+ - file1.jpg
+ - ...
+ - 2000-02
+ - file2.jpg
+ - ...
+ - ...
+ - by-day
+ - 2000
+ - 2000-01-01
+ - file1.jpg
+ - ...
+ - 2000-01-02
+ - file2.jpg
+ - ...
+ - ...
+- album
+ - album name
+ - album name/sub
+- shared-album
+ - album name
+ - album name/sub
+- feature
+ - favorites
+ - file1.jpg
+ - file2.jpg
+```
-Cutoff for switching to chunked upload.
+There are two writable parts of the tree, the `upload` directory and
+sub directories of the `album` directory.
-Properties:
+The `upload` directory is for uploading files you don't want to put
+into albums. This will be empty to start with and will contain the
+files you've uploaded for one rclone session only, becoming empty
+again when you restart rclone. The use case for this would be if you
+have a load of files you just want to once off dump into Google
+Photos. For repeated syncing, uploading to `album` will work better.
-- Config: upload_cutoff
-- Env Var: RCLONE_DRIVE_UPLOAD_CUTOFF
-- Type: SizeSuffix
-- Default: 8Mi
+Directories within the `album` directory are also writeable and you
+may create new directories (albums) under `album`. If you copy files
+with a directory hierarchy in there then rclone will create albums
+with the `/` character in them. For example if you do
-#### --drive-chunk-size
+ rclone copy /path/to/images remote:album/images
-Upload chunk size.
+and the images directory contains
-Must a power of 2 >= 256k.
+```
+images
+ - file1.jpg
+ dir
+ file2.jpg
+ dir2
+ dir3
+ file3.jpg
+```
-Making this larger will improve performance, but note that each chunk
-is buffered in memory one per transfer.
+Then rclone will create the following albums with the following files in
-Reducing this will reduce memory usage but decrease performance.
+- images
+ - file1.jpg
+- images/dir
+ - file2.jpg
+- images/dir2/dir3
+ - file3.jpg
-Properties:
+This means that you can use the `album` path pretty much like a normal
+filesystem and it is a good target for repeated syncing.
-- Config: chunk_size
-- Env Var: RCLONE_DRIVE_CHUNK_SIZE
-- Type: SizeSuffix
-- Default: 8Mi
+The `shared-album` directory shows albums shared with you or by you.
+This is similar to the Sharing tab in the Google Photos web interface.
-#### --drive-acknowledge-abuse
-Set to allow files which return cannotDownloadAbusiveFile to be downloaded.
+### Standard options
-If downloading a file returns the error "This file has been identified
-as malware or spam and cannot be downloaded" with the error code
-"cannotDownloadAbusiveFile" then supply this flag to rclone to
-indicate you acknowledge the risks of downloading the file and rclone
-will download it anyway.
+Here are the Standard options specific to google photos (Google Photos).
-Note that if you are using service account it will need Manager
-permission (not Content Manager) to for this flag to work. If the SA
-does not have the right permission, Google will just ignore the flag.
+#### --gphotos-client-id
-Properties:
+OAuth Client Id.
-- Config: acknowledge_abuse
-- Env Var: RCLONE_DRIVE_ACKNOWLEDGE_ABUSE
-- Type: bool
-- Default: false
+Leave blank normally.
-#### --drive-keep-revision-forever
+Properties:
-Keep new head revision of each file forever.
+- Config: client_id
+- Env Var: RCLONE_GPHOTOS_CLIENT_ID
+- Type: string
+- Required: false
-Properties:
+#### --gphotos-client-secret
-- Config: keep_revision_forever
-- Env Var: RCLONE_DRIVE_KEEP_REVISION_FOREVER
-- Type: bool
-- Default: false
+OAuth Client Secret.
-#### --drive-size-as-quota
+Leave blank normally.
-Show sizes as storage quota usage, not actual size.
+Properties:
-Show the size of a file as the storage quota used. This is the
-current version plus any older versions that have been set to keep
-forever.
+- Config: client_secret
+- Env Var: RCLONE_GPHOTOS_CLIENT_SECRET
+- Type: string
+- Required: false
-**WARNING**: This flag may have some unexpected consequences.
+#### --gphotos-read-only
-It is not recommended to set this flag in your config - the
-recommended usage is using the flag form --drive-size-as-quota when
-doing rclone ls/lsl/lsf/lsjson/etc only.
+Set to make the Google Photos backend read only.
-If you do use this flag for syncing (not recommended) then you will
-need to use --ignore size also.
+If you choose read only then rclone will only request read only access
+to your photos, otherwise rclone will request full access.
Properties:
-- Config: size_as_quota
-- Env Var: RCLONE_DRIVE_SIZE_AS_QUOTA
+- Config: read_only
+- Env Var: RCLONE_GPHOTOS_READ_ONLY
- Type: bool
- Default: false
-#### --drive-v2-download-min-size
+### Advanced options
-If Object's are greater, use drive v2 API to download.
+Here are the Advanced options specific to google photos (Google Photos).
+
+#### --gphotos-token
+
+OAuth Access Token as a JSON blob.
Properties:
-- Config: v2_download_min_size
-- Env Var: RCLONE_DRIVE_V2_DOWNLOAD_MIN_SIZE
-- Type: SizeSuffix
-- Default: off
+- Config: token
+- Env Var: RCLONE_GPHOTOS_TOKEN
+- Type: string
+- Required: false
-#### --drive-pacer-min-sleep
+#### --gphotos-auth-url
-Minimum time to sleep between API calls.
+Auth server URL.
+
+Leave blank to use the provider defaults.
Properties:
-- Config: pacer_min_sleep
-- Env Var: RCLONE_DRIVE_PACER_MIN_SLEEP
-- Type: Duration
-- Default: 100ms
+- Config: auth_url
+- Env Var: RCLONE_GPHOTOS_AUTH_URL
+- Type: string
+- Required: false
-#### --drive-pacer-burst
+#### --gphotos-token-url
-Number of API calls to allow without sleeping.
+Token server url.
-Properties:
+Leave blank to use the provider defaults.
-- Config: pacer_burst
-- Env Var: RCLONE_DRIVE_PACER_BURST
-- Type: int
-- Default: 100
+Properties:
-#### --drive-server-side-across-configs
+- Config: token_url
+- Env Var: RCLONE_GPHOTOS_TOKEN_URL
+- Type: string
+- Required: false
-Deprecated: use --server-side-across-configs instead.
+#### --gphotos-read-size
-Allow server-side operations (e.g. copy) to work across different drive configs.
+Set to read the size of media items.
-This can be useful if you wish to do a server-side copy between two
-different Google drives. Note that this isn't enabled by default
-because it isn't easy to tell if it will work between any two
-configurations.
+Normally rclone does not read the size of media items since this takes
+another transaction. This isn't necessary for syncing. However
+rclone mount needs to know the size of files in advance of reading
+them, so setting this flag when using rclone mount is recommended if
+you want to read the media.
Properties:
-- Config: server_side_across_configs
-- Env Var: RCLONE_DRIVE_SERVER_SIDE_ACROSS_CONFIGS
+- Config: read_size
+- Env Var: RCLONE_GPHOTOS_READ_SIZE
- Type: bool
- Default: false
-#### --drive-disable-http2
-
-Disable drive using http2.
-
-There is currently an unsolved issue with the google drive backend and
-HTTP/2. HTTP/2 is therefore disabled by default for the drive backend
-but can be re-enabled here. When the issue is solved this flag will
-be removed.
-
-See: https://github.com/rclone/rclone/issues/3631
-
+#### --gphotos-start-year
+Year limits the photos to be downloaded to those which are uploaded after the given year.
Properties:
-- Config: disable_http2
-- Env Var: RCLONE_DRIVE_DISABLE_HTTP2
-- Type: bool
-- Default: true
+- Config: start_year
+- Env Var: RCLONE_GPHOTOS_START_YEAR
+- Type: int
+- Default: 2000
-#### --drive-stop-on-upload-limit
+#### --gphotos-include-archived
-Make upload limit errors be fatal.
+Also view and download archived media.
-At the time of writing it is only possible to upload 750 GiB of data to
-Google Drive a day (this is an undocumented limit). When this limit is
-reached Google Drive produces a slightly different error message. When
-this flag is set it causes these errors to be fatal. These will stop
-the in-progress sync.
+By default, rclone does not request archived media. Thus, when syncing,
+archived media is not visible in directory listings or transferred.
-Note that this detection is relying on error message strings which
-Google don't document so it may break in the future.
+Note that media in albums is always visible and synced, no matter
+their archive status.
-See: https://github.com/rclone/rclone/issues/3857
+With this flag, archived media are always visible in directory
+listings and transferred.
+Without this flag, archived media will not be visible in directory
+listings and won't be transferred.
Properties:
-- Config: stop_on_upload_limit
-- Env Var: RCLONE_DRIVE_STOP_ON_UPLOAD_LIMIT
+- Config: include_archived
+- Env Var: RCLONE_GPHOTOS_INCLUDE_ARCHIVED
- Type: bool
- Default: false
-#### --drive-stop-on-download-limit
-
-Make download limit errors be fatal.
-
-At the time of writing it is only possible to download 10 TiB of data from
-Google Drive a day (this is an undocumented limit). When this limit is
-reached Google Drive produces a slightly different error message. When
-this flag is set it causes these errors to be fatal. These will stop
-the in-progress sync.
+#### --gphotos-encoding
-Note that this detection is relying on error message strings which
-Google don't document so it may break in the future.
+The encoding for the backend.
+See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info.
Properties:
-- Config: stop_on_download_limit
-- Env Var: RCLONE_DRIVE_STOP_ON_DOWNLOAD_LIMIT
-- Type: bool
-- Default: false
-
-#### --drive-skip-shortcuts
+- Config: encoding
+- Env Var: RCLONE_GPHOTOS_ENCODING
+- Type: Encoding
+- Default: Slash,CrLf,InvalidUtf8,Dot
-If set skip shortcut files.
+#### --gphotos-batch-mode
-Normally rclone dereferences shortcut files making them appear as if
-they are the original file (see [the shortcuts section](#shortcuts)).
-If this flag is set then rclone will ignore shortcut files completely.
+Upload file batching sync|async|off.
+This sets the batch mode used by rclone.
-Properties:
+This has 3 possible values
-- Config: skip_shortcuts
-- Env Var: RCLONE_DRIVE_SKIP_SHORTCUTS
-- Type: bool
-- Default: false
+- off - no batching
+- sync - batch uploads and check completion (default)
+- async - batch upload and don't check completion
-#### --drive-skip-dangling-shortcuts
+Rclone will close any outstanding batches when it exits which may make
+a delay on quit.
-If set skip dangling shortcut files.
-If this is set then rclone will not show any dangling shortcuts in listings.
+Properties:
+- Config: batch_mode
+- Env Var: RCLONE_GPHOTOS_BATCH_MODE
+- Type: string
+- Default: "sync"
-Properties:
+#### --gphotos-batch-size
-- Config: skip_dangling_shortcuts
-- Env Var: RCLONE_DRIVE_SKIP_DANGLING_SHORTCUTS
-- Type: bool
-- Default: false
+Max number of files in upload batch.
-#### --drive-resource-key
+This sets the batch size of files to upload. It has to be less than 50.
-Resource key for accessing a link-shared file.
+By default this is 0 which means rclone which calculate the batch size
+depending on the setting of batch_mode.
-If you need to access files shared with a link like this
+- batch_mode: async - default batch_size is 50
+- batch_mode: sync - default batch_size is the same as --transfers
+- batch_mode: off - not in use
- https://drive.google.com/drive/folders/XXX?resourcekey=YYY&usp=sharing
+Rclone will close any outstanding batches when it exits which may make
+a delay on quit.
-Then you will need to use the first part "XXX" as the "root_folder_id"
-and the second part "YYY" as the "resource_key" otherwise you will get
-404 not found errors when trying to access the directory.
+Setting this is a great idea if you are uploading lots of small files
+as it will make them a lot quicker. You can use --transfers 32 to
+maximise throughput.
-See: https://developers.google.com/drive/api/guides/resource-keys
-This resource key requirement only applies to a subset of old files.
+Properties:
-Note also that opening the folder once in the web interface (with the
-user you've authenticated rclone with) seems to be enough so that the
-resource key is not needed.
+- Config: batch_size
+- Env Var: RCLONE_GPHOTOS_BATCH_SIZE
+- Type: int
+- Default: 0
+#### --gphotos-batch-timeout
-Properties:
+Max time to allow an idle upload batch before uploading.
-- Config: resource_key
-- Env Var: RCLONE_DRIVE_RESOURCE_KEY
-- Type: string
-- Required: false
+If an upload batch is idle for more than this long then it will be
+uploaded.
-#### --drive-fast-list-bug-fix
+The default for this is 0 which means rclone will choose a sensible
+default based on the batch_mode in use.
-Work around a bug in Google Drive listing.
+- batch_mode: async - default batch_timeout is 10s
+- batch_mode: sync - default batch_timeout is 1s
+- batch_mode: off - not in use
-Normally rclone will work around a bug in Google Drive when using
---fast-list (ListR) where the search "(A in parents) or (B in
-parents)" returns nothing sometimes. See #3114, #4289 and
-https://issuetracker.google.com/issues/149522397
-Rclone detects this by finding no items in more than one directory
-when listing and retries them as lists of individual directories.
+Properties:
-This means that if you have a lot of empty directories rclone will end
-up listing them all individually and this can take many more API
-calls.
+- Config: batch_timeout
+- Env Var: RCLONE_GPHOTOS_BATCH_TIMEOUT
+- Type: Duration
+- Default: 0s
-This flag allows the work-around to be disabled. This is **not**
-recommended in normal use - only if you have a particular case you are
-having trouble with like many empty directories.
+#### --gphotos-batch-commit-timeout
+Max time to wait for a batch to finish committing
Properties:
-- Config: fast_list_bug_fix
-- Env Var: RCLONE_DRIVE_FAST_LIST_BUG_FIX
-- Type: bool
-- Default: true
+- Config: batch_commit_timeout
+- Env Var: RCLONE_GPHOTOS_BATCH_COMMIT_TIMEOUT
+- Type: Duration
+- Default: 10m0s
-#### --drive-encoding
-The encoding for the backend.
-See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info.
+## Limitations
-Properties:
+Only images and videos can be uploaded. If you attempt to upload non
+videos or images or formats that Google Photos doesn't understand,
+rclone will upload the file, then Google Photos will give an error
+when it is put turned into a media item.
-- Config: encoding
-- Env Var: RCLONE_DRIVE_ENCODING
-- Type: MultiEncoder
-- Default: InvalidUtf8
+Note that all media items uploaded to Google Photos through the API
+are stored in full resolution at "original quality" and **will** count
+towards your storage quota in your Google Account. The API does
+**not** offer a way to upload in "high quality" mode..
-#### --drive-env-auth
+`rclone about` is not supported by the Google Photos backend. Backends without
+this capability cannot determine free space for an rclone mount or
+use policy `mfs` (most free space) as a member of an rclone union
+remote.
-Get IAM credentials from runtime (environment variables or instance meta data if no env vars).
+See [List of backends that do not support rclone about](https://rclone.org/overview/#optional-features)
+See [rclone about](https://rclone.org/commands/rclone_about/)
-Only applies if service_account_file and service_account_credentials is blank.
+### Downloading Images
-Properties:
+When Images are downloaded this strips EXIF location (according to the
+docs and my tests). This is a limitation of the Google Photos API and
+is covered by [bug #112096115](https://issuetracker.google.com/issues/112096115).
-- Config: env_auth
-- Env Var: RCLONE_DRIVE_ENV_AUTH
-- Type: bool
-- Default: false
-- Examples:
- - "false"
- - Enter credentials in the next step.
- - "true"
- - Get GCP IAM credentials from the environment (env vars or IAM).
+**The current google API does not allow photos to be downloaded at original resolution. This is very important if you are, for example, relying on "Google Photos" as a backup of your photos. You will not be able to use rclone to redownload original images. You could use 'google takeout' to recover the original photos as a last resort**
-## Backend commands
+### Downloading Videos
-Here are the commands specific to the drive backend.
+When videos are downloaded they are downloaded in a really compressed
+version of the video compared to downloading it via the Google Photos
+web interface. This is covered by [bug #113672044](https://issuetracker.google.com/issues/113672044).
-Run them with
+### Duplicates
- rclone backend COMMAND remote:
+If a file name is duplicated in a directory then rclone will add the
+file ID into its name. So two files called `file.jpg` would then
+appear as `file {123456}.jpg` and `file {ABCDEF}.jpg` (the actual IDs
+are a lot longer alas!).
-The help below will explain what arguments each command takes.
+If you upload the same image (with the same binary data) twice then
+Google Photos will deduplicate it. However it will retain the
+filename from the first upload which may confuse rclone. For example
+if you uploaded an image to `upload` then uploaded the same image to
+`album/my_album` the filename of the image in `album/my_album` will be
+what it was uploaded with initially, not what you uploaded it with to
+`album`. In practise this shouldn't cause too many problems.
-See the [backend](https://rclone.org/commands/rclone_backend/) command for more
-info on how to pass options and arguments.
+### Modification times
-These can be run on a running backend using the rc command
-[backend/command](https://rclone.org/rc/#backend-command).
+The date shown of media in Google Photos is the creation date as
+determined by the EXIF information, or the upload date if that is not
+known.
-### get
+This is not changeable by rclone and is not the modification date of
+the media on local disk. This means that rclone cannot use the dates
+from Google Photos for syncing purposes.
-Get command for fetching the drive config parameters
+### Size
- rclone backend get remote: [options] [+]
+The Google Photos API does not return the size of media. This means
+that when syncing to Google Photos, rclone can only do a file
+existence check.
-This is a get command which will be used to fetch the various drive config parameters
+It is possible to read the size of the media, but this needs an extra
+HTTP HEAD request per media item so is **very slow** and uses up a lot of
+transactions. This can be enabled with the `--gphotos-read-size`
+option or the `read_size = true` config parameter.
-Usage Examples:
+If you want to use the backend with `rclone mount` you may need to
+enable this flag (depending on your OS and application using the
+photos) otherwise you may not be able to read media off the mount.
+You'll need to experiment to see if it works for you without the flag.
- rclone backend get drive: [-o service_account_file] [-o chunk_size]
- rclone rc backend/command command=get fs=drive: [-o service_account_file] [-o chunk_size]
+### Albums
+Rclone can only upload files to albums it created. This is a
+[limitation of the Google Photos API](https://developers.google.com/photos/library/guides/manage-albums).
-Options:
+Rclone can remove files it uploaded from albums it created only.
-- "chunk_size": show the current upload chunk size
-- "service_account_file": show the current service account file
+### Deleting files
-### set
+Rclone can remove files from albums it created, but note that the
+Google Photos API does not allow media to be deleted permanently so
+this media will still remain. See [bug #109759781](https://issuetracker.google.com/issues/109759781).
-Set command for updating the drive config parameters
+Rclone cannot delete files anywhere except under `album`.
- rclone backend set remote: [options] [+]
+### Deleting albums
-This is a set command which will be used to update the various drive config parameters
+The Google Photos API does not support deleting albums - see [bug #135714733](https://issuetracker.google.com/issues/135714733).
-Usage Examples:
+# Hasher
- rclone backend set drive: [-o service_account_file=sa.json] [-o chunk_size=67108864]
- rclone rc backend/command command=set fs=drive: [-o service_account_file=sa.json] [-o chunk_size=67108864]
+Hasher is a special overlay backend to create remotes which handle
+checksums for other remotes. It's main functions include:
+- Emulate hash types unimplemented by backends
+- Cache checksums to help with slow hashing of large local or (S)FTP files
+- Warm up checksum cache from external SUM files
+## Getting started
-Options:
+To use Hasher, first set up the underlying remote following the configuration
+instructions for that remote. You can also use a local pathname instead of
+a remote. Check that your base remote is working.
-- "chunk_size": update the current upload chunk size
-- "service_account_file": update the current service account file
+Let's call the base remote `myRemote:path` here. Note that anything inside
+`myRemote:path` will be handled by hasher and anything outside won't.
+This means that if you are using a bucket based remote (S3, B2, Swift)
+then you should put the bucket in the remote `s3:bucket`.
-### shortcut
+Now proceed to interactive or manual configuration.
-Create shortcuts from files or directories
+### Interactive configuration
- rclone backend shortcut remote: [options] [+]
+Run `rclone config`:
+```
+No remotes found, make a new one?
+n) New remote
+s) Set configuration password
+q) Quit config
+n/s/q> n
+name> Hasher1
+Type of storage to configure.
+Choose a number from below, or type in your own value
+[snip]
+XX / Handle checksums for other remotes
+ \ "hasher"
+[snip]
+Storage> hasher
+Remote to cache checksums for, like myremote:mypath.
+Enter a string value. Press Enter for the default ("").
+remote> myRemote:path
+Comma separated list of supported checksum types.
+Enter a string value. Press Enter for the default ("md5,sha1").
+hashsums> md5
+Maximum time to keep checksums in cache. 0 = no cache, off = cache forever.
+max_age> off
+Edit advanced config? (y/n)
+y) Yes
+n) No
+y/n> n
+Remote config
+--------------------
+[Hasher1]
+type = hasher
+remote = myRemote:path
+hashsums = md5
+max_age = off
+--------------------
+y) Yes this is OK
+e) Edit this remote
+d) Delete this remote
+y/e/d> y
+```
-This command creates shortcuts from files or directories.
+### Manual configuration
-Usage:
+Run `rclone config path` to see the path of current active config file,
+usually `YOURHOME/.config/rclone/rclone.conf`.
+Open it in your favorite text editor, find section for the base remote
+and create new section for hasher like in the following examples:
- rclone backend shortcut drive: source_item destination_shortcut
- rclone backend shortcut drive: source_item -o target=drive2: destination_shortcut
+```
+[Hasher1]
+type = hasher
+remote = myRemote:path
+hashes = md5
+max_age = off
-In the first example this creates a shortcut from the "source_item"
-which can be a file or a directory to the "destination_shortcut". The
-"source_item" and the "destination_shortcut" should be relative paths
-from "drive:"
+[Hasher2]
+type = hasher
+remote = /local/path
+hashes = dropbox,sha1
+max_age = 24h
+```
-In the second example this creates a shortcut from the "source_item"
-relative to "drive:" to the "destination_shortcut" relative to
-"drive2:". This may fail with a permission error if the user
-authenticated with "drive2:" can't read files from "drive:".
+Hasher takes basically the following parameters:
+- `remote` is required,
+- `hashes` is a comma separated list of supported checksums
+ (by default `md5,sha1`),
+- `max_age` - maximum time to keep a checksum value in the cache,
+ `0` will disable caching completely,
+ `off` will cache "forever" (that is until the files get changed).
+
+Make sure the `remote` has `:` (colon) in. If you specify the remote without
+a colon then rclone will use a local directory of that name. So if you use
+a remote of `/local/path` then rclone will handle hashes for that directory.
+If you use `remote = name` literally then rclone will put files
+**in a directory called `name` located under current directory**.
+## Usage
-Options:
+### Basic operations
+
+Now you can use it as `Hasher2:subdir/file` instead of base remote.
+Hasher will transparently update cache with new checksums when a file
+is fully read or overwritten, like:
+```
+rclone copy External:path/file Hasher:dest/path
-- "target": optional target remote for the shortcut destination
+rclone cat Hasher:path/to/file > /dev/null
+```
-### drives
+The way to refresh **all** cached checksums (even unsupported by the base backend)
+for a subtree is to **re-download** all files in the subtree. For example,
+use `hashsum --download` using **any** supported hashsum on the command line
+(we just care to re-read):
+```
+rclone hashsum MD5 --download Hasher:path/to/subtree > /dev/null
-List the Shared Drives available to this account
+rclone backend dump Hasher:path/to/subtree
+```
- rclone backend drives remote: [options] [+]
+You can print or drop hashsum cache using custom backend commands:
+```
+rclone backend dump Hasher:dir/subdir
-This command lists the Shared Drives (Team Drives) available to this
-account.
+rclone backend drop Hasher:
+```
-Usage:
+### Pre-Seed from a SUM File
- rclone backend [-o config] drives drive:
+Hasher supports two backend commands: generic SUM file `import` and faster
+but less consistent `stickyimport`.
-This will return a JSON list of objects like this
+```
+rclone backend import Hasher:dir/subdir SHA1 /path/to/SHA1SUM [--checkers 4]
+```
- [
- {
- "id": "0ABCDEF-01234567890",
- "kind": "drive#teamDrive",
- "name": "My Drive"
- },
- {
- "id": "0ABCDEFabcdefghijkl",
- "kind": "drive#teamDrive",
- "name": "Test Drive"
- }
- ]
+Instead of SHA1 it can be any hash supported by the remote. The last argument
+can point to either a local or an `other-remote:path` text file in SUM format.
+The command will parse the SUM file, then walk down the path given by the
+first argument, snapshot current fingerprints and fill in the cache entries
+correspondingly.
+- Paths in the SUM file are treated as relative to `hasher:dir/subdir`.
+- The command will **not** check that supplied values are correct.
+ You **must know** what you are doing.
+- This is a one-time action. The SUM file will not get "attached" to the
+ remote. Cache entries can still be overwritten later, should the object's
+ fingerprint change.
+- The tree walk can take long depending on the tree size. You can increase
+ `--checkers` to make it faster. Or use `stickyimport` if you don't care
+ about fingerprints and consistency.
-With the -o config parameter it will output the list in a format
-suitable for adding to a config file to make aliases for all the
-drives found and a combined drive.
+```
+rclone backend stickyimport hasher:path/to/data sha1 remote:/path/to/sum.sha1
+```
- [My Drive]
- type = alias
- remote = drive,team_drive=0ABCDEF-01234567890,root_folder_id=:
+`stickyimport` is similar to `import` but works much faster because it
+does not need to stat existing files and skips initial tree walk.
+Instead of binding cache entries to file fingerprints it creates _sticky_
+entries bound to the file name alone ignoring size, modification time etc.
+Such hash entries can be replaced only by `purge`, `delete`, `backend drop`
+or by full re-read/re-write of the files.
- [Test Drive]
- type = alias
- remote = drive,team_drive=0ABCDEFabcdefghijkl,root_folder_id=:
+## Configuration reference
- [AllDrives]
- type = combine
- upstreams = "My Drive=My Drive:" "Test Drive=Test Drive:"
-Adding this to the rclone config file will cause those team drives to
-be accessible with the aliases shown. Any illegal characters will be
-substituted with "_" and duplicate names will have numbers suffixed.
-It will also add a remote called AllDrives which shows all the shared
-drives combined into one directory tree.
+### Standard options
+Here are the Standard options specific to hasher (Better checksums for other remotes).
-### untrash
+#### --hasher-remote
-Untrash files and directories
+Remote to cache checksums for (e.g. myRemote:path).
- rclone backend untrash remote: [options] [+]
+Properties:
-This command untrashes all the files and directories in the directory
-passed in recursively.
+- Config: remote
+- Env Var: RCLONE_HASHER_REMOTE
+- Type: string
+- Required: true
-Usage:
+#### --hasher-hashes
-This takes an optional directory to trash which make this easier to
-use via the API.
+Comma separated list of supported checksum types.
- rclone backend untrash drive:directory
- rclone backend --interactive untrash drive:directory subdir
+Properties:
-Use the --interactive/-i or --dry-run flag to see what would be restored before restoring it.
+- Config: hashes
+- Env Var: RCLONE_HASHER_HASHES
+- Type: CommaSepList
+- Default: md5,sha1
-Result:
+#### --hasher-max-age
- {
- "Untrashed": 17,
- "Errors": 0
- }
+Maximum time to keep checksums in cache (0 = no cache, off = cache forever).
+Properties:
-### copyid
+- Config: max_age
+- Env Var: RCLONE_HASHER_MAX_AGE
+- Type: Duration
+- Default: off
-Copy files by ID
+### Advanced options
- rclone backend copyid remote: [options] [+]
+Here are the Advanced options specific to hasher (Better checksums for other remotes).
-This command copies files by ID
+#### --hasher-auto-size
-Usage:
+Auto-update checksum for files smaller than this size (disabled by default).
- rclone backend copyid drive: ID path
- rclone backend copyid drive: ID1 path1 ID2 path2
+Properties:
-It copies the drive file with ID given to the path (an rclone path which
-will be passed internally to rclone copyto). The ID and path pairs can be
-repeated.
+- Config: auto_size
+- Env Var: RCLONE_HASHER_AUTO_SIZE
+- Type: SizeSuffix
+- Default: 0
-The path should end with a / to indicate copy the file as named to
-this directory. If it doesn't end with a / then the last path
-component will be used as the file name.
+### Metadata
-If the destination is a drive backend then server-side copying will be
-attempted if possible.
+Any metadata supported by the underlying remote is read and written.
-Use the --interactive/-i or --dry-run flag to see what would be copied before copying.
+See the [metadata](https://rclone.org/docs/#metadata) docs for more info.
+## Backend commands
-### exportformats
+Here are the commands specific to the hasher backend.
-Dump the export formats for debug purposes
+Run them with
- rclone backend exportformats remote: [options] [+]
+ rclone backend COMMAND remote:
-### importformats
+The help below will explain what arguments each command takes.
-Dump the import formats for debug purposes
+See the [backend](https://rclone.org/commands/rclone_backend/) command for more
+info on how to pass options and arguments.
- rclone backend importformats remote: [options] [+]
+These can be run on a running backend using the rc command
+[backend/command](https://rclone.org/rc/#backend-command).
+### drop
+Drop cache
-## Limitations
+ rclone backend drop remote: [options] [+]
-Drive has quite a lot of rate limiting. This causes rclone to be
-limited to transferring about 2 files per second only. Individual
-files may be transferred much faster at 100s of MiB/s but lots of
-small files can take a long time.
+Completely drop checksum cache.
+Usage Example:
+ rclone backend drop hasher:
-Server side copies are also subject to a separate rate limit. If you
-see User rate limit exceeded errors, wait at least 24 hours and retry.
-You can disable server-side copies with `--disable copy` to download
-and upload the files if you prefer.
-### Limitations of Google Docs
+### dump
-Google docs will appear as size -1 in `rclone ls`, `rclone ncdu` etc,
-and as size 0 in anything which uses the VFS layer, e.g. `rclone mount`
-and `rclone serve`. When calculating directory totals, e.g. in
-`rclone size` and `rclone ncdu`, they will be counted in as empty
-files.
+Dump the database
-This is because rclone can't find out the size of the Google docs
-without downloading them.
+ rclone backend dump remote: [options] [+]
-Google docs will transfer correctly with `rclone sync`, `rclone copy`
-etc as rclone knows to ignore the size when doing the transfer.
+Dump cache records covered by the current remote
-However an unfortunate consequence of this is that you may not be able
-to download Google docs using `rclone mount`. If it doesn't work you
-will get a 0 sized file. If you try again the doc may gain its
-correct size and be downloadable. Whether it will work on not depends
-on the application accessing the mount and the OS you are running -
-experiment to find out if it does work for you!
+### fulldump
-### Duplicated files
+Full dump of the database
-Sometimes, for no reason I've been able to track down, drive will
-duplicate a file that rclone uploads. Drive unlike all the other
-remotes can have duplicated files.
+ rclone backend fulldump remote: [options] [+]
-Duplicated files cause problems with the syncing and you will see
-messages in the log about duplicates.
+Dump all cache records in the database
-Use `rclone dedupe` to fix duplicated files.
+### import
-Note that this isn't just a problem with rclone, even Google Photos on
-Android duplicates files on drive sometimes.
+Import a SUM file
-### Rclone appears to be re-copying files it shouldn't
+ rclone backend import remote: [options] [+]
-The most likely cause of this is the duplicated file issue above - run
-`rclone dedupe` and check your logs for duplicate object or directory
-messages.
+Amend hash cache from a SUM file and bind checksums to files by size/time.
+Usage Example:
+ rclone backend import hasher:subdir md5 /path/to/sum.md5
-This can also be caused by a delay/caching on google drive's end when
-comparing directory listings. Specifically with team drives used in
-combination with --fast-list. Files that were uploaded recently may
-not appear on the directory list sent to rclone when using --fast-list.
-Waiting a moderate period of time between attempts (estimated to be
-approximately 1 hour) and/or not using --fast-list both seem to be
-effective in preventing the problem.
+### stickyimport
-## Making your own client_id
+Perform fast import of a SUM file
-When you use rclone with Google drive in its default configuration you
-are using rclone's client_id. This is shared between all the rclone
-users. There is a global rate limit on the number of queries per
-second that each client_id can do set by Google. rclone already has a
-high quota and I will continue to make sure it is high enough by
-contacting Google.
+ rclone backend stickyimport remote: [options] [+]
-It is strongly recommended to use your own client ID as the default rclone ID is heavily used. If you have multiple services running, it is recommended to use an API key for each service. The default Google quota is 10 transactions per second so it is recommended to stay under that number as if you use more than that, it will cause rclone to rate limit and make things slower.
+Fill hash cache from a SUM file without verifying file fingerprints.
+Usage Example:
+ rclone backend stickyimport hasher:subdir md5 remote:path/to/sum.md5
-Here is how to create your own Google Drive client ID for rclone:
-1. Log into the [Google API
-Console](https://console.developers.google.com/) with your Google
-account. It doesn't matter what Google account you use. (It need not
-be the same account as the Google Drive you want to access)
-2. Select a project or create a new project.
-3. Under "ENABLE APIS AND SERVICES" search for "Drive", and enable the
-"Google Drive API".
+## Implementation details (advanced)
-4. Click "Credentials" in the left-side panel (not "Create
-credentials", which opens the wizard).
+This section explains how various rclone operations work on a hasher remote.
-5. If you already configured an "Oauth Consent Screen", then skip
-to the next step; if not, click on "CONFIGURE CONSENT SCREEN" button
-(near the top right corner of the right panel), then select "External"
-and click on "CREATE"; on the next screen, enter an "Application name"
-("rclone" is OK); enter "User Support Email" (your own email is OK);
-enter "Developer Contact Email" (your own email is OK); then click on
-"Save" (all other data is optional). You will also have to add some scopes,
-including `.../auth/docs` and `.../auth/drive` in order to be able to edit,
-create and delete files with RClone. You may also want to include the
-`../auth/drive.metadata.readonly` scope. After adding scopes, click
-"Save and continue" to add test users. Be sure to add your own account to
-the test users. Once you've added yourself as a test user and saved the
-changes, click again on "Credentials" on the left panel to go back to
-the "Credentials" screen.
+**Disclaimer. This section describes current implementation which can
+change in future rclone versions!.**
- (PS: if you are a GSuite user, you could also select "Internal" instead
-of "External" above, but this will restrict API use to Google Workspace
-users in your organisation).
+### Hashsum command
-6. Click on the "+ CREATE CREDENTIALS" button at the top of the screen,
-then select "OAuth client ID".
+The `rclone hashsum` (or `md5sum` or `sha1sum`) command will:
-7. Choose an application type of "Desktop app" and click "Create". (the default name is fine)
+1. if requested hash is supported by lower level, just pass it.
+2. if object size is below `auto_size` then download object and calculate
+ _requested_ hashes on the fly.
+3. if unsupported and the size is big enough, build object `fingerprint`
+ (including size, modtime if supported, first-found _other_ hash if any).
+4. if the strict match is found in cache for the requested remote, return
+ the stored hash.
+5. if remote found but fingerprint mismatched, then purge the entry and
+ proceed to step 6.
+6. if remote not found or had no requested hash type or after step 5:
+ download object, calculate all _supported_ hashes on the fly and store
+ in cache; return requested hash.
-8. It will show you a client ID and client secret. Make a note of these.
-
- (If you selected "External" at Step 5 continue to Step 9.
- If you chose "Internal" you don't need to publish and can skip straight to
- Step 10 but your destination drive must be part of the same Google Workspace.)
+### Other operations
-9. Go to "Oauth consent screen" and then click "PUBLISH APP" button and confirm.
- You will also want to add yourself as a test user.
+- whenever a file is uploaded or downloaded **in full**, capture the stream
+ to calculate all supported hashes on the fly and update database
+- server-side `move` will update keys of existing cache entries
+- `deletefile` will remove a single cache entry
+- `purge` will remove all cache entries under the purged path
-10. Provide the noted client ID and client secret to rclone.
+Note that setting `max_age = 0` will disable checksum caching completely.
-Be aware that, due to the "enhanced security" recently introduced by
-Google, you are theoretically expected to "submit your app for verification"
-and then wait a few weeks(!) for their response; in practice, you can go right
-ahead and use the client ID and client secret with rclone, the only issue will
-be a very scary confirmation screen shown when you connect via your browser
-for rclone to be able to get its token-id (but as this only happens during
-the remote configuration, it's not such a big deal). Keeping the application in
-"Testing" will work as well, but the limitation is that any grants will expire
-after a week, which can be annoying to refresh constantly. If, for whatever
-reason, a short grant time is not a problem, then keeping the application in
-testing mode would also be sufficient.
+If you set `max_age = off`, checksums in cache will never age, unless you
+fully rewrite or delete the file.
-(Thanks to @balazer on github for these instructions.)
+### Cache storage
-Sometimes, creation of an OAuth consent in Google API Console fails due to an error message
-“The request failed because changes to one of the field of the resource is not supportedâ€.
-As a convenient workaround, the necessary Google Drive API key can be created on the
-[Python Quickstart](https://developers.google.com/drive/api/v3/quickstart/python) page.
-Just push the Enable the Drive API button to receive the Client ID and Secret.
-Note that it will automatically create a new project in the API Console.
+Cached checksums are stored as `bolt` database files under rclone cache
+directory, usually `~/.cache/rclone/kv/`. Databases are maintained
+one per _base_ backend, named like `BaseRemote~hasher.bolt`.
+Checksums for multiple `alias`-es into a single base backend
+will be stored in the single database. All local paths are treated as
+aliases into the `local` backend (unless encrypted or chunked) and stored
+in `~/.cache/rclone/kv/local~hasher.bolt`.
+Databases can be shared between multiple rclone processes.
-# Google Photos
+# HDFS
-The rclone backend for [Google Photos](https://www.google.com/photos/about/) is
-a specialized backend for transferring photos and videos to and from
-Google Photos.
+[HDFS](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html) is a
+distributed file-system, part of the [Apache Hadoop](https://hadoop.apache.org/) framework.
-**NB** The Google Photos API which rclone uses has quite a few
-limitations, so please read the [limitations section](#limitations)
-carefully to make sure it is suitable for your use.
+Paths are specified as `remote:` or `remote:path/to/dir`.
## Configuration
-The initial setup for google cloud storage involves getting a token from Google Photos
-which you need to do in your browser. `rclone config` walks you
-through it.
-
-Here is an example of how to make a remote called `remote`. First run:
+Here is an example of how to make a remote called `remote`. First run:
rclone config
@@ -33852,793 +35207,691 @@ name> remote
Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
-[snip]
-XX / Google Photos
- \ "google photos"
-[snip]
-Storage> google photos
-** See help for google photos backend at: https://rclone.org/googlephotos/ **
+[skip]
+XX / Hadoop distributed file system
+ \ "hdfs"
+[skip]
+Storage> hdfs
+** See help for hdfs backend at: https://rclone.org/hdfs/ **
-Google Application Client Id
-Leave blank normally.
+hadoop name node and port
Enter a string value. Press Enter for the default ("").
-client_id>
-Google Application Client Secret
-Leave blank normally.
+Choose a number from below, or type in your own value
+ 1 / Connect to host namenode at port 8020
+ \ "namenode:8020"
+namenode> namenode.hadoop:8020
+hadoop user name
Enter a string value. Press Enter for the default ("").
-client_secret>
-Set to make the Google Photos backend read only.
-
-If you choose read only then rclone will only request read only access
-to your photos, otherwise rclone will request full access.
-Enter a boolean value (true or false). Press Enter for the default ("false").
-read_only>
+Choose a number from below, or type in your own value
+ 1 / Connect to hdfs as root
+ \ "root"
+username> root
Edit advanced config? (y/n)
y) Yes
-n) No
+n) No (default)
y/n> n
Remote config
-Use web browser to automatically authenticate rclone with remote?
- * Say Y if the machine running rclone has a web browser you can use
- * Say N if running rclone on a (remote) machine without web browser access
-If not sure try Y. If Y failed, try N.
-y) Yes
-n) No
-y/n> y
-If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth
-Log in and authorize rclone for access
-Waiting for code...
-Got code
-
-*** IMPORTANT: All media items uploaded to Google Photos with rclone
-*** are stored in full resolution at original quality. These uploads
-*** will count towards storage in your Google Account.
-
--------------------
[remote]
-type = google photos
-token = {"access_token":"XXX","token_type":"Bearer","refresh_token":"XXX","expiry":"2019-06-28T17:38:04.644930156+01:00"}
---------------------
-y) Yes this is OK
-e) Edit this remote
-d) Delete this remote
-y/e/d> y
-```
-
-See the [remote setup docs](https://rclone.org/remote_setup/) for how to set it up on a
-machine with no Internet browser available.
-
-Note that rclone runs a webserver on your local machine to collect the
-token as returned from Google if using web browser to automatically
-authenticate. This only
-runs from the moment it opens your browser to the moment you get back
-the verification code. This is on `http://127.0.0.1:53682/` and this
-may require you to unblock it temporarily if you are running a host
-firewall, or use manual mode.
-
-This remote is called `remote` and can now be used like this
-
-See all the albums in your photos
-
- rclone lsd remote:album
-
-Make a new album
-
- rclone mkdir remote:album/newAlbum
-
-List the contents of an album
-
- rclone ls remote:album/newAlbum
-
-Sync `/home/local/images` to the Google Photos, removing any excess
-files in the album.
-
- rclone sync --interactive /home/local/image remote:album/newAlbum
-
-### Layout
-
-As Google Photos is not a general purpose cloud storage system, the
-backend is laid out to help you navigate it.
-
-The directories under `media` show different ways of categorizing the
-media. Each file will appear multiple times. So if you want to make
-a backup of your google photos you might choose to backup
-`remote:media/by-month`. (**NB** `remote:media/by-day` is rather slow
-at the moment so avoid for syncing.)
-
-Note that all your photos and videos will appear somewhere under
-`media`, but they may not appear under `album` unless you've put them
-into albums.
-
-```
-/
-- upload
- - file1.jpg
- - file2.jpg
- - ...
-- media
- - all
- - file1.jpg
- - file2.jpg
- - ...
- - by-year
- - 2000
- - file1.jpg
- - ...
- - 2001
- - file2.jpg
- - ...
- - ...
- - by-month
- - 2000
- - 2000-01
- - file1.jpg
- - ...
- - 2000-02
- - file2.jpg
- - ...
- - ...
- - by-day
- - 2000
- - 2000-01-01
- - file1.jpg
- - ...
- - 2000-01-02
- - file2.jpg
- - ...
- - ...
-- album
- - album name
- - album name/sub
-- shared-album
- - album name
- - album name/sub
-- feature
- - favorites
- - file1.jpg
- - file2.jpg
+type = hdfs
+namenode = namenode.hadoop:8020
+username = root
+--------------------
+y) Yes this is OK (default)
+e) Edit this remote
+d) Delete this remote
+y/e/d> y
+Current remotes:
+
+Name Type
+==== ====
+hadoop hdfs
+
+e) Edit existing remote
+n) New remote
+d) Delete remote
+r) Rename remote
+c) Copy remote
+s) Set configuration password
+q) Quit config
+e/n/d/r/c/s/q> q
```
-There are two writable parts of the tree, the `upload` directory and
-sub directories of the `album` directory.
+This remote is called `remote` and can now be used like this
-The `upload` directory is for uploading files you don't want to put
-into albums. This will be empty to start with and will contain the
-files you've uploaded for one rclone session only, becoming empty
-again when you restart rclone. The use case for this would be if you
-have a load of files you just want to once off dump into Google
-Photos. For repeated syncing, uploading to `album` will work better.
+See all the top level directories
-Directories within the `album` directory are also writeable and you
-may create new directories (albums) under `album`. If you copy files
-with a directory hierarchy in there then rclone will create albums
-with the `/` character in them. For example if you do
+ rclone lsd remote:
- rclone copy /path/to/images remote:album/images
+List the contents of a directory
-and the images directory contains
+ rclone ls remote:directory
+
+Sync the remote `directory` to `/home/local/directory`, deleting any excess files.
+
+ rclone sync --interactive remote:directory /home/local/directory
+
+### Setting up your own HDFS instance for testing
+
+You may start with a [manual setup](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html)
+or use the docker image from the tests:
+
+If you want to build the docker image
```
-images
- - file1.jpg
- dir
- file2.jpg
- dir2
- dir3
- file3.jpg
+git clone https://github.com/rclone/rclone.git
+cd rclone/fstest/testserver/images/test-hdfs
+docker build --rm -t rclone/test-hdfs .
```
-Then rclone will create the following albums with the following files in
+Or you can just use the latest one pushed
-- images
- - file1.jpg
-- images/dir
- - file2.jpg
-- images/dir2/dir3
- - file3.jpg
+```
+docker run --rm --name "rclone-hdfs" -p 127.0.0.1:9866:9866 -p 127.0.0.1:8020:8020 --hostname "rclone-hdfs" rclone/test-hdfs
+```
-This means that you can use the `album` path pretty much like a normal
-filesystem and it is a good target for repeated syncing.
+**NB** it need few seconds to startup.
-The `shared-album` directory shows albums shared with you or by you.
-This is similar to the Sharing tab in the Google Photos web interface.
+For this docker image the remote needs to be configured like this:
+```
+[remote]
+type = hdfs
+namenode = 127.0.0.1:8020
+username = root
+```
-### Standard options
+You can stop this image with `docker kill rclone-hdfs` (**NB** it does not use volumes, so all data
+uploaded will be lost.)
-Here are the Standard options specific to google photos (Google Photos).
+### Modification times
-#### --gphotos-client-id
+Time accurate to 1 second is stored.
-OAuth Client Id.
+### Checksum
-Leave blank normally.
+No checksums are implemented.
-Properties:
+### Usage information
-- Config: client_id
-- Env Var: RCLONE_GPHOTOS_CLIENT_ID
-- Type: string
-- Required: false
+You can use the `rclone about remote:` command which will display filesystem size and current usage.
-#### --gphotos-client-secret
+### Restricted filename characters
-OAuth Client Secret.
+In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters)
+the following characters are also replaced:
-Leave blank normally.
+| Character | Value | Replacement |
+| --------- |:-----:|:-----------:|
+| : | 0x3A | ： |
-Properties:
+Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8).
-- Config: client_secret
-- Env Var: RCLONE_GPHOTOS_CLIENT_SECRET
-- Type: string
-- Required: false
-#### --gphotos-read-only
+### Standard options
-Set to make the Google Photos backend read only.
+Here are the Standard options specific to hdfs (Hadoop distributed file system).
-If you choose read only then rclone will only request read only access
-to your photos, otherwise rclone will request full access.
+#### --hdfs-namenode
-Properties:
+Hadoop name nodes and ports.
-- Config: read_only
-- Env Var: RCLONE_GPHOTOS_READ_ONLY
-- Type: bool
-- Default: false
+E.g. "namenode-1:8020,namenode-2:8020,..." to connect to host namenodes at port 8020.
-### Advanced options
+Properties:
-Here are the Advanced options specific to google photos (Google Photos).
+- Config: namenode
+- Env Var: RCLONE_HDFS_NAMENODE
+- Type: CommaSepList
+- Default:
-#### --gphotos-token
+#### --hdfs-username
-OAuth Access Token as a JSON blob.
+Hadoop user name.
Properties:
-- Config: token
-- Env Var: RCLONE_GPHOTOS_TOKEN
+- Config: username
+- Env Var: RCLONE_HDFS_USERNAME
- Type: string
- Required: false
+- Examples:
+ - "root"
+ - Connect to hdfs as root.
-#### --gphotos-auth-url
+### Advanced options
-Auth server URL.
+Here are the Advanced options specific to hdfs (Hadoop distributed file system).
-Leave blank to use the provider defaults.
+#### --hdfs-service-principal-name
+
+Kerberos service principal name for the namenode.
+
+Enables KERBEROS authentication. Specifies the Service Principal Name
+(SERVICE/FQDN) for the namenode. E.g. \"hdfs/namenode.hadoop.docker\"
+for namenode running as service 'hdfs' with FQDN 'namenode.hadoop.docker'.
Properties:
-- Config: auth_url
-- Env Var: RCLONE_GPHOTOS_AUTH_URL
+- Config: service_principal_name
+- Env Var: RCLONE_HDFS_SERVICE_PRINCIPAL_NAME
- Type: string
- Required: false
-#### --gphotos-token-url
+#### --hdfs-data-transfer-protection
-Token server url.
+Kerberos data transfer protection: authentication|integrity|privacy.
-Leave blank to use the provider defaults.
+Specifies whether or not authentication, data signature integrity
+checks, and wire encryption are required when communicating with
+the datanodes. Possible values are 'authentication', 'integrity'
+and 'privacy'. Used only with KERBEROS enabled.
Properties:
-- Config: token_url
-- Env Var: RCLONE_GPHOTOS_TOKEN_URL
+- Config: data_transfer_protection
+- Env Var: RCLONE_HDFS_DATA_TRANSFER_PROTECTION
- Type: string
- Required: false
+- Examples:
+ - "privacy"
+ - Ensure authentication, integrity and encryption enabled.
-#### --gphotos-read-size
+#### --hdfs-encoding
-Set to read the size of media items.
+The encoding for the backend.
-Normally rclone does not read the size of media items since this takes
-another transaction. This isn't necessary for syncing. However
-rclone mount needs to know the size of files in advance of reading
-them, so setting this flag when using rclone mount is recommended if
-you want to read the media.
+See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info.
Properties:
-- Config: read_size
-- Env Var: RCLONE_GPHOTOS_READ_SIZE
-- Type: bool
-- Default: false
-
-#### --gphotos-start-year
+- Config: encoding
+- Env Var: RCLONE_HDFS_ENCODING
+- Type: Encoding
+- Default: Slash,Colon,Del,Ctl,InvalidUtf8,Dot
-Year limits the photos to be downloaded to those which are uploaded after the given year.
-Properties:
-- Config: start_year
-- Env Var: RCLONE_GPHOTOS_START_YEAR
-- Type: int
-- Default: 2000
+## Limitations
-#### --gphotos-include-archived
+- No server-side `Move` or `DirMove`.
+- Checksums not implemented.
-Also view and download archived media.
+# HiDrive
-By default, rclone does not request archived media. Thus, when syncing,
-archived media is not visible in directory listings or transferred.
+Paths are specified as `remote:path`
-Note that media in albums is always visible and synced, no matter
-their archive status.
+Paths may be as deep as required, e.g. `remote:directory/subdirectory`.
-With this flag, archived media are always visible in directory
-listings and transferred.
+The initial setup for hidrive involves getting a token from HiDrive
+which you need to do in your browser.
+`rclone config` walks you through it.
-Without this flag, archived media will not be visible in directory
-listings and won't be transferred.
+## Configuration
-Properties:
+Here is an example of how to make a remote called `remote`. First run:
-- Config: include_archived
-- Env Var: RCLONE_GPHOTOS_INCLUDE_ARCHIVED
-- Type: bool
-- Default: false
+ rclone config
-#### --gphotos-encoding
+This will guide you through an interactive setup process:
-The encoding for the backend.
+```
+No remotes found - make a new one
+n) New remote
+s) Set configuration password
+q) Quit config
+n/s/q> n
+name> remote
+Type of storage to configure.
+Choose a number from below, or type in your own value
+[snip]
+XX / HiDrive
+ \ "hidrive"
+[snip]
+Storage> hidrive
+OAuth Client Id - Leave blank normally.
+client_id>
+OAuth Client Secret - Leave blank normally.
+client_secret>
+Access permissions that rclone should use when requesting access from HiDrive.
+Leave blank normally.
+scope_access>
+Edit advanced config?
+y/n> n
+Use web browser to automatically authenticate rclone with remote?
+ * Say Y if the machine running rclone has a web browser you can use
+ * Say N if running rclone on a (remote) machine without web browser access
+If not sure try Y. If Y failed, try N.
+y/n> y
+If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth?state=xxxxxxxxxxxxxxxxxxxxxx
+Log in and authorize rclone for access
+Waiting for code...
+Got code
+--------------------
+[remote]
+type = hidrive
+token = {"access_token":"xxxxxxxxxxxxxxxxxxxx","token_type":"Bearer","refresh_token":"xxxxxxxxxxxxxxxxxxxxxxx","expiry":"xxxxxxxxxxxxxxxxxxxxxxx"}
+--------------------
+y) Yes this is OK (default)
+e) Edit this remote
+d) Delete this remote
+y/e/d> y
+```
-See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info.
+**You should be aware that OAuth-tokens can be used to access your account
+and hence should not be shared with other persons.**
+See the [below section](#keeping-your-tokens-safe) for more information.
-Properties:
+See the [remote setup docs](https://rclone.org/remote_setup/) for how to set it up on a
+machine with no Internet browser available.
-- Config: encoding
-- Env Var: RCLONE_GPHOTOS_ENCODING
-- Type: MultiEncoder
-- Default: Slash,CrLf,InvalidUtf8,Dot
+Note that rclone runs a webserver on your local machine to collect the
+token as returned from HiDrive. This only runs from the moment it opens
+your browser to the moment you get back the verification code.
+The webserver runs on `http://127.0.0.1:53682/`.
+If local port `53682` is protected by a firewall you may need to temporarily
+unblock the firewall to complete authorization.
+Once configured you can then use `rclone` like this,
+List directories in top level of your HiDrive root folder
-## Limitations
+ rclone lsd remote:
-Only images and videos can be uploaded. If you attempt to upload non
-videos or images or formats that Google Photos doesn't understand,
-rclone will upload the file, then Google Photos will give an error
-when it is put turned into a media item.
+List all the files in your HiDrive filesystem
-Note that all media items uploaded to Google Photos through the API
-are stored in full resolution at "original quality" and **will** count
-towards your storage quota in your Google Account. The API does
-**not** offer a way to upload in "high quality" mode..
+ rclone ls remote:
-`rclone about` is not supported by the Google Photos backend. Backends without
-this capability cannot determine free space for an rclone mount or
-use policy `mfs` (most free space) as a member of an rclone union
-remote.
+To copy a local directory to a HiDrive directory called backup
-See [List of backends that do not support rclone about](https://rclone.org/overview/#optional-features)
-See [rclone about](https://rclone.org/commands/rclone_about/)
+ rclone copy /home/source remote:backup
-### Downloading Images
+### Keeping your tokens safe
-When Images are downloaded this strips EXIF location (according to the
-docs and my tests). This is a limitation of the Google Photos API and
-is covered by [bug #112096115](https://issuetracker.google.com/issues/112096115).
+Any OAuth-tokens will be stored by rclone in the remote's configuration file as unencrypted text.
+Anyone can use a valid refresh-token to access your HiDrive filesystem without knowing your password.
+Therefore you should make sure no one else can access your configuration.
-**The current google API does not allow photos to be downloaded at original resolution. This is very important if you are, for example, relying on "Google Photos" as a backup of your photos. You will not be able to use rclone to redownload original images. You could use 'google takeout' to recover the original photos as a last resort**
+It is possible to encrypt rclone's configuration file.
+You can find information on securing your configuration file by viewing the [configuration encryption docs](https://rclone.org/docs/#configuration-encryption).
-### Downloading Videos
+### Invalid refresh token
-When videos are downloaded they are downloaded in a really compressed
-version of the video compared to downloading it via the Google Photos
-web interface. This is covered by [bug #113672044](https://issuetracker.google.com/issues/113672044).
+As can be verified [here](https://developer.hidrive.com/basics-flows/),
+each `refresh_token` (for Native Applications) is valid for 60 days.
+If used to access HiDrivei, its validity will be automatically extended.
-### Duplicates
+This means that if you
-If a file name is duplicated in a directory then rclone will add the
-file ID into its name. So two files called `file.jpg` would then
-appear as `file {123456}.jpg` and `file {ABCDEF}.jpg` (the actual IDs
-are a lot longer alas!).
+ * Don't use the HiDrive remote for 60 days
-If you upload the same image (with the same binary data) twice then
-Google Photos will deduplicate it. However it will retain the
-filename from the first upload which may confuse rclone. For example
-if you uploaded an image to `upload` then uploaded the same image to
-`album/my_album` the filename of the image in `album/my_album` will be
-what it was uploaded with initially, not what you uploaded it with to
-`album`. In practise this shouldn't cause too many problems.
+then rclone will return an error which includes a text
+that implies the refresh token is *invalid* or *expired*.
-### Modified time
+To fix this you will need to authorize rclone to access your HiDrive account again.
-The date shown of media in Google Photos is the creation date as
-determined by the EXIF information, or the upload date if that is not
-known.
+Using
-This is not changeable by rclone and is not the modification date of
-the media on local disk. This means that rclone cannot use the dates
-from Google Photos for syncing purposes.
+ rclone config reconnect remote:
-### Size
+the process is very similar to the process of initial setup exemplified before.
-The Google Photos API does not return the size of media. This means
-that when syncing to Google Photos, rclone can only do a file
-existence check.
+### Modification times and hashes
-It is possible to read the size of the media, but this needs an extra
-HTTP HEAD request per media item so is **very slow** and uses up a lot of
-transactions. This can be enabled with the `--gphotos-read-size`
-option or the `read_size = true` config parameter.
+HiDrive allows modification times to be set on objects accurate to 1 second.
-If you want to use the backend with `rclone mount` you may need to
-enable this flag (depending on your OS and application using the
-photos) otherwise you may not be able to read media off the mount.
-You'll need to experiment to see if it works for you without the flag.
+HiDrive supports [its own hash type](https://static.hidrive.com/dev/0001)
+which is used to verify the integrity of file contents after successful transfers.
-### Albums
+### Restricted filename characters
-Rclone can only upload files to albums it created. This is a
-[limitation of the Google Photos API](https://developers.google.com/photos/library/guides/manage-albums).
+HiDrive cannot store files or folders that include
+`/` (0x2F) or null-bytes (0x00) in their name.
+Any other characters can be used in the names of files or folders.
+Additionally, files or folders cannot be named either of the following: `.` or `..`
-Rclone can remove files it uploaded from albums it created only.
+Therefore rclone will automatically replace these characters,
+if files or folders are stored or accessed with such names.
-### Deleting files
+You can read about how this filename encoding works in general
+[here](overview/#restricted-filenames).
-Rclone can remove files from albums it created, but note that the
-Google Photos API does not allow media to be deleted permanently so
-this media will still remain. See [bug #109759781](https://issuetracker.google.com/issues/109759781).
+Keep in mind that HiDrive only supports file or folder names
+with a length of 255 characters or less.
-Rclone cannot delete files anywhere except under `album`.
+### Transfers
-### Deleting albums
+HiDrive limits file sizes per single request to a maximum of 2 GiB.
+To allow storage of larger files and allow for better upload performance,
+the hidrive backend will use a chunked transfer for files larger than 96 MiB.
+Rclone will upload multiple parts/chunks of the file at the same time.
+Chunks in the process of being uploaded are buffered in memory,
+so you may want to restrict this behaviour on systems with limited resources.
-The Google Photos API does not support deleting albums - see [bug #135714733](https://issuetracker.google.com/issues/135714733).
+You can customize this behaviour using the following options:
-# Hasher
+* `chunk_size`: size of file parts
+* `upload_cutoff`: files larger or equal to this in size will use a chunked transfer
+* `upload_concurrency`: number of file-parts to upload at the same time
-Hasher is a special overlay backend to create remotes which handle
-checksums for other remotes. It's main functions include:
-- Emulate hash types unimplemented by backends
-- Cache checksums to help with slow hashing of large local or (S)FTP files
-- Warm up checksum cache from external SUM files
+See the below section about configuration options for more details.
-## Getting started
+### Root folder
-To use Hasher, first set up the underlying remote following the configuration
-instructions for that remote. You can also use a local pathname instead of
-a remote. Check that your base remote is working.
+You can set the root folder for rclone.
+This is the directory that rclone considers to be the root of your HiDrive.
-Let's call the base remote `myRemote:path` here. Note that anything inside
-`myRemote:path` will be handled by hasher and anything outside won't.
-This means that if you are using a bucket based remote (S3, B2, Swift)
-then you should put the bucket in the remote `s3:bucket`.
+Usually, you will leave this blank, and rclone will use the root of the account.
-Now proceed to interactive or manual configuration.
+However, you can set this to restrict rclone to a specific folder hierarchy.
-### Interactive configuration
+This works by prepending the contents of the `root_prefix` option
+to any paths accessed by rclone.
+For example, the following two ways to access the home directory are equivalent:
-Run `rclone config`:
-```
-No remotes found, make a new one?
-n) New remote
-s) Set configuration password
-q) Quit config
-n/s/q> n
-name> Hasher1
-Type of storage to configure.
-Choose a number from below, or type in your own value
-[snip]
-XX / Handle checksums for other remotes
- \ "hasher"
-[snip]
-Storage> hasher
-Remote to cache checksums for, like myremote:mypath.
-Enter a string value. Press Enter for the default ("").
-remote> myRemote:path
-Comma separated list of supported checksum types.
-Enter a string value. Press Enter for the default ("md5,sha1").
-hashsums> md5
-Maximum time to keep checksums in cache. 0 = no cache, off = cache forever.
-max_age> off
-Edit advanced config? (y/n)
-y) Yes
-n) No
-y/n> n
-Remote config
---------------------
-[Hasher1]
-type = hasher
-remote = myRemote:path
-hashsums = md5
-max_age = off
---------------------
-y) Yes this is OK
-e) Edit this remote
-d) Delete this remote
-y/e/d> y
-```
+ rclone lsd --hidrive-root-prefix="/users/test/" remote:path
-### Manual configuration
+ rclone lsd remote:/users/test/path
-Run `rclone config path` to see the path of current active config file,
-usually `YOURHOME/.config/rclone/rclone.conf`.
-Open it in your favorite text editor, find section for the base remote
-and create new section for hasher like in the following examples:
+See the below section about configuration options for more details.
-```
-[Hasher1]
-type = hasher
-remote = myRemote:path
-hashes = md5
-max_age = off
+### Directory member count
-[Hasher2]
-type = hasher
-remote = /local/path
-hashes = dropbox,sha1
-max_age = 24h
-```
+By default, rclone will know the number of directory members contained in a directory.
+For example, `rclone lsd` uses this information.
-Hasher takes basically the following parameters:
-- `remote` is required,
-- `hashes` is a comma separated list of supported checksums
- (by default `md5,sha1`),
-- `max_age` - maximum time to keep a checksum value in the cache,
- `0` will disable caching completely,
- `off` will cache "forever" (that is until the files get changed).
+The acquisition of this information will result in additional time costs for HiDrive's API.
+When dealing with large directory structures, it may be desirable to circumvent this time cost,
+especially when this information is not explicitly needed.
+For this, the `disable_fetching_member_count` option can be used.
-Make sure the `remote` has `:` (colon) in. If you specify the remote without
-a colon then rclone will use a local directory of that name. So if you use
-a remote of `/local/path` then rclone will handle hashes for that directory.
-If you use `remote = name` literally then rclone will put files
-**in a directory called `name` located under current directory**.
+See the below section about configuration options for more details.
-## Usage
-### Basic operations
+### Standard options
-Now you can use it as `Hasher2:subdir/file` instead of base remote.
-Hasher will transparently update cache with new checksums when a file
-is fully read or overwritten, like:
-```
-rclone copy External:path/file Hasher:dest/path
+Here are the Standard options specific to hidrive (HiDrive).
-rclone cat Hasher:path/to/file > /dev/null
-```
+#### --hidrive-client-id
-The way to refresh **all** cached checksums (even unsupported by the base backend)
-for a subtree is to **re-download** all files in the subtree. For example,
-use `hashsum --download` using **any** supported hashsum on the command line
-(we just care to re-read):
-```
-rclone hashsum MD5 --download Hasher:path/to/subtree > /dev/null
+OAuth Client Id.
-rclone backend dump Hasher:path/to/subtree
-```
+Leave blank normally.
-You can print or drop hashsum cache using custom backend commands:
-```
-rclone backend dump Hasher:dir/subdir
+Properties:
-rclone backend drop Hasher:
-```
+- Config: client_id
+- Env Var: RCLONE_HIDRIVE_CLIENT_ID
+- Type: string
+- Required: false
-### Pre-Seed from a SUM File
+#### --hidrive-client-secret
-Hasher supports two backend commands: generic SUM file `import` and faster
-but less consistent `stickyimport`.
+OAuth Client Secret.
-```
-rclone backend import Hasher:dir/subdir SHA1 /path/to/SHA1SUM [--checkers 4]
-```
+Leave blank normally.
-Instead of SHA1 it can be any hash supported by the remote. The last argument
-can point to either a local or an `other-remote:path` text file in SUM format.
-The command will parse the SUM file, then walk down the path given by the
-first argument, snapshot current fingerprints and fill in the cache entries
-correspondingly.
-- Paths in the SUM file are treated as relative to `hasher:dir/subdir`.
-- The command will **not** check that supplied values are correct.
- You **must know** what you are doing.
-- This is a one-time action. The SUM file will not get "attached" to the
- remote. Cache entries can still be overwritten later, should the object's
- fingerprint change.
-- The tree walk can take long depending on the tree size. You can increase
- `--checkers` to make it faster. Or use `stickyimport` if you don't care
- about fingerprints and consistency.
+Properties:
-```
-rclone backend stickyimport hasher:path/to/data sha1 remote:/path/to/sum.sha1
-```
+- Config: client_secret
+- Env Var: RCLONE_HIDRIVE_CLIENT_SECRET
+- Type: string
+- Required: false
-`stickyimport` is similar to `import` but works much faster because it
-does not need to stat existing files and skips initial tree walk.
-Instead of binding cache entries to file fingerprints it creates _sticky_
-entries bound to the file name alone ignoring size, modification time etc.
-Such hash entries can be replaced only by `purge`, `delete`, `backend drop`
-or by full re-read/re-write of the files.
+#### --hidrive-scope-access
-## Configuration reference
+Access permissions that rclone should use when requesting access from HiDrive.
+Properties:
-### Standard options
+- Config: scope_access
+- Env Var: RCLONE_HIDRIVE_SCOPE_ACCESS
+- Type: string
+- Default: "rw"
+- Examples:
+ - "rw"
+ - Read and write access to resources.
+ - "ro"
+ - Read-only access to resources.
-Here are the Standard options specific to hasher (Better checksums for other remotes).
+### Advanced options
-#### --hasher-remote
+Here are the Advanced options specific to hidrive (HiDrive).
-Remote to cache checksums for (e.g. myRemote:path).
+#### --hidrive-token
+
+OAuth Access Token as a JSON blob.
Properties:
-- Config: remote
-- Env Var: RCLONE_HASHER_REMOTE
+- Config: token
+- Env Var: RCLONE_HIDRIVE_TOKEN
- Type: string
-- Required: true
+- Required: false
-#### --hasher-hashes
+#### --hidrive-auth-url
-Comma separated list of supported checksum types.
+Auth server URL.
+
+Leave blank to use the provider defaults.
Properties:
-- Config: hashes
-- Env Var: RCLONE_HASHER_HASHES
-- Type: CommaSepList
-- Default: md5,sha1
+- Config: auth_url
+- Env Var: RCLONE_HIDRIVE_AUTH_URL
+- Type: string
+- Required: false
-#### --hasher-max-age
+#### --hidrive-token-url
-Maximum time to keep checksums in cache (0 = no cache, off = cache forever).
+Token server url.
+
+Leave blank to use the provider defaults.
Properties:
-- Config: max_age
-- Env Var: RCLONE_HASHER_MAX_AGE
-- Type: Duration
-- Default: off
+- Config: token_url
+- Env Var: RCLONE_HIDRIVE_TOKEN_URL
+- Type: string
+- Required: false
-### Advanced options
+#### --hidrive-scope-role
-Here are the Advanced options specific to hasher (Better checksums for other remotes).
+User-level that rclone should use when requesting access from HiDrive.
-#### --hasher-auto-size
+Properties:
-Auto-update checksum for files smaller than this size (disabled by default).
+- Config: scope_role
+- Env Var: RCLONE_HIDRIVE_SCOPE_ROLE
+- Type: string
+- Default: "user"
+- Examples:
+ - "user"
+ - User-level access to management permissions.
+ - This will be sufficient in most cases.
+ - "admin"
+ - Extensive access to management permissions.
+ - "owner"
+ - Full access to management permissions.
+
+#### --hidrive-root-prefix
+
+The root/parent folder for all paths.
+
+Fill in to use the specified folder as the parent for all paths given to the remote.
+This way rclone can use any folder as its starting point.
Properties:
-- Config: auto_size
-- Env Var: RCLONE_HASHER_AUTO_SIZE
-- Type: SizeSuffix
-- Default: 0
+- Config: root_prefix
+- Env Var: RCLONE_HIDRIVE_ROOT_PREFIX
+- Type: string
+- Default: "/"
+- Examples:
+ - "/"
+ - The topmost directory accessible by rclone.
+ - This will be equivalent with "root" if rclone uses a regular HiDrive user account.
+ - "root"
+ - The topmost directory of the HiDrive user account
+ - ""
+ - This specifies that there is no root-prefix for your paths.
+ - When using this you will always need to specify paths to this remote with a valid parent e.g. "remote:/path/to/dir" or "remote:root/path/to/dir".
-### Metadata
+#### --hidrive-endpoint
-Any metadata supported by the underlying remote is read and written.
+Endpoint for the service.
-See the [metadata](https://rclone.org/docs/#metadata) docs for more info.
+This is the URL that API-calls will be made to.
-## Backend commands
+Properties:
-Here are the commands specific to the hasher backend.
+- Config: endpoint
+- Env Var: RCLONE_HIDRIVE_ENDPOINT
+- Type: string
+- Default: "https://api.hidrive.strato.com/2.1"
-Run them with
+#### --hidrive-disable-fetching-member-count
- rclone backend COMMAND remote:
+Do not fetch number of objects in directories unless it is absolutely necessary.
-The help below will explain what arguments each command takes.
+Requests may be faster if the number of objects in subdirectories is not fetched.
-See the [backend](https://rclone.org/commands/rclone_backend/) command for more
-info on how to pass options and arguments.
+Properties:
-These can be run on a running backend using the rc command
-[backend/command](https://rclone.org/rc/#backend-command).
+- Config: disable_fetching_member_count
+- Env Var: RCLONE_HIDRIVE_DISABLE_FETCHING_MEMBER_COUNT
+- Type: bool
+- Default: false
-### drop
+#### --hidrive-chunk-size
-Drop cache
+Chunksize for chunked uploads.
- rclone backend drop remote: [options] [+]
+Any files larger than the configured cutoff (or files of unknown size) will be uploaded in chunks of this size.
-Completely drop checksum cache.
-Usage Example:
- rclone backend drop hasher:
+The upper limit for this is 2147483647 bytes (about 2.000Gi).
+That is the maximum amount of bytes a single upload-operation will support.
+Setting this above the upper limit or to a negative value will cause uploads to fail.
+Setting this to larger values may increase the upload speed at the cost of using more memory.
+It can be set to smaller values smaller to save on memory.
-### dump
+Properties:
-Dump the database
+- Config: chunk_size
+- Env Var: RCLONE_HIDRIVE_CHUNK_SIZE
+- Type: SizeSuffix
+- Default: 48Mi
- rclone backend dump remote: [options] [+]
+#### --hidrive-upload-cutoff
-Dump cache records covered by the current remote
+Cutoff/Threshold for chunked uploads.
-### fulldump
+Any files larger than this will be uploaded in chunks of the configured chunksize.
-Full dump of the database
+The upper limit for this is 2147483647 bytes (about 2.000Gi).
+That is the maximum amount of bytes a single upload-operation will support.
+Setting this above the upper limit will cause uploads to fail.
- rclone backend fulldump remote: [options] [+]
+Properties:
-Dump all cache records in the database
+- Config: upload_cutoff
+- Env Var: RCLONE_HIDRIVE_UPLOAD_CUTOFF
+- Type: SizeSuffix
+- Default: 96Mi
-### import
+#### --hidrive-upload-concurrency
-Import a SUM file
+Concurrency for chunked uploads.
- rclone backend import remote: [options] [+]
+This is the upper limit for how many transfers for the same file are running concurrently.
+Setting this above to a value smaller than 1 will cause uploads to deadlock.
-Amend hash cache from a SUM file and bind checksums to files by size/time.
-Usage Example:
- rclone backend import hasher:subdir md5 /path/to/sum.md5
+If you are uploading small numbers of large files over high-speed links
+and these uploads do not fully utilize your bandwidth, then increasing
+this may help to speed up the transfers.
+Properties:
-### stickyimport
+- Config: upload_concurrency
+- Env Var: RCLONE_HIDRIVE_UPLOAD_CONCURRENCY
+- Type: int
+- Default: 4
-Perform fast import of a SUM file
+#### --hidrive-encoding
- rclone backend stickyimport remote: [options] [+]
+The encoding for the backend.
-Fill hash cache from a SUM file without verifying file fingerprints.
-Usage Example:
- rclone backend stickyimport hasher:subdir md5 remote:path/to/sum.md5
+See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info.
+Properties:
+- Config: encoding
+- Env Var: RCLONE_HIDRIVE_ENCODING
+- Type: Encoding
+- Default: Slash,Dot
-## Implementation details (advanced)
-This section explains how various rclone operations work on a hasher remote.
+## Limitations
-**Disclaimer. This section describes current implementation which can
-change in future rclone versions!.**
+### Symbolic links
-### Hashsum command
+HiDrive is able to store symbolic links (*symlinks*) by design,
+for example, when unpacked from a zip archive.
-The `rclone hashsum` (or `md5sum` or `sha1sum`) command will:
+There exists no direct mechanism to manage native symlinks in remotes.
+As such this implementation has chosen to ignore any native symlinks present in the remote.
+rclone will not be able to access or show any symlinks stored in the hidrive-remote.
+This means symlinks cannot be individually removed, copied, or moved,
+except when removing, copying, or moving the parent folder.
-1. if requested hash is supported by lower level, just pass it.
-2. if object size is below `auto_size` then download object and calculate
- _requested_ hashes on the fly.
-3. if unsupported and the size is big enough, build object `fingerprint`
- (including size, modtime if supported, first-found _other_ hash if any).
-4. if the strict match is found in cache for the requested remote, return
- the stored hash.
-5. if remote found but fingerprint mismatched, then purge the entry and
- proceed to step 6.
-6. if remote not found or had no requested hash type or after step 5:
- download object, calculate all _supported_ hashes on the fly and store
- in cache; return requested hash.
+*This does not affect the `.rclonelink`-files
+that rclone uses to encode and store symbolic links.*
-### Other operations
+### Sparse files
-- whenever a file is uploaded or downloaded **in full**, capture the stream
- to calculate all supported hashes on the fly and update database
-- server-side `move` will update keys of existing cache entries
-- `deletefile` will remove a single cache entry
-- `purge` will remove all cache entries under the purged path
+It is possible to store sparse files in HiDrive.
-Note that setting `max_age = 0` will disable checksum caching completely.
+Note that copying a sparse file will expand the holes
+into null-byte (0x00) regions that will then consume disk space.
+Likewise, when downloading a sparse file,
+the resulting file will have null-byte regions in the place of file holes.
-If you set `max_age = off`, checksums in cache will never age, unless you
-fully rewrite or delete the file.
+# HTTP
-### Cache storage
+The HTTP remote is a read only remote for reading files of a
+webserver. The webserver should provide file listings which rclone
+will read and turn into a remote. This has been tested with common
+webservers such as Apache/Nginx/Caddy and will likely work with file
+listings from most web servers. (If it doesn't then please file an
+issue, or send a pull request!)
-Cached checksums are stored as `bolt` database files under rclone cache
-directory, usually `~/.cache/rclone/kv/`. Databases are maintained
-one per _base_ backend, named like `BaseRemote~hasher.bolt`.
-Checksums for multiple `alias`-es into a single base backend
-will be stored in the single database. All local paths are treated as
-aliases into the `local` backend (unless encrypted or chunked) and stored
-in `~/.cache/rclone/kv/local~hasher.bolt`.
-Databases can be shared between multiple rclone processes.
+Paths are specified as `remote:` or `remote:path`.
-# HDFS
+The `remote:` represents the configured [url](#http-url), and any path following
+it will be resolved relative to this url, according to the URL standard. This
+means with remote url `https://beta.rclone.org/branch` and path `fix`, the
+resolved URL will be `https://beta.rclone.org/branch/fix`, while with path
+`/fix` the resolved URL will be `https://beta.rclone.org/fix` as the absolute
+path is resolved from the root of the domain.
-[HDFS](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html) is a
-distributed file-system, part of the [Apache Hadoop](https://hadoop.apache.org/) framework.
+If the path following the `remote:` ends with `/` it will be assumed to point
+to a directory. If the path does not end with `/`, then a HEAD request is sent
+and the response used to decide if it it is treated as a file or a directory
+(run with `-vv` to see details). When [--http-no-head](#http-no-head) is
+specified, a path without ending `/` is always assumed to be a file. If rclone
+incorrectly assumes the path is a file, the solution is to specify the path with
+ending `/`. When you know the path is a directory, ending it with `/` is always
+better as it avoids the initial HEAD request.
-Paths are specified as `remote:` or `remote:path/to/dir`.
+To just download a single file it is easier to use
+[copyurl](https://rclone.org/commands/rclone_copyurl/).
## Configuration
-Here is an example of how to make a remote called `remote`. First run:
+Here is an example of how to make a remote called `remote`. First
+run:
rclone config
@@ -34652,39 +35905,23 @@ q) Quit config
n/s/q> n
name> remote
Type of storage to configure.
-Enter a string value. Press Enter for the default ("").
-Choose a number from below, or type in your own value
-[skip]
-XX / Hadoop distributed file system
- \ "hdfs"
-[skip]
-Storage> hdfs
-** See help for hdfs backend at: https://rclone.org/hdfs/ **
-
-hadoop name node and port
-Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
- 1 / Connect to host namenode at port 8020
- \ "namenode:8020"
-namenode> namenode.hadoop:8020
-hadoop user name
-Enter a string value. Press Enter for the default ("").
+[snip]
+XX / HTTP
+ \ "http"
+[snip]
+Storage> http
+URL of http host to connect to
Choose a number from below, or type in your own value
- 1 / Connect to hdfs as root
- \ "root"
-username> root
-Edit advanced config? (y/n)
-y) Yes
-n) No (default)
-y/n> n
+ 1 / Connect to example.com
+ \ "https://example.com"
+url> https://beta.rclone.org
Remote config
--------------------
[remote]
-type = hdfs
-namenode = namenode.hadoop:8020
-username = root
+url = https://beta.rclone.org
--------------------
-y) Yes this is OK (default)
+y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> y
@@ -34692,7 +35929,7 @@ Current remotes:
Name Type
==== ====
-hadoop hdfs
+remote http
e) Edit existing remote
n) New remote
@@ -34718,629 +35955,768 @@ Sync the remote `directory` to `/home/local/directory`, deleting any excess file
rclone sync --interactive remote:directory /home/local/directory
-### Setting up your own HDFS instance for testing
+### Read only
-You may start with a [manual setup](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html)
-or use the docker image from the tests:
+This remote is read only - you can't upload files to an HTTP server.
-If you want to build the docker image
+### Modification times
-```
-git clone https://github.com/rclone/rclone.git
-cd rclone/fstest/testserver/images/test-hdfs
-docker build --rm -t rclone/test-hdfs .
-```
+Most HTTP servers store time accurate to 1 second.
-Or you can just use the latest one pushed
+### Checksum
-```
-docker run --rm --name "rclone-hdfs" -p 127.0.0.1:9866:9866 -p 127.0.0.1:8020:8020 --hostname "rclone-hdfs" rclone/test-hdfs
-```
+No checksums are stored.
-**NB** it need few seconds to startup.
+### Usage without a config file
-For this docker image the remote needs to be configured like this:
+Since the http remote only has one config parameter it is easy to use
+without a config file:
-```
-[remote]
-type = hdfs
-namenode = 127.0.0.1:8020
-username = root
-```
+ rclone lsd --http-url https://beta.rclone.org :http:
-You can stop this image with `docker kill rclone-hdfs` (**NB** it does not use volumes, so all data
-uploaded will be lost.)
+or:
-### Modified time
+ rclone lsd :http,url='https://beta.rclone.org':
-Time accurate to 1 second is stored.
-### Checksum
+### Standard options
-No checksums are implemented.
+Here are the Standard options specific to http (HTTP).
-### Usage information
+#### --http-url
-You can use the `rclone about remote:` command which will display filesystem size and current usage.
+URL of HTTP host to connect to.
-### Restricted filename characters
+E.g. "https://example.com", or "https://user:pass@example.com" to use a username and password.
-In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters)
-the following characters are also replaced:
+Properties:
-| Character | Value | Replacement |
-| --------- |:-----:|:-----------:|
-| : | 0x3A | ： |
+- Config: url
+- Env Var: RCLONE_HTTP_URL
+- Type: string
+- Required: true
-Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8).
+### Advanced options
+
+Here are the Advanced options specific to http (HTTP).
+#### --http-headers
-### Standard options
+Set HTTP headers for all transactions.
-Here are the Standard options specific to hdfs (Hadoop distributed file system).
+Use this to set additional HTTP headers for all transactions.
-#### --hdfs-namenode
+The input format is comma separated list of key,value pairs. Standard
+[CSV encoding](https://godoc.org/encoding/csv) may be used.
-Hadoop name node and port.
+For example, to set a Cookie use 'Cookie,name=value', or '"Cookie","name=value"'.
-E.g. "namenode:8020" to connect to host namenode at port 8020.
+You can set multiple headers, e.g. '"Cookie","name=value","Authorization","xxx"'.
Properties:
-- Config: namenode
-- Env Var: RCLONE_HDFS_NAMENODE
-- Type: string
-- Required: true
+- Config: headers
+- Env Var: RCLONE_HTTP_HEADERS
+- Type: CommaSepList
+- Default:
-#### --hdfs-username
+#### --http-no-slash
-Hadoop user name.
+Set this if the site doesn't end directories with /.
+
+Use this if your target website does not use / on the end of
+directories.
+
+A / on the end of a path is how rclone normally tells the difference
+between files and directories. If this flag is set, then rclone will
+treat all files with Content-Type: text/html as directories and read
+URLs from them rather than downloading them.
+
+Note that this may cause rclone to confuse genuine HTML files with
+directories.
+
+Properties:
+
+- Config: no_slash
+- Env Var: RCLONE_HTTP_NO_SLASH
+- Type: bool
+- Default: false
+
+#### --http-no-head
+
+Don't use HEAD requests.
+
+HEAD requests are mainly used to find file sizes in dir listing.
+If your site is being very slow to load then you can try this option.
+Normally rclone does a HEAD request for each potential file in a
+directory listing to:
+
+- find its size
+- check it really exists
+- check to see if it is a directory
+
+If you set this option, rclone will not do the HEAD request. This will mean
+that directory listings are much quicker, but rclone won't have the times or
+sizes of any files, and some files that don't exist may be in the listing.
+
+Properties:
+
+- Config: no_head
+- Env Var: RCLONE_HTTP_NO_HEAD
+- Type: bool
+- Default: false
-Properties:
+## Backend commands
-- Config: username
-- Env Var: RCLONE_HDFS_USERNAME
-- Type: string
-- Required: false
-- Examples:
- - "root"
- - Connect to hdfs as root.
+Here are the commands specific to the http backend.
-### Advanced options
+Run them with
-Here are the Advanced options specific to hdfs (Hadoop distributed file system).
+ rclone backend COMMAND remote:
-#### --hdfs-service-principal-name
+The help below will explain what arguments each command takes.
-Kerberos service principal name for the namenode.
+See the [backend](https://rclone.org/commands/rclone_backend/) command for more
+info on how to pass options and arguments.
-Enables KERBEROS authentication. Specifies the Service Principal Name
-(SERVICE/FQDN) for the namenode. E.g. \"hdfs/namenode.hadoop.docker\"
-for namenode running as service 'hdfs' with FQDN 'namenode.hadoop.docker'.
+These can be run on a running backend using the rc command
+[backend/command](https://rclone.org/rc/#backend-command).
-Properties:
+### set
-- Config: service_principal_name
-- Env Var: RCLONE_HDFS_SERVICE_PRINCIPAL_NAME
-- Type: string
-- Required: false
+Set command for updating the config parameters.
-#### --hdfs-data-transfer-protection
+ rclone backend set remote: [options] [+]
-Kerberos data transfer protection: authentication|integrity|privacy.
+This set command can be used to update the config parameters
+for a running http backend.
-Specifies whether or not authentication, data signature integrity
-checks, and wire encryption are required when communicating with
-the datanodes. Possible values are 'authentication', 'integrity'
-and 'privacy'. Used only with KERBEROS enabled.
+Usage Examples:
-Properties:
+ rclone backend set remote: [-o opt_name=opt_value] [-o opt_name2=opt_value2]
+ rclone rc backend/command command=set fs=remote: [-o opt_name=opt_value] [-o opt_name2=opt_value2]
+ rclone rc backend/command command=set fs=remote: -o url=https://example.com
-- Config: data_transfer_protection
-- Env Var: RCLONE_HDFS_DATA_TRANSFER_PROTECTION
-- Type: string
-- Required: false
-- Examples:
- - "privacy"
- - Ensure authentication, integrity and encryption enabled.
+The option keys are named as they are in the config file.
-#### --hdfs-encoding
+This rebuilds the connection to the http backend when it is called with
+the new parameters. Only new parameters need be passed as the values
+will default to those currently in use.
-The encoding for the backend.
+It doesn't return anything.
-See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info.
-Properties:
-- Config: encoding
-- Env Var: RCLONE_HDFS_ENCODING
-- Type: MultiEncoder
-- Default: Slash,Colon,Del,Ctl,InvalidUtf8,Dot
+## Limitations
+`rclone about` is not supported by the HTTP backend. Backends without
+this capability cannot determine free space for an rclone mount or
+use policy `mfs` (most free space) as a member of an rclone union
+remote.
-## Limitations
+See [List of backends that do not support rclone about](https://rclone.org/overview/#optional-features) and [rclone about](https://rclone.org/commands/rclone_about/)
-- No server-side `Move` or `DirMove`.
-- Checksums not implemented.
+# ImageKit
+This is a backend for the [ImageKit.io](https://imagekit.io/) storage service.
-# HiDrive
+#### About ImageKit
+[ImageKit.io](https://imagekit.io/) provides real-time image and video optimizations, transformations, and CDN delivery. Over 1,000 businesses and 70,000 developers trust ImageKit with their images and videos on the web.
-Paths are specified as `remote:path`
-Paths may be as deep as required, e.g. `remote:directory/subdirectory`.
+#### Accounts & Pricing
-The initial setup for hidrive involves getting a token from HiDrive
-which you need to do in your browser.
-`rclone config` walks you through it.
+To use this backend, you need to [create an account](https://imagekit.io/registration/) on ImageKit. Start with a free plan with generous usage limits. Then, as your requirements grow, upgrade to a plan that best fits your needs. See [the pricing details](https://imagekit.io/plans).
## Configuration
-Here is an example of how to make a remote called `remote`. First run:
+Here is an example of making an imagekit configuration.
- rclone config
+Firstly create a [ImageKit.io](https://imagekit.io/) account and choose a plan.
+
+You will need to log in and get the `publicKey` and `privateKey` for your account from the developer section.
+
+Now run
+```
+rclone config
+```
This will guide you through an interactive setup process:
```
-No remotes found - make a new one
+No remotes found, make a new one?
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
-name> remote
+
+Enter the name for the new remote.
+name> imagekit-media-library
+
+Option Storage.
Type of storage to configure.
-Choose a number from below, or type in your own value
+Choose a number from below, or type in your own value.
[snip]
-XX / HiDrive
- \ "hidrive"
+XX / ImageKit.io
+\ (imagekit)
[snip]
-Storage> hidrive
-OAuth Client Id - Leave blank normally.
-client_id>
-OAuth Client Secret - Leave blank normally.
-client_secret>
-Access permissions that rclone should use when requesting access from HiDrive.
-Leave blank normally.
-scope_access>
+Storage> imagekit
+
+Option endpoint.
+You can find your ImageKit.io URL endpoint in your [dashboard](https://imagekit.io/dashboard/developer/api-keys)
+Enter a value.
+endpoint> https://ik.imagekit.io/imagekit_id
+
+Option public_key.
+You can find your ImageKit.io public key in your [dashboard](https://imagekit.io/dashboard/developer/api-keys)
+Enter a value.
+public_key> public_****************************
+
+Option private_key.
+You can find your ImageKit.io private key in your [dashboard](https://imagekit.io/dashboard/developer/api-keys)
+Enter a value.
+private_key> private_****************************
+
Edit advanced config?
+y) Yes
+n) No (default)
y/n> n
-Use web browser to automatically authenticate rclone with remote?
- * Say Y if the machine running rclone has a web browser you can use
- * Say N if running rclone on a (remote) machine without web browser access
-If not sure try Y. If Y failed, try N.
-y/n> y
-If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth?state=xxxxxxxxxxxxxxxxxxxxxx
-Log in and authorize rclone for access
-Waiting for code...
-Got code
---------------------
-[remote]
-type = hidrive
-token = {"access_token":"xxxxxxxxxxxxxxxxxxxx","token_type":"Bearer","refresh_token":"xxxxxxxxxxxxxxxxxxxxxxx","expiry":"xxxxxxxxxxxxxxxxxxxxxxx"}
---------------------
+
+Configuration complete.
+Options:
+- type: imagekit
+- endpoint: https://ik.imagekit.io/imagekit_id
+- public_key: public_****************************
+- private_key: private_****************************
+
+Keep this "imagekit-media-library" remote?
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y
```
+List directories in the top level of your Media Library
+```
+rclone lsd imagekit-media-library:
+```
+Make a new directory.
+```
+rclone mkdir imagekit-media-library:directory
+```
+List the contents of a directory.
+```
+rclone ls imagekit-media-library:directory
+```
-**You should be aware that OAuth-tokens can be used to access your account
-and hence should not be shared with other persons.**
-See the [below section](#keeping-your-tokens-safe) for more information.
+### Modified time and hashes
-See the [remote setup docs](https://rclone.org/remote_setup/) for how to set it up on a
-machine with no Internet browser available.
+ImageKit does not support modification times or hashes yet.
-Note that rclone runs a webserver on your local machine to collect the
-token as returned from HiDrive. This only runs from the moment it opens
-your browser to the moment you get back the verification code.
-The webserver runs on `http://127.0.0.1:53682/`.
-If local port `53682` is protected by a firewall you may need to temporarily
-unblock the firewall to complete authorization.
+### Checksums
-Once configured you can then use `rclone` like this,
+No checksums are supported.
-List directories in top level of your HiDrive root folder
- rclone lsd remote:
+### Standard options
-List all the files in your HiDrive filesystem
+Here are the Standard options specific to imagekit (ImageKit.io).
- rclone ls remote:
+#### --imagekit-endpoint
-To copy a local directory to a HiDrive directory called backup
+You can find your ImageKit.io URL endpoint in your [dashboard](https://imagekit.io/dashboard/developer/api-keys)
- rclone copy /home/source remote:backup
+Properties:
-### Keeping your tokens safe
+- Config: endpoint
+- Env Var: RCLONE_IMAGEKIT_ENDPOINT
+- Type: string
+- Required: true
-Any OAuth-tokens will be stored by rclone in the remote's configuration file as unencrypted text.
-Anyone can use a valid refresh-token to access your HiDrive filesystem without knowing your password.
-Therefore you should make sure no one else can access your configuration.
+#### --imagekit-public-key
-It is possible to encrypt rclone's configuration file.
-You can find information on securing your configuration file by viewing the [configuration encryption docs](https://rclone.org/docs/#configuration-encryption).
+You can find your ImageKit.io public key in your [dashboard](https://imagekit.io/dashboard/developer/api-keys)
-### Invalid refresh token
+Properties:
-As can be verified [here](https://developer.hidrive.com/basics-flows/),
-each `refresh_token` (for Native Applications) is valid for 60 days.
-If used to access HiDrivei, its validity will be automatically extended.
+- Config: public_key
+- Env Var: RCLONE_IMAGEKIT_PUBLIC_KEY
+- Type: string
+- Required: true
-This means that if you
+#### --imagekit-private-key
- * Don't use the HiDrive remote for 60 days
+You can find your ImageKit.io private key in your [dashboard](https://imagekit.io/dashboard/developer/api-keys)
-then rclone will return an error which includes a text
-that implies the refresh token is *invalid* or *expired*.
+Properties:
-To fix this you will need to authorize rclone to access your HiDrive account again.
+- Config: private_key
+- Env Var: RCLONE_IMAGEKIT_PRIVATE_KEY
+- Type: string
+- Required: true
-Using
+### Advanced options
- rclone config reconnect remote:
+Here are the Advanced options specific to imagekit (ImageKit.io).
-the process is very similar to the process of initial setup exemplified before.
+#### --imagekit-only-signed
-### Modified time and hashes
+If you have configured `Restrict unsigned image URLs` in your dashboard settings, set this to true.
-HiDrive allows modification times to be set on objects accurate to 1 second.
+Properties:
-HiDrive supports [its own hash type](https://static.hidrive.com/dev/0001)
-which is used to verify the integrity of file contents after successful transfers.
+- Config: only_signed
+- Env Var: RCLONE_IMAGEKIT_ONLY_SIGNED
+- Type: bool
+- Default: false
-### Restricted filename characters
+#### --imagekit-versions
-HiDrive cannot store files or folders that include
-`/` (0x2F) or null-bytes (0x00) in their name.
-Any other characters can be used in the names of files or folders.
-Additionally, files or folders cannot be named either of the following: `.` or `..`
+Include old versions in directory listings.
-Therefore rclone will automatically replace these characters,
-if files or folders are stored or accessed with such names.
+Properties:
-You can read about how this filename encoding works in general
-[here](overview/#restricted-filenames).
+- Config: versions
+- Env Var: RCLONE_IMAGEKIT_VERSIONS
+- Type: bool
+- Default: false
-Keep in mind that HiDrive only supports file or folder names
-with a length of 255 characters or less.
+#### --imagekit-upload-tags
-### Transfers
+Tags to add to the uploaded files, e.g. "tag1,tag2".
-HiDrive limits file sizes per single request to a maximum of 2 GiB.
-To allow storage of larger files and allow for better upload performance,
-the hidrive backend will use a chunked transfer for files larger than 96 MiB.
-Rclone will upload multiple parts/chunks of the file at the same time.
-Chunks in the process of being uploaded are buffered in memory,
-so you may want to restrict this behaviour on systems with limited resources.
+Properties:
-You can customize this behaviour using the following options:
+- Config: upload_tags
+- Env Var: RCLONE_IMAGEKIT_UPLOAD_TAGS
+- Type: string
+- Required: false
-* `chunk_size`: size of file parts
-* `upload_cutoff`: files larger or equal to this in size will use a chunked transfer
-* `upload_concurrency`: number of file-parts to upload at the same time
+#### --imagekit-encoding
-See the below section about configuration options for more details.
+The encoding for the backend.
-### Root folder
+See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info.
-You can set the root folder for rclone.
-This is the directory that rclone considers to be the root of your HiDrive.
+Properties:
-Usually, you will leave this blank, and rclone will use the root of the account.
+- Config: encoding
+- Env Var: RCLONE_IMAGEKIT_ENCODING
+- Type: Encoding
+- Default: Slash,LtGt,DoubleQuote,Dollar,Question,Hash,Percent,BackSlash,Del,Ctl,InvalidUtf8,Dot,SquareBracket
-However, you can set this to restrict rclone to a specific folder hierarchy.
+### Metadata
-This works by prepending the contents of the `root_prefix` option
-to any paths accessed by rclone.
-For example, the following two ways to access the home directory are equivalent:
+Any metadata supported by the underlying remote is read and written.
- rclone lsd --hidrive-root-prefix="/users/test/" remote:path
+Here are the possible system metadata items for the imagekit backend.
- rclone lsd remote:/users/test/path
+| Name | Help | Type | Example | Read Only |
+|------|------|------|---------|-----------|
+| aws-tags | AI generated tags by AWS Rekognition associated with the image | string | tag1,tag2 | **Y** |
+| btime | Time of file birth (creation) read from Last-Modified header | RFC 3339 | 2006-01-02T15:04:05.999999999Z07:00 | **Y** |
+| custom-coordinates | Custom coordinates of the file | string | 0,0,100,100 | **Y** |
+| file-type | Type of the file | string | image | **Y** |
+| google-tags | AI generated tags by Google Cloud Vision associated with the image | string | tag1,tag2 | **Y** |
+| has-alpha | Whether the image has alpha channel or not | bool | | **Y** |
+| height | Height of the image or video in pixels | int | | **Y** |
+| is-private-file | Whether the file is private or not | bool | | **Y** |
+| size | Size of the object in bytes | int64 | | **Y** |
+| tags | Tags associated with the file | string | tag1,tag2 | **Y** |
+| width | Width of the image or video in pixels | int | | **Y** |
-See the below section about configuration options for more details.
+See the [metadata](https://rclone.org/docs/#metadata) docs for more info.
-### Directory member count
-By default, rclone will know the number of directory members contained in a directory.
-For example, `rclone lsd` uses this information.
-The acquisition of this information will result in additional time costs for HiDrive's API.
-When dealing with large directory structures, it may be desirable to circumvent this time cost,
-especially when this information is not explicitly needed.
-For this, the `disable_fetching_member_count` option can be used.
+# Internet Archive
-See the below section about configuration options for more details.
+The Internet Archive backend utilizes Items on [archive.org](https://archive.org/)
+Refer to [IAS3 API documentation](https://archive.org/services/docs/api/ias3.html) for the API this backend uses.
-### Standard options
+Paths are specified as `remote:bucket` (or `remote:` for the `lsd`
+command.) You may put subdirectories in too, e.g. `remote:item/path/to/dir`.
-Here are the Standard options specific to hidrive (HiDrive).
+Unlike S3, listing up all items uploaded by you isn't supported.
-#### --hidrive-client-id
+Once you have made a remote, you can use it like this:
-OAuth Client Id.
+Make a new item
-Leave blank normally.
+ rclone mkdir remote:item
-Properties:
+List the contents of a item
-- Config: client_id
-- Env Var: RCLONE_HIDRIVE_CLIENT_ID
-- Type: string
-- Required: false
+ rclone ls remote:item
-#### --hidrive-client-secret
+Sync `/home/local/directory` to the remote item, deleting any excess
+files in the item.
-OAuth Client Secret.
+ rclone sync --interactive /home/local/directory remote:item
-Leave blank normally.
+## Notes
+Because of Internet Archive's architecture, it enqueues write operations (and extra post-processings) in a per-item queue. You can check item's queue at https://catalogd.archive.org/history/item-name-here . Because of that, all uploads/deletes will not show up immediately and takes some time to be available.
+The per-item queue is enqueued to an another queue, Item Deriver Queue. [You can check the status of Item Deriver Queue here.](https://catalogd.archive.org/catalog.php?whereami=1) This queue has a limit, and it may block you from uploading, or even deleting. You should avoid uploading a lot of small files for better behavior.
-Properties:
+You can optionally wait for the server's processing to finish, by setting non-zero value to `wait_archive` key.
+By making it wait, rclone can do normal file comparison.
+Make sure to set a large enough value (e.g. `30m0s` for smaller files) as it can take a long time depending on server's queue.
-- Config: client_secret
-- Env Var: RCLONE_HIDRIVE_CLIENT_SECRET
-- Type: string
-- Required: false
+## About metadata
+This backend supports setting, updating and reading metadata of each file.
+The metadata will appear as file metadata on Internet Archive.
+However, some fields are reserved by both Internet Archive and rclone.
-#### --hidrive-scope-access
+The following are reserved by Internet Archive:
+- `name`
+- `source`
+- `size`
+- `md5`
+- `crc32`
+- `sha1`
+- `format`
+- `old_version`
+- `viruscheck`
+- `summation`
-Access permissions that rclone should use when requesting access from HiDrive.
+Trying to set values to these keys is ignored with a warning.
+Only setting `mtime` is an exception. Doing so make it the identical behavior as setting ModTime.
-Properties:
+rclone reserves all the keys starting with `rclone-`. Setting value for these keys will give you warnings, but values are set according to request.
-- Config: scope_access
-- Env Var: RCLONE_HIDRIVE_SCOPE_ACCESS
-- Type: string
-- Default: "rw"
-- Examples:
- - "rw"
- - Read and write access to resources.
- - "ro"
- - Read-only access to resources.
+If there are multiple values for a key, only the first one is returned.
+This is a limitation of rclone, that supports one value per one key.
+It can be triggered when you did a server-side copy.
-### Advanced options
+Reading metadata will also provide custom (non-standard nor reserved) ones.
-Here are the Advanced options specific to hidrive (HiDrive).
+## Filtering auto generated files
-#### --hidrive-token
+The Internet Archive automatically creates metadata files after
+upload. These can cause problems when doing an `rclone sync` as rclone
+will try, and fail, to delete them. These metadata files are not
+changeable, as they are created by the Internet Archive automatically.
-OAuth Access Token as a JSON blob.
+These auto-created files can be excluded from the sync using [metadata
+filtering](https://rclone.org/filtering/#metadata).
-Properties:
+ rclone sync ... --metadata-exclude "source=metadata" --metadata-exclude "format=Metadata"
-- Config: token
-- Env Var: RCLONE_HIDRIVE_TOKEN
-- Type: string
-- Required: false
+Which excludes from the sync any files which have the
+`source=metadata` or `format=Metadata` flags which are added to
+Internet Archive auto-created files.
-#### --hidrive-auth-url
+## Configuration
-Auth server URL.
+Here is an example of making an internetarchive configuration.
+Most applies to the other providers as well, any differences are described [below](#providers).
-Leave blank to use the provider defaults.
+First run
-Properties:
+ rclone config
-- Config: auth_url
-- Env Var: RCLONE_HIDRIVE_AUTH_URL
-- Type: string
-- Required: false
+This will guide you through an interactive setup process.
-#### --hidrive-token-url
+```
+No remotes found, make a new one?
+n) New remote
+s) Set configuration password
+q) Quit config
+n/s/q> n
+name> remote
+Option Storage.
+Type of storage to configure.
+Choose a number from below, or type in your own value.
+XX / InternetArchive Items
+ \ (internetarchive)
+Storage> internetarchive
+Option access_key_id.
+IAS3 Access Key.
+Leave blank for anonymous access.
+You can find one here: https://archive.org/account/s3.php
+Enter a value. Press Enter to leave empty.
+access_key_id> XXXX
+Option secret_access_key.
+IAS3 Secret Key (password).
+Leave blank for anonymous access.
+Enter a value. Press Enter to leave empty.
+secret_access_key> XXXX
+Edit advanced config?
+y) Yes
+n) No (default)
+y/n> y
+Option endpoint.
+IAS3 Endpoint.
+Leave blank for default value.
+Enter a string value. Press Enter for the default (https://s3.us.archive.org).
+endpoint>
+Option front_endpoint.
+Host of InternetArchive Frontend.
+Leave blank for default value.
+Enter a string value. Press Enter for the default (https://archive.org).
+front_endpoint>
+Option disable_checksum.
+Don't store MD5 checksum with object metadata.
+Normally rclone will calculate the MD5 checksum of the input before
+uploading it so it can ask the server to check the object against checksum.
+This is great for data integrity checking but can cause long delays for
+large files to start uploading.
+Enter a boolean value (true or false). Press Enter for the default (true).
+disable_checksum> true
+Option encoding.
+The encoding for the backend.
+See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info.
+Enter a encoder.MultiEncoder value. Press Enter for the default (Slash,Question,Hash,Percent,Del,Ctl,InvalidUtf8,Dot).
+encoding>
+Edit advanced config?
+y) Yes
+n) No (default)
+y/n> n
+--------------------
+[remote]
+type = internetarchive
+access_key_id = XXXX
+secret_access_key = XXXX
+--------------------
+y) Yes this is OK (default)
+e) Edit this remote
+d) Delete this remote
+y/e/d> y
+```
-Token server url.
-Leave blank to use the provider defaults.
+### Standard options
+
+Here are the Standard options specific to internetarchive (Internet Archive).
+
+#### --internetarchive-access-key-id
+
+IAS3 Access Key.
+
+Leave blank for anonymous access.
+You can find one here: https://archive.org/account/s3.php
Properties:
-- Config: token_url
-- Env Var: RCLONE_HIDRIVE_TOKEN_URL
+- Config: access_key_id
+- Env Var: RCLONE_INTERNETARCHIVE_ACCESS_KEY_ID
- Type: string
- Required: false
-#### --hidrive-scope-role
+#### --internetarchive-secret-access-key
-User-level that rclone should use when requesting access from HiDrive.
+IAS3 Secret Key (password).
+
+Leave blank for anonymous access.
Properties:
-- Config: scope_role
-- Env Var: RCLONE_HIDRIVE_SCOPE_ROLE
+- Config: secret_access_key
+- Env Var: RCLONE_INTERNETARCHIVE_SECRET_ACCESS_KEY
- Type: string
-- Default: "user"
-- Examples:
- - "user"
- - User-level access to management permissions.
- - This will be sufficient in most cases.
- - "admin"
- - Extensive access to management permissions.
- - "owner"
- - Full access to management permissions.
+- Required: false
-#### --hidrive-root-prefix
+### Advanced options
-The root/parent folder for all paths.
+Here are the Advanced options specific to internetarchive (Internet Archive).
-Fill in to use the specified folder as the parent for all paths given to the remote.
-This way rclone can use any folder as its starting point.
+#### --internetarchive-endpoint
+
+IAS3 Endpoint.
+
+Leave blank for default value.
Properties:
-- Config: root_prefix
-- Env Var: RCLONE_HIDRIVE_ROOT_PREFIX
+- Config: endpoint
+- Env Var: RCLONE_INTERNETARCHIVE_ENDPOINT
- Type: string
-- Default: "/"
-- Examples:
- - "/"
- - The topmost directory accessible by rclone.
- - This will be equivalent with "root" if rclone uses a regular HiDrive user account.
- - "root"
- - The topmost directory of the HiDrive user account
- - ""
- - This specifies that there is no root-prefix for your paths.
- - When using this you will always need to specify paths to this remote with a valid parent e.g. "remote:/path/to/dir" or "remote:root/path/to/dir".
+- Default: "https://s3.us.archive.org"
-#### --hidrive-endpoint
+#### --internetarchive-front-endpoint
-Endpoint for the service.
+Host of InternetArchive Frontend.
-This is the URL that API-calls will be made to.
+Leave blank for default value.
Properties:
-- Config: endpoint
-- Env Var: RCLONE_HIDRIVE_ENDPOINT
+- Config: front_endpoint
+- Env Var: RCLONE_INTERNETARCHIVE_FRONT_ENDPOINT
- Type: string
-- Default: "https://api.hidrive.strato.com/2.1"
-
-#### --hidrive-disable-fetching-member-count
+- Default: "https://archive.org"
-Do not fetch number of objects in directories unless it is absolutely necessary.
+#### --internetarchive-disable-checksum
-Requests may be faster if the number of objects in subdirectories is not fetched.
+Don't ask the server to test against MD5 checksum calculated by rclone.
+Normally rclone will calculate the MD5 checksum of the input before
+uploading it so it can ask the server to check the object against checksum.
+This is great for data integrity checking but can cause long delays for
+large files to start uploading.
Properties:
-- Config: disable_fetching_member_count
-- Env Var: RCLONE_HIDRIVE_DISABLE_FETCHING_MEMBER_COUNT
+- Config: disable_checksum
+- Env Var: RCLONE_INTERNETARCHIVE_DISABLE_CHECKSUM
- Type: bool
-- Default: false
+- Default: true
-#### --hidrive-chunk-size
+#### --internetarchive-wait-archive
-Chunksize for chunked uploads.
+Timeout for waiting the server's processing tasks (specifically archive and book_op) to finish.
+Only enable if you need to be guaranteed to be reflected after write operations.
+0 to disable waiting. No errors to be thrown in case of timeout.
-Any files larger than the configured cutoff (or files of unknown size) will be uploaded in chunks of this size.
+Properties:
-The upper limit for this is 2147483647 bytes (about 2.000Gi).
-That is the maximum amount of bytes a single upload-operation will support.
-Setting this above the upper limit or to a negative value will cause uploads to fail.
+- Config: wait_archive
+- Env Var: RCLONE_INTERNETARCHIVE_WAIT_ARCHIVE
+- Type: Duration
+- Default: 0s
-Setting this to larger values may increase the upload speed at the cost of using more memory.
-It can be set to smaller values smaller to save on memory.
+#### --internetarchive-encoding
-Properties:
+The encoding for the backend.
-- Config: chunk_size
-- Env Var: RCLONE_HIDRIVE_CHUNK_SIZE
-- Type: SizeSuffix
-- Default: 48Mi
+See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info.
-#### --hidrive-upload-cutoff
+Properties:
-Cutoff/Threshold for chunked uploads.
+- Config: encoding
+- Env Var: RCLONE_INTERNETARCHIVE_ENCODING
+- Type: Encoding
+- Default: Slash,LtGt,CrLf,Del,Ctl,InvalidUtf8,Dot
-Any files larger than this will be uploaded in chunks of the configured chunksize.
+### Metadata
-The upper limit for this is 2147483647 bytes (about 2.000Gi).
-That is the maximum amount of bytes a single upload-operation will support.
-Setting this above the upper limit will cause uploads to fail.
+Metadata fields provided by Internet Archive.
+If there are multiple values for a key, only the first one is returned.
+This is a limitation of Rclone, that supports one value per one key.
-Properties:
+Owner is able to add custom keys. Metadata feature grabs all the keys including them.
-- Config: upload_cutoff
-- Env Var: RCLONE_HIDRIVE_UPLOAD_CUTOFF
-- Type: SizeSuffix
-- Default: 96Mi
+Here are the possible system metadata items for the internetarchive backend.
-#### --hidrive-upload-concurrency
+| Name | Help | Type | Example | Read Only |
+|------|------|------|---------|-----------|
+| crc32 | CRC32 calculated by Internet Archive | string | 01234567 | **Y** |
+| format | Name of format identified by Internet Archive | string | Comma-Separated Values | **Y** |
+| md5 | MD5 hash calculated by Internet Archive | string | 01234567012345670123456701234567 | **Y** |
+| mtime | Time of last modification, managed by Rclone | RFC 3339 | 2006-01-02T15:04:05.999999999Z | **Y** |
+| name | Full file path, without the bucket part | filename | backend/internetarchive/internetarchive.go | **Y** |
+| old_version | Whether the file was replaced and moved by keep-old-version flag | boolean | true | **Y** |
+| rclone-ia-mtime | Time of last modification, managed by Internet Archive | RFC 3339 | 2006-01-02T15:04:05.999999999Z | N |
+| rclone-mtime | Time of last modification, managed by Rclone | RFC 3339 | 2006-01-02T15:04:05.999999999Z | N |
+| rclone-update-track | Random value used by Rclone for tracking changes inside Internet Archive | string | aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa | N |
+| sha1 | SHA1 hash calculated by Internet Archive | string | 0123456701234567012345670123456701234567 | **Y** |
+| size | File size in bytes | decimal number | 123456 | **Y** |
+| source | The source of the file | string | original | **Y** |
+| summation | Check https://forum.rclone.org/t/31922 for how it is used | string | md5 | **Y** |
+| viruscheck | The last time viruscheck process was run for the file (?) | unixtime | 1654191352 | **Y** |
-Concurrency for chunked uploads.
+See the [metadata](https://rclone.org/docs/#metadata) docs for more info.
-This is the upper limit for how many transfers for the same file are running concurrently.
-Setting this above to a value smaller than 1 will cause uploads to deadlock.
-If you are uploading small numbers of large files over high-speed links
-and these uploads do not fully utilize your bandwidth, then increasing
-this may help to speed up the transfers.
-Properties:
+# Jottacloud
-- Config: upload_concurrency
-- Env Var: RCLONE_HIDRIVE_UPLOAD_CONCURRENCY
-- Type: int
-- Default: 4
+Jottacloud is a cloud storage service provider from a Norwegian company, using its own datacenters
+in Norway. In addition to the official service at [jottacloud.com](https://www.jottacloud.com/),
+it also provides white-label solutions to different companies, such as:
+* Telia
+ * Telia Cloud (cloud.telia.se)
+ * Telia Sky (sky.telia.no)
+* Tele2
+ * Tele2 Cloud (mittcloud.tele2.se)
+* Onlime
+ * Onlime Cloud Storage (onlime.dk)
+* Elkjøp (with subsidiaries):
+ * Elkjøp Cloud (cloud.elkjop.no)
+ * Elgiganten Sweden (cloud.elgiganten.se)
+ * Elgiganten Denmark (cloud.elgiganten.dk)
+ * Giganti Cloud (cloud.gigantti.fi)
+ * ELKO Cloud (cloud.elko.is)
-#### --hidrive-encoding
+Most of the white-label versions are supported by this backend, although may require different
+authentication setup - described below.
-The encoding for the backend.
+Paths are specified as `remote:path`
-See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info.
+Paths may be as deep as required, e.g. `remote:directory/subdirectory`.
-Properties:
+## Authentication types
-- Config: encoding
-- Env Var: RCLONE_HIDRIVE_ENCODING
-- Type: MultiEncoder
-- Default: Slash,Dot
+Some of the whitelabel versions uses a different authentication method than the official service,
+and you have to choose the correct one when setting up the remote.
+### Standard authentication
+The standard authentication method used by the official service (jottacloud.com), as well as
+some of the whitelabel services, requires you to generate a single-use personal login token
+from the account security settings in the service's web interface. Log in to your account,
+go to "Settings" and then "Security", or use the direct link presented to you by rclone when
+configuring the remote: . Scroll down to the section
+"Personal login token", and click the "Generate" button. Note that if you are using a
+whitelabel service you probably can't use the direct link, you need to find the same page in
+their dedicated web interface, and also it may be in a different location than described above.
-## Limitations
+To access your account from multiple instances of rclone, you need to configure each of them
+with a separate personal login token. E.g. you create a Jottacloud remote with rclone in one
+location, and copy the configuration file to a second location where you also want to run
+rclone and access the same remote. Then you need to replace the token for one of them, using
+the [config reconnect](https://rclone.org/commands/rclone_config_reconnect/) command, which
+requires you to generate a new personal login token and supply as input. If you do not
+do this, the token may easily end up being invalidated, resulting in both instances failing
+with an error message something along the lines of:
-### Symbolic links
+ oauth2: cannot fetch token: 400 Bad Request
+ Response: {"error":"invalid_grant","error_description":"Stale token"}
-HiDrive is able to store symbolic links (*symlinks*) by design,
-for example, when unpacked from a zip archive.
+When this happens, you need to replace the token as described above to be able to use your
+remote again.
-There exists no direct mechanism to manage native symlinks in remotes.
-As such this implementation has chosen to ignore any native symlinks present in the remote.
-rclone will not be able to access or show any symlinks stored in the hidrive-remote.
-This means symlinks cannot be individually removed, copied, or moved,
-except when removing, copying, or moving the parent folder.
+All personal login tokens you have taken into use will be listed in the web interface under
+"My logged in devices", and from the right side of that list you can click the "X" button to
+revoke individual tokens.
-*This does not affect the `.rclonelink`-files
-that rclone uses to encode and store symbolic links.*
+### Legacy authentication
-### Sparse files
+If you are using one of the whitelabel versions (e.g. from Elkjøp) you may not have the option
+to generate a CLI token. In this case you'll have to use the legacy authentication. To do this select
+yes when the setup asks for legacy authentication and enter your username and password.
+The rest of the setup is identical to the default setup.
-It is possible to store sparse files in HiDrive.
+### Telia Cloud authentication
-Note that copying a sparse file will expand the holes
-into null-byte (0x00) regions that will then consume disk space.
-Likewise, when downloading a sparse file,
-the resulting file will have null-byte regions in the place of file holes.
+Similar to other whitelabel versions Telia Cloud doesn't offer the option of creating a CLI token, and
+additionally uses a separate authentication flow where the username is generated internally. To setup
+rclone to use Telia Cloud, choose Telia Cloud authentication in the setup. The rest of the setup is
+identical to the default setup.
-# HTTP
+### Tele2 Cloud authentication
-The HTTP remote is a read only remote for reading files of a
-webserver. The webserver should provide file listings which rclone
-will read and turn into a remote. This has been tested with common
-webservers such as Apache/Nginx/Caddy and will likely work with file
-listings from most web servers. (If it doesn't then please file an
-issue, or send a pull request!)
+As Tele2-Com Hem merger was completed this authentication can be used for former Com Hem Cloud and
+Tele2 Cloud customers as no support for creating a CLI token exists, and additionally uses a separate
+authentication flow where the username is generated internally. To setup rclone to use Tele2 Cloud,
+choose Tele2 Cloud authentication in the setup. The rest of the setup is identical to the default setup.
-Paths are specified as `remote:` or `remote:path`.
+### Onlime Cloud Storage authentication
-The `remote:` represents the configured [url](#http-url), and any path following
-it will be resolved relative to this url, according to the URL standard. This
-means with remote url `https://beta.rclone.org/branch` and path `fix`, the
-resolved URL will be `https://beta.rclone.org/branch/fix`, while with path
-`/fix` the resolved URL will be `https://beta.rclone.org/fix` as the absolute
-path is resolved from the root of the domain.
+Onlime has sold access to Jottacloud proper, while providing localized support to Danish Customers, but
+have recently set up their own hosting, transferring their customers from Jottacloud servers to their
+own ones.
-If the path following the `remote:` ends with `/` it will be assumed to point
-to a directory. If the path does not end with `/`, then a HEAD request is sent
-and the response used to decide if it it is treated as a file or a directory
-(run with `-vv` to see details). When [--http-no-head](#http-no-head) is
-specified, a path without ending `/` is always assumed to be a file. If rclone
-incorrectly assumes the path is a file, the solution is to specify the path with
-ending `/`. When you know the path is a directory, ending it with `/` is always
-better as it avoids the initial HEAD request.
+This, of course, necessitates using their servers for authentication, but otherwise functionality and
+architecture seems equivalent to Jottacloud.
-To just download a single file it is easier to use
-[copyurl](https://rclone.org/commands/rclone_copyurl/).
+To setup rclone to use Onlime Cloud Storage, choose Onlime Cloud authentication in the setup. The rest
+of the setup is identical to the default setup.
## Configuration
-Here is an example of how to make a remote called `remote`. First
-run:
+Here is an example of how to make a remote called `remote` with the default setup. First run:
- rclone config
+ rclone config
This will guide you through an interactive setup process:
@@ -35351,267 +36727,390 @@ s) Set configuration password
q) Quit config
n/s/q> n
name> remote
+Option Storage.
Type of storage to configure.
-Choose a number from below, or type in your own value
+Choose a number from below, or type in your own value.
[snip]
-XX / HTTP
- \ "http"
+XX / Jottacloud
+ \ (jottacloud)
[snip]
-Storage> http
-URL of http host to connect to
-Choose a number from below, or type in your own value
- 1 / Connect to example.com
- \ "https://example.com"
-url> https://beta.rclone.org
-Remote config
+Storage> jottacloud
+Edit advanced config?
+y) Yes
+n) No (default)
+y/n> n
+Option config_type.
+Select authentication type.
+Choose a number from below, or type in an existing string value.
+Press Enter for the default (standard).
+ / Standard authentication.
+ 1 | Use this if you're a normal Jottacloud user.
+ \ (standard)
+ / Legacy authentication.
+ 2 | This is only required for certain whitelabel versions of Jottacloud and not recommended for normal users.
+ \ (legacy)
+ / Telia Cloud authentication.
+ 3 | Use this if you are using Telia Cloud.
+ \ (telia)
+ / Tele2 Cloud authentication.
+ 4 | Use this if you are using Tele2 Cloud.
+ \ (tele2)
+ / Onlime Cloud authentication.
+ 5 | Use this if you are using Onlime Cloud.
+ \ (onlime)
+config_type> 1
+Personal login token.
+Generate here: https://www.jottacloud.com/web/secure
+Login Token>
+Use a non-standard device/mountpoint?
+Choosing no, the default, will let you access the storage used for the archive
+section of the official Jottacloud client. If you instead want to access the
+sync or the backup section, for example, you must choose yes.
+y) Yes
+n) No (default)
+y/n> y
+Option config_device.
+The device to use. In standard setup the built-in Jotta device is used,
+which contains predefined mountpoints for archive, sync etc. All other devices
+are treated as backup devices by the official Jottacloud client. You may create
+a new by entering a unique name.
+Choose a number from below, or type in your own string value.
+Press Enter for the default (DESKTOP-3H31129).
+ 1 > DESKTOP-3H31129
+ 2 > Jotta
+config_device> 2
+Option config_mountpoint.
+The mountpoint to use for the built-in device Jotta.
+The standard setup is to use the Archive mountpoint. Most other mountpoints
+have very limited support in rclone and should generally be avoided.
+Choose a number from below, or type in an existing string value.
+Press Enter for the default (Archive).
+ 1 > Archive
+ 2 > Shared
+ 3 > Sync
+config_mountpoint> 1
--------------------
[remote]
-url = https://beta.rclone.org
+type = jottacloud
+configVersion = 1
+client_id = jottacli
+client_secret =
+tokenURL = https://id.jottacloud.com/auth/realms/jottacloud/protocol/openid-connect/token
+token = {........}
+username = 2940e57271a93d987d6f8a21
+device = Jotta
+mountpoint = Archive
--------------------
-y) Yes this is OK
+y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y
-Current remotes:
-
-Name Type
-==== ====
-remote http
-
-e) Edit existing remote
-n) New remote
-d) Delete remote
-r) Rename remote
-c) Copy remote
-s) Set configuration password
-q) Quit config
-e/n/d/r/c/s/q> q
```
-This remote is called `remote` and can now be used like this
+Once configured you can then use `rclone` like this,
-See all the top level directories
+List directories in top level of your Jottacloud
rclone lsd remote:
-List the contents of a directory
+List all the files in your Jottacloud
- rclone ls remote:directory
+ rclone ls remote:
-Sync the remote `directory` to `/home/local/directory`, deleting any excess files.
+To copy a local directory to an Jottacloud directory called backup
- rclone sync --interactive remote:directory /home/local/directory
+ rclone copy /home/source remote:backup
-### Read only
+### Devices and Mountpoints
-This remote is read only - you can't upload files to an HTTP server.
+The official Jottacloud client registers a device for each computer you install
+it on, and shows them in the backup section of the user interface. For each
+folder you select for backup it will create a mountpoint within this device.
+A built-in device called Jotta is special, and contains mountpoints Archive,
+Sync and some others, used for corresponding features in official clients.
-### Modified time
+With rclone you'll want to use the standard Jotta/Archive device/mountpoint in
+most cases. However, you may for example want to access files from the sync or
+backup functionality provided by the official clients, and rclone therefore
+provides the option to select other devices and mountpoints during config.
-Most HTTP servers store time accurate to 1 second.
+You are allowed to create new devices and mountpoints. All devices except the
+built-in Jotta device are treated as backup devices by official Jottacloud
+clients, and the mountpoints on them are individual backup sets.
-### Checksum
+With the built-in Jotta device, only existing, built-in, mountpoints can be
+selected. In addition to the mentioned Archive and Sync, it may contain
+several other mountpoints such as: Latest, Links, Shared and Trash. All of
+these are special mountpoints with a different internal representation than
+the "regular" mountpoints. Rclone will only to a very limited degree support
+them. Generally you should avoid these, unless you know what you are doing.
-No checksums are stored.
+### --fast-list
-### Usage without a config file
+This backend supports `--fast-list` which allows you to use fewer
+transactions in exchange for more memory. See the [rclone
+docs](https://rclone.org/docs/#fast-list) for more details.
-Since the http remote only has one config parameter it is easy to use
-without a config file:
+Note that the implementation in Jottacloud always uses only a single
+API request to get the entire list, so for large folders this could
+lead to long wait time before the first results are shown.
- rclone lsd --http-url https://beta.rclone.org :http:
+Note also that with rclone version 1.58 and newer, information about
+[MIME types](https://rclone.org/overview/#mime-type) and metadata item [utime](#metadata)
+are not available when using `--fast-list`.
-or:
+### Modification times and hashes
- rclone lsd :http,url='https://beta.rclone.org':
+Jottacloud allows modification times to be set on objects accurate to 1
+second. These will be used to detect whether objects need syncing or
+not.
+
+Jottacloud supports MD5 type hashes, so you can use the `--checksum`
+flag.
+
+Note that Jottacloud requires the MD5 hash before upload so if the
+source does not have an MD5 checksum then the file will be cached
+temporarily on disk (in location given by
+[--temp-dir](https://rclone.org/docs/#temp-dir-dir)) before it is uploaded.
+Small files will be cached in memory - see the
+[--jottacloud-md5-memory-limit](#jottacloud-md5-memory-limit) flag.
+When uploading from local disk the source checksum is always available,
+so this does not apply. Starting with rclone version 1.52 the same is
+true for encrypted remotes (in older versions the crypt backend would not
+calculate hashes for uploads from local disk, so the Jottacloud
+backend had to do it as described above).
+
+### Restricted filename characters
+
+In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters)
+the following characters are also replaced:
+
+| Character | Value | Replacement |
+| --------- |:-----:|:-----------:|
+| " | 0x22 | ＂ |
+| * | 0x2A | ＊ |
+| : | 0x3A | ： |
+| < | 0x3C | ＜ |
+| > | 0x3E | ＞ |
+| ? | 0x3F | ？ |
+| \| | 0x7C | ｜ |
+
+Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8),
+as they can't be used in XML strings.
+
+### Deleting files
+
+By default, rclone will send all files to the trash when deleting files. They will be permanently
+deleted automatically after 30 days. You may bypass the trash and permanently delete files immediately
+by using the [--jottacloud-hard-delete](#jottacloud-hard-delete) flag, or set the equivalent environment variable.
+Emptying the trash is supported by the [cleanup](https://rclone.org/commands/rclone_cleanup/) command.
+
+### Versions
+
+Jottacloud supports file versioning. When rclone uploads a new version of a file it creates a new version of it.
+Currently rclone only supports retrieving the current version but older versions can be accessed via the Jottacloud Website.
+
+Versioning can be disabled by `--jottacloud-no-versions` option. This is achieved by deleting the remote file prior to uploading
+a new version. If the upload the fails no version of the file will be available in the remote.
+
+### Quota information
+
+To view your current quota you can use the `rclone about remote:`
+command which will display your usage limit (unless it is unlimited)
+and the current usage.
### Standard options
-Here are the Standard options specific to http (HTTP).
+Here are the Standard options specific to jottacloud (Jottacloud).
-#### --http-url
+#### --jottacloud-client-id
-URL of HTTP host to connect to.
+OAuth Client Id.
-E.g. "https://example.com", or "https://user:pass@example.com" to use a username and password.
+Leave blank normally.
Properties:
-- Config: url
-- Env Var: RCLONE_HTTP_URL
+- Config: client_id
+- Env Var: RCLONE_JOTTACLOUD_CLIENT_ID
- Type: string
-- Required: true
+- Required: false
+
+#### --jottacloud-client-secret
+
+OAuth Client Secret.
+
+Leave blank normally.
+
+Properties:
+
+- Config: client_secret
+- Env Var: RCLONE_JOTTACLOUD_CLIENT_SECRET
+- Type: string
+- Required: false
### Advanced options
-Here are the Advanced options specific to http (HTTP).
+Here are the Advanced options specific to jottacloud (Jottacloud).
-#### --http-headers
+#### --jottacloud-token
-Set HTTP headers for all transactions.
+OAuth Access Token as a JSON blob.
-Use this to set additional HTTP headers for all transactions.
+Properties:
-The input format is comma separated list of key,value pairs. Standard
-[CSV encoding](https://godoc.org/encoding/csv) may be used.
+- Config: token
+- Env Var: RCLONE_JOTTACLOUD_TOKEN
+- Type: string
+- Required: false
-For example, to set a Cookie use 'Cookie,name=value', or '"Cookie","name=value"'.
+#### --jottacloud-auth-url
-You can set multiple headers, e.g. '"Cookie","name=value","Authorization","xxx"'.
+Auth server URL.
+
+Leave blank to use the provider defaults.
Properties:
-- Config: headers
-- Env Var: RCLONE_HTTP_HEADERS
-- Type: CommaSepList
-- Default:
+- Config: auth_url
+- Env Var: RCLONE_JOTTACLOUD_AUTH_URL
+- Type: string
+- Required: false
-#### --http-no-slash
+#### --jottacloud-token-url
-Set this if the site doesn't end directories with /.
+Token server url.
-Use this if your target website does not use / on the end of
-directories.
+Leave blank to use the provider defaults.
-A / on the end of a path is how rclone normally tells the difference
-between files and directories. If this flag is set, then rclone will
-treat all files with Content-Type: text/html as directories and read
-URLs from them rather than downloading them.
+Properties:
-Note that this may cause rclone to confuse genuine HTML files with
-directories.
+- Config: token_url
+- Env Var: RCLONE_JOTTACLOUD_TOKEN_URL
+- Type: string
+- Required: false
+
+#### --jottacloud-md5-memory-limit
+
+Files bigger than this will be cached on disk to calculate the MD5 if required.
Properties:
-- Config: no_slash
-- Env Var: RCLONE_HTTP_NO_SLASH
-- Type: bool
-- Default: false
+- Config: md5_memory_limit
+- Env Var: RCLONE_JOTTACLOUD_MD5_MEMORY_LIMIT
+- Type: SizeSuffix
+- Default: 10Mi
-#### --http-no-head
+#### --jottacloud-trashed-only
-Don't use HEAD requests.
+Only show files that are in the trash.
-HEAD requests are mainly used to find file sizes in dir listing.
-If your site is being very slow to load then you can try this option.
-Normally rclone does a HEAD request for each potential file in a
-directory listing to:
+This will show trashed files in their original directory structure.
-- find its size
-- check it really exists
-- check to see if it is a directory
+Properties:
-If you set this option, rclone will not do the HEAD request. This will mean
-that directory listings are much quicker, but rclone won't have the times or
-sizes of any files, and some files that don't exist may be in the listing.
+- Config: trashed_only
+- Env Var: RCLONE_JOTTACLOUD_TRASHED_ONLY
+- Type: bool
+- Default: false
+
+#### --jottacloud-hard-delete
+
+Delete files permanently rather than putting them into the trash.
Properties:
-- Config: no_head
-- Env Var: RCLONE_HTTP_NO_HEAD
+- Config: hard_delete
+- Env Var: RCLONE_JOTTACLOUD_HARD_DELETE
- Type: bool
- Default: false
+#### --jottacloud-upload-resume-limit
+Files bigger than this can be resumed if the upload fail's.
-## Limitations
-
-`rclone about` is not supported by the HTTP backend. Backends without
-this capability cannot determine free space for an rclone mount or
-use policy `mfs` (most free space) as a member of an rclone union
-remote.
+Properties:
-See [List of backends that do not support rclone about](https://rclone.org/overview/#optional-features) and [rclone about](https://rclone.org/commands/rclone_about/)
+- Config: upload_resume_limit
+- Env Var: RCLONE_JOTTACLOUD_UPLOAD_RESUME_LIMIT
+- Type: SizeSuffix
+- Default: 10Mi
-# Internet Archive
+#### --jottacloud-no-versions
-The Internet Archive backend utilizes Items on [archive.org](https://archive.org/)
+Avoid server side versioning by deleting files and recreating files instead of overwriting them.
-Refer to [IAS3 API documentation](https://archive.org/services/docs/api/ias3.html) for the API this backend uses.
+Properties:
-Paths are specified as `remote:bucket` (or `remote:` for the `lsd`
-command.) You may put subdirectories in too, e.g. `remote:item/path/to/dir`.
+- Config: no_versions
+- Env Var: RCLONE_JOTTACLOUD_NO_VERSIONS
+- Type: bool
+- Default: false
-Unlike S3, listing up all items uploaded by you isn't supported.
+#### --jottacloud-encoding
-Once you have made a remote, you can use it like this:
+The encoding for the backend.
-Make a new item
+See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info.
- rclone mkdir remote:item
+Properties:
-List the contents of a item
+- Config: encoding
+- Env Var: RCLONE_JOTTACLOUD_ENCODING
+- Type: Encoding
+- Default: Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,Del,Ctl,InvalidUtf8,Dot
- rclone ls remote:item
+### Metadata
-Sync `/home/local/directory` to the remote item, deleting any excess
-files in the item.
+Jottacloud has limited support for metadata, currently an extended set of timestamps.
- rclone sync --interactive /home/local/directory remote:item
+Here are the possible system metadata items for the jottacloud backend.
-## Notes
-Because of Internet Archive's architecture, it enqueues write operations (and extra post-processings) in a per-item queue. You can check item's queue at https://catalogd.archive.org/history/item-name-here . Because of that, all uploads/deletes will not show up immediately and takes some time to be available.
-The per-item queue is enqueued to an another queue, Item Deriver Queue. [You can check the status of Item Deriver Queue here.](https://catalogd.archive.org/catalog.php?whereami=1) This queue has a limit, and it may block you from uploading, or even deleting. You should avoid uploading a lot of small files for better behavior.
+| Name | Help | Type | Example | Read Only |
+|------|------|------|---------|-----------|
+| btime | Time of file birth (creation), read from rclone metadata | RFC 3339 | 2006-01-02T15:04:05.999999999Z07:00 | N |
+| content-type | MIME type, also known as media type | string | text/plain | **Y** |
+| mtime | Time of last modification, read from rclone metadata | RFC 3339 | 2006-01-02T15:04:05.999999999Z07:00 | N |
+| utime | Time of last upload, when current revision was created, generated by backend | RFC 3339 | 2006-01-02T15:04:05.999999999Z07:00 | **Y** |
-You can optionally wait for the server's processing to finish, by setting non-zero value to `wait_archive` key.
-By making it wait, rclone can do normal file comparison.
-Make sure to set a large enough value (e.g. `30m0s` for smaller files) as it can take a long time depending on server's queue.
+See the [metadata](https://rclone.org/docs/#metadata) docs for more info.
-## About metadata
-This backend supports setting, updating and reading metadata of each file.
-The metadata will appear as file metadata on Internet Archive.
-However, some fields are reserved by both Internet Archive and rclone.
-The following are reserved by Internet Archive:
-- `name`
-- `source`
-- `size`
-- `md5`
-- `crc32`
-- `sha1`
-- `format`
-- `old_version`
-- `viruscheck`
-- `summation`
-Trying to set values to these keys is ignored with a warning.
-Only setting `mtime` is an exception. Doing so make it the identical behavior as setting ModTime.
+## Limitations
-rclone reserves all the keys starting with `rclone-`. Setting value for these keys will give you warnings, but values are set according to request.
+Note that Jottacloud is case insensitive so you can't have a file called
+"Hello.doc" and one called "hello.doc".
-If there are multiple values for a key, only the first one is returned.
-This is a limitation of rclone, that supports one value per one key.
-It can be triggered when you did a server-side copy.
+There are quite a few characters that can't be in Jottacloud file names. Rclone will map these names to and from an identical
+looking unicode equivalent. For example if a file has a ? in it will be mapped to ？ instead.
-Reading metadata will also provide custom (non-standard nor reserved) ones.
+Jottacloud only supports filenames up to 255 characters in length.
-## Filtering auto generated files
+## Troubleshooting
-The Internet Archive automatically creates metadata files after
-upload. These can cause problems when doing an `rclone sync` as rclone
-will try, and fail, to delete them. These metadata files are not
-changeable, as they are created by the Internet Archive automatically.
+Jottacloud exhibits some inconsistent behaviours regarding deleted files and folders which may cause Copy, Move and DirMove
+operations to previously deleted paths to fail. Emptying the trash should help in such cases.
-These auto-created files can be excluded from the sync using [metadata
-filtering](https://rclone.org/filtering/#metadata).
+# Koofr
- rclone sync ... --metadata-exclude "source=metadata" --metadata-exclude "format=Metadata"
+Paths are specified as `remote:path`
-Which excludes from the sync any files which have the
-`source=metadata` or `format=Metadata` flags which are added to
-Internet Archive auto-created files.
+Paths may be as deep as required, e.g. `remote:directory/subdirectory`.
## Configuration
-Here is an example of making an internetarchive configuration.
-Most applies to the other providers as well, any differences are described [below](#providers).
+The initial setup for Koofr involves creating an application password for
+rclone. You can do that by opening the Koofr
+[web application](https://app.koofr.net/app/admin/preferences/password),
+giving the password a nice name like `rclone` and clicking on generate.
-First run
+Here is an example of how to make a remote called `koofr`. First run:
- rclone config
+ rclone config
-This will guide you through an interactive setup process.
+This will guide you through an interactive setup process:
```
No remotes found, make a new one?
@@ -35619,60 +37118,51 @@ n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
-name> remote
+name> koofr
Option Storage.
Type of storage to configure.
Choose a number from below, or type in your own value.
-XX / InternetArchive Items
- \ (internetarchive)
-Storage> internetarchive
-Option access_key_id.
-IAS3 Access Key.
-Leave blank for anonymous access.
-You can find one here: https://archive.org/account/s3.php
-Enter a value. Press Enter to leave empty.
-access_key_id> XXXX
-Option secret_access_key.
-IAS3 Secret Key (password).
-Leave blank for anonymous access.
-Enter a value. Press Enter to leave empty.
-secret_access_key> XXXX
-Edit advanced config?
-y) Yes
-n) No (default)
-y/n> y
-Option endpoint.
-IAS3 Endpoint.
-Leave blank for default value.
-Enter a string value. Press Enter for the default (https://s3.us.archive.org).
-endpoint>
-Option front_endpoint.
-Host of InternetArchive Frontend.
-Leave blank for default value.
-Enter a string value. Press Enter for the default (https://archive.org).
-front_endpoint>
-Option disable_checksum.
-Don't store MD5 checksum with object metadata.
-Normally rclone will calculate the MD5 checksum of the input before
-uploading it so it can ask the server to check the object against checksum.
-This is great for data integrity checking but can cause long delays for
-large files to start uploading.
-Enter a boolean value (true or false). Press Enter for the default (true).
-disable_checksum> true
-Option encoding.
-The encoding for the backend.
-See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info.
-Enter a encoder.MultiEncoder value. Press Enter for the default (Slash,Question,Hash,Percent,Del,Ctl,InvalidUtf8,Dot).
-encoding>
+[snip]
+22 / Koofr, Digi Storage and other Koofr-compatible storage providers
+ \ (koofr)
+[snip]
+Storage> koofr
+Option provider.
+Choose your storage provider.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+ 1 / Koofr, https://app.koofr.net/
+ \ (koofr)
+ 2 / Digi Storage, https://storage.rcs-rds.ro/
+ \ (digistorage)
+ 3 / Any other Koofr API compatible storage service
+ \ (other)
+provider> 1
+Option user.
+Your user name.
+Enter a value.
+user> USERNAME
+Option password.
+Your password for rclone (generate one at https://app.koofr.net/app/admin/preferences/password).
+Choose an alternative below.
+y) Yes, type in my own password
+g) Generate random password
+y/g> y
+Enter the password:
+password:
+Confirm the password:
+password:
Edit advanced config?
y) Yes
n) No (default)
y/n> n
+Remote config
--------------------
-[remote]
-type = internetarchive
-access_key_id = XXXX
-secret_access_key = XXXX
+[koofr]
+type = koofr
+provider = koofr
+user = USERNAME
+password = *** ENCRYPTED ***
--------------------
y) Yes this is OK (default)
e) Edit this remote
@@ -35680,97 +37170,127 @@ d) Delete this remote
y/e/d> y
```
+You can choose to edit advanced config in order to enter your own service URL
+if you use an on-premise or white label Koofr instance, or choose an alternative
+mount instead of your primary storage.
-### Standard options
+Once configured you can then use `rclone` like this,
-Here are the Standard options specific to internetarchive (Internet Archive).
+List directories in top level of your Koofr
-#### --internetarchive-access-key-id
+ rclone lsd koofr:
-IAS3 Access Key.
+List all the files in your Koofr
-Leave blank for anonymous access.
-You can find one here: https://archive.org/account/s3.php
+ rclone ls koofr:
-Properties:
+To copy a local directory to an Koofr directory called backup
-- Config: access_key_id
-- Env Var: RCLONE_INTERNETARCHIVE_ACCESS_KEY_ID
-- Type: string
-- Required: false
+ rclone copy /home/source koofr:backup
-#### --internetarchive-secret-access-key
+### Restricted filename characters
-IAS3 Secret Key (password).
+In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters)
+the following characters are also replaced:
-Leave blank for anonymous access.
+| Character | Value | Replacement |
+| --------- |:-----:|:-----------:|
+| \ | 0x5C | ï¼¼ |
+
+Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8),
+as they can't be used in XML strings.
+
+
+### Standard options
+
+Here are the Standard options specific to koofr (Koofr, Digi Storage and other Koofr-compatible storage providers).
+
+#### --koofr-provider
+
+Choose your storage provider.
Properties:
-- Config: secret_access_key
-- Env Var: RCLONE_INTERNETARCHIVE_SECRET_ACCESS_KEY
+- Config: provider
+- Env Var: RCLONE_KOOFR_PROVIDER
- Type: string
- Required: false
+- Examples:
+ - "koofr"
+ - Koofr, https://app.koofr.net/
+ - "digistorage"
+ - Digi Storage, https://storage.rcs-rds.ro/
+ - "other"
+ - Any other Koofr API compatible storage service
-### Advanced options
+#### --koofr-endpoint
-Here are the Advanced options specific to internetarchive (Internet Archive).
+The Koofr API endpoint to use.
-#### --internetarchive-endpoint
+Properties:
-IAS3 Endpoint.
+- Config: endpoint
+- Env Var: RCLONE_KOOFR_ENDPOINT
+- Provider: other
+- Type: string
+- Required: true
-Leave blank for default value.
+#### --koofr-user
+
+Your user name.
Properties:
-- Config: endpoint
-- Env Var: RCLONE_INTERNETARCHIVE_ENDPOINT
+- Config: user
+- Env Var: RCLONE_KOOFR_USER
- Type: string
-- Default: "https://s3.us.archive.org"
+- Required: true
-#### --internetarchive-front-endpoint
+#### --koofr-password
-Host of InternetArchive Frontend.
+Your password for rclone (generate one at https://app.koofr.net/app/admin/preferences/password).
-Leave blank for default value.
+**NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/).
Properties:
-- Config: front_endpoint
-- Env Var: RCLONE_INTERNETARCHIVE_FRONT_ENDPOINT
+- Config: password
+- Env Var: RCLONE_KOOFR_PASSWORD
+- Provider: koofr
- Type: string
-- Default: "https://archive.org"
+- Required: true
-#### --internetarchive-disable-checksum
+### Advanced options
-Don't ask the server to test against MD5 checksum calculated by rclone.
-Normally rclone will calculate the MD5 checksum of the input before
-uploading it so it can ask the server to check the object against checksum.
-This is great for data integrity checking but can cause long delays for
-large files to start uploading.
+Here are the Advanced options specific to koofr (Koofr, Digi Storage and other Koofr-compatible storage providers).
+
+#### --koofr-mountid
+
+Mount ID of the mount to use.
+
+If omitted, the primary mount is used.
Properties:
-- Config: disable_checksum
-- Env Var: RCLONE_INTERNETARCHIVE_DISABLE_CHECKSUM
-- Type: bool
-- Default: true
+- Config: mountid
+- Env Var: RCLONE_KOOFR_MOUNTID
+- Type: string
+- Required: false
-#### --internetarchive-wait-archive
+#### --koofr-setmtime
-Timeout for waiting the server's processing tasks (specifically archive and book_op) to finish.
-Only enable if you need to be guaranteed to be reflected after write operations.
-0 to disable waiting. No errors to be thrown in case of timeout.
+Does the backend support setting modification time.
+
+Set this to false if you use a mount ID that points to a Dropbox or Amazon Drive backend.
Properties:
-- Config: wait_archive
-- Env Var: RCLONE_INTERNETARCHIVE_WAIT_ARCHIVE
-- Type: Duration
-- Default: 0s
+- Config: setmtime
+- Env Var: RCLONE_KOOFR_SETMTIME
+- Type: bool
+- Default: true
-#### --internetarchive-encoding
+#### --koofr-encoding
The encoding for the backend.
@@ -35779,138 +37299,268 @@ See the [encoding section in the overview](https://rclone.org/overview/#encoding
Properties:
- Config: encoding
-- Env Var: RCLONE_INTERNETARCHIVE_ENCODING
-- Type: MultiEncoder
-- Default: Slash,LtGt,CrLf,Del,Ctl,InvalidUtf8,Dot
+- Env Var: RCLONE_KOOFR_ENCODING
+- Type: Encoding
+- Default: Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot
-### Metadata
-Metadata fields provided by Internet Archive.
-If there are multiple values for a key, only the first one is returned.
-This is a limitation of Rclone, that supports one value per one key.
-Owner is able to add custom keys. Metadata feature grabs all the keys including them.
+## Limitations
-Here are the possible system metadata items for the internetarchive backend.
+Note that Koofr is case insensitive so you can't have a file called
+"Hello.doc" and one called "hello.doc".
-| Name | Help | Type | Example | Read Only |
-|------|------|------|---------|-----------|
-| crc32 | CRC32 calculated by Internet Archive | string | 01234567 | **Y** |
-| format | Name of format identified by Internet Archive | string | Comma-Separated Values | **Y** |
-| md5 | MD5 hash calculated by Internet Archive | string | 01234567012345670123456701234567 | **Y** |
-| mtime | Time of last modification, managed by Rclone | RFC 3339 | 2006-01-02T15:04:05.999999999Z | **Y** |
-| name | Full file path, without the bucket part | filename | backend/internetarchive/internetarchive.go | **Y** |
-| old_version | Whether the file was replaced and moved by keep-old-version flag | boolean | true | **Y** |
-| rclone-ia-mtime | Time of last modification, managed by Internet Archive | RFC 3339 | 2006-01-02T15:04:05.999999999Z | N |
-| rclone-mtime | Time of last modification, managed by Rclone | RFC 3339 | 2006-01-02T15:04:05.999999999Z | N |
-| rclone-update-track | Random value used by Rclone for tracking changes inside Internet Archive | string | aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa | N |
-| sha1 | SHA1 hash calculated by Internet Archive | string | 0123456701234567012345670123456701234567 | **Y** |
-| size | File size in bytes | decimal number | 123456 | **Y** |
-| source | The source of the file | string | original | **Y** |
-| summation | Check https://forum.rclone.org/t/31922 for how it is used | string | md5 | **Y** |
-| viruscheck | The last time viruscheck process was run for the file (?) | unixtime | 1654191352 | **Y** |
+## Providers
-See the [metadata](https://rclone.org/docs/#metadata) docs for more info.
+### Koofr
+
+This is the original [Koofr](https://koofr.eu) storage provider used as main example and described in the [configuration](#configuration) section above.
+
+### Digi Storage
+
+[Digi Storage](https://www.digi.ro/servicii/online/digi-storage) is a cloud storage service run by [Digi.ro](https://www.digi.ro/) that
+provides a Koofr API.
+
+Here is an example of how to make a remote called `ds`. First run:
+
+ rclone config
+
+This will guide you through an interactive setup process:
+
+```
+No remotes found, make a new one?
+n) New remote
+s) Set configuration password
+q) Quit config
+n/s/q> n
+name> ds
+Option Storage.
+Type of storage to configure.
+Choose a number from below, or type in your own value.
+[snip]
+22 / Koofr, Digi Storage and other Koofr-compatible storage providers
+ \ (koofr)
+[snip]
+Storage> koofr
+Option provider.
+Choose your storage provider.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+ 1 / Koofr, https://app.koofr.net/
+ \ (koofr)
+ 2 / Digi Storage, https://storage.rcs-rds.ro/
+ \ (digistorage)
+ 3 / Any other Koofr API compatible storage service
+ \ (other)
+provider> 2
+Option user.
+Your user name.
+Enter a value.
+user> USERNAME
+Option password.
+Your password for rclone (generate one at https://storage.rcs-rds.ro/app/admin/preferences/password).
+Choose an alternative below.
+y) Yes, type in my own password
+g) Generate random password
+y/g> y
+Enter the password:
+password:
+Confirm the password:
+password:
+Edit advanced config?
+y) Yes
+n) No (default)
+y/n> n
+--------------------
+[ds]
+type = koofr
+provider = digistorage
+user = USERNAME
+password = *** ENCRYPTED ***
+--------------------
+y) Yes this is OK (default)
+e) Edit this remote
+d) Delete this remote
+y/e/d> y
+```
+
+### Other
+
+You may also want to use another, public or private storage provider that runs a Koofr API compatible service, by simply providing the base URL to connect to.
+
+Here is an example of how to make a remote called `other`. First run:
+
+ rclone config
+
+This will guide you through an interactive setup process:
+
+```
+No remotes found, make a new one?
+n) New remote
+s) Set configuration password
+q) Quit config
+n/s/q> n
+name> other
+Option Storage.
+Type of storage to configure.
+Choose a number from below, or type in your own value.
+[snip]
+22 / Koofr, Digi Storage and other Koofr-compatible storage providers
+ \ (koofr)
+[snip]
+Storage> koofr
+Option provider.
+Choose your storage provider.
+Choose a number from below, or type in your own value.
+Press Enter to leave empty.
+ 1 / Koofr, https://app.koofr.net/
+ \ (koofr)
+ 2 / Digi Storage, https://storage.rcs-rds.ro/
+ \ (digistorage)
+ 3 / Any other Koofr API compatible storage service
+ \ (other)
+provider> 3
+Option endpoint.
+The Koofr API endpoint to use.
+Enter a value.
+endpoint> https://koofr.other.org
+Option user.
+Your user name.
+Enter a value.
+user> USERNAME
+Option password.
+Your password for rclone (generate one at your service's settings page).
+Choose an alternative below.
+y) Yes, type in my own password
+g) Generate random password
+y/g> y
+Enter the password:
+password:
+Confirm the password:
+password:
+Edit advanced config?
+y) Yes
+n) No (default)
+y/n> n
+--------------------
+[other]
+type = koofr
+provider = other
+endpoint = https://koofr.other.org
+user = USERNAME
+password = *** ENCRYPTED ***
+--------------------
+y) Yes this is OK (default)
+e) Edit this remote
+d) Delete this remote
+y/e/d> y
+```
+# Linkbox
+
+Linkbox is [a private cloud drive](https://linkbox.to/).
+
+## Configuration
+
+Here is an example of making a remote for Linkbox.
+
+First run:
+
+ rclone config
+
+This will guide you through an interactive setup process:
+
+```
+No remotes found, make a new one?
+n) New remote
+s) Set configuration password
+q) Quit config
+n/s/q> n
+Enter name for new remote.
+name> remote
-# Jottacloud
+Option Storage.
+Type of storage to configure.
+Choose a number from below, or type in your own value.
+XX / Linkbox
+ \ (linkbox)
+Storage> XX
-Jottacloud is a cloud storage service provider from a Norwegian company, using its own datacenters
-in Norway. In addition to the official service at [jottacloud.com](https://www.jottacloud.com/),
-it also provides white-label solutions to different companies, such as:
-* Telia
- * Telia Cloud (cloud.telia.se)
- * Telia Sky (sky.telia.no)
-* Tele2
- * Tele2 Cloud (mittcloud.tele2.se)
-* Onlime
- * Onlime Cloud Storage (onlime.dk)
-* Elkjøp (with subsidiaries):
- * Elkjøp Cloud (cloud.elkjop.no)
- * Elgiganten Sweden (cloud.elgiganten.se)
- * Elgiganten Denmark (cloud.elgiganten.dk)
- * Giganti Cloud (cloud.gigantti.fi)
- * ELKO Cloud (cloud.elko.is)
+Option token.
+Token from https://www.linkbox.to/admin/account
+Enter a value.
+token> testFromCLToken
-Most of the white-label versions are supported by this backend, although may require different
-authentication setup - described below.
+Configuration complete.
+Options:
+- type: linkbox
+- token: XXXXXXXXXXX
+Keep this "linkbox" remote?
+y) Yes this is OK (default)
+e) Edit this remote
+d) Delete this remote
+y/e/d> y
-Paths are specified as `remote:path`
+```
-Paths may be as deep as required, e.g. `remote:directory/subdirectory`.
-## Authentication types
+### Standard options
-Some of the whitelabel versions uses a different authentication method than the official service,
-and you have to choose the correct one when setting up the remote.
+Here are the Standard options specific to linkbox (Linkbox).
-### Standard authentication
+#### --linkbox-token
-The standard authentication method used by the official service (jottacloud.com), as well as
-some of the whitelabel services, requires you to generate a single-use personal login token
-from the account security settings in the service's web interface. Log in to your account,
-go to "Settings" and then "Security", or use the direct link presented to you by rclone when
-configuring the remote: . Scroll down to the section
-"Personal login token", and click the "Generate" button. Note that if you are using a
-whitelabel service you probably can't use the direct link, you need to find the same page in
-their dedicated web interface, and also it may be in a different location than described above.
+Token from https://www.linkbox.to/admin/account
-To access your account from multiple instances of rclone, you need to configure each of them
-with a separate personal login token. E.g. you create a Jottacloud remote with rclone in one
-location, and copy the configuration file to a second location where you also want to run
-rclone and access the same remote. Then you need to replace the token for one of them, using
-the [config reconnect](https://rclone.org/commands/rclone_config_reconnect/) command, which
-requires you to generate a new personal login token and supply as input. If you do not
-do this, the token may easily end up being invalidated, resulting in both instances failing
-with an error message something along the lines of:
+Properties:
- oauth2: cannot fetch token: 400 Bad Request
- Response: {"error":"invalid_grant","error_description":"Stale token"}
+- Config: token
+- Env Var: RCLONE_LINKBOX_TOKEN
+- Type: string
+- Required: true
-When this happens, you need to replace the token as described above to be able to use your
-remote again.
-All personal login tokens you have taken into use will be listed in the web interface under
-"My logged in devices", and from the right side of that list you can click the "X" button to
-revoke individual tokens.
-### Legacy authentication
+## Limitations
-If you are using one of the whitelabel versions (e.g. from Elkjøp) you may not have the option
-to generate a CLI token. In this case you'll have to use the legacy authentication. To do this select
-yes when the setup asks for legacy authentication and enter your username and password.
-The rest of the setup is identical to the default setup.
+Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8),
+as they can't be used in JSON strings.
-### Telia Cloud authentication
+# Mail.ru Cloud
-Similar to other whitelabel versions Telia Cloud doesn't offer the option of creating a CLI token, and
-additionally uses a separate authentication flow where the username is generated internally. To setup
-rclone to use Telia Cloud, choose Telia Cloud authentication in the setup. The rest of the setup is
-identical to the default setup.
+[Mail.ru Cloud](https://cloud.mail.ru/) is a cloud storage provided by a Russian internet company [Mail.Ru Group](https://mail.ru). The official desktop client is [Disk-O:](https://disk-o.cloud/en), available on Windows and Mac OS.
-### Tele2 Cloud authentication
+## Features highlights
-As Tele2-Com Hem merger was completed this authentication can be used for former Com Hem Cloud and
-Tele2 Cloud customers as no support for creating a CLI token exists, and additionally uses a separate
-authentication flow where the username is generated internally. To setup rclone to use Tele2 Cloud,
-choose Tele2 Cloud authentication in the setup. The rest of the setup is identical to the default setup.
+- Paths may be as deep as required, e.g. `remote:directory/subdirectory`
+- Files have a `last modified time` property, directories don't
+- Deleted files are by default moved to the trash
+- Files and directories can be shared via public links
+- Partial uploads or streaming are not supported, file size must be known before upload
+- Maximum file size is limited to 2G for a free account, unlimited for paid accounts
+- Storage keeps hash for all files and performs transparent deduplication,
+ the hash algorithm is a modified SHA1
+- If a particular file is already present in storage, one can quickly submit file hash
+ instead of long file upload (this optimization is supported by rclone)
-### Onlime Cloud Storage authentication
+## Configuration
-Onlime has sold access to Jottacloud proper, while providing localized support to Danish Customers, but
-have recently set up their own hosting, transferring their customers from Jottacloud servers to their
-own ones.
+Here is an example of making a mailru configuration.
-This, of course, necessitates using their servers for authentication, but otherwise functionality and
-architecture seems equivalent to Jottacloud.
+First create a Mail.ru Cloud account and choose a tariff.
-To setup rclone to use Onlime Cloud Storage, choose Onlime Cloud authentication in the setup. The rest
-of the setup is identical to the default setup.
+You will need to log in and create an app password for rclone. Rclone
+**will not work** with your normal username and password - it will
+give an error like `oauth2: server response missing access_token`.
-## Configuration
+- Click on your user icon in the top right
+- Go to Security / "Пароль и безопаÑноÑÑ‚ÑŒ"
+- Click password for apps / "Пароли Ð´Ð»Ñ Ð²Ð½ÐµÑˆÐ½Ð¸Ñ… приложений"
+- Add the password - give it a name - eg "rclone"
+- Copy the password and use this password below - your normal login password won't work.
-Here is an example of how to make a remote called `remote` with the default setup. First run:
+Now run
rclone config
@@ -35923,157 +37573,103 @@ s) Set configuration password
q) Quit config
n/s/q> n
name> remote
-Option Storage.
Type of storage to configure.
-Choose a number from below, or type in your own value.
+Type of storage to configure.
+Enter a string value. Press Enter for the default ("").
+Choose a number from below, or type in your own value
[snip]
-XX / Jottacloud
- \ (jottacloud)
+XX / Mail.ru Cloud
+ \ "mailru"
[snip]
-Storage> jottacloud
-Edit advanced config?
+Storage> mailru
+User name (usually email)
+Enter a string value. Press Enter for the default ("").
+user> username@mail.ru
+Password
+
+This must be an app password - rclone will not work with your normal
+password. See the Configuration section in the docs for how to make an
+app password.
+y) Yes type in my own password
+g) Generate random password
+y/g> y
+Enter the password:
+password:
+Confirm the password:
+password:
+Skip full upload if there is another file with same data hash.
+This feature is called "speedup" or "put by hash". It is especially efficient
+in case of generally available files like popular books, video or audio clips
+[snip]
+Enter a boolean value (true or false). Press Enter for the default ("true").
+Choose a number from below, or type in your own value
+ 1 / Enable
+ \ "true"
+ 2 / Disable
+ \ "false"
+speedup_enable> 1
+Edit advanced config? (y/n)
y) Yes
-n) No (default)
+n) No
y/n> n
-Option config_type.
-Select authentication type.
-Choose a number from below, or type in an existing string value.
-Press Enter for the default (standard).
- / Standard authentication.
- 1 | Use this if you're a normal Jottacloud user.
- \ (standard)
- / Legacy authentication.
- 2 | This is only required for certain whitelabel versions of Jottacloud and not recommended for normal users.
- \ (legacy)
- / Telia Cloud authentication.
- 3 | Use this if you are using Telia Cloud.
- \ (telia)
- / Tele2 Cloud authentication.
- 4 | Use this if you are using Tele2 Cloud.
- \ (tele2)
- / Onlime Cloud authentication.
- 5 | Use this if you are using Onlime Cloud.
- \ (onlime)
-config_type> 1
-Personal login token.
-Generate here: https://www.jottacloud.com/web/secure
-Login Token>
-Use a non-standard device/mountpoint?
-Choosing no, the default, will let you access the storage used for the archive
-section of the official Jottacloud client. If you instead want to access the
-sync or the backup section, for example, you must choose yes.
-y) Yes
-n) No (default)
-y/n> y
-Option config_device.
-The device to use. In standard setup the built-in Jotta device is used,
-which contains predefined mountpoints for archive, sync etc. All other devices
-are treated as backup devices by the official Jottacloud client. You may create
-a new by entering a unique name.
-Choose a number from below, or type in your own string value.
-Press Enter for the default (DESKTOP-3H31129).
- 1 > DESKTOP-3H31129
- 2 > Jotta
-config_device> 2
-Option config_mountpoint.
-The mountpoint to use for the built-in device Jotta.
-The standard setup is to use the Archive mountpoint. Most other mountpoints
-have very limited support in rclone and should generally be avoided.
-Choose a number from below, or type in an existing string value.
-Press Enter for the default (Archive).
- 1 > Archive
- 2 > Shared
- 3 > Sync
-config_mountpoint> 1
+Remote config
--------------------
[remote]
-type = jottacloud
-configVersion = 1
-client_id = jottacli
-client_secret =
-tokenURL = https://id.jottacloud.com/auth/realms/jottacloud/protocol/openid-connect/token
-token = {........}
-username = 2940e57271a93d987d6f8a21
-device = Jotta
-mountpoint = Archive
+type = mailru
+user = username@mail.ru
+pass = *** ENCRYPTED ***
+speedup_enable = true
--------------------
-y) Yes this is OK (default)
+y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> y
```
-Once configured you can then use `rclone` like this,
+Configuration of this backend does not require a local web browser.
+You can use the configured backend as shown below:
-List directories in top level of your Jottacloud
+See top level directories
rclone lsd remote:
-List all the files in your Jottacloud
-
- rclone ls remote:
-
-To copy a local directory to an Jottacloud directory called backup
-
- rclone copy /home/source remote:backup
-
-### Devices and Mountpoints
+Make a new directory
-The official Jottacloud client registers a device for each computer you install
-it on, and shows them in the backup section of the user interface. For each
-folder you select for backup it will create a mountpoint within this device.
-A built-in device called Jotta is special, and contains mountpoints Archive,
-Sync and some others, used for corresponding features in official clients.
+ rclone mkdir remote:directory
-With rclone you'll want to use the standard Jotta/Archive device/mountpoint in
-most cases. However, you may for example want to access files from the sync or
-backup functionality provided by the official clients, and rclone therefore
-provides the option to select other devices and mountpoints during config.
+List the contents of a directory
-You are allowed to create new devices and mountpoints. All devices except the
-built-in Jotta device are treated as backup devices by official Jottacloud
-clients, and the mountpoints on them are individual backup sets.
+ rclone ls remote:directory
-With the built-in Jotta device, only existing, built-in, mountpoints can be
-selected. In addition to the mentioned Archive and Sync, it may contain
-several other mountpoints such as: Latest, Links, Shared and Trash. All of
-these are special mountpoints with a different internal representation than
-the "regular" mountpoints. Rclone will only to a very limited degree support
-them. Generally you should avoid these, unless you know what you are doing.
+Sync `/home/local/directory` to the remote path, deleting any
+excess files in the path.
-### --fast-list
+ rclone sync --interactive /home/local/directory remote:directory
-This remote supports `--fast-list` which allows you to use fewer
-transactions in exchange for more memory. See the [rclone
-docs](https://rclone.org/docs/#fast-list) for more details.
+### Modification times and hashes
-Note that the implementation in Jottacloud always uses only a single
-API request to get the entire list, so for large folders this could
-lead to long wait time before the first results are shown.
+Files support a modification time attribute with up to 1 second precision.
+Directories do not have a modification time, which is shown as "Jan 1 1970".
-Note also that with rclone version 1.58 and newer information about
-[MIME types](https://rclone.org/overview/#mime-type) are not available when using `--fast-list`.
+File hashes are supported, with a custom Mail.ru algorithm based on SHA1.
+If file size is less than or equal to the SHA1 block size (20 bytes),
+its hash is simply its data right-padded with zero bytes.
+Hashes of a larger file is computed as a SHA1 of the file data
+bytes concatenated with a decimal representation of the data length.
-### Modified time and hashes
+### Emptying Trash
-Jottacloud allows modification times to be set on objects accurate to 1
-second. These will be used to detect whether objects need syncing or
-not.
+Removing a file or directory actually moves it to the trash, which is not
+visible to rclone but can be seen in a web browser. The trashed file
+still occupies part of total quota. If you wish to empty your trash
+and free some quota, you can use the `rclone cleanup remote:` command,
+which will permanently delete all your trashed files.
+This command does not take any path arguments.
-Jottacloud supports MD5 type hashes, so you can use the `--checksum`
-flag.
+### Quota information
-Note that Jottacloud requires the MD5 hash before upload so if the
-source does not have an MD5 checksum then the file will be cached
-temporarily on disk (in location given by
-[--temp-dir](https://rclone.org/docs/#temp-dir-dir)) before it is uploaded.
-Small files will be cached in memory - see the
-[--jottacloud-md5-memory-limit](#jottacloud-md5-memory-limit) flag.
-When uploading from local disk the source checksum is always available,
-so this does not apply. Starting with rclone version 1.52 the same is
-true for encrypted remotes (in older versions the crypt backend would not
-calculate hashes for uploads from local disk, so the Jottacloud
-backend had to do it as described above).
+To view your current quota you can use the `rclone about remote:`
+command which will display your usage limit (quota) and the current usage.
### Restricted filename characters
@@ -36088,38 +37684,18 @@ the following characters are also replaced:
| < | 0x3C | ＜ |
| > | 0x3E | ＞ |
| ? | 0x3F | ？ |
+| \ | 0x5C | ï¼¼ |
| \| | 0x7C | ｜ |
Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8),
-as they can't be used in XML strings.
-
-### Deleting files
-
-By default, rclone will send all files to the trash when deleting files. They will be permanently
-deleted automatically after 30 days. You may bypass the trash and permanently delete files immediately
-by using the [--jottacloud-hard-delete](#jottacloud-hard-delete) flag, or set the equivalent environment variable.
-Emptying the trash is supported by the [cleanup](https://rclone.org/commands/rclone_cleanup/) command.
-
-### Versions
-
-Jottacloud supports file versioning. When rclone uploads a new version of a file it creates a new version of it.
-Currently rclone only supports retrieving the current version but older versions can be accessed via the Jottacloud Website.
-
-Versioning can be disabled by `--jottacloud-no-versions` option. This is achieved by deleting the remote file prior to uploading
-a new version. If the upload the fails no version of the file will be available in the remote.
-
-### Quota information
-
-To view your current quota you can use the `rclone about remote:`
-command which will display your usage limit (unless it is unlimited)
-and the current usage.
+as they can't be used in JSON strings.
### Standard options
-Here are the Standard options specific to jottacloud (Jottacloud).
+Here are the Standard options specific to mailru (Mail.ru Cloud).
-#### --jottacloud-client-id
+#### --mailru-client-id
OAuth Client Id.
@@ -36128,11 +37704,11 @@ Leave blank normally.
Properties:
- Config: client_id
-- Env Var: RCLONE_JOTTACLOUD_CLIENT_ID
+- Env Var: RCLONE_MAILRU_CLIENT_ID
- Type: string
- Required: false
-#### --jottacloud-client-secret
+#### --mailru-client-secret
OAuth Client Secret.
@@ -36141,26 +37717,80 @@ Leave blank normally.
Properties:
- Config: client_secret
-- Env Var: RCLONE_JOTTACLOUD_CLIENT_SECRET
+- Env Var: RCLONE_MAILRU_CLIENT_SECRET
- Type: string
- Required: false
+#### --mailru-user
+
+User name (usually email).
+
+Properties:
+
+- Config: user
+- Env Var: RCLONE_MAILRU_USER
+- Type: string
+- Required: true
+
+#### --mailru-pass
+
+Password.
+
+This must be an app password - rclone will not work with your normal
+password. See the Configuration section in the docs for how to make an
+app password.
+
+
+**NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/).
+
+Properties:
+
+- Config: pass
+- Env Var: RCLONE_MAILRU_PASS
+- Type: string
+- Required: true
+
+#### --mailru-speedup-enable
+
+Skip full upload if there is another file with same data hash.
+
+This feature is called "speedup" or "put by hash". It is especially efficient
+in case of generally available files like popular books, video or audio clips,
+because files are searched by hash in all accounts of all mailru users.
+It is meaningless and ineffective if source file is unique or encrypted.
+Please note that rclone may need local memory and disk space to calculate
+content hash in advance and decide whether full upload is required.
+Also, if rclone does not know file size in advance (e.g. in case of
+streaming or partial uploads), it will not even try this optimization.
+
+Properties:
+
+- Config: speedup_enable
+- Env Var: RCLONE_MAILRU_SPEEDUP_ENABLE
+- Type: bool
+- Default: true
+- Examples:
+ - "true"
+ - Enable
+ - "false"
+ - Disable
+
### Advanced options
-Here are the Advanced options specific to jottacloud (Jottacloud).
+Here are the Advanced options specific to mailru (Mail.ru Cloud).
-#### --jottacloud-token
+#### --mailru-token
OAuth Access Token as a JSON blob.
Properties:
- Config: token
-- Env Var: RCLONE_JOTTACLOUD_TOKEN
+- Env Var: RCLONE_MAILRU_TOKEN
- Type: string
- Required: false
-#### --jottacloud-auth-url
+#### --mailru-auth-url
Auth server URL.
@@ -36169,11 +37799,11 @@ Leave blank to use the provider defaults.
Properties:
- Config: auth_url
-- Env Var: RCLONE_JOTTACLOUD_AUTH_URL
+- Env Var: RCLONE_MAILRU_AUTH_URL
- Type: string
- Required: false
-#### --jottacloud-token-url
+#### --mailru-token-url
Token server url.
@@ -36182,68 +37812,117 @@ Leave blank to use the provider defaults.
Properties:
- Config: token_url
-- Env Var: RCLONE_JOTTACLOUD_TOKEN_URL
+- Env Var: RCLONE_MAILRU_TOKEN_URL
- Type: string
- Required: false
-#### --jottacloud-md5-memory-limit
+#### --mailru-speedup-file-patterns
-Files bigger than this will be cached on disk to calculate the MD5 if required.
+Comma separated list of file name patterns eligible for speedup (put by hash).
+
+Patterns are case insensitive and can contain '*' or '?' meta characters.
Properties:
-- Config: md5_memory_limit
-- Env Var: RCLONE_JOTTACLOUD_MD5_MEMORY_LIMIT
-- Type: SizeSuffix
-- Default: 10Mi
+- Config: speedup_file_patterns
+- Env Var: RCLONE_MAILRU_SPEEDUP_FILE_PATTERNS
+- Type: string
+- Default: "*.mkv,*.avi,*.mp4,*.mp3,*.zip,*.gz,*.rar,*.pdf"
+- Examples:
+ - ""
+ - Empty list completely disables speedup (put by hash).
+ - "*"
+ - All files will be attempted for speedup.
+ - "*.mkv,*.avi,*.mp4,*.mp3"
+ - Only common audio/video files will be tried for put by hash.
+ - "*.zip,*.gz,*.rar,*.pdf"
+ - Only common archives or PDF books will be tried for speedup.
-#### --jottacloud-trashed-only
+#### --mailru-speedup-max-disk
-Only show files that are in the trash.
+This option allows you to disable speedup (put by hash) for large files.
-This will show trashed files in their original directory structure.
+Reason is that preliminary hashing can exhaust your RAM or disk space.
Properties:
-- Config: trashed_only
-- Env Var: RCLONE_JOTTACLOUD_TRASHED_ONLY
-- Type: bool
-- Default: false
+- Config: speedup_max_disk
+- Env Var: RCLONE_MAILRU_SPEEDUP_MAX_DISK
+- Type: SizeSuffix
+- Default: 3Gi
+- Examples:
+ - "0"
+ - Completely disable speedup (put by hash).
+ - "1G"
+ - Files larger than 1Gb will be uploaded directly.
+ - "3G"
+ - Choose this option if you have less than 3Gb free on local disk.
-#### --jottacloud-hard-delete
+#### --mailru-speedup-max-memory
-Delete files permanently rather than putting them into the trash.
+Files larger than the size given below will always be hashed on disk.
Properties:
-- Config: hard_delete
-- Env Var: RCLONE_JOTTACLOUD_HARD_DELETE
+- Config: speedup_max_memory
+- Env Var: RCLONE_MAILRU_SPEEDUP_MAX_MEMORY
+- Type: SizeSuffix
+- Default: 32Mi
+- Examples:
+ - "0"
+ - Preliminary hashing will always be done in a temporary disk location.
+ - "32M"
+ - Do not dedicate more than 32Mb RAM for preliminary hashing.
+ - "256M"
+ - You have at most 256Mb RAM free for hash calculations.
+
+#### --mailru-check-hash
+
+What should copy do if file checksum is mismatched or invalid.
+
+Properties:
+
+- Config: check_hash
+- Env Var: RCLONE_MAILRU_CHECK_HASH
- Type: bool
-- Default: false
+- Default: true
+- Examples:
+ - "true"
+ - Fail with error.
+ - "false"
+ - Ignore and continue.
-#### --jottacloud-upload-resume-limit
+#### --mailru-user-agent
-Files bigger than this can be resumed if the upload fail's.
+HTTP user agent used internally by client.
+
+Defaults to "rclone/VERSION" or "--user-agent" provided on command line.
Properties:
-- Config: upload_resume_limit
-- Env Var: RCLONE_JOTTACLOUD_UPLOAD_RESUME_LIMIT
-- Type: SizeSuffix
-- Default: 10Mi
+- Config: user_agent
+- Env Var: RCLONE_MAILRU_USER_AGENT
+- Type: string
+- Required: false
-#### --jottacloud-no-versions
+#### --mailru-quirks
-Avoid server side versioning by deleting files and recreating files instead of overwriting them.
+Comma separated list of internal maintenance flags.
+
+This option must not be used by an ordinary user. It is intended only to
+facilitate remote troubleshooting of backend issues. Strict meaning of
+flags is not documented and not guaranteed to persist between releases.
+Quirks will be removed when the backend grows stable.
+Supported quirks: atomicmkdir binlist unknowndirs
Properties:
-- Config: no_versions
-- Env Var: RCLONE_JOTTACLOUD_NO_VERSIONS
-- Type: bool
-- Default: false
+- Config: quirks
+- Env Var: RCLONE_MAILRU_QUIRKS
+- Type: string
+- Required: false
-#### --jottacloud-encoding
+#### --mailru-encoding
The encoding for the backend.
@@ -36252,28 +37931,31 @@ See the [encoding section in the overview](https://rclone.org/overview/#encoding
Properties:
- Config: encoding
-- Env Var: RCLONE_JOTTACLOUD_ENCODING
-- Type: MultiEncoder
-- Default: Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,Del,Ctl,InvalidUtf8,Dot
+- Env Var: RCLONE_MAILRU_ENCODING
+- Type: Encoding
+- Default: Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Del,Ctl,InvalidUtf8,Dot
## Limitations
-Note that Jottacloud is case insensitive so you can't have a file called
-"Hello.doc" and one called "hello.doc".
-
-There are quite a few characters that can't be in Jottacloud file names. Rclone will map these names to and from an identical
-looking unicode equivalent. For example if a file has a ? in it will be mapped to ？ instead.
+File size limits depend on your account. A single file size is limited by 2G
+for a free account and unlimited for paid tariffs. Please refer to the Mail.ru
+site for the total uploaded size limits.
-Jottacloud only supports filenames up to 255 characters in length.
+Note that Mailru is case insensitive so you can't have a file called
+"Hello.doc" and one called "hello.doc".
-## Troubleshooting
+# Mega
-Jottacloud exhibits some inconsistent behaviours regarding deleted files and folders which may cause Copy, Move and DirMove
-operations to previously deleted paths to fail. Emptying the trash should help in such cases.
+[Mega](https://mega.nz/) is a cloud storage and file hosting service
+known for its security feature where all files are encrypted locally
+before they are uploaded. This prevents anyone (including employees of
+Mega) from accessing the files without knowledge of the key used for
+encryption.
-# Koofr
+This is an rclone backend for Mega which supports the file transfer
+features of Mega using the same client side encryption.
Paths are specified as `remote:path`
@@ -36281,12 +37963,7 @@ Paths may be as deep as required, e.g. `remote:directory/subdirectory`.
## Configuration
-The initial setup for Koofr involves creating an application password for
-rclone. You can do that by opening the Koofr
-[web application](https://app.koofr.net/app/admin/preferences/password),
-giving the password a nice name like `rclone` and clicking on generate.
-
-Here is an example of how to make a remote called `koofr`. First run:
+Here is an example of how to make a remote called `remote`. First run:
rclone config
@@ -36298,207 +37975,243 @@ n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
-name> koofr
-Option Storage.
+name> remote
Type of storage to configure.
-Choose a number from below, or type in your own value.
+Choose a number from below, or type in your own value
[snip]
-22 / Koofr, Digi Storage and other Koofr-compatible storage providers
- \ (koofr)
+XX / Mega
+ \ "mega"
[snip]
-Storage> koofr
-Option provider.
-Choose your storage provider.
-Choose a number from below, or type in your own value.
-Press Enter to leave empty.
- 1 / Koofr, https://app.koofr.net/
- \ (koofr)
- 2 / Digi Storage, https://storage.rcs-rds.ro/
- \ (digistorage)
- 3 / Any other Koofr API compatible storage service
- \ (other)
-provider> 1
-Option user.
-Your user name.
-Enter a value.
-user> USERNAME
-Option password.
-Your password for rclone (generate one at https://app.koofr.net/app/admin/preferences/password).
-Choose an alternative below.
-y) Yes, type in my own password
+Storage> mega
+User name
+user> you@example.com
+Password.
+y) Yes type in my own password
g) Generate random password
-y/g> y
+n) No leave this optional password blank
+y/g/n> y
Enter the password:
password:
Confirm the password:
password:
-Edit advanced config?
-y) Yes
-n) No (default)
-y/n> n
Remote config
--------------------
-[koofr]
-type = koofr
-provider = koofr
-user = USERNAME
-password = *** ENCRYPTED ***
+[remote]
+type = mega
+user = you@example.com
+pass = *** ENCRYPTED ***
--------------------
-y) Yes this is OK (default)
+y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> y
```
-You can choose to edit advanced config in order to enter your own service URL
-if you use an on-premise or white label Koofr instance, or choose an alternative
-mount instead of your primary storage.
+**NOTE:** The encryption keys need to have been already generated after a regular login
+via the browser, otherwise attempting to use the credentials in `rclone` will fail.
Once configured you can then use `rclone` like this,
-List directories in top level of your Koofr
+List directories in top level of your Mega
- rclone lsd koofr:
+ rclone lsd remote:
-List all the files in your Koofr
+List all the files in your Mega
- rclone ls koofr:
+ rclone ls remote:
-To copy a local directory to an Koofr directory called backup
+To copy a local directory to an Mega directory called backup
- rclone copy /home/source koofr:backup
+ rclone copy /home/source remote:backup
-### Restricted filename characters
+### Modification times and hashes
-In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters)
-the following characters are also replaced:
+Mega does not support modification times or hashes yet.
+
+### Restricted filename characters
| Character | Value | Replacement |
| --------- |:-----:|:-----------:|
-| \ | 0x5C | ï¼¼ |
+| NUL | 0x00 | †|
+| / | 0x2F | ï¼ |
Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8),
-as they can't be used in XML strings.
+as they can't be used in JSON strings.
+### Duplicated files
-### Standard options
+Mega can have two files with exactly the same name and path (unlike a
+normal file system).
-Here are the Standard options specific to koofr (Koofr, Digi Storage and other Koofr-compatible storage providers).
+Duplicated files cause problems with the syncing and you will see
+messages in the log about duplicates.
-#### --koofr-provider
+Use `rclone dedupe` to fix duplicated files.
-Choose your storage provider.
+### Failure to log-in
-Properties:
+#### Object not found
-- Config: provider
-- Env Var: RCLONE_KOOFR_PROVIDER
-- Type: string
-- Required: false
-- Examples:
- - "koofr"
- - Koofr, https://app.koofr.net/
- - "digistorage"
- - Digi Storage, https://storage.rcs-rds.ro/
- - "other"
- - Any other Koofr API compatible storage service
+If you are connecting to your Mega remote for the first time,
+to test access and synchronization, you may receive an error such as
-#### --koofr-endpoint
+```
+Failed to create file system for "my-mega-remote:":
+couldn't login: Object (typically, node or user) not found
+```
-The Koofr API endpoint to use.
+The diagnostic steps often recommended in the [rclone forum](https://forum.rclone.org/search?q=mega)
+start with the **MEGAcmd** utility. Note that this refers to
+the official C++ command from https://github.com/meganz/MEGAcmd
+and not the go language built command from t3rm1n4l/megacmd
+that is no longer maintained.
-Properties:
+Follow the instructions for installing MEGAcmd and try accessing
+your remote as they recommend. You can establish whether or not
+you can log in using MEGAcmd, and obtain diagnostic information
+to help you, and search or work with others in the forum.
-- Config: endpoint
-- Env Var: RCLONE_KOOFR_ENDPOINT
-- Provider: other
-- Type: string
-- Required: true
+```
+MEGA CMD> login me@example.com
+Password:
+Fetching nodes ...
+Loading transfers from local cache
+Login complete as me@example.com
+me@example.com:/$
+```
-#### --koofr-user
+Note that some have found issues with passwords containing special
+characters. If you can not log on with rclone, but MEGAcmd logs on
+just fine, then consider changing your password temporarily to
+pure alphanumeric characters, in case that helps.
-Your user name.
-Properties:
+#### Repeated commands blocks access
-- Config: user
-- Env Var: RCLONE_KOOFR_USER
-- Type: string
-- Required: true
+Mega remotes seem to get blocked (reject logins) under "heavy use".
+We haven't worked out the exact blocking rules but it seems to be
+related to fast paced, successive rclone commands.
-#### --koofr-password
+For example, executing this command 90 times in a row `rclone link
+remote:file` will cause the remote to become "blocked". This is not an
+abnormal situation, for example if you wish to get the public links of
+a directory with hundred of files... After more or less a week, the
+remote will remote accept rclone logins normally again.
-Your password for rclone (generate one at https://app.koofr.net/app/admin/preferences/password).
+You can mitigate this issue by mounting the remote it with `rclone
+mount`. This will log-in when mounting and a log-out when unmounting
+only. You can also run `rclone rcd` and then use `rclone rc` to run
+the commands over the API to avoid logging in each time.
-**NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/).
+Rclone does not currently close mega sessions (you can see them in the
+web interface), however closing the sessions does not solve the issue.
-Properties:
+If you space rclone commands by 3 seconds it will avoid blocking the
+remote. We haven't identified the exact blocking rules, so perhaps one
+could execute the command 80 times without waiting and avoid blocking
+by waiting 3 seconds, then continuing...
-- Config: password
-- Env Var: RCLONE_KOOFR_PASSWORD
-- Provider: koofr
-- Type: string
-- Required: true
+Note that this has been observed by trial and error and might not be
+set in stone.
-#### --koofr-password
+Other tools seem not to produce this blocking effect, as they use a
+different working approach (state-based, using sessionIDs instead of
+log-in) which isn't compatible with the current stateless rclone
+approach.
-Your password for rclone (generate one at https://storage.rcs-rds.ro/app/admin/preferences/password).
+Note that once blocked, the use of other tools (such as megacmd) is
+not a sure workaround: following megacmd login times have been
+observed in succession for blocked remote: 7 minutes, 20 min, 30min, 30
+min, 30min. Web access looks unaffected though.
-**NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/).
+Investigation is continuing in relation to workarounds based on
+timeouts, pacers, retrials and tpslimits - if you discover something
+relevant, please post on the forum.
+
+So, if rclone was working nicely and suddenly you are unable to log-in
+and you are sure the user and the password are correct, likely you
+have got the remote blocked for a while.
+
+
+### Standard options
+
+Here are the Standard options specific to mega (Mega).
+
+#### --mega-user
+
+User name.
Properties:
-- Config: password
-- Env Var: RCLONE_KOOFR_PASSWORD
-- Provider: digistorage
+- Config: user
+- Env Var: RCLONE_MEGA_USER
- Type: string
- Required: true
-#### --koofr-password
+#### --mega-pass
-Your password for rclone (generate one at your service's settings page).
+Password.
**NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/).
Properties:
-- Config: password
-- Env Var: RCLONE_KOOFR_PASSWORD
-- Provider: other
+- Config: pass
+- Env Var: RCLONE_MEGA_PASS
- Type: string
- Required: true
### Advanced options
-Here are the Advanced options specific to koofr (Koofr, Digi Storage and other Koofr-compatible storage providers).
+Here are the Advanced options specific to mega (Mega).
-#### --koofr-mountid
+#### --mega-debug
-Mount ID of the mount to use.
+Output more debug from Mega.
-If omitted, the primary mount is used.
+If this flag is set (along with -vv) it will print further debugging
+information from the mega backend.
Properties:
-- Config: mountid
-- Env Var: RCLONE_KOOFR_MOUNTID
-- Type: string
-- Required: false
+- Config: debug
+- Env Var: RCLONE_MEGA_DEBUG
+- Type: bool
+- Default: false
-#### --koofr-setmtime
+#### --mega-hard-delete
-Does the backend support setting modification time.
+Delete files permanently rather than putting them into the trash.
-Set this to false if you use a mount ID that points to a Dropbox or Amazon Drive backend.
+Normally the mega backend will put all deletions into the trash rather
+than permanently deleting them. If you specify this then rclone will
+permanently delete objects instead.
Properties:
-- Config: setmtime
-- Env Var: RCLONE_KOOFR_SETMTIME
+- Config: hard_delete
+- Env Var: RCLONE_MEGA_HARD_DELETE
- Type: bool
-- Default: true
+- Default: false
-#### --koofr-encoding
+#### --mega-use-https
+
+Use HTTPS for transfers.
+
+MEGA uses plain text HTTP connections by default.
+Some ISPs throttle HTTP connections, this causes transfers to become very slow.
+Enabling this will force MEGA to use HTTPS for all transfers.
+HTTPS is normally not necessary since all data is already encrypted anyway.
+Enabling it will increase CPU usage and add network overhead.
+
+Properties:
+
+- Config: use_https
+- Env Var: RCLONE_MEGA_USE_HTTPS
+- Type: bool
+- Default: false
+
+#### --mega-encoding
The encoding for the backend.
@@ -36507,33 +38220,38 @@ See the [encoding section in the overview](https://rclone.org/overview/#encoding
Properties:
- Config: encoding
-- Env Var: RCLONE_KOOFR_ENCODING
-- Type: MultiEncoder
-- Default: Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot
+- Env Var: RCLONE_MEGA_ENCODING
+- Type: Encoding
+- Default: Slash,InvalidUtf8,Dot
-## Limitations
+### Process `killed`
-Note that Koofr is case insensitive so you can't have a file called
-"Hello.doc" and one called "hello.doc".
+On accounts with large files or something else, memory usage can significantly increase when executing list/sync instructions. When running on cloud providers (like AWS with EC2), check if the instance type has sufficient memory/CPU to execute the commands. Use the resource monitoring tools to inspect after sending the commands. Look [at this issue](https://forum.rclone.org/t/rclone-with-mega-appears-to-work-only-in-some-accounts/40233/4).
-## Providers
+## Limitations
-### Koofr
+This backend uses the [go-mega go library](https://github.com/t3rm1n4l/go-mega) which is an opensource
+go library implementing the Mega API. There doesn't appear to be any
+documentation for the mega protocol beyond the [mega C++ SDK](https://github.com/meganz/sdk) source code
+so there are likely quite a few errors still remaining in this library.
-This is the original [Koofr](https://koofr.eu) storage provider used as main example and described in the [configuration](#configuration) section above.
+Mega allows duplicate files which may confuse rclone.
-### Digi Storage
+# Memory
-[Digi Storage](https://www.digi.ro/servicii/online/digi-storage) is a cloud storage service run by [Digi.ro](https://www.digi.ro/) that
-provides a Koofr API.
+The memory backend is an in RAM backend. It does not persist its
+data - use the local backend for that.
-Here is an example of how to make a remote called `ds`. First run:
+The memory backend behaves like a bucket-based remote (e.g. like
+s3). Because it has no parameters you can just use it with the
+`:memory:` remote name.
- rclone config
+## Configuration
-This will guide you through an interactive setup process:
+You can configure it as a remote like this with `rclone config` too if
+you want to:
```
No remotes found, make a new one?
@@ -36541,50 +38259,22 @@ n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
-name> ds
-Option Storage.
+name> remote
Type of storage to configure.
-Choose a number from below, or type in your own value.
+Enter a string value. Press Enter for the default ("").
+Choose a number from below, or type in your own value
[snip]
-22 / Koofr, Digi Storage and other Koofr-compatible storage providers
- \ (koofr)
+XX / Memory
+ \ "memory"
[snip]
-Storage> koofr
-Option provider.
-Choose your storage provider.
-Choose a number from below, or type in your own value.
-Press Enter to leave empty.
- 1 / Koofr, https://app.koofr.net/
- \ (koofr)
- 2 / Digi Storage, https://storage.rcs-rds.ro/
- \ (digistorage)
- 3 / Any other Koofr API compatible storage service
- \ (other)
-provider> 2
-Option user.
-Your user name.
-Enter a value.
-user> USERNAME
-Option password.
-Your password for rclone (generate one at https://storage.rcs-rds.ro/app/admin/preferences/password).
-Choose an alternative below.
-y) Yes, type in my own password
-g) Generate random password
-y/g> y
-Enter the password:
-password:
-Confirm the password:
-password:
-Edit advanced config?
-y) Yes
-n) No (default)
-y/n> n
+Storage> memory
+** See help for memory backend at: https://rclone.org/memory/ **
+
+Remote config
+
--------------------
-[ds]
-type = koofr
-provider = digistorage
-user = USERNAME
-password = *** ENCRYPTED ***
+[remote]
+type = memory
--------------------
y) Yes this is OK (default)
e) Edit this remote
@@ -36592,141 +38282,106 @@ d) Delete this remote
y/e/d> y
```
-### Other
+Because the memory backend isn't persistent it is most useful for
+testing or with an rclone server or rclone mount, e.g.
-You may also want to use another, public or private storage provider that runs a Koofr API compatible service, by simply providing the base URL to connect to.
+ rclone mount :memory: /mnt/tmp
+ rclone serve webdav :memory:
+ rclone serve sftp :memory:
-Here is an example of how to make a remote called `other`. First run:
+### Modification times and hashes
- rclone config
+The memory backend supports MD5 hashes and modification times accurate to 1 nS.
-This will guide you through an interactive setup process:
+### Restricted filename characters
-```
-No remotes found, make a new one?
-n) New remote
-s) Set configuration password
-q) Quit config
-n/s/q> n
-name> other
-Option Storage.
-Type of storage to configure.
-Choose a number from below, or type in your own value.
-[snip]
-22 / Koofr, Digi Storage and other Koofr-compatible storage providers
- \ (koofr)
-[snip]
-Storage> koofr
-Option provider.
-Choose your storage provider.
-Choose a number from below, or type in your own value.
-Press Enter to leave empty.
- 1 / Koofr, https://app.koofr.net/
- \ (koofr)
- 2 / Digi Storage, https://storage.rcs-rds.ro/
- \ (digistorage)
- 3 / Any other Koofr API compatible storage service
- \ (other)
-provider> 3
-Option endpoint.
-The Koofr API endpoint to use.
-Enter a value.
-endpoint> https://koofr.other.org
-Option user.
-Your user name.
-Enter a value.
-user> USERNAME
-Option password.
-Your password for rclone (generate one at your service's settings page).
-Choose an alternative below.
-y) Yes, type in my own password
-g) Generate random password
-y/g> y
-Enter the password:
-password:
-Confirm the password:
-password:
-Edit advanced config?
-y) Yes
-n) No (default)
-y/n> n
---------------------
-[other]
-type = koofr
-provider = other
-endpoint = https://koofr.other.org
-user = USERNAME
-password = *** ENCRYPTED ***
---------------------
-y) Yes this is OK (default)
-e) Edit this remote
-d) Delete this remote
-y/e/d> y
-```
+The memory backend replaces the [default restricted characters
+set](https://rclone.org/overview/#restricted-characters).
-# Mail.ru Cloud
-[Mail.ru Cloud](https://cloud.mail.ru/) is a cloud storage provided by a Russian internet company [Mail.Ru Group](https://mail.ru). The official desktop client is [Disk-O:](https://disk-o.cloud/en), available on Windows and Mac OS.
-## Features highlights
-- Paths may be as deep as required, e.g. `remote:directory/subdirectory`
-- Files have a `last modified time` property, directories don't
-- Deleted files are by default moved to the trash
-- Files and directories can be shared via public links
-- Partial uploads or streaming are not supported, file size must be known before upload
-- Maximum file size is limited to 2G for a free account, unlimited for paid accounts
-- Storage keeps hash for all files and performs transparent deduplication,
- the hash algorithm is a modified SHA1
-- If a particular file is already present in storage, one can quickly submit file hash
- instead of long file upload (this optimization is supported by rclone)
+# Akamai NetStorage
+
+Paths are specified as `remote:`
+You may put subdirectories in too, e.g. `remote:/path/to/dir`.
+If you have a CP code you can use that as the folder after the domain such as \\/\\/\.
+
+For example, this is commonly configured with or without a CP code:
+* **With a CP code**. `[your-domain-prefix]-nsu.akamaihd.net/123456/subdirectory/`
+* **Without a CP code**. `[your-domain-prefix]-nsu.akamaihd.net`
+
+
+See all buckets
+ rclone lsd remote:
+The initial setup for Netstorage involves getting an account and secret. Use `rclone config` to walk you through the setup process.
+
+## Configuration
+
+Here's an example of how to make a remote called `ns1`.
+
+1. To begin the interactive configuration process, enter this command:
+
+```
+rclone config
+```
+
+2. Type `n` to create a new remote.
+
+```
+n) New remote
+d) Delete remote
+q) Quit config
+e/n/d/q> n
+```
-## Configuration
+3. For this example, enter `ns1` when you reach the name> prompt.
-Here is an example of making a mailru configuration.
+```
+name> ns1
+```
-First create a Mail.ru Cloud account and choose a tariff.
+4. Enter `netstorage` as the type of storage to configure.
-You will need to log in and create an app password for rclone. Rclone
-**will not work** with your normal username and password - it will
-give an error like `oauth2: server response missing access_token`.
+```
+Type of storage to configure.
+Enter a string value. Press Enter for the default ("").
+Choose a number from below, or type in your own value
+XX / NetStorage
+ \ "netstorage"
+Storage> netstorage
+```
-- Click on your user icon in the top right
-- Go to Security / "Пароль и безопаÑноÑÑ‚ÑŒ"
-- Click password for apps / "Пароли Ð´Ð»Ñ Ð²Ð½ÐµÑˆÐ½Ð¸Ñ… приложений"
-- Add the password - give it a name - eg "rclone"
-- Copy the password and use this password below - your normal login password won't work.
+5. Select between the HTTP or HTTPS protocol. Most users should choose HTTPS, which is the default. HTTP is provided primarily for debugging purposes.
-Now run
- rclone config
+```
+Enter a string value. Press Enter for the default ("").
+Choose a number from below, or type in your own value
+ 1 / HTTP protocol
+ \ "http"
+ 2 / HTTPS protocol
+ \ "https"
+protocol> 1
+```
-This will guide you through an interactive setup process:
+6. Specify your NetStorage host, CP code, and any necessary content paths using this format: `///`
```
-No remotes found, make a new one?
-n) New remote
-s) Set configuration password
-q) Quit config
-n/s/q> n
-name> remote
-Type of storage to configure.
-Type of storage to configure.
Enter a string value. Press Enter for the default ("").
-Choose a number from below, or type in your own value
-[snip]
-XX / Mail.ru Cloud
- \ "mailru"
-[snip]
-Storage> mailru
-User name (usually email)
+host> baseball-nsu.akamaihd.net/123456/content/
+```
+
+7. Set the netstorage account name
+```
Enter a string value. Press Enter for the default ("").
-user> username@mail.ru
-Password
+account> username
+```
-This must be an app password - rclone will not work with your normal
-password. See the Configuration section in the docs for how to make an
-app password.
+8. Set the Netstorage account secret/G2O key which will be used for authentication purposes. Select the `y` option to set your own password then enter your secret.
+Note: The secret is stored in the `rclone.conf` file with hex-encoded encryption.
+
+```
y) Yes type in my own password
g) Generate random password
y/g> y
@@ -36734,997 +38389,1072 @@ Enter the password:
password:
Confirm the password:
password:
-Skip full upload if there is another file with same data hash.
-This feature is called "speedup" or "put by hash". It is especially efficient
-in case of generally available files like popular books, video or audio clips
-[snip]
-Enter a boolean value (true or false). Press Enter for the default ("true").
-Choose a number from below, or type in your own value
- 1 / Enable
- \ "true"
- 2 / Disable
- \ "false"
-speedup_enable> 1
-Edit advanced config? (y/n)
-y) Yes
-n) No
-y/n> n
-Remote config
---------------------
-[remote]
-type = mailru
-user = username@mail.ru
-pass = *** ENCRYPTED ***
-speedup_enable = true
+```
+
+9. View the summary and confirm your remote configuration.
+
+```
+[ns1]
+type = netstorage
+protocol = http
+host = baseball-nsu.akamaihd.net/123456/content/
+account = username
+secret = *** ENCRYPTED ***
--------------------
-y) Yes this is OK
+y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y
```
-Configuration of this backend does not require a local web browser.
-You can use the configured backend as shown below:
+This remote is called `ns1` and can now be used.
-See top level directories
+## Example operations
- rclone lsd remote:
+Get started with rclone and NetStorage with these examples. For additional rclone commands, visit https://rclone.org/commands/.
-Make a new directory
+### See contents of a directory in your project
- rclone mkdir remote:directory
+ rclone lsd ns1:/974012/testing/
-List the contents of a directory
+### Sync the contents local with remote
- rclone ls remote:directory
+ rclone sync . ns1:/974012/testing/
-Sync `/home/local/directory` to the remote path, deleting any
-excess files in the path.
+### Upload local content to remote
+ rclone copy notes.txt ns1:/974012/testing/
- rclone sync --interactive /home/local/directory remote:directory
+### Delete content on remote
+ rclone delete ns1:/974012/testing/notes.txt
-### Modified time
+### Move or copy content between CP codes.
-Files support a modification time attribute with up to 1 second precision.
-Directories do not have a modification time, which is shown as "Jan 1 1970".
+Your credentials must have access to two CP codes on the same remote. You can't perform operations between different remotes.
-### Hash checksums
+ rclone move ns1:/974012/testing/notes.txt ns1:/974450/testing2/
-Hash sums use a custom Mail.ru algorithm based on SHA1.
-If file size is less than or equal to the SHA1 block size (20 bytes),
-its hash is simply its data right-padded with zero bytes.
-Hash sum of a larger file is computed as a SHA1 sum of the file data
-bytes concatenated with a decimal representation of the data length.
+## Features
-### Emptying Trash
+### Symlink Support
-Removing a file or directory actually moves it to the trash, which is not
-visible to rclone but can be seen in a web browser. The trashed file
-still occupies part of total quota. If you wish to empty your trash
-and free some quota, you can use the `rclone cleanup remote:` command,
-which will permanently delete all your trashed files.
-This command does not take any path arguments.
+The Netstorage backend changes the rclone `--links, -l` behavior. When uploading, instead of creating the .rclonelink file, use the "symlink" API in order to create the corresponding symlink on the remote. The .rclonelink file will not be created, the upload will be intercepted and only the symlink file that matches the source file name with no suffix will be created on the remote.
-### Quota information
+This will effectively allow commands like copy/copyto, move/moveto and sync to upload from local to remote and download from remote to local directories with symlinks. Due to internal rclone limitations, it is not possible to upload an individual symlink file to any remote backend. You can always use the "backend symlink" command to create a symlink on the NetStorage server, refer to "symlink" section below.
-To view your current quota you can use the `rclone about remote:`
-command which will display your usage limit (quota) and the current usage.
+Individual symlink files on the remote can be used with the commands like "cat" to print the destination name, or "delete" to delete symlink, or copy, copy/to and move/moveto to download from the remote to local. Note: individual symlink files on the remote should be specified including the suffix .rclonelink.
-### Restricted filename characters
+**Note**: No file with the suffix .rclonelink should ever exist on the server since it is not possible to actually upload/create a file with .rclonelink suffix with rclone, it can only exist if it is manually created through a non-rclone method on the remote.
-In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters)
-the following characters are also replaced:
+### Implicit vs. Explicit Directories
-| Character | Value | Replacement |
-| --------- |:-----:|:-----------:|
-| " | 0x22 | ＂ |
-| * | 0x2A | ＊ |
-| : | 0x3A | ： |
-| < | 0x3C | ＜ |
-| > | 0x3E | ＞ |
-| ? | 0x3F | ？ |
-| \ | 0x5C | ï¼¼ |
-| \| | 0x7C | ｜ |
+With NetStorage, directories can exist in one of two forms:
-Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8),
-as they can't be used in JSON strings.
+1. **Explicit Directory**. This is an actual, physical directory that you have created in a storage group.
+2. **Implicit Directory**. This refers to a directory within a path that has not been physically created. For example, during upload of a file, nonexistent subdirectories can be specified in the target path. NetStorage creates these as "implicit." While the directories aren't physically created, they exist implicitly and the noted path is connected with the uploaded file.
+Rclone will intercept all file uploads and mkdir commands for the NetStorage remote and will explicitly issue the mkdir command for each directory in the uploading path. This will help with the interoperability with the other Akamai services such as SFTP and the Content Management Shell (CMShell). Rclone will not guarantee correctness of operations with implicit directories which might have been created as a result of using an upload API directly.
-### Standard options
+### `--fast-list` / ListR support
-Here are the Standard options specific to mailru (Mail.ru Cloud).
+NetStorage remote supports the ListR feature by using the "list" NetStorage API action to return a lexicographical list of all objects within the specified CP code, recursing into subdirectories as they're encountered.
-#### --mailru-client-id
+* **Rclone will use the ListR method for some commands by default**. Commands such as `lsf -R` will use ListR by default. To disable this, include the `--disable listR` option to use the non-recursive method of listing objects.
-OAuth Client Id.
+* **Rclone will not use the ListR method for some commands**. Commands such as `sync` don't use ListR by default. To force using the ListR method, include the `--fast-list` option.
-Leave blank normally.
+There are pros and cons of using the ListR method, refer to [rclone documentation](https://rclone.org/docs/#fast-list). In general, the sync command over an existing deep tree on the remote will run faster with the "--fast-list" flag but with extra memory usage as a side effect. It might also result in higher CPU utilization but the whole task can be completed faster.
-Properties:
+**Note**: There is a known limitation that "lsf -R" will display number of files in the directory and directory size as -1 when ListR method is used. The workaround is to pass "--disable listR" flag if these numbers are important in the output.
-- Config: client_id
-- Env Var: RCLONE_MAILRU_CLIENT_ID
-- Type: string
-- Required: false
+### Purge
-#### --mailru-client-secret
+NetStorage remote supports the purge feature by using the "quick-delete" NetStorage API action. The quick-delete action is disabled by default for security reasons and can be enabled for the account through the Akamai portal. Rclone will first try to use quick-delete action for the purge command and if this functionality is disabled then will fall back to a standard delete method.
-OAuth Client Secret.
+**Note**: Read the [NetStorage Usage API](https://learn.akamai.com/en-us/webhelp/netstorage/netstorage-http-api-developer-guide/GUID-15836617-9F50-405A-833C-EA2556756A30.html) for considerations when using "quick-delete". In general, using quick-delete method will not delete the tree immediately and objects targeted for quick-delete may still be accessible.
-Leave blank normally.
+
+### Standard options
+
+Here are the Standard options specific to netstorage (Akamai NetStorage).
+
+#### --netstorage-host
+
+Domain+path of NetStorage host to connect to.
+
+Format should be `/`
Properties:
-- Config: client_secret
-- Env Var: RCLONE_MAILRU_CLIENT_SECRET
+- Config: host
+- Env Var: RCLONE_NETSTORAGE_HOST
- Type: string
-- Required: false
+- Required: true
-#### --mailru-user
+#### --netstorage-account
-User name (usually email).
+Set the NetStorage account name
Properties:
-- Config: user
-- Env Var: RCLONE_MAILRU_USER
+- Config: account
+- Env Var: RCLONE_NETSTORAGE_ACCOUNT
- Type: string
- Required: true
-#### --mailru-pass
-
-Password.
+#### --netstorage-secret
-This must be an app password - rclone will not work with your normal
-password. See the Configuration section in the docs for how to make an
-app password.
+Set the NetStorage account secret/G2O key for authentication.
+Please choose the 'y' option to set your own password then enter your secret.
**NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/).
Properties:
-- Config: pass
-- Env Var: RCLONE_MAILRU_PASS
+- Config: secret
+- Env Var: RCLONE_NETSTORAGE_SECRET
- Type: string
- Required: true
-#### --mailru-speedup-enable
+### Advanced options
-Skip full upload if there is another file with same data hash.
+Here are the Advanced options specific to netstorage (Akamai NetStorage).
-This feature is called "speedup" or "put by hash". It is especially efficient
-in case of generally available files like popular books, video or audio clips,
-because files are searched by hash in all accounts of all mailru users.
-It is meaningless and ineffective if source file is unique or encrypted.
-Please note that rclone may need local memory and disk space to calculate
-content hash in advance and decide whether full upload is required.
-Also, if rclone does not know file size in advance (e.g. in case of
-streaming or partial uploads), it will not even try this optimization.
+#### --netstorage-protocol
+
+Select between HTTP or HTTPS protocol.
+
+Most users should choose HTTPS, which is the default.
+HTTP is provided primarily for debugging purposes.
Properties:
-- Config: speedup_enable
-- Env Var: RCLONE_MAILRU_SPEEDUP_ENABLE
-- Type: bool
-- Default: true
+- Config: protocol
+- Env Var: RCLONE_NETSTORAGE_PROTOCOL
+- Type: string
+- Default: "https"
- Examples:
- - "true"
- - Enable
- - "false"
- - Disable
+ - "http"
+ - HTTP protocol
+ - "https"
+ - HTTPS protocol
-### Advanced options
+## Backend commands
-Here are the Advanced options specific to mailru (Mail.ru Cloud).
+Here are the commands specific to the netstorage backend.
-#### --mailru-token
+Run them with
-OAuth Access Token as a JSON blob.
+ rclone backend COMMAND remote:
-Properties:
+The help below will explain what arguments each command takes.
-- Config: token
-- Env Var: RCLONE_MAILRU_TOKEN
-- Type: string
-- Required: false
+See the [backend](https://rclone.org/commands/rclone_backend/) command for more
+info on how to pass options and arguments.
-#### --mailru-auth-url
+These can be run on a running backend using the rc command
+[backend/command](https://rclone.org/rc/#backend-command).
-Auth server URL.
+### du
-Leave blank to use the provider defaults.
+Return disk usage information for a specified directory
-Properties:
+ rclone backend du remote: [options] [+]
-- Config: auth_url
-- Env Var: RCLONE_MAILRU_AUTH_URL
-- Type: string
-- Required: false
+The usage information returned, includes the targeted directory as well as all
+files stored in any sub-directories that may exist.
-#### --mailru-token-url
+### symlink
-Token server url.
+You can create a symbolic link in ObjectStore with the symlink action.
-Leave blank to use the provider defaults.
+ rclone backend symlink remote: [options] [+]
-Properties:
+The desired path location (including applicable sub-directories) ending in
+the object that will be the target of the symlink (for example, /links/mylink).
+Include the file extension for the object, if applicable.
+`rclone backend symlink `
-- Config: token_url
-- Env Var: RCLONE_MAILRU_TOKEN_URL
-- Type: string
-- Required: false
-#### --mailru-speedup-file-patterns
-Comma separated list of file name patterns eligible for speedup (put by hash).
+# Microsoft Azure Blob Storage
-Patterns are case insensitive and can contain '*' or '?' meta characters.
+Paths are specified as `remote:container` (or `remote:` for the `lsd`
+command.) You may put subdirectories in too, e.g.
+`remote:container/path/to/dir`.
-Properties:
+## Configuration
-- Config: speedup_file_patterns
-- Env Var: RCLONE_MAILRU_SPEEDUP_FILE_PATTERNS
-- Type: string
-- Default: "*.mkv,*.avi,*.mp4,*.mp3,*.zip,*.gz,*.rar,*.pdf"
-- Examples:
- - ""
- - Empty list completely disables speedup (put by hash).
- - "*"
- - All files will be attempted for speedup.
- - "*.mkv,*.avi,*.mp4,*.mp3"
- - Only common audio/video files will be tried for put by hash.
- - "*.zip,*.gz,*.rar,*.pdf"
- - Only common archives or PDF books will be tried for speedup.
+Here is an example of making a Microsoft Azure Blob Storage
+configuration. For a remote called `remote`. First run:
-#### --mailru-speedup-max-disk
+ rclone config
-This option allows you to disable speedup (put by hash) for large files.
+This will guide you through an interactive setup process:
-Reason is that preliminary hashing can exhaust your RAM or disk space.
+```
+No remotes found, make a new one?
+n) New remote
+s) Set configuration password
+q) Quit config
+n/s/q> n
+name> remote
+Type of storage to configure.
+Choose a number from below, or type in your own value
+[snip]
+XX / Microsoft Azure Blob Storage
+ \ "azureblob"
+[snip]
+Storage> azureblob
+Storage Account Name
+account> account_name
+Storage Account Key
+key> base64encodedkey==
+Endpoint for the service - leave blank normally.
+endpoint>
+Remote config
+--------------------
+[remote]
+account = account_name
+key = base64encodedkey==
+endpoint =
+--------------------
+y) Yes this is OK
+e) Edit this remote
+d) Delete this remote
+y/e/d> y
+```
-Properties:
+See all containers
-- Config: speedup_max_disk
-- Env Var: RCLONE_MAILRU_SPEEDUP_MAX_DISK
-- Type: SizeSuffix
-- Default: 3Gi
-- Examples:
- - "0"
- - Completely disable speedup (put by hash).
- - "1G"
- - Files larger than 1Gb will be uploaded directly.
- - "3G"
- - Choose this option if you have less than 3Gb free on local disk.
+ rclone lsd remote:
+
+Make a new container
+
+ rclone mkdir remote:container
+
+List the contents of a container
+
+ rclone ls remote:container
+
+Sync `/home/local/directory` to the remote container, deleting any excess
+files in the container.
+
+ rclone sync --interactive /home/local/directory remote:container
+
+### --fast-list
+
+This remote supports `--fast-list` which allows you to use fewer
+transactions in exchange for more memory. See the [rclone
+docs](https://rclone.org/docs/#fast-list) for more details.
+
+### Modification times and hashes
+
+The modification time is stored as metadata on the object with the
+`mtime` key. It is stored using RFC3339 Format time with nanosecond
+precision. The metadata is supplied during directory listings so
+there is no performance overhead to using it.
+
+If you wish to use the Azure standard `LastModified` time stored on
+the object as the modified time, then use the `--use-server-modtime`
+flag. Note that rclone can't set `LastModified`, so using the
+`--update` flag when syncing is recommended if using
+`--use-server-modtime`.
+
+MD5 hashes are stored with blobs. However blobs that were uploaded in
+chunks only have an MD5 if the source remote was capable of MD5
+hashes, e.g. the local disk.
+
+### Performance
+
+When uploading large files, increasing the value of
+`--azureblob-upload-concurrency` will increase performance at the cost
+of using more memory. The default of 16 is set quite conservatively to
+use less memory. It maybe be necessary raise it to 64 or higher to
+fully utilize a 1 GBit/s link with a single file transfer.
+
+### Restricted filename characters
+
+In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters)
+the following characters are also replaced:
+
+| Character | Value | Replacement |
+| --------- |:-----:|:-----------:|
+| / | 0x2F | ï¼ |
+| \ | 0x5C | ï¼¼ |
+
+File names can also not end with the following characters.
+These only get replaced if they are the last character in the name:
+
+| Character | Value | Replacement |
+| --------- |:-----:|:-----------:|
+| . | 0x2E | ． |
+
+Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8),
+as they can't be used in JSON strings.
+
+### Authentication {#authentication}
+
+There are a number of ways of supplying credentials for Azure Blob
+Storage. Rclone tries them in the order of the sections below.
+
+#### Env Auth
+
+If the `env_auth` config parameter is `true` then rclone will pull
+credentials from the environment or runtime.
+
+It tries these authentication methods in this order:
+
+1. Environment Variables
+2. Managed Service Identity Credentials
+3. Azure CLI credentials (as used by the az tool)
+
+These are described in the following sections
+
+##### Env Auth: 1. Environment Variables
+
+If `env_auth` is set and environment variables are present rclone
+authenticates a service principal with a secret or certificate, or a
+user with a password, depending on which environment variable are set.
+It reads configuration from these variables, in the following order:
+
+1. Service principal with client secret
+ - `AZURE_TENANT_ID`: ID of the service principal's tenant. Also called its "directory" ID.
+ - `AZURE_CLIENT_ID`: the service principal's client ID
+ - `AZURE_CLIENT_SECRET`: one of the service principal's client secrets
+2. Service principal with certificate
+ - `AZURE_TENANT_ID`: ID of the service principal's tenant. Also called its "directory" ID.
+ - `AZURE_CLIENT_ID`: the service principal's client ID
+ - `AZURE_CLIENT_CERTIFICATE_PATH`: path to a PEM or PKCS12 certificate file including the private key.
+ - `AZURE_CLIENT_CERTIFICATE_PASSWORD`: (optional) password for the certificate file.
+ - `AZURE_CLIENT_SEND_CERTIFICATE_CHAIN`: (optional) Specifies whether an authentication request will include an x5c header to support subject name / issuer based authentication. When set to "true" or "1", authentication requests include the x5c header.
+3. User with username and password
+ - `AZURE_TENANT_ID`: (optional) tenant to authenticate in. Defaults to "organizations".
+ - `AZURE_CLIENT_ID`: client ID of the application the user will authenticate to
+ - `AZURE_USERNAME`: a username (usually an email address)
+ - `AZURE_PASSWORD`: the user's password
+4. Workload Identity
+ - `AZURE_TENANT_ID`: Tenant to authenticate in.
+ - `AZURE_CLIENT_ID`: Client ID of the application the user will authenticate to.
+ - `AZURE_FEDERATED_TOKEN_FILE`: Path to projected service account token file.
+ - `AZURE_AUTHORITY_HOST`: Authority of an Azure Active Directory endpoint (default: login.microsoftonline.com).
+
+
+##### Env Auth: 2. Managed Service Identity Credentials
+
+When using Managed Service Identity if the VM(SS) on which this
+program is running has a system-assigned identity, it will be used by
+default. If the resource has no system-assigned but exactly one
+user-assigned identity, the user-assigned identity will be used by
+default.
+
+If the resource has multiple user-assigned identities you will need to
+unset `env_auth` and set `use_msi` instead. See the [`use_msi`
+section](#use_msi).
+
+##### Env Auth: 3. Azure CLI credentials (as used by the az tool)
+
+Credentials created with the `az` tool can be picked up using `env_auth`.
+
+For example if you were to login with a service principal like this:
+
+ az login --service-principal -u XXX -p XXX --tenant XXX
-#### --mailru-speedup-max-memory
+Then you could access rclone resources like this:
-Files larger than the size given below will always be hashed on disk.
+ rclone lsf :azureblob,env_auth,account=ACCOUNT:CONTAINER
-Properties:
+Or
-- Config: speedup_max_memory
-- Env Var: RCLONE_MAILRU_SPEEDUP_MAX_MEMORY
-- Type: SizeSuffix
-- Default: 32Mi
-- Examples:
- - "0"
- - Preliminary hashing will always be done in a temporary disk location.
- - "32M"
- - Do not dedicate more than 32Mb RAM for preliminary hashing.
- - "256M"
- - You have at most 256Mb RAM free for hash calculations.
+ rclone lsf --azureblob-env-auth --azureblob-account=ACCOUNT :azureblob:CONTAINER
-#### --mailru-check-hash
+Which is analogous to using the `az` tool:
-What should copy do if file checksum is mismatched or invalid.
+ az storage blob list --container-name CONTAINER --account-name ACCOUNT --auth-mode login
-Properties:
+#### Account and Shared Key
-- Config: check_hash
-- Env Var: RCLONE_MAILRU_CHECK_HASH
-- Type: bool
-- Default: true
-- Examples:
- - "true"
- - Fail with error.
- - "false"
- - Ignore and continue.
+This is the most straight forward and least flexible way. Just fill
+in the `account` and `key` lines and leave the rest blank.
-#### --mailru-user-agent
+#### SAS URL
-HTTP user agent used internally by client.
+This can be an account level SAS URL or container level SAS URL.
-Defaults to "rclone/VERSION" or "--user-agent" provided on command line.
+To use it leave `account` and `key` blank and fill in `sas_url`.
-Properties:
+An account level SAS URL or container level SAS URL can be obtained
+from the Azure portal or the Azure Storage Explorer. To get a
+container level SAS URL right click on a container in the Azure Blob
+explorer in the Azure portal.
-- Config: user_agent
-- Env Var: RCLONE_MAILRU_USER_AGENT
-- Type: string
-- Required: false
+If you use a container level SAS URL, rclone operations are permitted
+only on a particular container, e.g.
-#### --mailru-quirks
+ rclone ls azureblob:container
-Comma separated list of internal maintenance flags.
+You can also list the single container from the root. This will only
+show the container specified by the SAS URL.
-This option must not be used by an ordinary user. It is intended only to
-facilitate remote troubleshooting of backend issues. Strict meaning of
-flags is not documented and not guaranteed to persist between releases.
-Quirks will be removed when the backend grows stable.
-Supported quirks: atomicmkdir binlist unknowndirs
+ $ rclone lsd azureblob:
+ container/
-Properties:
+Note that you can't see or access any other containers - this will
+fail
-- Config: quirks
-- Env Var: RCLONE_MAILRU_QUIRKS
-- Type: string
-- Required: false
+ rclone ls azureblob:othercontainer
-#### --mailru-encoding
+Container level SAS URLs are useful for temporarily allowing third
+parties access to a single container or putting credentials into an
+untrusted environment such as a CI build server.
-The encoding for the backend.
+#### Service principal with client secret
-See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info.
+If these variables are set, rclone will authenticate with a service principal with a client secret.
-Properties:
+- `tenant`: ID of the service principal's tenant. Also called its "directory" ID.
+- `client_id`: the service principal's client ID
+- `client_secret`: one of the service principal's client secrets
-- Config: encoding
-- Env Var: RCLONE_MAILRU_ENCODING
-- Type: MultiEncoder
-- Default: Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Del,Ctl,InvalidUtf8,Dot
+The credentials can also be placed in a file using the
+`service_principal_file` configuration option.
+#### Service principal with certificate
+If these variables are set, rclone will authenticate with a service principal with certificate.
-## Limitations
+- `tenant`: ID of the service principal's tenant. Also called its "directory" ID.
+- `client_id`: the service principal's client ID
+- `client_certificate_path`: path to a PEM or PKCS12 certificate file including the private key.
+- `client_certificate_password`: (optional) password for the certificate file.
+- `client_send_certificate_chain`: (optional) Specifies whether an authentication request will include an x5c header to support subject name / issuer based authentication. When set to "true" or "1", authentication requests include the x5c header.
-File size limits depend on your account. A single file size is limited by 2G
-for a free account and unlimited for paid tariffs. Please refer to the Mail.ru
-site for the total uploaded size limits.
+**NB** `client_certificate_password` must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/).
-Note that Mailru is case insensitive so you can't have a file called
-"Hello.doc" and one called "hello.doc".
+#### User with username and password
-# Mega
+If these variables are set, rclone will authenticate with username and password.
-[Mega](https://mega.nz/) is a cloud storage and file hosting service
-known for its security feature where all files are encrypted locally
-before they are uploaded. This prevents anyone (including employees of
-Mega) from accessing the files without knowledge of the key used for
-encryption.
+- `tenant`: (optional) tenant to authenticate in. Defaults to "organizations".
+- `client_id`: client ID of the application the user will authenticate to
+- `username`: a username (usually an email address)
+- `password`: the user's password
-This is an rclone backend for Mega which supports the file transfer
-features of Mega using the same client side encryption.
+Microsoft doesn't recommend this kind of authentication, because it's
+less secure than other authentication flows. This method is not
+interactive, so it isn't compatible with any form of multi-factor
+authentication, and the application must already have user or admin
+consent. This credential can only authenticate work and school
+accounts; it can't authenticate Microsoft accounts.
-Paths are specified as `remote:path`
+**NB** `password` must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/).
-Paths may be as deep as required, e.g. `remote:directory/subdirectory`.
+#### Managed Service Identity Credentials {#use_msi}
-## Configuration
+If `use_msi` is set then managed service identity credentials are
+used. This authentication only works when running in an Azure service.
+`env_auth` needs to be unset to use this.
-Here is an example of how to make a remote called `remote`. First run:
+However if you have multiple user identities to choose from these must
+be explicitly specified using exactly one of the `msi_object_id`,
+`msi_client_id`, or `msi_mi_res_id` parameters.
- rclone config
+If none of `msi_object_id`, `msi_client_id`, or `msi_mi_res_id` is
+set, this is is equivalent to using `env_auth`.
-This will guide you through an interactive setup process:
-```
-No remotes found, make a new one?
-n) New remote
-s) Set configuration password
-q) Quit config
-n/s/q> n
-name> remote
-Type of storage to configure.
-Choose a number from below, or type in your own value
-[snip]
-XX / Mega
- \ "mega"
-[snip]
-Storage> mega
-User name
-user> you@example.com
-Password.
-y) Yes type in my own password
-g) Generate random password
-n) No leave this optional password blank
-y/g/n> y
-Enter the password:
-password:
-Confirm the password:
-password:
-Remote config
---------------------
-[remote]
-type = mega
-user = you@example.com
-pass = *** ENCRYPTED ***
---------------------
-y) Yes this is OK
-e) Edit this remote
-d) Delete this remote
-y/e/d> y
-```
+### Standard options
-**NOTE:** The encryption keys need to have been already generated after a regular login
-via the browser, otherwise attempting to use the credentials in `rclone` will fail.
+Here are the Standard options specific to azureblob (Microsoft Azure Blob Storage).
-Once configured you can then use `rclone` like this,
+#### --azureblob-account
-List directories in top level of your Mega
+Azure Storage Account Name.
- rclone lsd remote:
+Set this to the Azure Storage Account Name in use.
-List all the files in your Mega
+Leave blank to use SAS URL or Emulator, otherwise it needs to be set.
- rclone ls remote:
+If this is blank and if env_auth is set it will be read from the
+environment variable `AZURE_STORAGE_ACCOUNT_NAME` if possible.
-To copy a local directory to an Mega directory called backup
- rclone copy /home/source remote:backup
+Properties:
-### Modified time and hashes
+- Config: account
+- Env Var: RCLONE_AZUREBLOB_ACCOUNT
+- Type: string
+- Required: false
-Mega does not support modification times or hashes yet.
+#### --azureblob-env-auth
-### Restricted filename characters
+Read credentials from runtime (environment variables, CLI or MSI).
-| Character | Value | Replacement |
-| --------- |:-----:|:-----------:|
-| NUL | 0x00 | †|
-| / | 0x2F | ï¼ |
+See the [authentication docs](/azureblob#authentication) for full info.
-Invalid UTF-8 bytes will also be [replaced](https://rclone.org/overview/#invalid-utf8),
-as they can't be used in JSON strings.
+Properties:
-### Duplicated files
+- Config: env_auth
+- Env Var: RCLONE_AZUREBLOB_ENV_AUTH
+- Type: bool
+- Default: false
-Mega can have two files with exactly the same name and path (unlike a
-normal file system).
+#### --azureblob-key
-Duplicated files cause problems with the syncing and you will see
-messages in the log about duplicates.
+Storage Account Shared Key.
-Use `rclone dedupe` to fix duplicated files.
+Leave blank to use SAS URL or Emulator.
-### Failure to log-in
+Properties:
-#### Object not found
+- Config: key
+- Env Var: RCLONE_AZUREBLOB_KEY
+- Type: string
+- Required: false
-If you are connecting to your Mega remote for the first time,
-to test access and synchronization, you may receive an error such as
+#### --azureblob-sas-url
-```
-Failed to create file system for "my-mega-remote:":
-couldn't login: Object (typically, node or user) not found
-```
+SAS URL for container level access only.
-The diagnostic steps often recommended in the [rclone forum](https://forum.rclone.org/search?q=mega)
-start with the **MEGAcmd** utility. Note that this refers to
-the official C++ command from https://github.com/meganz/MEGAcmd
-and not the go language built command from t3rm1n4l/megacmd
-that is no longer maintained.
+Leave blank if using account/key or Emulator.
-Follow the instructions for installing MEGAcmd and try accessing
-your remote as they recommend. You can establish whether or not
-you can log in using MEGAcmd, and obtain diagnostic information
-to help you, and search or work with others in the forum.
+Properties:
-```
-MEGA CMD> login me@example.com
-Password:
-Fetching nodes ...
-Loading transfers from local cache
-Login complete as me@example.com
-me@example.com:/$
-```
+- Config: sas_url
+- Env Var: RCLONE_AZUREBLOB_SAS_URL
+- Type: string
+- Required: false
-Note that some have found issues with passwords containing special
-characters. If you can not log on with rclone, but MEGAcmd logs on
-just fine, then consider changing your password temporarily to
-pure alphanumeric characters, in case that helps.
+#### --azureblob-tenant
+ID of the service principal's tenant. Also called its directory ID.
-#### Repeated commands blocks access
+Set this if using
+- Service principal with client secret
+- Service principal with certificate
+- User with username and password
-Mega remotes seem to get blocked (reject logins) under "heavy use".
-We haven't worked out the exact blocking rules but it seems to be
-related to fast paced, successive rclone commands.
-For example, executing this command 90 times in a row `rclone link
-remote:file` will cause the remote to become "blocked". This is not an
-abnormal situation, for example if you wish to get the public links of
-a directory with hundred of files... After more or less a week, the
-remote will remote accept rclone logins normally again.
+Properties:
-You can mitigate this issue by mounting the remote it with `rclone
-mount`. This will log-in when mounting and a log-out when unmounting
-only. You can also run `rclone rcd` and then use `rclone rc` to run
-the commands over the API to avoid logging in each time.
+- Config: tenant
+- Env Var: RCLONE_AZUREBLOB_TENANT
+- Type: string
+- Required: false
-Rclone does not currently close mega sessions (you can see them in the
-web interface), however closing the sessions does not solve the issue.
+#### --azureblob-client-id
-If you space rclone commands by 3 seconds it will avoid blocking the
-remote. We haven't identified the exact blocking rules, so perhaps one
-could execute the command 80 times without waiting and avoid blocking
-by waiting 3 seconds, then continuing...
+The ID of the client in use.
-Note that this has been observed by trial and error and might not be
-set in stone.
+Set this if using
+- Service principal with client secret
+- Service principal with certificate
+- User with username and password
-Other tools seem not to produce this blocking effect, as they use a
-different working approach (state-based, using sessionIDs instead of
-log-in) which isn't compatible with the current stateless rclone
-approach.
-Note that once blocked, the use of other tools (such as megacmd) is
-not a sure workaround: following megacmd login times have been
-observed in succession for blocked remote: 7 minutes, 20 min, 30min, 30
-min, 30min. Web access looks unaffected though.
+Properties:
-Investigation is continuing in relation to workarounds based on
-timeouts, pacers, retrials and tpslimits - if you discover something
-relevant, please post on the forum.
+- Config: client_id
+- Env Var: RCLONE_AZUREBLOB_CLIENT_ID
+- Type: string
+- Required: false
-So, if rclone was working nicely and suddenly you are unable to log-in
-and you are sure the user and the password are correct, likely you
-have got the remote blocked for a while.
+#### --azureblob-client-secret
+One of the service principal's client secrets
-### Standard options
+Set this if using
+- Service principal with client secret
-Here are the Standard options specific to mega (Mega).
-#### --mega-user
+Properties:
+
+- Config: client_secret
+- Env Var: RCLONE_AZUREBLOB_CLIENT_SECRET
+- Type: string
+- Required: false
+
+#### --azureblob-client-certificate-path
+
+Path to a PEM or PKCS12 certificate file including the private key.
+
+Set this if using
+- Service principal with certificate
-User name.
Properties:
-- Config: user
-- Env Var: RCLONE_MEGA_USER
+- Config: client_certificate_path
+- Env Var: RCLONE_AZUREBLOB_CLIENT_CERTIFICATE_PATH
- Type: string
-- Required: true
+- Required: false
-#### --mega-pass
+#### --azureblob-client-certificate-password
+
+Password for the certificate file (optional).
+
+Optionally set this if using
+- Service principal with certificate
+
+And the certificate has a password.
-Password.
**NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/).
Properties:
-- Config: pass
-- Env Var: RCLONE_MEGA_PASS
+- Config: client_certificate_password
+- Env Var: RCLONE_AZUREBLOB_CLIENT_CERTIFICATE_PASSWORD
- Type: string
-- Required: true
+- Required: false
### Advanced options
-Here are the Advanced options specific to mega (Mega).
+Here are the Advanced options specific to azureblob (Microsoft Azure Blob Storage).
-#### --mega-debug
+#### --azureblob-client-send-certificate-chain
-Output more debug from Mega.
+Send the certificate chain when using certificate auth.
+
+Specifies whether an authentication request will include an x5c header
+to support subject name / issuer based authentication. When set to
+true, authentication requests include the x5c header.
+
+Optionally set this if using
+- Service principal with certificate
-If this flag is set (along with -vv) it will print further debugging
-information from the mega backend.
Properties:
-- Config: debug
-- Env Var: RCLONE_MEGA_DEBUG
+- Config: client_send_certificate_chain
+- Env Var: RCLONE_AZUREBLOB_CLIENT_SEND_CERTIFICATE_CHAIN
- Type: bool
- Default: false
-#### --mega-hard-delete
+#### --azureblob-username
-Delete files permanently rather than putting them into the trash.
+User name (usually an email address)
+
+Set this if using
+- User with username and password
-Normally the mega backend will put all deletions into the trash rather
-than permanently deleting them. If you specify this then rclone will
-permanently delete objects instead.
Properties:
-- Config: hard_delete
-- Env Var: RCLONE_MEGA_HARD_DELETE
-- Type: bool
-- Default: false
+- Config: username
+- Env Var: RCLONE_AZUREBLOB_USERNAME
+- Type: string
+- Required: false
-#### --mega-use-https
+#### --azureblob-password
-Use HTTPS for transfers.
+The user's password
-MEGA uses plain text HTTP connections by default.
-Some ISPs throttle HTTP connections, this causes transfers to become very slow.
-Enabling this will force MEGA to use HTTPS for all transfers.
-HTTPS is normally not necessary since all data is already encrypted anyway.
-Enabling it will increase CPU usage and add network overhead.
+Set this if using
+- User with username and password
+
+
+**NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/).
Properties:
-- Config: use_https
-- Env Var: RCLONE_MEGA_USE_HTTPS
-- Type: bool
-- Default: false
+- Config: password
+- Env Var: RCLONE_AZUREBLOB_PASSWORD
+- Type: string
+- Required: false
-#### --mega-encoding
+#### --azureblob-service-principal-file
-The encoding for the backend.
+Path to file containing credentials for use with a service principal.
-See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info.
+Leave blank normally. Needed only if you want to use a service principal instead of interactive login.
-Properties:
+ $ az ad sp create-for-rbac --name "" \
+ --role "Storage Blob Data Owner" \
+ --scopes "/subscriptions//resourceGroups//providers/Microsoft.Storage/storageAccounts//blobServices/default/containers/" \
+ > azure-principal.json
-- Config: encoding
-- Env Var: RCLONE_MEGA_ENCODING
-- Type: MultiEncoder
-- Default: Slash,InvalidUtf8,Dot
+See ["Create an Azure service principal"](https://docs.microsoft.com/en-us/cli/azure/create-an-azure-service-principal-azure-cli) and ["Assign an Azure role for access to blob data"](https://docs.microsoft.com/en-us/azure/storage/common/storage-auth-aad-rbac-cli) pages for more details.
+It may be more convenient to put the credentials directly into the
+rclone config file under the `client_id`, `tenant` and `client_secret`
+keys instead of setting `service_principal_file`.
-### Process `killed`
+Properties:
-On accounts with large files or something else, memory usage can significantly increase when executing list/sync instructions. When running on cloud providers (like AWS with EC2), check if the instance type has sufficient memory/CPU to execute the commands. Use the resource monitoring tools to inspect after sending the commands. Look [at this issue](https://forum.rclone.org/t/rclone-with-mega-appears-to-work-only-in-some-accounts/40233/4).
+- Config: service_principal_file
+- Env Var: RCLONE_AZUREBLOB_SERVICE_PRINCIPAL_FILE
+- Type: string
+- Required: false
-## Limitations
+#### --azureblob-use-msi
-This backend uses the [go-mega go library](https://github.com/t3rm1n4l/go-mega) which is an opensource
-go library implementing the Mega API. There doesn't appear to be any
-documentation for the mega protocol beyond the [mega C++ SDK](https://github.com/meganz/sdk) source code
-so there are likely quite a few errors still remaining in this library.
+Use a managed service identity to authenticate (only works in Azure).
-Mega allows duplicate files which may confuse rclone.
+When true, use a [managed service identity](https://docs.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/)
+to authenticate to Azure Storage instead of a SAS token or account key.
-# Memory
+If the VM(SS) on which this program is running has a system-assigned identity, it will
+be used by default. If the resource has no system-assigned but exactly one user-assigned identity,
+the user-assigned identity will be used by default. If the resource has multiple user-assigned
+identities, the identity to use must be explicitly specified using exactly one of the msi_object_id,
+msi_client_id, or msi_mi_res_id parameters.
-The memory backend is an in RAM backend. It does not persist its
-data - use the local backend for that.
+Properties:
-The memory backend behaves like a bucket-based remote (e.g. like
-s3). Because it has no parameters you can just use it with the
-`:memory:` remote name.
+- Config: use_msi
+- Env Var: RCLONE_AZUREBLOB_USE_MSI
+- Type: bool
+- Default: false
-## Configuration
+#### --azureblob-msi-object-id
-You can configure it as a remote like this with `rclone config` too if
-you want to:
+Object ID of the user-assigned MSI to use, if any.
-```
-No remotes found, make a new one?
-n) New remote
-s) Set configuration password
-q) Quit config
-n/s/q> n
-name> remote
-Type of storage to configure.
-Enter a string value. Press Enter for the default ("").
-Choose a number from below, or type in your own value
-[snip]
-XX / Memory
- \ "memory"
-[snip]
-Storage> memory
-** See help for memory backend at: https://rclone.org/memory/ **
+Leave blank if msi_client_id or msi_mi_res_id specified.
-Remote config
+Properties:
---------------------
-[remote]
-type = memory
---------------------
-y) Yes this is OK (default)
-e) Edit this remote
-d) Delete this remote
-y/e/d> y
-```
+- Config: msi_object_id
+- Env Var: RCLONE_AZUREBLOB_MSI_OBJECT_ID
+- Type: string
+- Required: false
-Because the memory backend isn't persistent it is most useful for
-testing or with an rclone server or rclone mount, e.g.
+#### --azureblob-msi-client-id
- rclone mount :memory: /mnt/tmp
- rclone serve webdav :memory:
- rclone serve sftp :memory:
+Object ID of the user-assigned MSI to use, if any.
-### Modified time and hashes
+Leave blank if msi_object_id or msi_mi_res_id specified.
-The memory backend supports MD5 hashes and modification times accurate to 1 nS.
+Properties:
-### Restricted filename characters
+- Config: msi_client_id
+- Env Var: RCLONE_AZUREBLOB_MSI_CLIENT_ID
+- Type: string
+- Required: false
-The memory backend replaces the [default restricted characters
-set](https://rclone.org/overview/#restricted-characters).
+#### --azureblob-msi-mi-res-id
+Azure resource ID of the user-assigned MSI to use, if any.
+Leave blank if msi_client_id or msi_object_id specified.
+Properties:
-# Akamai NetStorage
+- Config: msi_mi_res_id
+- Env Var: RCLONE_AZUREBLOB_MSI_MI_RES_ID
+- Type: string
+- Required: false
-Paths are specified as `remote:`
-You may put subdirectories in too, e.g. `remote:/path/to/dir`.
-If you have a CP code you can use that as the folder after the domain such as \\/\\/\.
+#### --azureblob-use-emulator
-For example, this is commonly configured with or without a CP code:
-* **With a CP code**. `[your-domain-prefix]-nsu.akamaihd.net/123456/subdirectory/`
-* **Without a CP code**. `[your-domain-prefix]-nsu.akamaihd.net`
+Uses local storage emulator if provided as 'true'.
+Leave blank if using real azure storage endpoint.
-See all buckets
- rclone lsd remote:
-The initial setup for Netstorage involves getting an account and secret. Use `rclone config` to walk you through the setup process.
+Properties:
-## Configuration
+- Config: use_emulator
+- Env Var: RCLONE_AZUREBLOB_USE_EMULATOR
+- Type: bool
+- Default: false
-Here's an example of how to make a remote called `ns1`.
+#### --azureblob-endpoint
-1. To begin the interactive configuration process, enter this command:
+Endpoint for the service.
-```
-rclone config
-```
+Leave blank normally.
-2. Type `n` to create a new remote.
+Properties:
-```
-n) New remote
-d) Delete remote
-q) Quit config
-e/n/d/q> n
-```
+- Config: endpoint
+- Env Var: RCLONE_AZUREBLOB_ENDPOINT
+- Type: string
+- Required: false
-3. For this example, enter `ns1` when you reach the name> prompt.
+#### --azureblob-upload-cutoff
-```
-name> ns1
-```
+Cutoff for switching to chunked upload (<= 256 MiB) (deprecated).
-4. Enter `netstorage` as the type of storage to configure.
+Properties:
-```
-Type of storage to configure.
-Enter a string value. Press Enter for the default ("").
-Choose a number from below, or type in your own value
-XX / NetStorage
- \ "netstorage"
-Storage> netstorage
-```
+- Config: upload_cutoff
+- Env Var: RCLONE_AZUREBLOB_UPLOAD_CUTOFF
+- Type: string
+- Required: false
-5. Select between the HTTP or HTTPS protocol. Most users should choose HTTPS, which is the default. HTTP is provided primarily for debugging purposes.
+#### --azureblob-chunk-size
+Upload chunk size.
-```
-Enter a string value. Press Enter for the default ("").
-Choose a number from below, or type in your own value
- 1 / HTTP protocol
- \ "http"
- 2 / HTTPS protocol
- \ "https"
-protocol> 1
-```
+Note that this is stored in memory and there may be up to
+"--transfers" * "--azureblob-upload-concurrency" chunks stored at once
+in memory.
-6. Specify your NetStorage host, CP code, and any necessary content paths using this format: `///`
+Properties:
-```
-Enter a string value. Press Enter for the default ("").
-host> baseball-nsu.akamaihd.net/123456/content/
-```
+- Config: chunk_size
+- Env Var: RCLONE_AZUREBLOB_CHUNK_SIZE
+- Type: SizeSuffix
+- Default: 4Mi
-7. Set the netstorage account name
-```
-Enter a string value. Press Enter for the default ("").
-account> username
-```
+#### --azureblob-upload-concurrency
-8. Set the Netstorage account secret/G2O key which will be used for authentication purposes. Select the `y` option to set your own password then enter your secret.
-Note: The secret is stored in the `rclone.conf` file with hex-encoded encryption.
+Concurrency for multipart uploads.
-```
-y) Yes type in my own password
-g) Generate random password
-y/g> y
-Enter the password:
-password:
-Confirm the password:
-password:
-```
+This is the number of chunks of the same file that are uploaded
+concurrently.
-9. View the summary and confirm your remote configuration.
+If you are uploading small numbers of large files over high-speed
+links and these uploads do not fully utilize your bandwidth, then
+increasing this may help to speed up the transfers.
-```
-[ns1]
-type = netstorage
-protocol = http
-host = baseball-nsu.akamaihd.net/123456/content/
-account = username
-secret = *** ENCRYPTED ***
---------------------
-y) Yes this is OK (default)
-e) Edit this remote
-d) Delete this remote
-y/e/d> y
-```
+In tests, upload speed increases almost linearly with upload
+concurrency. For example to fill a gigabit pipe it may be necessary to
+raise this to 64. Note that this will use more memory.
-This remote is called `ns1` and can now be used.
+Note that chunks are stored in memory and there may be up to
+"--transfers" * "--azureblob-upload-concurrency" chunks stored at once
+in memory.
-## Example operations
+Properties:
-Get started with rclone and NetStorage with these examples. For additional rclone commands, visit https://rclone.org/commands/.
+- Config: upload_concurrency
+- Env Var: RCLONE_AZUREBLOB_UPLOAD_CONCURRENCY
+- Type: int
+- Default: 16
-### See contents of a directory in your project
+#### --azureblob-list-chunk
- rclone lsd ns1:/974012/testing/
+Size of blob list.
-### Sync the contents local with remote
+This sets the number of blobs requested in each listing chunk. Default
+is the maximum, 5000. "List blobs" requests are permitted 2 minutes
+per megabyte to complete. If an operation is taking longer than 2
+minutes per megabyte on average, it will time out (
+[source](https://docs.microsoft.com/en-us/rest/api/storageservices/setting-timeouts-for-blob-service-operations#exceptions-to-default-timeout-interval)
+). This can be used to limit the number of blobs items to return, to
+avoid the time out.
- rclone sync . ns1:/974012/testing/
+Properties:
-### Upload local content to remote
- rclone copy notes.txt ns1:/974012/testing/
+- Config: list_chunk
+- Env Var: RCLONE_AZUREBLOB_LIST_CHUNK
+- Type: int
+- Default: 5000
-### Delete content on remote
- rclone delete ns1:/974012/testing/notes.txt
+#### --azureblob-access-tier
-### Move or copy content between CP codes.
+Access tier of blob: hot, cool, cold or archive.
-Your credentials must have access to two CP codes on the same remote. You can't perform operations between different remotes.
+Archived blobs can be restored by setting access tier to hot, cool or
+cold. Leave blank if you intend to use default access tier, which is
+set at account level
- rclone move ns1:/974012/testing/notes.txt ns1:/974450/testing2/
+If there is no "access tier" specified, rclone doesn't apply any tier.
+rclone performs "Set Tier" operation on blobs while uploading, if objects
+are not modified, specifying "access tier" to new one will have no effect.
+If blobs are in "archive tier" at remote, trying to perform data transfer
+operations from remote will not be allowed. User should first restore by
+tiering blob to "Hot", "Cool" or "Cold".
-## Features
+Properties:
-### Symlink Support
+- Config: access_tier
+- Env Var: RCLONE_AZUREBLOB_ACCESS_TIER
+- Type: string
+- Required: false
-The Netstorage backend changes the rclone `--links, -l` behavior. When uploading, instead of creating the .rclonelink file, use the "symlink" API in order to create the corresponding symlink on the remote. The .rclonelink file will not be created, the upload will be intercepted and only the symlink file that matches the source file name with no suffix will be created on the remote.
+#### --azureblob-archive-tier-delete
-This will effectively allow commands like copy/copyto, move/moveto and sync to upload from local to remote and download from remote to local directories with symlinks. Due to internal rclone limitations, it is not possible to upload an individual symlink file to any remote backend. You can always use the "backend symlink" command to create a symlink on the NetStorage server, refer to "symlink" section below.
+Delete archive tier blobs before overwriting.
-Individual symlink files on the remote can be used with the commands like "cat" to print the destination name, or "delete" to delete symlink, or copy, copy/to and move/moveto to download from the remote to local. Note: individual symlink files on the remote should be specified including the suffix .rclonelink.
+Archive tier blobs cannot be updated. So without this flag, if you
+attempt to update an archive tier blob, then rclone will produce the
+error:
-**Note**: No file with the suffix .rclonelink should ever exist on the server since it is not possible to actually upload/create a file with .rclonelink suffix with rclone, it can only exist if it is manually created through a non-rclone method on the remote.
+ can't update archive tier blob without --azureblob-archive-tier-delete
-### Implicit vs. Explicit Directories
+With this flag set then before rclone attempts to overwrite an archive
+tier blob, it will delete the existing blob before uploading its
+replacement. This has the potential for data loss if the upload fails
+(unlike updating a normal blob) and also may cost more since deleting
+archive tier blobs early may be chargable.
-With NetStorage, directories can exist in one of two forms:
-1. **Explicit Directory**. This is an actual, physical directory that you have created in a storage group.
-2. **Implicit Directory**. This refers to a directory within a path that has not been physically created. For example, during upload of a file, nonexistent subdirectories can be specified in the target path. NetStorage creates these as "implicit." While the directories aren't physically created, they exist implicitly and the noted path is connected with the uploaded file.
+Properties:
-Rclone will intercept all file uploads and mkdir commands for the NetStorage remote and will explicitly issue the mkdir command for each directory in the uploading path. This will help with the interoperability with the other Akamai services such as SFTP and the Content Management Shell (CMShell). Rclone will not guarantee correctness of operations with implicit directories which might have been created as a result of using an upload API directly.
+- Config: archive_tier_delete
+- Env Var: RCLONE_AZUREBLOB_ARCHIVE_TIER_DELETE
+- Type: bool
+- Default: false
-### `--fast-list` / ListR support
+#### --azureblob-disable-checksum
-NetStorage remote supports the ListR feature by using the "list" NetStorage API action to return a lexicographical list of all objects within the specified CP code, recursing into subdirectories as they're encountered.
+Don't store MD5 checksum with object metadata.
-* **Rclone will use the ListR method for some commands by default**. Commands such as `lsf -R` will use ListR by default. To disable this, include the `--disable listR` option to use the non-recursive method of listing objects.
+Normally rclone will calculate the MD5 checksum of the input before
+uploading it so it can add it to metadata on the object. This is great
+for data integrity checking but can cause long delays for large files
+to start uploading.
-* **Rclone will not use the ListR method for some commands**. Commands such as `sync` don't use ListR by default. To force using the ListR method, include the `--fast-list` option.
+Properties:
-There are pros and cons of using the ListR method, refer to [rclone documentation](https://rclone.org/docs/#fast-list). In general, the sync command over an existing deep tree on the remote will run faster with the "--fast-list" flag but with extra memory usage as a side effect. It might also result in higher CPU utilization but the whole task can be completed faster.
+- Config: disable_checksum
+- Env Var: RCLONE_AZUREBLOB_DISABLE_CHECKSUM
+- Type: bool
+- Default: false
-**Note**: There is a known limitation that "lsf -R" will display number of files in the directory and directory size as -1 when ListR method is used. The workaround is to pass "--disable listR" flag if these numbers are important in the output.
+#### --azureblob-memory-pool-flush-time
-### Purge
+How often internal memory buffer pools will be flushed. (no longer used)
-NetStorage remote supports the purge feature by using the "quick-delete" NetStorage API action. The quick-delete action is disabled by default for security reasons and can be enabled for the account through the Akamai portal. Rclone will first try to use quick-delete action for the purge command and if this functionality is disabled then will fall back to a standard delete method.
+Properties:
-**Note**: Read the [NetStorage Usage API](https://learn.akamai.com/en-us/webhelp/netstorage/netstorage-http-api-developer-guide/GUID-15836617-9F50-405A-833C-EA2556756A30.html) for considerations when using "quick-delete". In general, using quick-delete method will not delete the tree immediately and objects targeted for quick-delete may still be accessible.
+- Config: memory_pool_flush_time
+- Env Var: RCLONE_AZUREBLOB_MEMORY_POOL_FLUSH_TIME
+- Type: Duration
+- Default: 1m0s
+#### --azureblob-memory-pool-use-mmap
-### Standard options
+Whether to use mmap buffers in internal memory pool. (no longer used)
-Here are the Standard options specific to netstorage (Akamai NetStorage).
+Properties:
-#### --netstorage-host
+- Config: memory_pool_use_mmap
+- Env Var: RCLONE_AZUREBLOB_MEMORY_POOL_USE_MMAP
+- Type: bool
+- Default: false
-Domain+path of NetStorage host to connect to.
+#### --azureblob-encoding
-Format should be `/`
+The encoding for the backend.
+
+See the [encoding section in the overview](https://rclone.org/overview/#encoding) for more info.
Properties:
-- Config: host
-- Env Var: RCLONE_NETSTORAGE_HOST
-- Type: string
-- Required: true
+- Config: encoding
+- Env Var: RCLONE_AZUREBLOB_ENCODING
+- Type: Encoding
+- Default: Slash,BackSlash,Del,Ctl,RightPeriod,InvalidUtf8
-#### --netstorage-account
+#### --azureblob-public-access
-Set the NetStorage account name
+Public access level of a container: blob or container.
Properties:
-- Config: account
-- Env Var: RCLONE_NETSTORAGE_ACCOUNT
+- Config: public_access
+- Env Var: RCLONE_AZUREBLOB_PUBLIC_ACCESS
- Type: string
-- Required: true
+- Required: false
+- Examples:
+ - ""
+ - The container and its blobs can be accessed only with an authorized request.
+ - It's a default value.
+ - "blob"
+ - Blob data within this container can be read via anonymous request.
+ - "container"
+ - Allow full public read access for container and blob data.
-#### --netstorage-secret
+#### --azureblob-directory-markers
-Set the NetStorage account secret/G2O key for authentication.
+Upload an empty object with a trailing slash when a new directory is created
-Please choose the 'y' option to set your own password then enter your secret.
+Empty folders are unsupported for bucket based remotes, this option
+creates an empty object ending with "/", to persist the folder.
-**NB** Input to this must be obscured - see [rclone obscure](https://rclone.org/commands/rclone_obscure/).
+This object also has the metadata "hdi_isfolder = true" to conform to
+the Microsoft standard.
+
Properties:
-- Config: secret
-- Env Var: RCLONE_NETSTORAGE_SECRET
-- Type: string
-- Required: true
-
-### Advanced options
+- Config: directory_markers
+- Env Var: RCLONE_AZUREBLOB_DIRECTORY_MARKERS
+- Type: bool
+- Default: false
-Here are the Advanced options specific to netstorage (Akamai NetStorage).
+#### --azureblob-no-check-container
-#### --netstorage-protocol
+If set, don't attempt to check the container exists or create it.
-Select between HTTP or HTTPS protocol.
+This can be useful when trying to minimise the number of transactions
+rclone does if you know the container exists already.
-Most users should choose HTTPS, which is the default.
-HTTP is provided primarily for debugging purposes.
Properties:
-- Config: protocol
-- Env Var: RCLONE_NETSTORAGE_PROTOCOL
-- Type: string
-- Default: "https"
-- Examples:
- - "http"
- - HTTP protocol
- - "https"
- - HTTPS protocol
+- Config: no_check_container
+- Env Var: RCLONE_AZUREBLOB_NO_CHECK_CONTAINER
+- Type: bool
+- Default: false
-## Backend commands
+#### --azureblob-no-head-object
-Here are the commands specific to the netstorage backend.
+If set, do not do HEAD before GET when getting objects.
-Run them with
+Properties:
- rclone backend COMMAND remote:
+- Config: no_head_object
+- Env Var: RCLONE_AZUREBLOB_NO_HEAD_OBJECT
+- Type: bool
+- Default: false
-The help below will explain what arguments each command takes.
-See the [backend](https://rclone.org/commands/rclone_backend/) command for more
-info on how to pass options and arguments.
-These can be run on a running backend using the rc command
-[backend/command](https://rclone.org/rc/#backend-command).
+### Custom upload headers
-### du
+You can set custom upload headers with the `--header-upload` flag.
-Return disk usage information for a specified directory
+- Cache-Control
+- Content-Disposition
+- Content-Encoding
+- Content-Language
+- Content-Type
- rclone backend du remote: [options] [+]
+Eg `--header-upload "Content-Type: text/potato"`
-The usage information returned, includes the targeted directory as well as all
-files stored in any sub-directories that may exist.
+## Limitations
-### symlink
+MD5 sums are only uploaded with chunked files if the source has an MD5
+sum. This will always be the case for a local to azure copy.
-You can create a symbolic link in ObjectStore with the symlink action.
+`rclone about` is not supported by the Microsoft Azure Blob storage backend. Backends without
+this capability cannot determine free space for an rclone mount or
+use policy `mfs` (most free space) as a member of an rclone union
+remote.
- rclone backend symlink remote: [options] [+]
+See [List of backends that do not support rclone about](https://rclone.org/overview/#optional-features) and [rclone about](https://rclone.org/commands/rclone_about/)
-The desired path location (including applicable sub-directories) ending in
-the object that will be the target of the symlink (for example, /links/mylink).
-Include the file extension for the object, if applicable.
-`rclone backend symlink `
+## Azure Storage Emulator Support
+You can run rclone with the storage emulator (usually _azurite_).
+
+To do this, just set up a new remote with `rclone config` following
+the instructions in the introduction and set `use_emulator` in the
+advanced settings as `true`. You do not need to provide a default
+account name nor an account key. But you can override them in the
+`account` and `key` options. (Prior to v1.61 they were hard coded to
+_azurite_'s `devstoreaccount1`.)
+Also, if you want to access a storage emulator instance running on a
+different machine, you can override the `endpoint` parameter in the
+advanced settings, setting it to
+`http(s)://:/devstoreaccount1`
+(e.g. `http://10.254.2.5:10000/devstoreaccount1`).
-# Microsoft Azure Blob Storage
+# Microsoft Azure Files Storage
-Paths are specified as `remote:container` (or `remote:` for the `lsd`
-command.) You may put subdirectories in too, e.g.
-`remote:container/path/to/dir`.
+Paths are specified as `remote:` You may put subdirectories in too,
+e.g. `remote:path/to/dir`.
## Configuration
-Here is an example of making a Microsoft Azure Blob Storage
+Here is an example of making a Microsoft Azure Files Storage
configuration. For a remote called `remote`. First run:
rclone config
@@ -37741,69 +39471,90 @@ name> remote
Type of storage to configure.
Choose a number from below, or type in your own value
[snip]
-XX / Microsoft Azure Blob Storage
- \ "azureblob"
+XX / Microsoft Azure Files Storage
+ \ "azurefiles"
[snip]
-Storage> azureblob
-Storage Account Name
+
+Option account.
+Azure Storage Account Name.
+Set this to the Azure Storage Account Name in use.
+Leave blank to use SAS URL or connection string, otherwise it needs to be set.
+If this is blank and if env_auth is set it will be read from the
+environment variable `AZURE_STORAGE_ACCOUNT_NAME` if possible.
+Enter a value. Press Enter to leave empty.
account> account_name
-Storage Account Key
+
+Option share_name.
+Azure Files Share Name.
+This is required and is the name of the share to access.
+Enter a value. Press Enter to leave empty.
+share_name> share_name
+
+Option env_auth.
+Read credentials from runtime (environment variables, CLI or MSI).
+See the [authentication docs](/azurefiles#authentication) for full info.
+Enter a boolean value (true or false). Press Enter for the default (false).
+env_auth>
+
+Option key.
+Storage Account Shared Key.
+Leave blank to use SAS URL or connection string.
+Enter a value. Press Enter to leave empty.
key> base64encodedkey==
-Endpoint for the service - leave blank normally.
-endpoint>
-Remote config
---------------------
-[remote]
-account = account_name
-key = base64encodedkey==
-endpoint =
---------------------
-y) Yes this is OK
+
+Option sas_url.
+SAS URL.
+Leave blank if using account/key or connection string.
+Enter a value. Press Enter to leave empty.
+sas_url>
+
+Option connection_string.
+Azure Files Connection String.
+Enter a value. Press Enter to leave empty.
+connection_string>
+[snip]
+
+Configuration complete.
+Options:
+- type: azurefiles
+- account: account_name
+- share_name: share_name
+- key: base64encodedkey==
+Keep this "remote" remote?
+y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
-y/e/d> y
+y/e/d>
```
-See all containers
-
- rclone lsd remote:
+Once configured you can use rclone.
-Make a new container
+See all files in the top level:
- rclone mkdir remote:container
+ rclone lsf remote:
-List the contents of a container
+Make a new directory in the root:
- rclone ls remote:container
+ rclone mkdir remote:dir
-Sync `/home/local/directory` to the remote container, deleting any excess
-files in the container.
+Recursively List the contents:
- rclone sync --interactive /home/local/directory remote:container
+ rclone ls remote:
-### --fast-list
+Sync `/home/local/directory` to the remote directory, deleting any
+excess files in the directory.
-This remote supports `--fast-list` which allows you to use fewer
-transactions in exchange for more memory. See the [rclone
-docs](https://rclone.org/docs/#fast-list) for more details.
+ rclone sync --interactive /home/local/directory remote:dir
### Modified time
-The modified time is stored as metadata on the object with the `mtime`
-key. It is stored using RFC3339 Format time with nanosecond
-precision. The metadata is supplied during directory listings so
-there is no performance overhead to using it.
-
-If you wish to use the Azure standard `LastModified` time stored on
-the object as the modified time, then use the `--use-server-modtime`
-flag. Note that rclone can't set `LastModified`, so using the
-`--update` flag when syncing is recommended if using
-`--use-server-modtime`.
+The modified time is stored as Azure standard `LastModified` time on
+files
### Performance
When uploading large files, increasing the value of
-`--azureblob-upload-concurrency` will increase performance at the cost
+`--azurefiles-upload-concurrency` will increase performance at the cost
of using more memory. The default of 16 is set quite conservatively to
use less memory. It maybe be necessary raise it to 64 or higher to
fully utilize a 1 GBit/s link with a single file transfer.
@@ -37815,8 +39566,14 @@ the following characters are also replaced:
| Character | Value | Replacement |
| --------- |:-----:|:-----------:|
-| / | 0x2F | ï¼ |
-| \ | 0x5C | ï¼¼ |
+| " | 0x22 | ＂ |
+| * | 0x2A | ＊ |
+| : | 0x3A | ： |
+| < | 0x3C | ＜ |
+| > | 0x3E | ＞ |
+| ? | 0x3F | ？ |
+| \ | 0x5C | ï¼¼ |
+| \| | 0x7C | ｜ |
File names can also not end with the following characters.
These only get replaced if they are the last character in the name:
@@ -37830,13 +39587,12 @@ as they can't be used in JSON strings.
### Hashes
-MD5 hashes are stored with blobs. However blobs that were uploaded in
-chunks only have an MD5 if the source remote was capable of MD5
-hashes, e.g. the local disk.
+MD5 hashes are stored with files. Not all files will have MD5 hashes
+as these have to be uploaded with the file.
### Authentication {#authentication}
-There are a number of ways of supplying credentials for Azure Blob
+There are a number of ways of supplying credentials for Azure Files
Storage. Rclone tries them in the order of the sections below.
#### Env Auth
@@ -37903,15 +39659,11 @@ For example if you were to login with a service principal like this:
Then you could access rclone resources like this:
- rclone lsf :azureblob,env_auth,account=ACCOUNT:CONTAINER
+ rclone lsf :azurefiles,env_auth,account=ACCOUNT:
Or
- rclone lsf --azureblob-env-auth --azureblob-account=ACCOUNT :azureblob:CONTAINER
-
-Which is analogous to using the `az` tool:
-
- az storage blob list --container-name CONTAINER --account-name ACCOUNT --auth-mode login
+ rclone lsf --azurefiles-env-auth --azurefiles-account=ACCOUNT :azurefiles:
#### Account and Shared Key
@@ -37920,34 +39672,11 @@ in the `account` and `key` lines and leave the rest blank.
#### SAS URL
-This can be an account level SAS URL or container level SAS URL.
-
-To use it leave `account` and `key` blank and fill in `sas_url`.
-
-An account level SAS URL or container level SAS URL can be obtained
-from the Azure portal or the Azure Storage Explorer. To get a
-container level SAS URL right click on a container in the Azure Blob
-explorer in the Azure portal.
-
-If you use a container level SAS URL, rclone operations are permitted
-only on a particular container, e.g.
-
- rclone ls azureblob:container
-
-You can also list the single container from the root. This will only
-show the container specified by the SAS URL.
-
- $ rclone lsd azureblob:
- container/
-
-Note that you can't see or access any other containers - this will
-fail
+To use it leave `account`, `key` and `connection_string` blank and fill in `sas_url`.
- rclone ls azureblob:othercontainer
+#### Connection String
-Container level SAS URLs are useful for temporarily allowing third
-parties access to a single container or putting credentials into an
-untrusted environment such as a CI build server.
+To use it leave `account`, `key` and "sas_url" blank and fill in `connection_string`.
#### Service principal with client secret
@@ -38006,15 +39735,15 @@ set, this is is equivalent to using `env_auth`.
### Standard options
-Here are the Standard options specific to azureblob (Microsoft Azure Blob Storage).
+Here are the Standard options specific to azurefiles (Microsoft Azure Files).
-#### --azureblob-account
+#### --azurefiles-account
Azure Storage Account Name.
Set this to the Azure Storage Account Name in use.
-Leave blank to use SAS URL or Emulator, otherwise it needs to be set.
+Leave blank to use SAS URL or connection string, otherwise it needs to be set.
If this is blank and if env_auth is set it will be read from the
environment variable `AZURE_STORAGE_ACCOUNT_NAME` if possible.
@@ -38023,50 +39752,75 @@ environment variable `AZURE_STORAGE_ACCOUNT_NAME` if possible.
Properties:
- Config: account
-- Env Var: RCLONE_AZUREBLOB_ACCOUNT
+- Env Var: RCLONE_AZUREFILES_ACCOUNT
- Type: string
- Required: false
-#### --azureblob-env-auth
+#### --azurefiles-share-name
+
+Azure Files Share Name.
+
+This is required and is the name of the share to access.
+
+
+Properties:
+
+- Config: share_name
+- Env Var: RCLONE_AZUREFILES_SHARE_NAME
+- Type: string
+- Required: false
+
+#### --azurefiles-env-auth
Read credentials from runtime (environment variables, CLI or MSI).
-See the [authentication docs](/azureblob#authentication) for full info.
+See the [authentication docs](/azurefiles#authentication) for full info.
Properties:
- Config: env_auth
-- Env Var: RCLONE_AZUREBLOB_ENV_AUTH
+- Env Var: RCLONE_AZUREFILES_ENV_AUTH
- Type: bool
- Default: false
-#### --azureblob-key
+#### --azurefiles-key
Storage Account Shared Key.
-Leave blank to use SAS URL or Emulator.
+Leave blank to use SAS URL or connection string.
Properties:
- Config: key
-- Env Var: RCLONE_AZUREBLOB_KEY
+- Env Var: RCLONE_AZUREFILES_KEY
- Type: string
- Required: false
-#### --azureblob-sas-url
+#### --azurefiles-sas-url
-SAS URL for container level access only.
+SAS URL.
-Leave blank if using account/key or Emulator.
+Leave blank if using account/key or connection string.
Properties:
- Config: sas_url
-- Env Var: RCLONE_AZUREBLOB_SAS_URL
+- Env Var: RCLONE_AZUREFILES_SAS_URL
- Type: string
- Required: false
-#### --azureblob-tenant
+#### --azurefiles-connection-string
+
+Azure Files Connection String.
+
+Properties:
+
+- Config: connection_string
+- Env Var: RCLONE_AZUREFILES_CONNECTION_STRING
+- Type: string
+- Required: false
+
+#### --azurefiles-tenant
ID of the service principal's tenant. Also called its directory ID.
@@ -38079,11 +39833,11 @@ Set this if using
Properties:
- Config: tenant
-- Env Var: RCLONE_AZUREBLOB_TENANT
+- Env Var: RCLONE_AZUREFILES_TENANT
- Type: string
- Required: false
-#### --azureblob-client-id
+#### --azurefiles-client-id
The ID of the client in use.
@@ -38096,11 +39850,11 @@ Set this if using
Properties:
- Config: client_id
-- Env Var: RCLONE_AZUREBLOB_CLIENT_ID
+- Env Var: RCLONE_AZUREFILES_CLIENT_ID
- Type: string
- Required: false
-#### --azureblob-client-secret
+#### --azurefiles-client-secret
One of the service principal's client secrets
@@ -38111,11 +39865,11 @@ Set this if using
Properties:
- Config: client_secret
-- Env Var: RCLONE_AZUREBLOB_CLIENT_SECRET
+- Env Var: RCLONE_AZUREFILES_CLIENT_SECRET
- Type: string
- Required: false
-#### --azureblob-client-certificate-path
+#### --azurefiles-client-certificate-path
Path to a PEM or PKCS12 certificate file including the private key.
@@ -38126,11 +39880,11 @@ Set this if using
Properties:
- Config: client_certificate_path
-- Env Var: RCLONE_AZUREBLOB_CLIENT_CERTIFICATE_PATH
+- Env Var: RCLONE_AZUREFILES_CLIENT_CERTIFICATE_PATH
- Type: string
- Required: false
-#### --azureblob-client-certificate-password
+#### --azurefiles-client-certificate-password
Password for the certificate file (optional).
@@ -38145,15 +39899,15 @@ And the certificate has a password.
Properties:
- Config: client_certificate_password
-- Env Var: RCLONE_AZUREBLOB_CLIENT_CERTIFICATE_PASSWORD
+- Env Var: RCLONE_AZUREFILES_CLIENT_CERTIFICATE_PASSWORD
- Type: string
- Required: false
### Advanced options
-Here are the Advanced options specific to azureblob (Microsoft Azure Blob Storage).
+Here are the Advanced options specific to azurefiles (Microsoft Azure Files).
-#### --azureblob-client-send-certificate-chain
+#### --azurefiles-client-send-certificate-chain
Send the certificate chain when using certificate auth.
@@ -38168,11 +39922,11 @@ Optionally set this if using
Properties:
- Config: client_send_certificate_chain
-- Env Var: RCLONE_AZUREBLOB_CLIENT_SEND_CERTIFICATE_CHAIN
+- Env Var: RCLONE_AZUREFILES_CLIENT_SEND_CERTIFICATE_CHAIN
- Type: bool
- Default: false
-#### --azureblob-username
+#### --azurefiles-username
User name (usually an email address)
@@ -38183,11 +39937,11 @@ Set this if using
Properties:
- Config: username
-- Env Var: RCLONE_AZUREBLOB_USERNAME
+- Env Var: RCLONE_AZUREFILES_USERNAME
- Type: string
- Required: false
-#### --azureblob-password
+#### --azurefiles-password
The user's password
@@ -38200,22 +39954,24 @@ Set this if using
Properties:
- Config: password
-- Env Var: RCLONE_AZUREBLOB_PASSWORD
+- Env Var: RCLONE_AZUREFILES_PASSWORD
- Type: string
- Required: false
-#### --azureblob-service-principal-file
+#### --azurefiles-service-principal-file
Path to file containing credentials for use with a service principal.
Leave blank normally. Needed only if you want to use a service principal instead of interactive login.
$ az ad sp create-for-rbac --name "" \
- --role "Storage Blob Data Owner" \
+ --role "Storage Files Data Owner" \
--scopes "/subscriptions//resourceGroups//providers/Microsoft.Storage/storageAccounts//blobServices/default/containers/" \
> azure-principal.json
-See ["Create an Azure service principal"](https://docs.microsoft.com/en-us/cli/azure/create-an-azure-service-principal-azure-cli) and ["Assign an Azure role for access to blob data"](https://docs.microsoft.com/en-us/azure/storage/common/storage-auth-aad-rbac-cli) pages for more details.
+See ["Create an Azure service principal"](https://docs.microsoft.com/en-us/cli/azure/create-an-azure-service-principal-azure-cli) and ["Assign an Azure role for access to files data"](https://docs.microsoft.com/en-us/azure/storage/common/storage-auth-aad-rbac-cli) pages for more details.
+
+**NB** this section needs updating for Azure Files - pull requests appreciated!
It may be more convenient to put the credentials directly into the
rclone config file under the `client_id`, `tenant` and `client_secret`
@@ -38225,11 +39981,11 @@ keys instead of setting `service_principal_file`.
Properties:
- Config: service_principal_file
-- Env Var: RCLONE_AZUREBLOB_SERVICE_PRINCIPAL_FILE
+- Env Var: RCLONE_AZUREFILES_SERVICE_PRINCIPAL_FILE
- Type: string
- Required: false
-#### --azureblob-use-msi
+#### --azurefiles-use-msi
Use a managed service identity to authenticate (only works in Azure).
@@ -38245,11 +40001,11 @@ msi_client_id, or msi_mi_res_id parameters.
Properties:
- Config: use_msi
-- Env Var: RCLONE_AZUREBLOB_USE_MSI
+- Env Var: RCLONE_AZUREFILES_USE_MSI
- Type: bool
- Default: false
-#### --azureblob-msi-object-id
+#### --azurefiles-msi-object-id
Object ID of the user-assigned MSI to use, if any.
@@ -38258,11 +40014,11 @@ Leave blank if msi_client_id or msi_mi_res_id specified.
Properties:
- Config: msi_object_id
-- Env Var: RCLONE_AZUREBLOB_MSI_OBJECT_ID
+- Env Var: RCLONE_AZUREFILES_MSI_OBJECT_ID
- Type: string
- Required: false
-#### --azureblob-msi-client-id
+#### --azurefiles-msi-client-id
Object ID of the user-assigned MSI to use, if any.
@@ -38271,11 +40027,11 @@ Leave blank if msi_object_id or msi_mi_res_id specified.
Properties:
- Config: msi_client_id
-- Env Var: RCLONE_AZUREBLOB_MSI_CLIENT_ID
+- Env Var: RCLONE_AZUREFILES_MSI_CLIENT_ID
- Type: string
- Required: false
-#### --azureblob-msi-mi-res-id
+#### --azurefiles-msi-mi-res-id
Azure resource ID of the user-assigned MSI to use, if any.
@@ -38284,24 +40040,11 @@ Leave blank if msi_client_id or msi_object_id specified.
Properties:
- Config: msi_mi_res_id
-- Env Var: RCLONE_AZUREBLOB_MSI_MI_RES_ID
+- Env Var: RCLONE_AZUREFILES_MSI_MI_RES_ID
- Type: string
- Required: false
-#### --azureblob-use-emulator
-
-Uses local storage emulator if provided as 'true'.
-
-Leave blank if using real azure storage endpoint.
-
-Properties:
-
-- Config: use_emulator
-- Env Var: RCLONE_AZUREBLOB_USE_EMULATOR
-- Type: bool
-- Default: false
-
-#### --azureblob-endpoint
+#### --azurefiles-endpoint
Endpoint for the service.
@@ -38310,37 +40053,26 @@ Leave blank normally.
Properties:
- Config: endpoint
-- Env Var: RCLONE_AZUREBLOB_ENDPOINT
-- Type: string
-- Required: false
-
-#### --azureblob-upload-cutoff
-
-Cutoff for switching to chunked upload (<= 256 MiB) (deprecated).
-
-Properties:
-
-- Config: upload_cutoff
-- Env Var: RCLONE_AZUREBLOB_UPLOAD_CUTOFF
+- Env Var: RCLONE_AZUREFILES_ENDPOINT
- Type: string
- Required: false
-#### --azureblob-chunk-size
+#### --azurefiles-chunk-size
Upload chunk size.
Note that this is stored in memory and there may be up to
-"--transfers" * "--azureblob-upload-concurrency" chunks stored at once
+"--transfers" * "--azurefile-upload-concurrency" chunks stored at once
in memory.
Properties:
- Config: chunk_size
-- Env Var: RCLONE_AZUREBLOB_CHUNK_SIZE
+- Env Var: RCLONE_AZUREFILES_CHUNK_SIZE
- Type: SizeSuffix
- Default: 4Mi
-#### --azureblob-upload-concurrency
+#### --azurefiles-upload-concurrency
Concurrency for multipart uploads.
@@ -38351,125 +40083,41 @@ If you are uploading small numbers of large files over high-speed
links and these uploads do not fully utilize your bandwidth, then
increasing this may help to speed up the transfers.
-In tests, upload speed increases almost linearly with upload
-concurrency. For example to fill a gigabit pipe it may be necessary to
-raise this to 64. Note that this will use more memory.
-
Note that chunks are stored in memory and there may be up to
-"--transfers" * "--azureblob-upload-concurrency" chunks stored at once
+"--transfers" * "--azurefile-upload-concurrency" chunks stored at once
in memory.
Properties:
- Config: upload_concurrency
-- Env Var: RCLONE_AZUREBLOB_UPLOAD_CONCURRENCY
+- Env Var: RCLONE_AZUREFILES_UPLOAD_CONCURRENCY
- Type: int
- Default: 16
-#### --azureblob-list-chunk
-
-Size of blob list.
-
-This sets the number of blobs requested in each listing chunk. Default
-is the maximum, 5000. "List blobs" requests are permitted 2 minutes
-per megabyte to complete. If an operation is taking longer than 2
-minutes per megabyte on average, it will time out (
-[source](https://docs.microsoft.com/en-us/rest/api/storageservices/setting-timeouts-for-blob-service-operations#exceptions-to-default-timeout-interval)
-). This can be used to limit the number of blobs items to return, to
-avoid the time out.
-
-Properties:
-
-- Config: list_chunk
-- Env Var: RCLONE_AZUREBLOB_LIST_CHUNK
-- Type: int
-- Default: 5000
-
-#### --azureblob-access-tier
-
-Access tier of blob: hot, cool or archive.
-
-Archived blobs can be restored by setting access tier to hot or
-cool. Leave blank if you intend to use default access tier, which is
-set at account level
-
-If there is no "access tier" specified, rclone doesn't apply any tier.
-rclone performs "Set Tier" operation on blobs while uploading, if objects
-are not modified, specifying "access tier" to new one will have no effect.
-If blobs are in "archive tier" at remote, trying to perform data transfer
-operations from remote will not be allowed. User should first restore by
-tiering blob to "Hot" or "Cool".
-
-Properties:
-
-- Config: access_tier
-- Env Var: RCLONE_AZUREBLOB_ACCESS_TIER
-- Type: string
-- Required: false
-
-#### --azureblob-archive-tier-delete
-
-Delete archive tier blobs before overwriting.
-
-Archive tier blobs cannot be updated. So without this flag, if you
-attempt to update an archive tier blob, then rclone will produce the
-error:
-
- can't update archive tier blob without --azureblob-archive-tier-delete
-
-With this flag set then before rclone attempts to overwrite an archive
-tier blob, it will delete the existing blob before uploading its
-replacement. This has the potential for data loss if the upload fails
-(unlike updating a normal blob) and also may cost more since deleting
-archive tier blobs early may be chargable.
+#### --azurefiles-max-stream-size
+Max size for streamed files.
-Properties:
+Azure files needs to know in advance how big the file will be. When
+rclone doesn't know it uses this value instead.
-- Config: archive_tier_delete
-- Env Var: RCLONE_AZUREBLOB_ARCHIVE_TIER_DELETE
-- Type: bool
-- Default: false
+This will be used when rclone is streaming data, the most common uses are:
-#### --azureblob-disable-checksum
+- Uploading files with `--vfs-cache-mode off` with `rclone mount`
+- Using `rclone rcat`
+- Copying files with unknown length
-Don't store MD5 checksum with object metadata.
+You will need this much free space in the share as the file will be this size temporarily.
-Normally rclone will calculate the MD5 checksum of the input before
-uploading it so it can add it to metadata on the object. This is great
-for data integrity checking but can cause long delays for large files
-to start uploading.
Properties:
-- Config: disable_checksum
-- Env Var: RCLONE_AZUREBLOB_DISABLE_CHECKSUM
-- Type: bool
-- Default: false
-
-#### --azureblob-memory-pool-flush-time
-
-How often internal memory buffer pools will be flushed. (no longer used)
-
-Properties:
-
-- Config: memory_pool_flush_time
-- Env Var: RCLONE_AZUREBLOB_MEMORY_POOL_FLUSH_TIME
-- Type: Duration
-- Default: 1m0s
-
-#### --azureblob-memory-pool-use-mmap
-
-Whether to use mmap buffers in internal memory pool. (no longer used)
-
-Properties:
-
-- Config: memory_pool_use_mmap
-- Env Var: RCLONE_AZUREBLOB_MEMORY_POOL_USE_MMAP
-- Type: bool
-- Default: false
+- Config: max_stream_size
+- Env Var: RCLONE_AZUREFILES_MAX_STREAM_SIZE
+- Type: SizeSuffix
+- Default: 10Gi
-#### --azureblob-encoding
+#### --azurefiles-encoding
The encoding for the backend.
@@ -38478,72 +40126,9 @@ See the [encoding section in the overview](https://rclone.org/overview/#encoding
Properties:
- Config: encoding
-- Env Var: RCLONE_AZUREBLOB_ENCODING
-- Type: MultiEncoder
-- Default: Slash,BackSlash,Del,Ctl,RightPeriod,InvalidUtf8
-
-#### --azureblob-public-access
-
-Public access level of a container: blob or container.
-
-Properties:
-
-- Config: public_access
-- Env Var: RCLONE_AZUREBLOB_PUBLIC_ACCESS
-- Type: string
-- Required: false
-- Examples:
- - ""
- - The container and its blobs can be accessed only with an authorized request.
- - It's a default value.
- - "blob"
- - Blob data within this container can be read via anonymous request.
- - "container"
- - Allow full public read access for container and blob data.
-
-#### --azureblob-directory-markers
-
-Upload an empty object with a trailing slash when a new directory is created
-
-Empty folders are unsupported for bucket based remotes, this option
-creates an empty object ending with "/", to persist the folder.
-
-This object also has the metadata "hdi_isfolder = true" to conform to
-the Microsoft standard.
-
-
-Properties:
-
-- Config: directory_markers
-- Env Var: RCLONE_AZUREBLOB_DIRECTORY_MARKERS
-- Type: bool
-- Default: false
-
-#### --azureblob-no-check-container
-
-If set, don't attempt to check the container exists or create it.
-
-This can be useful when trying to minimise the number of transactions
-rclone does if you know the container exists already.
-
-
-Properties:
-
-- Config: no_check_container
-- Env Var: RCLONE_AZUREBLOB_NO_CHECK_CONTAINER
-- Type: bool
-- Default: false
-
-#### --azureblob-no-head-object
-
-If set, do not do HEAD before GET when getting objects.
-
-Properties:
-
-- Config: no_head_object
-- Env Var: RCLONE_AZUREBLOB_NO_HEAD_OBJECT
-- Type: bool
-- Default: false
+- Env Var: RCLONE_AZUREFILES_ENCODING
+- Type: Encoding
+- Default: Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Del,Ctl,RightPeriod,InvalidUtf8,Dot
@@ -38564,30 +40149,6 @@ Eg `--header-upload "Content-Type: text/potato"`
MD5 sums are only uploaded with chunked files if the source has an MD5
sum. This will always be the case for a local to azure copy.
-`rclone about` is not supported by the Microsoft Azure Blob storage backend. Backends without
-this capability cannot determine free space for an rclone mount or
-use policy `mfs` (most free space) as a member of an rclone union
-remote.
-
-See [List of backends that do not support rclone about](https://rclone.org/overview/#optional-features) and [rclone about](https://rclone.org/commands/rclone_about/)
-
-## Azure Storage Emulator Support
-
-You can run rclone with the storage emulator (usually _azurite_).
-
-To do this, just set up a new remote with `rclone config` following
-the instructions in the introduction and set `use_emulator` in the
-advanced settings as `true`. You do not need to provide a default
-account name nor an account key. But you can override them in the
-`account` and `key` options. (Prior to v1.61 they were hard coded to
-_azurite_'s `devstoreaccount1`.)
-
-Also, if you want to access a storage emulator instance running on a
-different machine, you can override the `endpoint` parameter in the
-advanced settings, setting it to
-`http(s)://:/devstoreaccount1`
-(e.g. `http://10.254.2.5:10000/devstoreaccount1`).
-
# Microsoft OneDrive
Paths are specified as `remote:path`
@@ -38746,7 +40307,7 @@ You may try to [verify you account](https://docs.microsoft.com/en-us/azure/activ
Note: If you have a special region, you may need a different host in step 4 and 5. Here are [some hints](https://github.com/rclone/rclone/blob/bc23bf11db1c78c6ebbf8ea538fbebf7058b4176/backend/onedrive/onedrive.go#L86).
-### Modification time and hashes
+### Modification times and hashes
OneDrive allows modification times to be set on objects accurate to 1
second. These will be used to detect whether objects need syncing or
@@ -38767,6 +40328,32 @@ your workflow.
For all types of OneDrive you can use the `--checksum` flag.
+### --fast-list
+
+This remote supports `--fast-list` which allows you to use fewer
+transactions in exchange for more memory. See the [rclone
+docs](https://rclone.org/docs/#fast-list) for more details.
+
+This must be enabled with the `--onedrive-delta` flag (or `delta =
+true` in the config file) as it can cause performance degradation.
+
+It does this by using the delta listing facilities of OneDrive which
+returns all the files in the remote very efficiently. This is much
+more efficient than listing directories recursively and is Microsoft's
+recommended way of reading all the file information from a drive.
+
+This can be useful with `rclone mount` and [rclone rc vfs/refresh
+recursive=true](https://rclone.org/rc/#vfs-refresh)) to very quickly fill the mount with
+information about all the files.
+
+The API used for the recursive listing (`ListR`) only supports listing
+from the root of the drive. This will become increasingly inefficient
+the further away you get from the root as rclone will have to discard
+files outside of the directory you are using.
+
+Some commands (like `rclone lsf -R`) will use `ListR` by default - you
+can turn this off with `--disable ListR` if you need to.
+
### Restricted filename characters
In addition to the [default restricted characters set](https://rclone.org/overview/#restricted-characters)
@@ -39178,6 +40765,43 @@ Properties:
- Type: bool
- Default: false
+#### --onedrive-delta
+
+If set rclone will use delta listing to implement recursive listings.
+
+If this flag is set the the onedrive backend will advertise `ListR`
+support for recursive listings.
+
+Setting this flag speeds up these things greatly:
+
+ rclone lsf -R onedrive:
+ rclone size onedrive:
+ rclone rc vfs/refresh recursive=true
+
+**However** the delta listing API **only** works at the root of the
+drive. If you use it not at the root then it recurses from the root
+and discards all the data that is not under the directory you asked
+for. So it will be correct but may not be very efficient.
+
+This is why this flag is not set as the default.
+
+As a rule of thumb if nearly all of your data is under rclone's root
+directory (the `root/directory` in `onedrive:root/directory`) then
+using this flag will be be a big performance win. If your data is
+mostly not under the root then using this flag will be a big
+performance loss.
+
+It is recommended if you are mounting your onedrive at the root
+(or near the root when using crypt) and using rclone `rc vfs/refresh`.
+
+
+Properties:
+
+- Config: delta
+- Env Var: RCLONE_ONEDRIVE_DELTA
+- Type: bool
+- Default: false
+
#### --onedrive-encoding
The encoding for the backend.
@@ -39188,7 +40812,7 @@ Properties:
- Config: encoding
- Env Var: RCLONE_ONEDRIVE_ENCODING
-- Type: MultiEncoder
+- Type: Encoding
- Default: Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Del,Ctl,LeftSpace,LeftTilde,RightSpace,RightPeriod,InvalidUtf8,Dot
@@ -39482,12 +41106,14 @@ To copy a local directory to an OpenDrive directory called backup
rclone copy /home/source remote:backup
-### Modified time and MD5SUMs
+### Modification times and hashes
OpenDrive allows modification times to be set on objects accurate to 1
second. These will be used to detect whether objects need syncing or
not.
+The MD5 hash algorithm is supported.
+
### Restricted filename characters
| Character | Value | Replacement |
@@ -39561,7 +41187,7 @@ Properties:
- Config: encoding
- Env Var: RCLONE_OPENDRIVE_ENCODING
-- Type: MultiEncoder
+- Type: Encoding
- Default: Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,LeftSpace,LeftCrLfHtVt,RightSpace,RightCrLfHtVt,InvalidUtf8,Dot
#### --opendrive-chunk-size
@@ -39747,6 +41373,7 @@ Rclone supports the following OCI authentication provider.
No authentication
### User Principal
+
Sample rclone config file for Authentication Provider User Principal:
[oos]
@@ -39767,6 +41394,7 @@ Considerations:
- If the user is deleted, the config file will no longer work and may cause automation regressions that use the user's credentials.
### Instance Principal
+
An OCI compute instance can be authorized to use rclone by using it's identity and certificates as an instance principal.
With this approach no credentials have to be stored and managed.
@@ -39796,6 +41424,7 @@ Considerations:
- It is applicable for oci compute instances only. It cannot be used on external instance or resources.
### Resource Principal
+
Resource principal auth is very similar to instance principal auth but used for resources that are not
compute instances such as [serverless functions](https://docs.oracle.com/en-us/iaas/Content/Functions/Concepts/functionsoverview.htm).
To use resource principal ensure Rclone process is started with these environment variables set in its process.
@@ -39815,6 +41444,7 @@ Sample rclone configuration file for Authentication Provider Resource Principal:
provider = resource_principal_auth
### No authentication
+
Public buckets do not require any authentication mechanism to read objects.
Sample rclone configuration file for No authentication:
@@ -39825,10 +41455,9 @@ Sample rclone configuration file for No authentication:
region = us-ashburn-1
provider = no_auth
-## Options
-### Modified time
+### Modification times and hashes
-The modified time is stored as metadata on the object as
+The modification time is stored as metadata on the object as
`opc-meta-mtime` as floating point since the epoch, accurate to 1 ns.
If the modification time needs to be updated rclone will attempt to perform a server
@@ -39838,6 +41467,8 @@ In the case the object is larger than 5Gb, the object will be uploaded rather th
Note that reading this from the object takes an additional `HEAD` request as the metadata
isn't returned in object listings.
+The MD5 hash algorithm is supported.
+
### Multipart uploads
rclone supports multipart uploads with OOS which means that it can
@@ -40140,7 +41771,7 @@ Properties:
- Config: encoding
- Env Var: RCLONE_OOS_ENCODING
-- Type: MultiEncoder
+- Type: Encoding
- Default: Slash,InvalidUtf8,Dot
#### --oos-leave-parts-on-error
@@ -40667,7 +42298,7 @@ Properties:
- Config: encoding
- Env Var: RCLONE_QINGSTOR_ENCODING
-- Type: MultiEncoder
+- Type: Encoding
- Default: Slash,Ctl,InvalidUtf8
@@ -40798,7 +42429,7 @@ d) Delete this remote
y/e/d> y
```
-### Modified time and hashes
+### Modification times and hashes
Quatrix allows modification times to be set on objects accurate to 1 microsecond.
These will be used to detect whether objects need syncing or not.
@@ -40866,7 +42497,7 @@ Properties:
- Config: encoding
- Env Var: RCLONE_QUATRIX_ENCODING
-- Type: MultiEncoder
+- Type: Encoding
- Default: Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot
#### --quatrix-effective-upload-time
@@ -41111,7 +42742,7 @@ Properties:
- Config: encoding
- Env Var: RCLONE_SIA_ENCODING
-- Type: MultiEncoder
+- Type: Encoding
- Default: Slash,Question,Hash,Percent,Del,Ctl,InvalidUtf8,Dot
@@ -41350,7 +42981,7 @@ sufficient to determine if it is "dirty". By using `--update` along with
`--use-server-modtime`, you can avoid the extra API call and simply upload
files whose local modtime is newer than the time it was last uploaded.
-### Modified time
+### Modification times and hashes
The modified time is stored as metadata on the object as
`X-Object-Meta-Mtime` as floating point since the epoch accurate to 1
@@ -41359,6 +42990,8 @@ ns.
This is a de facto standard (used in the official python-swiftclient
amongst others) for storing the modification time for an object.
+The MD5 hash algorithm is supported.
+
### Restricted filename characters
| Character | Value | Replacement |
@@ -41705,7 +43338,7 @@ Properties:
- Config: encoding
- Env Var: RCLONE_SWIFT_ENCODING
-- Type: MultiEncoder
+- Type: Encoding
- Default: Slash,InvalidUtf8
@@ -41833,7 +43466,7 @@ To copy a local directory to a pCloud directory called backup
rclone copy /home/source remote:backup
-### Modified time and hashes ###
+### Modification times and hashes
pCloud allows modification times to be set on objects accurate to 1
second. These will be used to detect whether objects need syncing or
@@ -41972,7 +43605,7 @@ Properties:
- Config: encoding
- Env Var: RCLONE_PCLOUD_ENCODING
-- Type: MultiEncoder
+- Type: Encoding
- Default: Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot
#### --pcloud-root-folder-id
@@ -42104,6 +43737,13 @@ d) Delete this remote
y/e/d> y
```
+### Modification times and hashes
+
+PikPak keeps modification times on objects, and updates them when uploading objects,
+but it does not support changing only the modification time
+
+The MD5 hash algorithm is supported.
+
### Standard options
@@ -42263,7 +43903,7 @@ Properties:
- Config: encoding
- Env Var: RCLONE_PIKPAK_ENCODING
-- Type: MultiEncoder
+- Type: Encoding
- Default: Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,LeftSpace,RightSpace,RightPeriod,InvalidUtf8,Dot
## Backend commands
@@ -42327,15 +43967,16 @@ Result:
-## Limitations ##
+## Limitations
-### Hashes ###
+### Hashes may be empty
PikPak supports MD5 hash, but sometimes given empty especially for user-uploaded files.
-### Deleted files ###
+### Deleted files still visible with trashed-only
-Deleted files will still be visible with `--pikpak-trashed-only` even after the trash emptied. This goes away after few days.
+Deleted files will still be visible with `--pikpak-trashed-only` even after the
+trash emptied. This goes away after few days.
# premiumize.me
@@ -42417,7 +44058,7 @@ To copy a local directory to an premiumize.me directory called backup
rclone copy /home/source remote:backup
-### Modified time and hashes
+### Modification times and hashes
premiumize.me does not support modification times or hashes, therefore
syncing will default to `--size-only` checking. Note that using
@@ -42532,7 +44173,7 @@ Properties:
- Config: encoding
- Env Var: RCLONE_PREMIUMIZEME_ENCODING
-- Type: MultiEncoder
+- Type: Encoding
- Default: Slash,DoubleQuote,BackSlash,Del,Ctl,InvalidUtf8,Dot
@@ -42638,10 +44279,12 @@ To copy a local directory to an Proton Drive directory called backup
rclone copy /home/source remote:backup
-### Modified time
+### Modification times and hashes
Proton Drive Bridge does not support updating modification times yet.
+The SHA1 hash algorithm is supported.
+
### Restricted filename characters
Invalid UTF-8 bytes will be [replaced](https://rclone.org/overview/#invalid-utf8), also left and
@@ -42787,7 +44430,7 @@ Properties:
- Config: encoding
- Env Var: RCLONE_PROTONDRIVE_ENCODING
-- Type: MultiEncoder
+- Type: Encoding
- Default: Slash,LeftSpace,RightSpace,InvalidUtf8,Dot
#### --protondrive-original-file-size
@@ -43091,7 +44734,7 @@ Properties:
- Config: encoding
- Env Var: RCLONE_PUTIO_ENCODING
-- Type: MultiEncoder
+- Type: Encoding
- Default: Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot
@@ -43195,10 +44838,12 @@ To copy a local directory to an Proton Drive directory called backup
rclone copy /home/source remote:backup
-### Modified time
+### Modification times and hashes
Proton Drive Bridge does not support updating modification times yet.
+The SHA1 hash algorithm is supported.
+
### Restricted filename characters
Invalid UTF-8 bytes will be [replaced](https://rclone.org/overview/#invalid-utf8), also left and
@@ -43344,7 +44989,7 @@ Properties:
- Config: encoding
- Env Var: RCLONE_PROTONDRIVE_ENCODING
-- Type: MultiEncoder
+- Type: Encoding
- Default: Slash,LeftSpace,RightSpace,InvalidUtf8,Dot
#### --protondrive-original-file-size
@@ -43838,7 +45483,7 @@ Properties:
- Config: encoding
- Env Var: RCLONE_SEAFILE_ENCODING
-- Type: MultiEncoder
+- Type: Encoding
- Default: Slash,DoubleQuote,BackSlash,Ctl,InvalidUtf8
@@ -44198,7 +45843,7 @@ commands is prohibited. Set the configuration option `disable_hashcheck`
to `true` to disable checksumming entirely, or set `shell_type` to `none`
to disable all functionality based on remote shell command execution.
-### Modified time
+### Modification times and hashes
Modified times are stored on the server to 1 second precision.
@@ -44855,6 +46500,32 @@ Properties:
- Type: string
- Required: false
+#### --sftp-copy-is-hardlink
+
+Set to enable server side copies using hardlinks.
+
+The SFTP protocol does not define a copy command so normally server
+side copies are not allowed with the sftp backend.
+
+However the SFTP protocol does support hardlinking, and if you enable
+this flag then the sftp backend will support server side copies. These
+will be implemented by doing a hardlink from the source to the
+destination.
+
+Not all sftp servers support this.
+
+Note that hardlinking two files together will use no additional space
+as the source and the destination will be the same file.
+
+This feature may be useful backups made with --copy-dest.
+
+Properties:
+
+- Config: copy_is_hardlink
+- Env Var: RCLONE_SFTP_COPY_IS_HARDLINK
+- Type: bool
+- Default: false
+
## Limitations
@@ -45133,7 +46804,7 @@ Properties:
- Config: encoding
- Env Var: RCLONE_SMB_ENCODING
-- Type: MultiEncoder
+- Type: Encoding
- Default: Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot
@@ -45652,7 +47323,7 @@ Paths may be as deep as required, e.g. `remote:directory/subdirectory`.
create a folder, which rclone will create as a "Sync Folder" with
SugarSync.
-### Modified time and hashes
+### Modification times and hashes
SugarSync does not support modification times or hashes, therefore
syncing will default to `--size-only` checking. Note that using
@@ -45823,7 +47494,7 @@ Properties:
- Config: encoding
- Env Var: RCLONE_SUGARSYNC_ENCODING
-- Type: MultiEncoder
+- Type: Encoding
- Default: Slash,Ctl,InvalidUtf8,Dot
@@ -45920,7 +47591,7 @@ To copy a local directory to an Uptobox directory called backup
rclone copy /home/source remote:backup
-### Modified time and hashes
+### Modification times and hashes
Uptobox supports neither modified times nor checksums. All timestamps
will read as that set by `--default-time`.
@@ -45981,7 +47652,7 @@ Properties:
- Config: encoding
- Env Var: RCLONE_UPTOBOX_ENCODING
-- Type: MultiEncoder
+- Type: Encoding
- Default: Slash,LtGt,DoubleQuote,BackQuote,Del,Ctl,LeftSpace,InvalidUtf8,Dot
@@ -46001,7 +47672,7 @@ During the initial setup with `rclone config` you will specify the upstream
remotes as a space separated list. The upstream remotes can either be a local
paths or other remotes.
-The attributes `:ro`, `:nc` and `:nc` can be attached to the end of the remote
+The attributes `:ro`, `:nc` and `:writeback` can be attached to the end of the remote
to tag the remote as **read only**, **no create** or **writeback**, e.g.
`remote:directory/subdirectory:ro` or `remote:directory/subdirectory:nc`.
@@ -46333,7 +48004,9 @@ Choose a number from below, or type in your own value
\ (sharepoint)
5 / Sharepoint with NTLM authentication, usually self-hosted or on-premises
\ (sharepoint-ntlm)
- 6 / Other site/service or software
+ 6 / rclone WebDAV server to serve a remote over HTTP via the WebDAV protocol
+ \ (rclone)
+ 7 / Other site/service or software
\ (other)
vendor> 2
User name
@@ -46379,7 +48052,7 @@ To copy a local directory to an WebDAV directory called backup
rclone copy /home/source remote:backup
-### Modified time and hashes ###
+### Modification times and hashes
Plain WebDAV does not support modified times. However when used with
Fastmail Files, Owncloud or Nextcloud rclone will support modified times.
@@ -46429,6 +48102,8 @@ Properties:
- Sharepoint Online, authenticated by Microsoft account
- "sharepoint-ntlm"
- Sharepoint with NTLM authentication, usually self-hosted or on-premises
+ - "rclone"
+ - rclone WebDAV server to serve a remote over HTTP via the WebDAV protocol
- "other"
- Other site/service or software
@@ -46659,6 +48334,14 @@ For Rclone calls copying files (especially Office files such as .docx, .xlsx, et
--ignore-size --ignore-checksum --update
```
+## Rclone
+
+Use this option if you are hosting remotes over WebDAV provided by rclone.
+Read [rclone serve webdav](commands/rclone_serve_webdav/) for more details.
+
+rclone serve supports modified times using the `X-OC-Mtime` header.
+
+
### dCache
dCache is a storage system that supports many protocols and
@@ -46819,14 +48502,12 @@ excess files in the path.
Yandex paths may be as deep as required, e.g. `remote:directory/subdirectory`.
-### Modified time
+### Modification times and hashes
Modified times are supported and are stored accurate to 1 ns in custom
metadata called `rclone_modified` in RFC3339 with nanoseconds format.
-### MD5 checksums
-
-MD5 checksums are natively supported by Yandex Disk.
+The MD5 hash algorithm is natively supported by Yandex Disk.
### Emptying Trash
@@ -46940,7 +48621,7 @@ Properties:
- Config: encoding
- Env Var: RCLONE_YANDEX_ENCODING
-- Type: MultiEncoder
+- Type: Encoding
- Default: Slash,Del,Ctl,InvalidUtf8,Dot
@@ -47067,13 +48748,11 @@ excess files in the path.
Zoho paths may be as deep as required, eg `remote:directory/subdirectory`.
-### Modified time
+### Modification times and hashes
Modified times are currently not supported for Zoho Workdrive
-### Checksums
-
-No checksums are supported.
+No hash algorithms are supported.
### Usage information
@@ -47196,7 +48875,7 @@ Properties:
- Config: encoding
- Env Var: RCLONE_ZOHO_ENCODING
-- Type: MultiEncoder
+- Type: Encoding
- Default: Del,Ctl,InvalidUtf8
@@ -47228,10 +48907,10 @@ For consistencies sake one can also configure a remote of type
rclone remote paths, e.g. `remote:path/to/wherever`, but it is probably
easier not to.
-### Modified time ###
+### Modification times
-Rclone reads and writes the modified time using an accuracy determined by
-the OS. Typically this is 1ns on Linux, 10 ns on Windows and 1 Second
+Rclone reads and writes the modification times using an accuracy determined
+by the OS. Typically this is 1ns on Linux, 10 ns on Windows and 1 Second
on OS X.
### Filenames ###
@@ -47660,6 +49339,11 @@ time we:
- Only checksum the size that stat gave
- Don't update the stat info for the file
+**NB** do not use this flag on a Windows Volume Shadow (VSS). For some
+unknown reason, files in a VSS sometimes show different sizes from the
+directory listing (where the initial stat value comes from on Windows)
+and when stat is called on them directly. Other copy tools always use
+the direct stat value and setting this flag will disable that.
Properties:
@@ -47770,7 +49454,7 @@ Properties:
- Config: encoding
- Env Var: RCLONE_LOCAL_ENCODING
-- Type: MultiEncoder
+- Type: Encoding
- Default: Slash,Dot
### Metadata
@@ -47832,6 +49516,153 @@ Options:
# Changelog
+## v1.65.0 - 2023-11-26
+
+[See commits](https://github.com/rclone/rclone/compare/v1.64.0...v1.65.0)
+
+* New backends
+ * Azure Files (karan, moongdal, Nick Craig-Wood)
+ * ImageKit (Abhinav Dhiman)
+ * Linkbox (viktor, Nick Craig-Wood)
+* New commands
+ * `serve s3`: Let rclone act as an S3 compatible server (Mikubill, Artur Neumann, Saw-jan, Nick Craig-Wood)
+ * `nfsmount`: mount command to provide mount mechanism on macOS without FUSE (Saleh Dindar)
+ * `serve nfs`: to serve a remote for use by `nfsmount` (Saleh Dindar)
+* New Features
+ * install.sh: Clean up temp files in install script (Jacob Hands)
+ * build
+ * Update all dependencies (Nick Craig-Wood)
+ * Refactor version info and icon resource handling on windows (albertony)
+ * doc updates (albertony, alfish2000, asdffdsazqqq, Dimitri Papadopoulos, Herby Gillot, Joda Stößer, Manoj Ghosh, Nick Craig-Wood)
+ * Implement `--metadata-mapper` to transform metatadata with a user supplied program (Nick Craig-Wood)
+ * Add `ChunkWriterDoesntSeek` feature flag and set it for b2 (Nick Craig-Wood)
+ * lib/http: Export basic go string functions for use in `--template` (Gabriel Espinoza)
+ * makefile: Use POSIX compatible install arguments (Mina Galić)
+ * operations
+ * Use less memory when doing multithread uploads (Nick Craig-Wood)
+ * Implement `--partial-suffix` to control extension of temporary file names (Volodymyr)
+ * rc
+ * Add `operations/check` to the rc API (Nick Craig-Wood)
+ * Always report an error as JSON (Nick Craig-Wood)
+ * Set `Last-Modified` header for files served by `--rc-serve` (Nikita Shoshin)
+ * size: Dont show duplicate object count when less than 1k (albertony)
+* Bug Fixes
+ * fshttp: Fix `--contimeout` being ignored (ä½ çŸ¥é“未æ¥å—)
+ * march: Fix excessive parallelism when using `--no-traverse` (Nick Craig-Wood)
+ * ncdu: Fix crash when re-entering changed directory after rescan (Nick Craig-Wood)
+ * operations
+ * Fix overwrite of destination when multi-thread transfer fails (Nick Craig-Wood)
+ * Fix invalid UTF-8 when truncating file names when not using `--inplace` (Nick Craig-Wood)
+ * serve dnla: Fix crash on graceful exit (wuxingzhong)
+* Mount
+ * Disable mount for freebsd and alias cmount as mount on that platform (Nick Craig-Wood)
+* VFS
+ * Add `--vfs-refresh` flag to read all the directories on start (Beyond Meat)
+ * Implement Name() method in WriteFileHandle and ReadFileHandle (Saleh Dindar)
+ * Add go-billy dependency and make sure vfs.Handle implements billy.File (Saleh Dindar)
+ * Error out early if can't upload 0 length file (Nick Craig-Wood)
+* Local
+ * Fix copying from Windows Volume Shadows (Nick Craig-Wood)
+* Azure Blob
+ * Add support for cold tier (Ivan Yanitra)
+* B2
+ * Implement "rclone backend lifecycle" to read and set bucket lifecycles (Nick Craig-Wood)
+ * Implement `--b2-lifecycle` to control lifecycle when creating buckets (Nick Craig-Wood)
+ * Fix listing all buckets when not needed (Nick Craig-Wood)
+ * Fix multi-thread upload with copyto going to wrong name (Nick Craig-Wood)
+ * Fix server side chunked copy when file size was exactly `--b2-copy-cutoff` (Nick Craig-Wood)
+ * Fix streaming chunked files an exact multiple of chunk size (Nick Craig-Wood)
+* Box
+ * Filter more EventIDs when polling (David Sze)
+ * Add more logging for polling (David Sze)
+ * Fix performance problem reading metadata for single files (Nick Craig-Wood)
+* Drive
+ * Add read/write metadata support (Nick Craig-Wood)
+ * Add support for SHA-1 and SHA-256 checksums (rinsuki)
+ * Add `--drive-show-all-gdocs` to allow unexportable gdocs to be server side copied (Nick Craig-Wood)
+ * Add a note that `--drive-scope` accepts comma-separated list of scopes (Keigo Imai)
+ * Fix error updating created time metadata on existing object (Nick Craig-Wood)
+ * Fix integration tests by enabling metadata support from the context (Nick Craig-Wood)
+* Dropbox
+ * Factor batcher into lib/batcher (Nick Craig-Wood)
+ * Fix missing encoding for rclone purge (Nick Craig-Wood)
+* Google Cloud Storage
+ * Fix 400 Bad request errors when using multi-thread copy (Nick Craig-Wood)
+* Googlephotos
+ * Implement batcher for uploads (Nick Craig-Wood)
+* Hdfs
+ * Added support for list of namenodes in hdfs remote config (Tayo-pasedaRJ)
+* HTTP
+ * Implement set backend command to update running backend (Nick Craig-Wood)
+ * Enable methods used with WebDAV (Alen Å iljak)
+* Jottacloud
+ * Add support for reading and writing metadata (albertony)
+* Onedrive
+ * Implement ListR method which gives `--fast-list` support (Nick Craig-Wood)
+ * This must be enabled with the `--onedrive-delta` flag
+* Quatrix
+ * Add partial upload support (Oksana Zhykina)
+ * Overwrite files on conflict during server-side move (Oksana Zhykina)
+* S3
+ * Add Linode provider (Nick Craig-Wood)
+ * Add docs on how to add a new provider (Nick Craig-Wood)
+ * Fix no error being returned when creating a bucket we don't own (Nick Craig-Wood)
+ * Emit a debug message if anonymous credentials are in use (Nick Craig-Wood)
+ * Add `--s3-disable-multipart-uploads` flag (Nick Craig-Wood)
+ * Detect looping when using gcs and versions (Nick Craig-Wood)
+* SFTP
+ * Implement `--sftp-copy-is-hardlink` to server side copy as hardlink (Nick Craig-Wood)
+* Smb
+ * Fix incorrect `about` size by switching to `github.com/cloudsoda/go-smb2` fork (Nick Craig-Wood)
+ * Fix modtime of multithread uploads by setting PartialUploads (Nick Craig-Wood)
+* WebDAV
+ * Added an rclone vendor to work with `rclone serve webdav` (Adithya Kumar)
+
+## v1.64.2 - 2023-10-19
+
+[See commits](https://github.com/rclone/rclone/compare/v1.64.1...v1.64.2)
+
+* Bug Fixes
+ * selfupdate: Fix "invalid hashsum signature" error (Nick Craig-Wood)
+ * build: Fix docker build running out of space (Nick Craig-Wood)
+
+## v1.64.1 - 2023-10-17
+
+[See commits](https://github.com/rclone/rclone/compare/v1.64.0...v1.64.1)
+
+* Bug Fixes
+ * cmd: Make `--progress` output logs in the same format as without (Nick Craig-Wood)
+ * docs fixes (Dimitri Papadopoulos Orfanos, Herby Gillot, Manoj Ghosh, Nick Craig-Wood)
+ * lsjson: Make sure we set the global metadata flag too (Nick Craig-Wood)
+ * operations
+ * Ensure concurrency is no greater than the number of chunks (Pat Patterson)
+ * Fix OpenOptions ignored in copy if operation was a multiThreadCopy (Vitor Gomes)
+ * Fix error message on delete to have file name (Nick Craig-Wood)
+ * serve sftp: Return not supported error for not supported commands (Nick Craig-Wood)
+ * build: Upgrade golang.org/x/net to v0.17.0 to fix HTTP/2 rapid reset (Nick Craig-Wood)
+ * pacer: Fix b2 deadlock by defaulting max connections to unlimited (Nick Craig-Wood)
+* Mount
+ * Fix automount not detecting drive is ready (Nick Craig-Wood)
+* VFS
+ * Fix update dir modification time (Saleh Dindar)
+* Azure Blob
+ * Fix "fatal error: concurrent map writes" (Nick Craig-Wood)
+* B2
+ * Fix multipart upload: corrupted on transfer: sizes differ XXX vs 0 (Nick Craig-Wood)
+ * Fix locking window when getting mutipart upload URL (Nick Craig-Wood)
+ * Fix server side copies greater than 4GB (Nick Craig-Wood)
+ * Fix chunked streaming uploads (Nick Craig-Wood)
+ * Reduce default `--b2-upload-concurrency` to 4 to reduce memory usage (Nick Craig-Wood)
+* Onedrive
+ * Fix the configurator to allow `/teams/ID` in the config (Nick Craig-Wood)
+* Oracleobjectstorage
+ * Fix OpenOptions being ignored in uploadMultipart with chunkWriter (Nick Craig-Wood)
+* S3
+ * Fix slice bounds out of range error when listing (Nick Craig-Wood)
+ * Fix OpenOptions being ignored in uploadMultipart with chunkWriter (Vitor Gomes)
+* Storj
+ * Update storj.io/uplink to v1.12.0 (Kaloyan Raev)
+
## v1.64.0 - 2023-09-11
[See commits](https://github.com/rclone/rclone/compare/v1.63.0...v1.64.0)
@@ -47932,14 +49763,14 @@ Options:
* Fix 425 "TLS session of data connection not resumed" errors (Nick Craig-Wood)
* Hdfs
* Retry "replication in progress" errors when uploading (Nick Craig-Wood)
- * Fix uploading to the wrong object on Update with overriden remote name (Nick Craig-Wood)
+ * Fix uploading to the wrong object on Update with overridden remote name (Nick Craig-Wood)
* HTTP
* CORS should not be sent if not set (yuudi)
* Fix webdav OPTIONS response (yuudi)
* Opendrive
* Fix List on a just deleted and remade directory (Nick Craig-Wood)
* Oracleobjectstorage
- * Use rclone's rate limiter in mutipart transfers (Manoj Ghosh)
+ * Use rclone's rate limiter in multipart transfers (Manoj Ghosh)
* Implement `OpenChunkWriter` and multi-thread uploads (Manoj Ghosh)
* S3
* Refactor multipart upload to use `OpenChunkWriter` and `ChunkWriter` (Vitor Gomes)
@@ -48112,14 +49943,14 @@ Options:
* Fix quickxorhash on 32 bit architectures (Nick Craig-Wood)
* Report any list errors during `rclone cleanup` (albertony)
* Putio
- * Fix uploading to the wrong object on Update with overriden remote name (Nick Craig-Wood)
+ * Fix uploading to the wrong object on Update with overridden remote name (Nick Craig-Wood)
* Fix modification times not being preserved for server side copy and move (Nick Craig-Wood)
* Fix server side copy failures (400 errors) (Nick Craig-Wood)
* S3
* Empty directory markers (JÄnis BebrÄ«tis, Nick Craig-Wood)
* Update Scaleway storage classes (Brian Starkey)
* Fix `--s3-versions` on individual objects (Nick Craig-Wood)
- * Fix hang on aborting multpart upload with iDrive e2 (Nick Craig-Wood)
+ * Fix hang on aborting multipart upload with iDrive e2 (Nick Craig-Wood)
* Fix missing "tier" metadata (Nick Craig-Wood)
* Fix V3sign: add missing subresource delete (cc)
* Fix Arvancloud Domain and region changes and alphabetise the provider (Ehsan Tadayon)
@@ -48136,7 +49967,7 @@ Options:
* Code cleanup to avoid overwriting ctx before first use (fixes issue reported by the staticcheck linter) (albertony)
* Storj
* Fix "uplink: too many requests" errors when uploading to the same file (Nick Craig-Wood)
- * Fix uploading to the wrong object on Update with overriden remote name (Nick Craig-Wood)
+ * Fix uploading to the wrong object on Update with overridden remote name (Nick Craig-Wood)
* Swift
* Ignore 404 error when deleting an object (Nick Craig-Wood)
* Union
@@ -51765,7 +53596,7 @@ Point release to fix hubic and azureblob backends.
* Revert to copy when moving file across file system boundaries
* `--skip-links` to suppress symlink warnings (thanks Zhiming Wang)
* Mount
- * Re-use `rcat` internals to support uploads from all remotes
+ * Reuse `rcat` internals to support uploads from all remotes
* Dropbox
* Fix "entry doesn't belong in directory" error
* Stop using deprecated API methods
@@ -53437,7 +55268,7 @@ put them back in again.` >}}
* HNGamingUK
* Jonta <359397+Jonta@users.noreply.github.com>
* YenForYang
- * Joda Stößer
+ * SimJoSt / Joda Stößer
* Logeshwaran
* Rajat Goel
* r0kk3rz
@@ -53684,6 +55515,38 @@ put them back in again.` >}}
* Volodymyr Kit
* David Pedersen
* Drew Stinnett
+ * Pat Patterson
+ * Herby Gillot
+ * Nikita Shoshin
+ * rinsuki <428rinsuki+git@gmail.com>
+ * Beyond Meat <51850644+beyondmeat@users.noreply.github.com>
+ * Saleh Dindar
+ * Volodymyr <142890760+vkit-maytech@users.noreply.github.com>
+ * Gabriel Espinoza <31670639+gspinoza@users.noreply.github.com>
+ * Keigo Imai
+ * Ivan Yanitra
+ * alfish2000
+ * wuxingzhong
+ * Adithya Kumar
+ * Tayo-pasedaRJ <138471223+Tayo-pasedaRJ@users.noreply.github.com>
+ * Peter Kreuser
+ * Piyush
+ * fotile96
+ * Luc Ritchie
+ * cynful
+ * wjielai
+ * Jack Deng
+ * Mikubill <31246794+Mikubill@users.noreply.github.com>
+ * Artur Neumann
+ * Saw-jan
+ * Oksana Zhykina
+ * karan
+ * viktor
+ * moongdal
+ * Mina Galić
+ * Alen Å iljak
+ * ä½ çŸ¥é“未æ¥å—
+ * Abhinav Dhiman <8640877+ahnv@users.noreply.github.com>
# Contact the rclone project
diff --git a/MANUAL.txt b/MANUAL.txt
index 2d0f92187f59c..11a38590b18d9 100644
--- a/MANUAL.txt
+++ b/MANUAL.txt
@@ -1,6 +1,6 @@
rclone(1) User Manual
Nick Craig-Wood
-Sep 11, 2023
+Nov 26, 2023
Rclone syncs your files to cloud storage
@@ -124,11 +124,14 @@ S3, that work out of the box.)
- Koofr
- Leviia Object Storage
- Liara Object Storage
+- Linkbox
+- Linode Object Storage
- Mail.ru Cloud
- Memset Memstore
- Mega
- Memory
- Microsoft Azure Blob Storage
+- Microsoft Azure Files Storage
- Microsoft OneDrive
- Minio
- Nextcloud
@@ -265,6 +268,19 @@ developers so it may be out of date. Its current version is as below.
[Homebrew package]
+Installation with MacPorts (#macos-macports)
+
+On macOS, rclone can also be installed via MacPorts:
+
+ sudo port install rclone
+
+Note that this is a third party installer not controlled by the rclone
+developers so it may be out of date. Its current version is as below.
+
+[MacPorts port]
+
+More information here.
+
Precompiled binary, using curl
To avoid problems with macOS gatekeeper enforcing the binary to be
@@ -483,7 +499,7 @@ Make sure you have Snapd installed
$ sudo snap install rclone
-Due to the strict confinement of Snap, rclone snap cannot acess real
+Due to the strict confinement of Snap, rclone snap cannot access real
/home/$USER/.config/rclone directory, default config path is as below.
- Default config directory:
@@ -502,8 +518,8 @@ developers so it may be out of date. Its current version is as below.
Source installation
-Make sure you have git and Go installed. Go version 1.17 or newer is
-required, latest release is recommended. You can get it from your
+Make sure you have git and Go installed. Go version 1.18 or newer is
+required, the latest release is recommended. You can get it from your
package manager, or download it from golang.org/dl. Then you can run the
following:
@@ -531,22 +547,51 @@ cgo on Windows as well, by using the MinGW port of GCC, e.g. by
installing it in a MSYS2 distribution (make sure you install it in the
classic mingw64 subsystem, the ucrt64 version is not compatible).
-Additionally, on Windows, you must install the third party utility
-WinFsp, with the "Developer" feature selected. If building with cgo, you
-must also set environment variable CPATH pointing to the fuse include
-directory within the WinFsp installation (normally
+Additionally, to build with mount on Windows, you must install the third
+party utility WinFsp, with the "Developer" feature selected. If building
+with cgo, you must also set environment variable CPATH pointing to the
+fuse include directory within the WinFsp installation (normally
C:\Program Files (x86)\WinFsp\inc\fuse).
-You may also add arguments -ldflags -s (with or without -tags cmount),
-to omit symbol table and debug information, making the executable file
-smaller, and -trimpath to remove references to local file system paths.
-This is how the official rclone releases are built.
+You may add arguments -ldflags -s to omit symbol table and debug
+information, making the executable file smaller, and -trimpath to remove
+references to local file system paths. The official rclone releases are
+built with both of these.
go build -trimpath -ldflags -s -tags cmount
+If you want to customize the version string, as reported by the
+rclone version command, you can set one of the variables fs.Version,
+fs.VersionTag (to keep default suffix but customize the number), or
+fs.VersionSuffix (to keep default number but customize the suffix). This
+can be done from the build command, by adding to the -ldflags argument
+value as shown below.
+
+ go build -trimpath -ldflags "-s -X github.com/rclone/rclone/fs.Version=v9.9.9-test" -tags cmount
+
+On Windows, the official executables also have the version information,
+as well as a file icon, embedded as binary resources. To get that with
+your own build you need to run the following command before the build
+command. It generates a Windows resource system object file, with
+extension .syso, e.g. resource_windows_amd64.syso, that will be
+automatically picked up by future build commands.
+
+ go run bin/resource_windows.go
+
+The above command will generate a resource file containing version
+information based on the fs.Version variable in source at the time you
+run the command, which means if the value of this variable changes you
+need to re-run the command for it to be reflected in the version
+information. Also, if you override this version variable in the build
+command as described above, you need to do that also when generating the
+resource file, or else it will still use the value from the source.
+
+ go run bin/resource_windows.go -version v9.9.9-test
+
Instead of executing the go build command directly, you can run it via
-the Makefile. It changes the version number suffix from "-DEV" to
-"-beta" and appends commit details. It also copies the resulting rclone
+the Makefile. The default target changes the version suffix from "-DEV"
+to "-beta" followed by additional commit details, embeds version
+information binary resources on Windows, and copies the resulting rclone
executable into your GOPATH bin folder ($(go env GOPATH)/bin, which
corresponds to ~/go/bin/rclone by default).
@@ -557,27 +602,18 @@ To include mount command on macOS and Windows with Makefile build:
make GOTAGS=cmount
There are other make targets that can be used for more advanced builds,
-such as cross-compiling for all supported os/architectures, embedding
-icon and version info resources into windows executable, and packaging
-results into release artifacts. See Makefile and cross-compile.go for
-details.
-
-Another alternative is to download the source, build and install rclone
-in one operation, as a regular Go package. The source will be stored it
-in the Go module cache, and the resulting executable will be in your
-GOPATH bin folder ($(go env GOPATH)/bin, which corresponds to
-~/go/bin/rclone by default).
+such as cross-compiling for all supported os/architectures, and
+packaging results into release artifacts. See Makefile and
+cross-compile.go for details.
-With Go version 1.17 or newer:
+Another alternative method for source installation is to download the
+source, build and install rclone - all in one operation, as a regular Go
+package. The source will be stored it in the Go module cache, and the
+resulting executable will be in your GOPATH bin folder
+($(go env GOPATH)/bin, which corresponds to ~/go/bin/rclone by default).
go install github.com/rclone/rclone@latest
-With Go versions older than 1.17 (do not use the -u flag, it causes Go
-to try to update the dependencies that rclone uses and sometimes these
-don't work with the current version):
-
- go get github.com/rclone/rclone
-
Ansible installation
This can be done with Stefan Weichinger's ansible role.
@@ -739,7 +775,7 @@ also provides alternative standalone distributions which includes
necessary runtime (.NET 5). WinSW is a command-line only utility, where
you have to manually create an XML file with service configuration. This
may be a drawback for some, but it can also be an advantage as it is
-easy to back up and re-use the configuration settings, without having go
+easy to back up and reuse the configuration settings, without having go
through manual steps in a GUI. One thing to note is that by default it
does not restart the service on error, one have to explicit enable this
in the configuration file (via the "onfailure" parameter).
@@ -808,10 +844,12 @@ See the following for detailed instructions for
- Internet Archive
- Jottacloud
- Koofr
+- Linkbox
- Mail.ru Cloud
- Mega
- Memory
- Microsoft Azure Blob Storage
+- Microsoft Azure Files Storage
- Microsoft OneDrive
- OpenStack Swift / Rackspace Cloudfiles / Blomp Cloud Storage /
Memset Memstore
@@ -980,11 +1018,11 @@ Flags for anything which can Copy a file.
-c, --checksum Check for changes with size & checksum (if available, or fallback to size only).
--compare-dest stringArray Include additional comma separated server-side paths during comparison
--copy-dest stringArray Implies --compare-dest but also copies files from paths into destination
- --cutoff-mode string Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default "HARD")
+ --cutoff-mode HARD|SOFT|CAUTIOUS Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default HARD)
--ignore-case-sync Ignore case when synchronizing
--ignore-checksum Skip post copy check of checksums
--ignore-existing Skip all files that exist on destination
- --ignore-size Ignore size when skipping use mod-time or checksum
+ --ignore-size Ignore size when skipping use modtime or checksum
-I, --ignore-times Don't skip files that match size and time - transfer all files
--immutable Do not modify files, fail if existing files have been modified
--inplace Download directly to destination file instead of atomic download to temp/rename
@@ -999,11 +1037,12 @@ Flags for anything which can Copy a file.
--multi-thread-write-buffer-size SizeSuffix In memory buffer size for writing when in multi-thread mode (default 128Ki)
--no-check-dest Don't check the destination, copy regardless
--no-traverse Don't traverse destination file system on copy
- --no-update-modtime Don't update destination mod-time if files identical
+ --no-update-modtime Don't update destination modtime if files identical
--order-by string Instructions on how to order the transfers, e.g. 'size,descending'
+ --partial-suffix string Add partial-suffix to temporary file name when --inplace is not used (default ".partial")
--refresh-times Refresh the modtime of remote files
--server-side-across-configs Allow server-side operations (e.g. copy) to work across different configs
- --size-only Skip based on size only, not mod-time or checksum
+ --size-only Skip based on size only, not modtime or checksum
--streaming-upload-cutoff SizeSuffix Cutoff for switching to chunked upload if file size is unknown, upload starts after reaching cutoff or when file ends (default 100Ki)
-u, --update Skip files that are newer on the destination
@@ -1111,11 +1150,11 @@ Flags for anything which can Copy a file.
-c, --checksum Check for changes with size & checksum (if available, or fallback to size only).
--compare-dest stringArray Include additional comma separated server-side paths during comparison
--copy-dest stringArray Implies --compare-dest but also copies files from paths into destination
- --cutoff-mode string Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default "HARD")
+ --cutoff-mode HARD|SOFT|CAUTIOUS Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default HARD)
--ignore-case-sync Ignore case when synchronizing
--ignore-checksum Skip post copy check of checksums
--ignore-existing Skip all files that exist on destination
- --ignore-size Ignore size when skipping use mod-time or checksum
+ --ignore-size Ignore size when skipping use modtime or checksum
-I, --ignore-times Don't skip files that match size and time - transfer all files
--immutable Do not modify files, fail if existing files have been modified
--inplace Download directly to destination file instead of atomic download to temp/rename
@@ -1130,11 +1169,12 @@ Flags for anything which can Copy a file.
--multi-thread-write-buffer-size SizeSuffix In memory buffer size for writing when in multi-thread mode (default 128Ki)
--no-check-dest Don't check the destination, copy regardless
--no-traverse Don't traverse destination file system on copy
- --no-update-modtime Don't update destination mod-time if files identical
+ --no-update-modtime Don't update destination modtime if files identical
--order-by string Instructions on how to order the transfers, e.g. 'size,descending'
+ --partial-suffix string Add partial-suffix to temporary file name when --inplace is not used (default ".partial")
--refresh-times Refresh the modtime of remote files
--server-side-across-configs Allow server-side operations (e.g. copy) to work across different configs
- --size-only Skip based on size only, not mod-time or checksum
+ --size-only Skip based on size only, not modtime or checksum
--streaming-upload-cutoff SizeSuffix Cutoff for switching to chunked upload if file size is unknown, upload starts after reaching cutoff or when file ends (default 100Ki)
-u, --update Skip files that are newer on the destination
@@ -1250,11 +1290,11 @@ Flags for anything which can Copy a file.
-c, --checksum Check for changes with size & checksum (if available, or fallback to size only).
--compare-dest stringArray Include additional comma separated server-side paths during comparison
--copy-dest stringArray Implies --compare-dest but also copies files from paths into destination
- --cutoff-mode string Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default "HARD")
+ --cutoff-mode HARD|SOFT|CAUTIOUS Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default HARD)
--ignore-case-sync Ignore case when synchronizing
--ignore-checksum Skip post copy check of checksums
--ignore-existing Skip all files that exist on destination
- --ignore-size Ignore size when skipping use mod-time or checksum
+ --ignore-size Ignore size when skipping use modtime or checksum
-I, --ignore-times Don't skip files that match size and time - transfer all files
--immutable Do not modify files, fail if existing files have been modified
--inplace Download directly to destination file instead of atomic download to temp/rename
@@ -1269,11 +1309,12 @@ Flags for anything which can Copy a file.
--multi-thread-write-buffer-size SizeSuffix In memory buffer size for writing when in multi-thread mode (default 128Ki)
--no-check-dest Don't check the destination, copy regardless
--no-traverse Don't traverse destination file system on copy
- --no-update-modtime Don't update destination mod-time if files identical
+ --no-update-modtime Don't update destination modtime if files identical
--order-by string Instructions on how to order the transfers, e.g. 'size,descending'
+ --partial-suffix string Add partial-suffix to temporary file name when --inplace is not used (default ".partial")
--refresh-times Refresh the modtime of remote files
--server-side-across-configs Allow server-side operations (e.g. copy) to work across different configs
- --size-only Skip based on size only, not mod-time or checksum
+ --size-only Skip based on size only, not modtime or checksum
--streaming-upload-cutoff SizeSuffix Cutoff for switching to chunked upload if file size is unknown, upload starts after reaching cutoff or when file ends (default 100Ki)
-u, --update Skip files that are newer on the destination
@@ -2525,11 +2566,11 @@ Flags for anything which can Copy a file.
-c, --checksum Check for changes with size & checksum (if available, or fallback to size only).
--compare-dest stringArray Include additional comma separated server-side paths during comparison
--copy-dest stringArray Implies --compare-dest but also copies files from paths into destination
- --cutoff-mode string Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default "HARD")
+ --cutoff-mode HARD|SOFT|CAUTIOUS Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default HARD)
--ignore-case-sync Ignore case when synchronizing
--ignore-checksum Skip post copy check of checksums
--ignore-existing Skip all files that exist on destination
- --ignore-size Ignore size when skipping use mod-time or checksum
+ --ignore-size Ignore size when skipping use modtime or checksum
-I, --ignore-times Don't skip files that match size and time - transfer all files
--immutable Do not modify files, fail if existing files have been modified
--inplace Download directly to destination file instead of atomic download to temp/rename
@@ -2544,11 +2585,12 @@ Flags for anything which can Copy a file.
--multi-thread-write-buffer-size SizeSuffix In memory buffer size for writing when in multi-thread mode (default 128Ki)
--no-check-dest Don't check the destination, copy regardless
--no-traverse Don't traverse destination file system on copy
- --no-update-modtime Don't update destination mod-time if files identical
+ --no-update-modtime Don't update destination modtime if files identical
--order-by string Instructions on how to order the transfers, e.g. 'size,descending'
+ --partial-suffix string Add partial-suffix to temporary file name when --inplace is not used (default ".partial")
--refresh-times Refresh the modtime of remote files
--server-side-across-configs Allow server-side operations (e.g. copy) to work across different configs
- --size-only Skip based on size only, not mod-time or checksum
+ --size-only Skip based on size only, not modtime or checksum
--streaming-upload-cutoff SizeSuffix Cutoff for switching to chunked upload if file size is unknown, upload starts after reaching cutoff or when file ends (default 100Ki)
-u, --update Skip files that are newer on the destination
@@ -2684,16 +2726,19 @@ SEE ALSO
rclone checksum
-Checks the files in the source against a SUM file.
+Checks the files in the destination against a SUM file.
Synopsis
-Checks that hashsums of source files match the SUM file. It compares
-hashes (MD5, SHA1, etc) and logs a report of files which don't match. It
-doesn't alter the file system.
+Checks that hashsums of destination files match the SUM file. It
+compares hashes (MD5, SHA1, etc) and logs a report of files which don't
+match. It doesn't alter the file system.
-If you supply the --download flag, it will download the data from remote
-and calculate the contents hash on the fly. This can be useful for
+The sumfile is treated as the source and the dst:path is treated as the
+destination for the purposes of the output.
+
+If you supply the --download flag, it will download the data from the
+remote and calculate the content hash on the fly. This can be useful for
remotes that don't support hashes or if you really want to check all the
data.
@@ -2728,7 +2773,7 @@ what happened to it. These are reminiscent of diff files.
The default number of parallel checks is 8. See the --checkers=N option
for more information.
- rclone checksum sumfile src:path [flags]
+ rclone checksum sumfile dst:path [flags]
Options
@@ -3500,11 +3545,11 @@ Flags for anything which can Copy a file.
-c, --checksum Check for changes with size & checksum (if available, or fallback to size only).
--compare-dest stringArray Include additional comma separated server-side paths during comparison
--copy-dest stringArray Implies --compare-dest but also copies files from paths into destination
- --cutoff-mode string Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default "HARD")
+ --cutoff-mode HARD|SOFT|CAUTIOUS Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default HARD)
--ignore-case-sync Ignore case when synchronizing
--ignore-checksum Skip post copy check of checksums
--ignore-existing Skip all files that exist on destination
- --ignore-size Ignore size when skipping use mod-time or checksum
+ --ignore-size Ignore size when skipping use modtime or checksum
-I, --ignore-times Don't skip files that match size and time - transfer all files
--immutable Do not modify files, fail if existing files have been modified
--inplace Download directly to destination file instead of atomic download to temp/rename
@@ -3519,11 +3564,12 @@ Flags for anything which can Copy a file.
--multi-thread-write-buffer-size SizeSuffix In memory buffer size for writing when in multi-thread mode (default 128Ki)
--no-check-dest Don't check the destination, copy regardless
--no-traverse Don't traverse destination file system on copy
- --no-update-modtime Don't update destination mod-time if files identical
+ --no-update-modtime Don't update destination modtime if files identical
--order-by string Instructions on how to order the transfers, e.g. 'size,descending'
+ --partial-suffix string Add partial-suffix to temporary file name when --inplace is not used (default ".partial")
--refresh-times Refresh the modtime of remote files
--server-side-across-configs Allow server-side operations (e.g. copy) to work across different configs
- --size-only Skip based on size only, not mod-time or checksum
+ --size-only Skip based on size only, not modtime or checksum
--streaming-upload-cutoff SizeSuffix Cutoff for switching to chunked upload if file size is unknown, upload starts after reaching cutoff or when file ends (default 100Ki)
-u, --update Skip files that are newer on the destination
@@ -3984,10 +4030,6 @@ Run without a hash to see the list of all supported hashes, e.g.
* whirlpool
* crc32
* sha256
- * dropbox
- * hidrive
- * mailru
- * quickxor
Then
@@ -3996,7 +4038,7 @@ Then
Note that hash names are case insensitive and values are output in lower
case.
- rclone hashsum remote:path [flags]
+ rclone hashsum [ remote:path] [flags]
Options
@@ -4712,10 +4754,17 @@ not suffer from the same limitations.
Mounting on macOS
-Mounting on macOS can be done either via macFUSE (also known as osxfuse)
-or FUSE-T. macFUSE is a traditional FUSE driver utilizing a macOS kernel
-extension (kext). FUSE-T is an alternative FUSE system which "mounts"
-via an NFSv4 local server.
+Mounting on macOS can be done either via built-in NFS server, macFUSE
+(also known as osxfuse) or FUSE-T. macFUSE is a traditional FUSE driver
+utilizing a macOS kernel extension (kext). FUSE-T is an alternative FUSE
+system which "mounts" via an NFSv4 local server.
+
+NFS mount
+
+This method spins up an NFS server using serve nfs command and mounts it
+to the specified mountpoint. If you run this in background mode using
+|--daemon|, you will need to send SIGTERM signal to the rclone process
+using |kill| command to stop the mount.
macFUSE Notes
@@ -4764,7 +4813,8 @@ Without the use of --vfs-cache-mode this can only write files
sequentially, it can only seek when reading. This means that many
applications won't work with their files on an rclone mount without
--vfs-cache-mode writes or --vfs-cache-mode full. See the VFS File
-Caching section for more info.
+Caching section for more info. When using NFS mount on macOS, if you
+don't specify |--vfs-cache-mode| the mount point will be read-only.
The bucket-based remotes (e.g. Swift, S3, Google Compute Storage, B2) do
not support the concept of empty directories, so empty directories will
@@ -4904,9 +4954,8 @@ Mount option syntax includes a few extra options treated specially:
pgrep.
- vv... will be transformed into appropriate --verbose=N
- standard mount options like x-systemd.automount, _netdev, nosuid and
- alike are intended only for Automountd and ignored by rclone.
-
-VFS - Virtual File System
+ alike are intended only for Automountd and ignored by rclone. ##
+ VFS - Virtual File System
This command uses the VFS layer. This adapts the cloud storage objects
that rclone uses into something which looks much more like a disk filing
@@ -5283,6 +5332,7 @@ Options
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi)
--vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off)
--vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms)
+ --vfs-refresh Refreshes the directory cache recursively on start
--vfs-used-is-size rclone size Use the rclone size algorithm for Used size
--vfs-write-back Duration Time to writeback files after last use when using cache (default 5s)
--vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s)
@@ -5373,11 +5423,11 @@ Flags for anything which can Copy a file.
-c, --checksum Check for changes with size & checksum (if available, or fallback to size only).
--compare-dest stringArray Include additional comma separated server-side paths during comparison
--copy-dest stringArray Implies --compare-dest but also copies files from paths into destination
- --cutoff-mode string Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default "HARD")
+ --cutoff-mode HARD|SOFT|CAUTIOUS Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default HARD)
--ignore-case-sync Ignore case when synchronizing
--ignore-checksum Skip post copy check of checksums
--ignore-existing Skip all files that exist on destination
- --ignore-size Ignore size when skipping use mod-time or checksum
+ --ignore-size Ignore size when skipping use modtime or checksum
-I, --ignore-times Don't skip files that match size and time - transfer all files
--immutable Do not modify files, fail if existing files have been modified
--inplace Download directly to destination file instead of atomic download to temp/rename
@@ -5392,11 +5442,12 @@ Flags for anything which can Copy a file.
--multi-thread-write-buffer-size SizeSuffix In memory buffer size for writing when in multi-thread mode (default 128Ki)
--no-check-dest Don't check the destination, copy regardless
--no-traverse Don't traverse destination file system on copy
- --no-update-modtime Don't update destination mod-time if files identical
+ --no-update-modtime Don't update destination modtime if files identical
--order-by string Instructions on how to order the transfers, e.g. 'size,descending'
+ --partial-suffix string Add partial-suffix to temporary file name when --inplace is not used (default ".partial")
--refresh-times Refresh the modtime of remote files
--server-side-across-configs Allow server-side operations (e.g. copy) to work across different configs
- --size-only Skip based on size only, not mod-time or checksum
+ --size-only Skip based on size only, not modtime or checksum
--streaming-upload-cutoff SizeSuffix Cutoff for switching to chunked upload if file size is unknown, upload starts after reaching cutoff or when file ends (default 100Ki)
-u, --update Skip files that are newer on the destination
@@ -5844,6 +5895,27 @@ be used within the template to server pages:
-- .ModTime The UTC timestamp of an entry.
-----------------------------------------------------------------------
+The server also makes the following functions available so that they can
+be used within the template. These functions help extend the options for
+dynamic rendering of HTML. They can be used to render HTML based on
+specific conditions.
+
+ -----------------------------------------------------------------------
+ Function Description
+ ----------------------------------- -----------------------------------
+ afterEpoch Returns the time since the epoch
+ for the given time.
+
+ contains Checks whether a given substring is
+ present or not in a given string.
+
+ hasPrefix Checks whether the given string
+ begins with the specified prefix.
+
+ hasSuffix Checks whether the given string end
+ with the specified suffix.
+ -----------------------------------------------------------------------
+
Authentication
By default this will serve files without needing a login.
@@ -6063,7 +6135,9 @@ SEE ALSO
API.
- rclone serve ftp - Serve remote:path over FTP.
- rclone serve http - Serve the remote over HTTP.
+- rclone serve nfs - Serve the remote as an NFS mount
- rclone serve restic - Serve the remote for restic's REST API.
+- rclone serve s3 - Serve remote:path over s3.
- rclone serve sftp - Serve the remote over SFTP.
- rclone serve webdav - Serve remote:path over WebDAV.
@@ -6093,9 +6167,7 @@ Use --name to choose the friendly server name, which is by default
"rclone (hostname)".
Use --log-trace in conjunction with -vv to enable additional debug
-logging of all UPNP traffic.
-
-VFS - Virtual File System
+logging of all UPNP traffic. ## VFS - Virtual File System
This command uses the VFS layer. This adapts the cloud storage objects
that rclone uses into something which looks much more like a disk filing
@@ -6459,6 +6531,7 @@ Options
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi)
--vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off)
--vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms)
+ --vfs-refresh Refreshes the directory cache recursively on start
--vfs-used-is-size rclone size Use the rclone size algorithm for Used size
--vfs-write-back Duration Time to writeback files after last use when using cache (default 5s)
--vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s)
@@ -6539,8 +6612,7 @@ directory with book-keeping records of created and mounted volumes.
All mount and VFS options are submitted by the docker daemon via API,
but you can also provide defaults on the command line as well as set
path to the config file and cache directory or adjust logging verbosity.
-
-VFS - Virtual File System
+## VFS - Virtual File System
This command uses the VFS layer. This adapts the cloud storage objects
that rclone uses into something which looks much more like a disk filing
@@ -6922,6 +6994,7 @@ Options
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi)
--vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off)
--vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms)
+ --vfs-refresh Refreshes the directory cache recursively on start
--vfs-used-is-size rclone size Use the rclone size algorithm for Used size
--vfs-write-back Duration Time to writeback files after last use when using cache (default 5s)
--vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s)
@@ -6986,9 +7059,7 @@ Authentication
By default this will serve files without needing a login.
You can set a single username and password with the --user and --pass
-flags.
-
-VFS - Virtual File System
+flags. ## VFS - Virtual File System
This command uses the VFS layer. This adapts the cloud storage objects
that rclone uses into something which looks much more like a disk filing
@@ -7426,6 +7497,7 @@ Options
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi)
--vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off)
--vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms)
+ --vfs-refresh Refreshes the directory cache recursively on start
--vfs-used-is-size rclone size Use the rclone size algorithm for Used size
--vfs-write-back Duration Time to writeback files after last use when using cache (default 5s)
--vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s)
@@ -7579,6 +7651,27 @@ be used within the template to server pages:
-- .ModTime The UTC timestamp of an entry.
-----------------------------------------------------------------------
+The server also makes the following functions available so that they can
+be used within the template. These functions help extend the options for
+dynamic rendering of HTML. They can be used to render HTML based on
+specific conditions.
+
+ -----------------------------------------------------------------------
+ Function Description
+ ----------------------------------- -----------------------------------
+ afterEpoch Returns the time since the epoch
+ for the given time.
+
+ contains Checks whether a given substring is
+ present or not in a given string.
+
+ hasPrefix Checks whether the given string
+ begins with the specified prefix.
+
+ hasSuffix Checks whether the given string end
+ with the specified suffix.
+ -----------------------------------------------------------------------
+
Authentication
By default this will serve files without needing a login.
@@ -7605,9 +7698,8 @@ The password file can be updated while rclone is running.
Use --realm to set the authentication realm.
-Use --salt to change the password hashing salt from the default.
-
-VFS - Virtual File System
+Use --salt to change the password hashing salt from the default. ## VFS
+- Virtual File System
This command uses the VFS layer. This adapts the cloud storage objects
that rclone uses into something which looks much more like a disk filing
@@ -8054,6 +8146,7 @@ Options
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi)
--vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off)
--vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms)
+ --vfs-refresh Refreshes the directory cache recursively on start
--vfs-used-is-size rclone size Use the rclone size algorithm for Used size
--vfs-write-back Duration Time to writeback files after last use when using cache (default 5s)
--vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s)
@@ -8091,254 +8184,44 @@ SEE ALSO
- rclone serve - Serve a remote over a protocol.
-rclone serve restic
+rclone serve nfs
-Serve the remote for restic's REST API.
+Serve the remote as an NFS mount
Synopsis
-Run a basic web server to serve a remote over restic's REST backend API
-over HTTP. This allows restic to use rclone as a data storage mechanism
-for cloud providers that restic does not support directly.
-
-Restic is a command-line program for doing backups.
-
-The server will log errors. Use -v to see access logs.
-
---bwlimit will be respected for file transfers. Use --stats to control
-the stats printing.
-
-Setting up rclone for use by restic
-
-First set up a remote for your chosen cloud provider.
-
-Once you have set up the remote, check it is working with, for example
-"rclone lsd remote:". You may have called the remote something other
-than "remote:" - just substitute whatever you called it in the following
-instructions.
-
-Now start the rclone restic server
-
- rclone serve restic -v remote:backup
-
-Where you can replace "backup" in the above by whatever path in the
-remote you wish to use.
-
-By default this will serve on "localhost:8080" you can change this with
-use of the --addr flag.
+Create an NFS server that serves the given remote over the network.
-You might wish to start this server on boot.
+The primary purpose for this command is to enable mount command on
+recent macOS versions where installing FUSE is very cumbersome.
-Adding --cache-objects=false will cause rclone to stop caching objects
-returned from the List call. Caching is normally desirable as it speeds
-up downloading objects, saves transactions and uses very little memory.
+Since this is running on NFSv3, no authentication method is available.
+Any client will be able to access the data. To limit access, you can use
+serve NFS on loopback address and rely on secure tunnels (such as SSH).
+For this reason, by default, a random TCP port is chosen and loopback
+interface is used for the listening address; meaning that it is only
+available to the local machine. If you want other machines to access the
+NFS mount over local network, you need to specify the listening address
+and port using --addr flag.
-Setting up restic to use rclone
+Modifying files through NFS protocol requires VFS caching. Usually you
+will need to specify --vfs-cache-mode in order to be able to write to
+the mountpoint (full is recommended). If you don't specify VFS cache
+mode, the mount will be read-only.
-Now you can follow the restic instructions on setting up restic.
-
-Note that you will need restic 0.8.2 or later to interoperate with
-rclone.
+To serve NFS over the network use following command:
-For the example above you will want to use "http://localhost:8080/" as
-the URL for the REST server.
+ rclone serve nfs remote: --addr 0.0.0.0:$PORT --vfs-cache-mode=full
-For example:
+We specify a specific port that we can use in the mount command:
- $ export RESTIC_REPOSITORY=rest:http://localhost:8080/
- $ export RESTIC_PASSWORD=yourpassword
- $ restic init
- created restic backend 8b1a4b56ae at rest:http://localhost:8080/
+To mount the server under Linux/macOS, use the following command:
- Please note that knowledge of your password is required to access
- the repository. Losing your password means that your data is
- irrecoverably lost.
- $ restic backup /path/to/files/to/backup
- scan [/path/to/files/to/backup]
- scanned 189 directories, 312 files in 0:00
- [0:00] 100.00% 38.128 MiB / 38.128 MiB 501 / 501 items 0 errors ETA 0:00
- duration: 0:00
- snapshot 45c8fdd8 saved
+ mount -oport=$PORT,mountport=$PORT $HOSTNAME: path/to/mountpoint
-Multiple repositories
+Where $PORT is the same port number we used in the serve nfs command.
-Note that you can use the endpoint to host multiple repositories. Do
-this by adding a directory name or path after the URL. Note that these
-must end with /. Eg
-
- $ export RESTIC_REPOSITORY=rest:http://localhost:8080/user1repo/
- # backup user1 stuff
- $ export RESTIC_REPOSITORY=rest:http://localhost:8080/user2repo/
- # backup user2 stuff
-
-Private repositories
-
-The--private-repos flag can be used to limit users to repositories
-starting with a path of //.
-
-Server options
-
-Use --addr to specify which IP address and port the server should listen
-on, eg --addr 1.2.3.4:8000 or --addr :8080 to listen to all IPs. By
-default it only listens on localhost. You can use port :0 to let the OS
-choose an available port.
-
-If you set --addr to listen on a public or LAN accessible IP address
-then using Authentication is advised - see the next section for info.
-
-You can use a unix socket by setting the url to unix:///path/to/socket
-or just by using an absolute path name. Note that unix sockets bypass
-the authentication - this is expected to be done with file system
-permissions.
-
---addr may be repeated to listen on multiple IPs/ports/sockets.
-
---server-read-timeout and --server-write-timeout can be used to control
-the timeouts on the server. Note that this is the total time for a
-transfer.
-
---max-header-bytes controls the maximum number of bytes the server will
-accept in the HTTP header.
-
---baseurl controls the URL prefix that rclone serves from. By default
-rclone will serve from the root. If you used --baseurl "/rclone" then
-rclone would serve from a URL starting with "/rclone/". This is useful
-if you wish to proxy rclone serve. Rclone automatically inserts leading
-and trailing "/" on --baseurl, so --baseurl "rclone",
---baseurl "/rclone" and --baseurl "/rclone/" are all treated
-identically.
-
-TLS (SSL)
-
-By default this will serve over http. If you want you can serve over
-https. You will need to supply the --cert and --key flags. If you wish
-to do client side certificate validation then you will need to supply
---client-ca also.
-
---cert should be a either a PEM encoded certificate or a concatenation
-of that with the CA certificate. --key should be the PEM encoded private
-key and --client-ca should be the PEM encoded client certificate
-authority certificate.
-
---min-tls-version is minimum TLS version that is acceptable. Valid
-values are "tls1.0", "tls1.1", "tls1.2" and "tls1.3" (default "tls1.0").
-
-Authentication
-
-By default this will serve files without needing a login.
-
-You can either use an htpasswd file which can take lots of users, or set
-a single username and password with the --user and --pass flags.
-
-If no static users are configured by either of the above methods, and
-client certificates are required by the --client-ca flag passed to the
-server, the client certificate common name will be considered as the
-username.
-
-Use --htpasswd /path/to/htpasswd to provide an htpasswd file. This is in
-standard apache format and supports MD5, SHA1 and BCrypt for basic
-authentication. Bcrypt is recommended.
-
-To create an htpasswd file:
-
- touch htpasswd
- htpasswd -B htpasswd user
- htpasswd -B htpasswd anotherUser
-
-The password file can be updated while rclone is running.
-
-Use --realm to set the authentication realm.
-
-Use --salt to change the password hashing salt from the default.
-
- rclone serve restic remote:path [flags]
-
-Options
-
- --addr stringArray IPaddress:Port or :Port to bind server to (default [127.0.0.1:8080])
- --allow-origin string Origin which cross-domain request (CORS) can be executed from
- --append-only Disallow deletion of repository data
- --baseurl string Prefix for URLs - leave blank for root
- --cache-objects Cache listed objects (default true)
- --cert string TLS PEM key (concatenation of certificate and CA certificate)
- --client-ca string Client certificate authority to verify clients with
- -h, --help help for restic
- --htpasswd string A htpasswd file - if not provided no authentication is done
- --key string TLS PEM Private key
- --max-header-bytes int Maximum size of request header (default 4096)
- --min-tls-version string Minimum TLS version that is acceptable (default "tls1.0")
- --pass string Password for authentication
- --private-repos Users can only access their private repo
- --realm string Realm for authentication
- --salt string Password hashing salt (default "dlPL2MqE")
- --server-read-timeout Duration Timeout for server reading data (default 1h0m0s)
- --server-write-timeout Duration Timeout for server writing data (default 1h0m0s)
- --stdio Run an HTTP2 server on stdin/stdout
- --user string User name for authentication
-
-See the global flags page for global options not listed here.
-
-SEE ALSO
-
-- rclone serve - Serve a remote over a protocol.
-
-rclone serve sftp
-
-Serve the remote over SFTP.
-
-Synopsis
-
-Run an SFTP server to serve a remote over SFTP. This can be used with an
-SFTP client or you can make a remote of type sftp to use with it.
-
-You can use the filter flags (e.g. --include, --exclude) to control what
-is served.
-
-The server will respond to a small number of shell commands, mainly
-md5sum, sha1sum and df, which enable it to provide support for checksums
-and the about feature when accessed from an sftp remote.
-
-Note that this server uses standard 32 KiB packet payload size, which
-means you must not configure the client to expect anything else, e.g.
-with the chunk_size option on an sftp remote.
-
-The server will log errors. Use -v to see access logs.
-
---bwlimit will be respected for file transfers. Use --stats to control
-the stats printing.
-
-You must provide some means of authentication, either with
---user/--pass, an authorized keys file (specify location with
---authorized-keys - the default is the same as ssh), an --auth-proxy, or
-set the --no-auth flag for no authentication when logging in.
-
-If you don't supply a host --key then rclone will generate rsa, ecdsa
-and ed25519 variants, and cache them for later use in rclone's cache
-directory (see rclone help flags cache-dir) in the "serve-sftp"
-directory.
-
-By default the server binds to localhost:2022 - if you want it to be
-reachable externally then supply --addr :2022 for example.
-
-Note that the default of --vfs-cache-mode off is fine for the rclone
-sftp backend, but it may not be with other SFTP clients.
-
-If --stdio is specified, rclone will serve SFTP over stdio, which can be
-used with sshd via ~/.ssh/authorized_keys, for example:
-
- restrict,command="rclone serve sftp --stdio ./photos" ssh-rsa ...
-
-On the client you need to set --transfers 1 when using --stdio.
-Otherwise multiple instances of the rclone server are started by OpenSSH
-which can lead to "corrupted on transfer" errors. This is the case
-because the client chooses indiscriminately which server to send
-commands to while the servers all have different views of the state of
-the filing system.
-
-The "restrict" in authorized_keys prevents SHA1SUMs and MD5SUMs from
-being used. Omitting "restrict" and using --sftp-path-override to enable
-checksumming is possible but less secure and you could use the SFTP
-server provided by OpenSSH in this case.
+This feature is only available on Unix platforms.
VFS - Virtual File System
@@ -8671,101 +8554,23 @@ result is accurate. However, this is very inefficient and may cost lots
of API calls resulting in extra charges. Use it as a last resort and
only with caching.
-Auth Proxy
-
-If you supply the parameter --auth-proxy /path/to/program then rclone
-will use that program to generate backends on the fly which then are
-used to authenticate incoming requests. This uses a simple JSON based
-protocol with input on STDIN and output on STDOUT.
-
-PLEASE NOTE: --auth-proxy and --authorized-keys cannot be used together,
-if --auth-proxy is set the authorized keys option will be ignored.
-
-There is an example program bin/test_proxy.py in the rclone source code.
-
-The program's job is to take a user and pass on the input and turn those
-into the config for a backend on STDOUT in JSON format. This config will
-have any default parameters for the backend added, but it won't use
-configuration from environment variables or command line options - it is
-the job of the proxy program to make a complete config.
-
-This config generated must have this extra parameter - _root - root to
-use for the backend
-
-And it may have this parameter - _obscure - comma separated strings for
-parameters to obscure
-
-If password authentication was used by the client, input to the proxy
-process (on STDIN) would look similar to this:
-
- {
- "user": "me",
- "pass": "mypassword"
- }
-
-If public-key authentication was used by the client, input to the proxy
-process (on STDIN) would look similar to this:
-
- {
- "user": "me",
- "public_key": "AAAAB3NzaC1yc2EAAAADAQABAAABAQDuwESFdAe14hVS6omeyX7edc...JQdf"
- }
-
-And as an example return this on STDOUT
-
- {
- "type": "sftp",
- "_root": "",
- "_obscure": "pass",
- "user": "me",
- "pass": "mypassword",
- "host": "sftp.example.com"
- }
-
-This would mean that an SFTP backend would be created on the fly for the
-user and pass/public_key returned in the output to the host given. Note
-that since _obscure is set to pass, rclone will obscure the pass
-parameter before creating the backend (which is required for sftp
-backends).
-
-The program can manipulate the supplied user in any way, for example to
-make proxy to many different sftp backends, you could make the user be
-user@example.com and then set the host to example.com in the output and
-the user to user. For security you'd probably want to restrict the host
-to a limited list.
-
-Note that an internal cache is keyed on user so only use that for
-configuration, don't use pass or public_key. This also means that if a
-user's password or public-key is changed the cache will need to expire
-(which takes 5 mins) before it takes effect.
-
-This can be used to build general purpose proxies to any kind of backend
-that rclone supports.
-
- rclone serve sftp remote:path [flags]
+ rclone serve nfs remote:path [flags]
Options
- --addr string IPaddress:Port or :Port to bind server to (default "localhost:2022")
- --auth-proxy string A program to use to create the backend from the auth
- --authorized-keys string Authorized keys file (default "~/.ssh/authorized_keys")
+ --addr string IPaddress:Port or :Port to bind server to
--dir-cache-time Duration Time to cache directory entries for (default 5m0s)
--dir-perms FileMode Directory permissions (default 0777)
--file-perms FileMode File permissions (default 0666)
--gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000)
- -h, --help help for sftp
- --key stringArray SSH private host key file (Can be multi-valued, leave blank to auto generate)
- --no-auth Allow connections with no authentication if set
+ -h, --help help for nfs
--no-checksum Don't compare checksums on up/download
--no-modtime Don't read/write the modification time (can speed things up)
--no-seek Don't allow seeking in files
- --pass string Password for authentication
--poll-interval Duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s)
--read-only Only allow read-only access
- --stdio Run an sftp server on stdin/stdout
--uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000)
--umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2)
- --user string User name for authentication
--vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s)
--vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off)
--vfs-cache-min-free-space SizeSuffix Target minimum free space on the disk containing the cache (default off)
@@ -8778,6 +8583,7 @@ Options
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi)
--vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off)
--vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms)
+ --vfs-refresh Refreshes the directory cache recursively on start
--vfs-used-is-size rclone size Use the rclone size algorithm for Used size
--vfs-write-back Duration Time to writeback files after last use when using cache (default 5s)
--vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s)
@@ -8815,52 +8621,90 @@ SEE ALSO
- rclone serve - Serve a remote over a protocol.
-rclone serve webdav
+rclone serve restic
-Serve remote:path over WebDAV.
+Serve the remote for restic's REST API.
Synopsis
-Run a basic WebDAV server to serve a remote over HTTP via the WebDAV
-protocol. This can be viewed with a WebDAV client, through a web
-browser, or you can make a remote of type WebDAV to read and write it.
+Run a basic web server to serve a remote over restic's REST backend API
+over HTTP. This allows restic to use rclone as a data storage mechanism
+for cloud providers that restic does not support directly.
-WebDAV options
+Restic is a command-line program for doing backups.
---etag-hash
+The server will log errors. Use -v to see access logs.
-This controls the ETag header. Without this flag the ETag will be based
-on the ModTime and Size of the object.
+--bwlimit will be respected for file transfers. Use --stats to control
+the stats printing.
-If this flag is set to "auto" then rclone will choose the first
-supported hash on the backend or you can use a named hash such as "MD5"
-or "SHA-1". Use the hashsum command to see the full list.
+Setting up rclone for use by restic
-Access WebDAV on Windows
+First set up a remote for your chosen cloud provider.
-WebDAV shared folder can be mapped as a drive on Windows, however the
-default settings prevent it. Windows will fail to connect to the server
-using insecure Basic authentication. It will not even display any login
-dialog. Windows requires SSL / HTTPS connection to be used with Basic.
-If you try to connect via Add Network Location Wizard you will get the
-following error: "The folder you entered does not appear to be valid.
-Please choose another". However, you still can connect if you set the
-following registry key on a client machine: HKEY_LOCAL_MACHINEto 2. The
-BasicAuthLevel can be set to the following values: 0 - Basic
-authentication disabled 1 - Basic authentication enabled for SSL
-connections only 2 - Basic authentication enabled for SSL connections
-and for non-SSL connections If required, increase the
-FileSizeLimitInBytes to a higher value. Navigate to the Services
-interface, then restart the WebClient service.
+Once you have set up the remote, check it is working with, for example
+"rclone lsd remote:". You may have called the remote something other
+than "remote:" - just substitute whatever you called it in the following
+instructions.
-Access Office applications on WebDAV
+Now start the rclone restic server
-Navigate to following registry HKEY_CURRENT_USER[14.0/15.0/16.0] Create
-a new DWORD BasicAuthLevel with value 2. 0 - Basic authentication
-disabled 1 - Basic authentication enabled for SSL connections only 2 -
-Basic authentication enabled for SSL and for non-SSL connections
+ rclone serve restic -v remote:backup
-https://learn.microsoft.com/en-us/office/troubleshoot/powerpoint/office-opens-blank-from-sharepoint
+Where you can replace "backup" in the above by whatever path in the
+remote you wish to use.
+
+By default this will serve on "localhost:8080" you can change this with
+use of the --addr flag.
+
+You might wish to start this server on boot.
+
+Adding --cache-objects=false will cause rclone to stop caching objects
+returned from the List call. Caching is normally desirable as it speeds
+up downloading objects, saves transactions and uses very little memory.
+
+Setting up restic to use rclone
+
+Now you can follow the restic instructions on setting up restic.
+
+Note that you will need restic 0.8.2 or later to interoperate with
+rclone.
+
+For the example above you will want to use "http://localhost:8080/" as
+the URL for the REST server.
+
+For example:
+
+ $ export RESTIC_REPOSITORY=rest:http://localhost:8080/
+ $ export RESTIC_PASSWORD=yourpassword
+ $ restic init
+ created restic backend 8b1a4b56ae at rest:http://localhost:8080/
+
+ Please note that knowledge of your password is required to access
+ the repository. Losing your password means that your data is
+ irrecoverably lost.
+ $ restic backup /path/to/files/to/backup
+ scan [/path/to/files/to/backup]
+ scanned 189 directories, 312 files in 0:00
+ [0:00] 100.00% 38.128 MiB / 38.128 MiB 501 / 501 items 0 errors ETA 0:00
+ duration: 0:00
+ snapshot 45c8fdd8 saved
+
+Multiple repositories
+
+Note that you can use the endpoint to host multiple repositories. Do
+this by adding a directory name or path after the URL. Note that these
+must end with /. Eg
+
+ $ export RESTIC_REPOSITORY=rest:http://localhost:8080/user1repo/
+ # backup user1 stuff
+ $ export RESTIC_REPOSITORY=rest:http://localhost:8080/user2repo/
+ # backup user2 stuff
+
+Private repositories
+
+The--private-repos flag can be used to limit users to repositories
+starting with a path of //.
Server options
@@ -8909,58 +8753,6 @@ authority certificate.
--min-tls-version is minimum TLS version that is acceptable. Valid
values are "tls1.0", "tls1.1", "tls1.2" and "tls1.3" (default "tls1.0").
-Template
-
---template allows a user to specify a custom markup template for HTTP
-and WebDAV serve functions. The server exports the following markup to
-be used within the template to server pages:
-
- -----------------------------------------------------------------------
- Parameter Description
- ----------------------------------- -----------------------------------
- .Name The full path of a file/directory.
-
- .Title Directory listing of .Name
-
- .Sort The current sort used. This is
- changeable via ?sort= parameter
-
- Sort Options:
- namedirfirst,name,size,time
- (default namedirfirst)
-
- .Order The current ordering used. This is
- changeable via ?order= parameter
-
- Order Options: asc,desc (default
- asc)
-
- .Query Currently unused.
-
- .Breadcrumb Allows for creating a relative
- navigation
-
- -- .Link The relative to the root link of
- the Text.
-
- -- .Text The Name of the directory.
-
- .Entries Information about a specific
- file/directory.
-
- -- .URL The 'url' of an entry.
-
- -- .Leaf Currently same as 'URL' but
- intended to be 'just' the name.
-
- -- .IsDir Boolean for if an entry is a
- directory or not.
-
- -- .Size Size in Bytes of the entry.
-
- -- .ModTime The UTC timestamp of an entry.
- -----------------------------------------------------------------------
-
Authentication
By default this will serve files without needing a login.
@@ -8989,7 +8781,196 @@ Use --realm to set the authentication realm.
Use --salt to change the password hashing salt from the default.
-VFS - Virtual File System
+ rclone serve restic remote:path [flags]
+
+Options
+
+ --addr stringArray IPaddress:Port or :Port to bind server to (default [127.0.0.1:8080])
+ --allow-origin string Origin which cross-domain request (CORS) can be executed from
+ --append-only Disallow deletion of repository data
+ --baseurl string Prefix for URLs - leave blank for root
+ --cache-objects Cache listed objects (default true)
+ --cert string TLS PEM key (concatenation of certificate and CA certificate)
+ --client-ca string Client certificate authority to verify clients with
+ -h, --help help for restic
+ --htpasswd string A htpasswd file - if not provided no authentication is done
+ --key string TLS PEM Private key
+ --max-header-bytes int Maximum size of request header (default 4096)
+ --min-tls-version string Minimum TLS version that is acceptable (default "tls1.0")
+ --pass string Password for authentication
+ --private-repos Users can only access their private repo
+ --realm string Realm for authentication
+ --salt string Password hashing salt (default "dlPL2MqE")
+ --server-read-timeout Duration Timeout for server reading data (default 1h0m0s)
+ --server-write-timeout Duration Timeout for server writing data (default 1h0m0s)
+ --stdio Run an HTTP2 server on stdin/stdout
+ --user string User name for authentication
+
+See the global flags page for global options not listed here.
+
+SEE ALSO
+
+- rclone serve - Serve a remote over a protocol.
+
+rclone serve s3
+
+Serve remote:path over s3.
+
+Synopsis
+
+serve s3 implements a basic s3 server that serves a remote via s3. This
+can be viewed with an s3 client, or you can make an s3 type remote to
+read and write to it with rclone.
+
+serve s3 is considered Experimental so use with care.
+
+S3 server supports Signature Version 4 authentication. Just use
+--auth-key accessKey,secretKey and set the Authorization header
+correctly in the request. (See the AWS docs).
+
+--auth-key can be repeated for multiple auth pairs. If --auth-key is not
+provided then serve s3 will allow anonymous access.
+
+Please note that some clients may require HTTPS endpoints. See the SSL
+docs for more information.
+
+This command uses the VFS directory cache. All the functionality will
+work with --vfs-cache-mode off. Using --vfs-cache-mode full (or writes)
+can be used to cache objects locally to improve performance.
+
+Use --force-path-style=false if you want to use the bucket name as a
+part of the hostname (such as mybucket.local)
+
+Use --etag-hash if you want to change the hash uses for the ETag. Note
+that using anything other than MD5 (the default) is likely to cause
+problems for S3 clients which rely on the Etag being the MD5.
+
+Quickstart
+
+For a simple set up, to serve remote:path over s3, run the server like
+this:
+
+ rclone serve s3 --auth-key ACCESS_KEY_ID,SECRET_ACCESS_KEY remote:path
+
+This will be compatible with an rclone remote which is defined like
+this:
+
+ [serves3]
+ type = s3
+ provider = Rclone
+ endpoint = http://127.0.0.1:8080/
+ access_key_id = ACCESS_KEY_ID
+ secret_access_key = SECRET_ACCESS_KEY
+ use_multipart_uploads = false
+
+Note that setting disable_multipart_uploads = true is to work around a
+bug which will be fixed in due course.
+
+Bugs
+
+When uploading multipart files serve s3 holds all the parts in memory
+(see #7453). This is a limitaton of the library rclone uses for serving
+S3 and will hopefully be fixed at some point.
+
+Multipart server side copies do not work (see #7454). These take a very
+long time and eventually fail. The default threshold for multipart
+server side copies is 5G which is the maximum it can be, so files above
+this side will fail to be server side copied.
+
+For a current list of serve s3 bugs see the serve s3 bug category on
+GitHub.
+
+Limitations
+
+serve s3 will treat all directories in the root as buckets and ignore
+all files in the root. You can use CreateBucket to create folders under
+the root, but you can't create empty folders under other folders not in
+the root.
+
+When using PutObject or DeleteObject, rclone will automatically create
+or clean up empty folders. If you don't want to clean up empty folders
+automatically, use --no-cleanup.
+
+When using ListObjects, rclone will use / when the delimiter is empty.
+This reduces backend requests with no effect on most operations, but if
+the delimiter is something other than / and empty, rclone will do a full
+recursive search of the backend, which can take some time.
+
+Versioning is not currently supported.
+
+Metadata will only be saved in memory other than the rclone mtime
+metadata which will be set as the modification time of the file.
+
+Supported operations
+
+serve s3 currently supports the following operations.
+
+- Bucket
+ - ListBuckets
+ - CreateBucket
+ - DeleteBucket
+- Object
+ - HeadObject
+ - ListObjects
+ - GetObject
+ - PutObject
+ - DeleteObject
+ - DeleteObjects
+ - CreateMultipartUpload
+ - CompleteMultipartUpload
+ - AbortMultipartUpload
+ - CopyObject
+ - UploadPart
+
+Other operations will return error Unimplemented.
+
+Server options
+
+Use --addr to specify which IP address and port the server should listen
+on, eg --addr 1.2.3.4:8000 or --addr :8080 to listen to all IPs. By
+default it only listens on localhost. You can use port :0 to let the OS
+choose an available port.
+
+If you set --addr to listen on a public or LAN accessible IP address
+then using Authentication is advised - see the next section for info.
+
+You can use a unix socket by setting the url to unix:///path/to/socket
+or just by using an absolute path name. Note that unix sockets bypass
+the authentication - this is expected to be done with file system
+permissions.
+
+--addr may be repeated to listen on multiple IPs/ports/sockets.
+
+--server-read-timeout and --server-write-timeout can be used to control
+the timeouts on the server. Note that this is the total time for a
+transfer.
+
+--max-header-bytes controls the maximum number of bytes the server will
+accept in the HTTP header.
+
+--baseurl controls the URL prefix that rclone serves from. By default
+rclone will serve from the root. If you used --baseurl "/rclone" then
+rclone would serve from a URL starting with "/rclone/". This is useful
+if you wish to proxy rclone serve. Rclone automatically inserts leading
+and trailing "/" on --baseurl, so --baseurl "rclone",
+--baseurl "/rclone" and --baseurl "/rclone/" are all treated
+identically.
+
+TLS (SSL)
+
+By default this will serve over http. If you want you can serve over
+https. You will need to supply the --cert and --key flags. If you wish
+to do client side certificate validation then you will need to supply
+--client-ca also.
+
+--cert should be a either a PEM encoded certificate or a concatenation
+of that with the CA certificate. --key should be the PEM encoded private
+key and --client-ca should be the PEM encoded client certificate
+authority certificate.
+
+--min-tls-version is minimum TLS version that is acceptable. Valid
+values are "tls1.0", "tls1.1", "tls1.2" and "tls1.3" (default "tls1.0").
+## VFS - Virtual File System
This command uses the VFS layer. This adapts the cloud storage objects
that rclone uses into something which looks much more like a disk filing
@@ -9320,112 +9301,36 @@ result is accurate. However, this is very inefficient and may cost lots
of API calls resulting in extra charges. Use it as a last resort and
only with caching.
-Auth Proxy
-
-If you supply the parameter --auth-proxy /path/to/program then rclone
-will use that program to generate backends on the fly which then are
-used to authenticate incoming requests. This uses a simple JSON based
-protocol with input on STDIN and output on STDOUT.
-
-PLEASE NOTE: --auth-proxy and --authorized-keys cannot be used together,
-if --auth-proxy is set the authorized keys option will be ignored.
-
-There is an example program bin/test_proxy.py in the rclone source code.
-
-The program's job is to take a user and pass on the input and turn those
-into the config for a backend on STDOUT in JSON format. This config will
-have any default parameters for the backend added, but it won't use
-configuration from environment variables or command line options - it is
-the job of the proxy program to make a complete config.
-
-This config generated must have this extra parameter - _root - root to
-use for the backend
-
-And it may have this parameter - _obscure - comma separated strings for
-parameters to obscure
-
-If password authentication was used by the client, input to the proxy
-process (on STDIN) would look similar to this:
-
- {
- "user": "me",
- "pass": "mypassword"
- }
-
-If public-key authentication was used by the client, input to the proxy
-process (on STDIN) would look similar to this:
-
- {
- "user": "me",
- "public_key": "AAAAB3NzaC1yc2EAAAADAQABAAABAQDuwESFdAe14hVS6omeyX7edc...JQdf"
- }
-
-And as an example return this on STDOUT
-
- {
- "type": "sftp",
- "_root": "",
- "_obscure": "pass",
- "user": "me",
- "pass": "mypassword",
- "host": "sftp.example.com"
- }
-
-This would mean that an SFTP backend would be created on the fly for the
-user and pass/public_key returned in the output to the host given. Note
-that since _obscure is set to pass, rclone will obscure the pass
-parameter before creating the backend (which is required for sftp
-backends).
-
-The program can manipulate the supplied user in any way, for example to
-make proxy to many different sftp backends, you could make the user be
-user@example.com and then set the host to example.com in the output and
-the user to user. For security you'd probably want to restrict the host
-to a limited list.
-
-Note that an internal cache is keyed on user so only use that for
-configuration, don't use pass or public_key. This also means that if a
-user's password or public-key is changed the cache will need to expire
-(which takes 5 mins) before it takes effect.
-
-This can be used to build general purpose proxies to any kind of backend
-that rclone supports.
-
- rclone serve webdav remote:path [flags]
+ rclone serve s3 remote:path [flags]
Options
--addr stringArray IPaddress:Port or :Port to bind server to (default [127.0.0.1:8080])
--allow-origin string Origin which cross-domain request (CORS) can be executed from
- --auth-proxy string A program to use to create the backend from the auth
+ --auth-key stringArray Set key pair for v4 authorization: access_key_id,secret_access_key
--baseurl string Prefix for URLs - leave blank for root
--cert string TLS PEM key (concatenation of certificate and CA certificate)
--client-ca string Client certificate authority to verify clients with
--dir-cache-time Duration Time to cache directory entries for (default 5m0s)
--dir-perms FileMode Directory permissions (default 0777)
- --disable-dir-list Disable HTML directory list on GET request for a directory
- --etag-hash string Which hash to use for the ETag, or auto or blank for off
+ --etag-hash string Which hash to use for the ETag, or auto or blank for off (default "MD5")
--file-perms FileMode File permissions (default 0666)
+ --force-path-style If true use path style access if false use virtual hosted style (default true) (default true)
--gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000)
- -h, --help help for webdav
- --htpasswd string A htpasswd file - if not provided no authentication is done
+ -h, --help help for s3
--key string TLS PEM Private key
--max-header-bytes int Maximum size of request header (default 4096)
--min-tls-version string Minimum TLS version that is acceptable (default "tls1.0")
--no-checksum Don't compare checksums on up/download
+ --no-cleanup Not to cleanup empty folder after object is deleted
--no-modtime Don't read/write the modification time (can speed things up)
--no-seek Don't allow seeking in files
- --pass string Password for authentication
--poll-interval Duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s)
--read-only Only allow read-only access
- --realm string Realm for authentication
- --salt string Password hashing salt (default "dlPL2MqE")
--server-read-timeout Duration Timeout for server reading data (default 1h0m0s)
--server-write-timeout Duration Timeout for server writing data (default 1h0m0s)
- --template string User-specified template
--uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000)
--umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2)
- --user string User name for authentication
--vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s)
--vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off)
--vfs-cache-min-free-space SizeSuffix Target minimum free space on the disk containing the cache (default off)
@@ -9438,6 +9343,7 @@ Options
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi)
--vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off)
--vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms)
+ --vfs-refresh Refreshes the directory cache recursively on start
--vfs-used-is-size rclone size Use the rclone size algorithm for Used size
--vfs-write-back Duration Time to writeback files after last use when using cache (default 5s)
--vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s)
@@ -9475,268 +9381,506 @@ SEE ALSO
- rclone serve - Serve a remote over a protocol.
-rclone settier
+rclone serve sftp
-Changes storage class/tier of objects in remote.
+Serve the remote over SFTP.
Synopsis
-rclone settier changes storage tier or class at remote if supported. Few
-cloud storage services provides different storage classes on objects,
-for example AWS S3 and Glacier, Azure Blob storage - Hot, Cool and
-Archive, Google Cloud Storage, Regional Storage, Nearline, Coldline etc.
-
-Note that, certain tier changes make objects not available to access
-immediately. For example tiering to archive in azure blob storage makes
-objects in frozen state, user can restore by setting tier to Hot/Cool,
-similarly S3 to Glacier makes object inaccessible.true
+Run an SFTP server to serve a remote over SFTP. This can be used with an
+SFTP client or you can make a remote of type sftp to use with it.
-You can use it to tier single object
+You can use the filter flags (e.g. --include, --exclude) to control what
+is served.
- rclone settier Cool remote:path/file
+The server will respond to a small number of shell commands, mainly
+md5sum, sha1sum and df, which enable it to provide support for checksums
+and the about feature when accessed from an sftp remote.
-Or use rclone filters to set tier on only specific files
+Note that this server uses standard 32 KiB packet payload size, which
+means you must not configure the client to expect anything else, e.g.
+with the chunk_size option on an sftp remote.
- rclone --include "*.txt" settier Hot remote:path/dir
+The server will log errors. Use -v to see access logs.
-Or just provide remote directory and all files in directory will be
-tiered
+--bwlimit will be respected for file transfers. Use --stats to control
+the stats printing.
- rclone settier tier remote:path/dir
+You must provide some means of authentication, either with
+--user/--pass, an authorized keys file (specify location with
+--authorized-keys - the default is the same as ssh), an --auth-proxy, or
+set the --no-auth flag for no authentication when logging in.
- rclone settier tier remote:path [flags]
+If you don't supply a host --key then rclone will generate rsa, ecdsa
+and ed25519 variants, and cache them for later use in rclone's cache
+directory (see rclone help flags cache-dir) in the "serve-sftp"
+directory.
-Options
+By default the server binds to localhost:2022 - if you want it to be
+reachable externally then supply --addr :2022 for example.
- -h, --help help for settier
+Note that the default of --vfs-cache-mode off is fine for the rclone
+sftp backend, but it may not be with other SFTP clients.
-See the global flags page for global options not listed here.
+If --stdio is specified, rclone will serve SFTP over stdio, which can be
+used with sshd via ~/.ssh/authorized_keys, for example:
-SEE ALSO
+ restrict,command="rclone serve sftp --stdio ./photos" ssh-rsa ...
-- rclone - Show help for rclone commands, flags and backends.
+On the client you need to set --transfers 1 when using --stdio.
+Otherwise multiple instances of the rclone server are started by OpenSSH
+which can lead to "corrupted on transfer" errors. This is the case
+because the client chooses indiscriminately which server to send
+commands to while the servers all have different views of the state of
+the filing system.
-rclone test
+The "restrict" in authorized_keys prevents SHA1SUMs and MD5SUMs from
+being used. Omitting "restrict" and using --sftp-path-override to enable
+checksumming is possible but less secure and you could use the SFTP
+server provided by OpenSSH in this case.
-Run a test command
+VFS - Virtual File System
-Synopsis
+This command uses the VFS layer. This adapts the cloud storage objects
+that rclone uses into something which looks much more like a disk filing
+system.
-Rclone test is used to run test commands.
+Cloud storage objects have lots of properties which aren't like disk
+files - you can't extend them or write to the middle of them, so the VFS
+layer has to deal with that. Because there is no one right way of doing
+this there are various options explained below.
-Select which test command you want with the subcommand, eg
+The VFS layer also implements a directory cache - this caches info about
+files and directories (but not the data) in memory.
- rclone test memory remote:
+VFS Directory Cache
-Each subcommand has its own options which you can see in their help.
+Using the --dir-cache-time flag, you can control how long a directory
+should be considered up to date and not refreshed from the backend.
+Changes made through the VFS will appear immediately or invalidate the
+cache.
-NB Be careful running these commands, they may do strange things so
-reading their documentation first is recommended.
+ --dir-cache-time duration Time to cache directory entries for (default 5m0s)
+ --poll-interval duration Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable (default 1m0s)
-Options
+However, changes made directly on the cloud storage by the web interface
+or a different copy of rclone will only be picked up once the directory
+cache expires if the backend configured does not support polling for
+changes. If the backend supports polling, changes will be picked up
+within the polling interval.
- -h, --help help for test
+You can send a SIGHUP signal to rclone for it to flush all directory
+caches, regardless of how old they are. Assuming only one rclone
+instance is running, you can reset the cache like this:
-See the global flags page for global options not listed here.
+ kill -SIGHUP $(pidof rclone)
-SEE ALSO
+If you configure rclone with a remote control then you can use rclone rc
+to flush the whole directory cache:
-- rclone - Show help for rclone commands, flags and backends.
-- rclone test changenotify - Log any change notify requests for the
- remote passed in.
-- rclone test histogram - Makes a histogram of file name characters.
-- rclone test info - Discovers file name or other limitations for
- paths.
-- rclone test makefile - Make files with random contents of the size
- given
-- rclone test makefiles - Make a random file hierarchy in a directory
-- rclone test memory - Load all the objects at remote:path into memory
- and report memory stats.
+ rclone rc vfs/forget
-rclone test changenotify
+Or individual files or directories:
-Log any change notify requests for the remote passed in.
+ rclone rc vfs/forget file=path/to/file dir=path/to/dir
- rclone test changenotify remote: [flags]
+VFS File Buffering
-Options
+The --buffer-size flag determines the amount of memory, that will be
+used to buffer data in advance.
- -h, --help help for changenotify
- --poll-interval Duration Time to wait between polling for changes (default 10s)
+Each open file will try to keep the specified amount of data in memory
+at all times. The buffered data is bound to one open file and won't be
+shared.
-See the global flags page for global options not listed here.
+This flag is a upper limit for the used memory per open file. The buffer
+will only use memory for data that is downloaded but not not yet read.
+If the buffer is empty, only a small amount of memory will be used.
-SEE ALSO
+The maximum memory used by rclone for buffering can be up to
+--buffer-size * open files.
-- rclone test - Run a test command
+VFS File Caching
-rclone test histogram
+These flags control the VFS file caching options. File caching is
+necessary to make the VFS layer appear compatible with a normal file
+system. It can be disabled at the cost of some compatibility.
-Makes a histogram of file name characters.
+For example you'll need to enable VFS caching if you want to read and
+write simultaneously to a file. See below for more details.
-Synopsis
+Note that the VFS cache is separate from the cache backend and you may
+find that you need one or the other or both.
-This command outputs JSON which shows the histogram of characters used
-in filenames in the remote:path specified.
+ --cache-dir string Directory rclone will use for caching.
+ --vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off)
+ --vfs-cache-max-age duration Max time since last access of objects in the cache (default 1h0m0s)
+ --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off)
+ --vfs-cache-min-free-space SizeSuffix Target minimum free space on the disk containing the cache (default off)
+ --vfs-cache-poll-interval duration Interval to poll the cache for stale objects (default 1m0s)
+ --vfs-write-back duration Time to writeback files after last use when using cache (default 5s)
-The data doesn't contain any identifying information but is useful for
-the rclone developers when developing filename compression.
+If run with -vv rclone will print the location of the file cache. The
+files are stored in the user cache file area which is OS dependent but
+can be controlled with --cache-dir or setting the appropriate
+environment variable.
- rclone test histogram [remote:path] [flags]
+The cache has 4 different modes selected by --vfs-cache-mode. The higher
+the cache mode the more compatible rclone becomes at the cost of using
+disk space.
-Options
+Note that files are written back to the remote only when they are closed
+and if they haven't been accessed for --vfs-write-back seconds. If
+rclone is quit or dies with files that haven't been uploaded, these will
+be uploaded next time rclone is run with the same flags.
- -h, --help help for histogram
+If using --vfs-cache-max-size or --vfs-cache-min-free-size note that the
+cache may exceed these quotas for two reasons. Firstly because it is
+only checked every --vfs-cache-poll-interval. Secondly because open
+files cannot be evicted from the cache. When --vfs-cache-max-size or
+--vfs-cache-min-free-size is exceeded, rclone will attempt to evict the
+least accessed files from the cache first. rclone will start with files
+that haven't been accessed for the longest. This cache flushing strategy
+is efficient and more relevant files are likely to remain cached.
-See the global flags page for global options not listed here.
+The --vfs-cache-max-age will evict files from the cache after the set
+time since last access has passed. The default value of 1 hour will
+start evicting files from cache that haven't been accessed for 1 hour.
+When a cached file is accessed the 1 hour timer is reset to 0 and will
+wait for 1 more hour before evicting. Specify the time with standard
+notation, s, m, h, d, w .
-SEE ALSO
+You should not run two copies of rclone using the same VFS cache with
+the same or overlapping remotes if using --vfs-cache-mode > off. This
+can potentially cause data corruption if you do. You can work around
+this by giving each rclone its own cache hierarchy with --cache-dir. You
+don't need to worry about this if the remotes in use don't overlap.
-- rclone test - Run a test command
+--vfs-cache-mode off
-rclone test info
+In this mode (the default) the cache will read directly from the remote
+and write directly to the remote without caching anything on disk.
-Discovers file name or other limitations for paths.
+This will mean some operations are not possible
-Synopsis
+- Files can't be opened for both read AND write
+- Files opened for write can't be seeked
+- Existing files opened for write must have O_TRUNC set
+- Files open for read with O_TRUNC will be opened write only
+- Files open for write only will behave as if O_TRUNC was supplied
+- Open modes O_APPEND, O_TRUNC are ignored
+- If an upload fails it can't be retried
-rclone info discovers what filenames and upload methods are possible to
-write to the paths passed in and how long they can be. It can take some
-time. It will write test files into the remote:path passed in. It
-outputs a bit of go code for each one.
+--vfs-cache-mode minimal
-NB this can create undeletable files and other hazards - use with care
+This is very similar to "off" except that files opened for read AND
+write will be buffered to disk. This means that files opened for write
+will be a lot more compatible, but uses the minimal disk space.
- rclone test info [remote:path]+ [flags]
+These operations are not possible
-Options
+- Files opened for write only can't be seeked
+- Existing files opened for write must have O_TRUNC set
+- Files opened for write only will ignore O_APPEND, O_TRUNC
+- If an upload fails it can't be retried
- --all Run all tests
- --check-base32768 Check can store all possible base32768 characters
- --check-control Check control characters
- --check-length Check max filename length
- --check-normalization Check UTF-8 Normalization
- --check-streaming Check uploads with indeterminate file size
- -h, --help help for info
- --upload-wait Duration Wait after writing a file (default 0s)
- --write-json string Write results to file
+--vfs-cache-mode writes
-See the global flags page for global options not listed here.
+In this mode files opened for read only are still read directly from the
+remote, write only and read/write files are buffered to disk first.
-SEE ALSO
+This mode should support all normal file system operations.
-- rclone test - Run a test command
+If an upload fails it will be retried at exponentially increasing
+intervals up to 1 minute.
-rclone test makefile
+--vfs-cache-mode full
-Make files with random contents of the size given
+In this mode all reads and writes are buffered to and from disk. When
+data is read from the remote this is buffered to disk as well.
- rclone test makefile []+ [flags]
+In this mode the files in the cache will be sparse files and rclone will
+keep track of which bits of the files it has downloaded.
-Options
+So if an application only reads the starts of each file, then rclone
+will only buffer the start of the file. These files will appear to be
+their full size in the cache, but they will be sparse files with only
+the data that has been downloaded present in them.
- --ascii Fill files with random ASCII printable bytes only
- --chargen Fill files with a ASCII chargen pattern
- -h, --help help for makefile
- --pattern Fill files with a periodic pattern
- --seed int Seed for the random number generator (0 for random) (default 1)
- --sparse Make the files sparse (appear to be filled with ASCII 0x00)
- --zero Fill files with ASCII 0x00
+This mode should support all normal file system operations and is
+otherwise identical to --vfs-cache-mode writes.
-See the global flags page for global options not listed here.
+When reading a file rclone will read --buffer-size plus --vfs-read-ahead
+bytes ahead. The --buffer-size is buffered in memory whereas the
+--vfs-read-ahead is buffered on disk.
-SEE ALSO
+When using this mode it is recommended that --buffer-size is not set too
+large and --vfs-read-ahead is set large if required.
-- rclone test - Run a test command
+IMPORTANT not all file systems support sparse files. In particular
+FAT/exFAT do not. Rclone will perform very badly if the cache directory
+is on a filesystem which doesn't support sparse files and it will log an
+ERROR message if one is detected.
-rclone test makefiles
+Fingerprinting
-Make a random file hierarchy in a directory
+Various parts of the VFS use fingerprinting to see if a local file copy
+has changed relative to a remote file. Fingerprints are made from:
- rclone test makefiles [flags]
+- size
+- modification time
+- hash
-Options
+where available on an object.
- --ascii Fill files with random ASCII printable bytes only
- --chargen Fill files with a ASCII chargen pattern
- --files int Number of files to create (default 1000)
- --files-per-directory int Average number of files per directory (default 10)
- -h, --help help for makefiles
- --max-depth int Maximum depth of directory hierarchy (default 10)
- --max-file-size SizeSuffix Maximum size of files to create (default 100)
- --max-name-length int Maximum size of file names (default 12)
- --min-file-size SizeSuffix Minimum size of file to create
- --min-name-length int Minimum size of file names (default 4)
- --pattern Fill files with a periodic pattern
- --seed int Seed for the random number generator (0 for random) (default 1)
- --sparse Make the files sparse (appear to be filled with ASCII 0x00)
- --zero Fill files with ASCII 0x00
+On some backends some of these attributes are slow to read (they take an
+extra API call per object, or extra work per object).
-See the global flags page for global options not listed here.
+For example hash is slow with the local and sftp backends as they have
+to read the entire file and hash it, and modtime is slow with the s3,
+swift, ftp and qinqstor backends because they need to do an extra API
+call to fetch it.
-SEE ALSO
+If you use the --vfs-fast-fingerprint flag then rclone will not include
+the slow operations in the fingerprint. This makes the fingerprinting
+less accurate but much faster and will improve the opening time of
+cached files.
-- rclone test - Run a test command
+If you are running a vfs cache over local, s3 or swift backends then
+using this flag is recommended.
-rclone test memory
+Note that if you change the value of this flag, the fingerprints of the
+files in the cache may be invalidated and the files will need to be
+downloaded again.
-Load all the objects at remote:path into memory and report memory stats.
+VFS Chunked Reading
- rclone test memory remote:path [flags]
+When rclone reads files from a remote it reads them in chunks. This
+means that rather than requesting the whole file rclone reads the chunk
+specified. This can reduce the used download quota for some remotes by
+requesting only chunks from the remote that are actually read, at the
+cost of an increased number of requests.
-Options
+These flags control the chunking:
- -h, --help help for memory
+ --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M)
+ --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off)
-See the global flags page for global options not listed here.
+Rclone will start reading a chunk of size --vfs-read-chunk-size, and
+then double the size for each read. When --vfs-read-chunk-size-limit is
+specified, and greater than --vfs-read-chunk-size, the chunk size for
+each open file will get doubled only until the specified value is
+reached. If the value is "off", which is the default, the limit is
+disabled and the chunk size will grow indefinitely.
-SEE ALSO
+With --vfs-read-chunk-size 100M and --vfs-read-chunk-size-limit 0 the
+following parts will be downloaded: 0-100M, 100M-200M, 200M-300M,
+300M-400M and so on. When --vfs-read-chunk-size-limit 500M is specified,
+the result would be 0-100M, 100M-300M, 300M-700M, 700M-1200M,
+1200M-1700M and so on.
-- rclone test - Run a test command
+Setting --vfs-read-chunk-size to 0 or "off" disables chunked reading.
-rclone touch
+VFS Performance
-Create new file or change file modification time.
+These flags may be used to enable/disable features of the VFS for
+performance or other reasons. See also the chunked reading feature.
-Synopsis
+In particular S3 and Swift benefit hugely from the --no-modtime flag (or
+use --use-server-modtime for a slightly different effect) as each read
+of the modification time takes a transaction.
-Set the modification time on file(s) as specified by remote:path to have
-the current time.
+ --no-checksum Don't compare checksums on up/download.
+ --no-modtime Don't read/write the modification time (can speed things up).
+ --no-seek Don't allow seeking in files.
+ --read-only Only allow read-only access.
-If remote:path does not exist then a zero sized file will be created,
-unless --no-create or --recursive is provided.
+Sometimes rclone is delivered reads or writes out of order. Rather than
+seeking rclone will wait a short time for the in sequence read or write
+to come in. These flags only come into effect when not using an on disk
+cache file.
-If --recursive is used then recursively sets the modification time on
-all existing files that is found under the path. Filters are supported,
-and you can test with the --dry-run or the --interactive/-i flag.
+ --vfs-read-wait duration Time to wait for in-sequence read before seeking (default 20ms)
+ --vfs-write-wait duration Time to wait for in-sequence write before giving error (default 1s)
-If --timestamp is used then sets the modification time to that time
-instead of the current time. Times may be specified as one of:
+When using VFS write caching (--vfs-cache-mode with value writes or
+full), the global flag --transfers can be set to adjust the number of
+parallel uploads of modified files from the cache (the related global
+flag --checkers has no effect on the VFS).
-- 'YYMMDD' - e.g. 17.10.30
-- 'YYYY-MM-DDTHH:MM:SS' - e.g. 2006-01-02T15:04:05
-- 'YYYY-MM-DDTHH:MM:SS.SSS' - e.g. 2006-01-02T15:04:05.123456789
+ --transfers int Number of file transfers to run in parallel (default 4)
-Note that value of --timestamp is in UTC. If you want local time then
-add the --localtime flag.
+VFS Case Sensitivity
- rclone touch remote:path [flags]
+Linux file systems are case-sensitive: two files can differ only by
+case, and the exact case must be used when opening a file.
-Options
+File systems in modern Windows are case-insensitive but case-preserving:
+although existing files can be opened using any case, the exact case
+used to create the file is preserved and available for programs to
+query. It is not allowed for two files in the same directory to differ
+only by case.
- -h, --help help for touch
- --localtime Use localtime for timestamp, not UTC
- -C, --no-create Do not create the file if it does not exist (implied with --recursive)
- -R, --recursive Recursively touch all files
- -t, --timestamp string Use specified time instead of the current time of day
+Usually file systems on macOS are case-insensitive. It is possible to
+make macOS file systems case-sensitive but that is not the default.
-Important Options
+The --vfs-case-insensitive VFS flag controls how rclone handles these
+two cases. If its value is "false", rclone passes file names to the
+remote as-is. If the flag is "true" (or appears without a value on the
+command line), rclone may perform a "fixup" as explained below.
-Important flags useful for most commands.
+The user may specify a file name to open/delete/rename/etc with a case
+different than what is stored on the remote. If an argument refers to an
+existing file with exactly the same name, then the case of the existing
+file on the disk will be used. However, if a file name with exactly the
+same name is not found but a name differing only by case exists, rclone
+will transparently fixup the name. This fixup happens only when an
+existing file is requested. Case sensitivity of file names created anew
+by rclone is controlled by the underlying remote.
- -n, --dry-run Do a trial run with no permanent changes
- -i, --interactive Enable interactive mode
- -v, --verbose count Print lots more stuff (repeat for more)
+Note that case sensitivity of the operating system running rclone (the
+target) may differ from case sensitivity of a file system presented by
+rclone (the source). The flag controls whether "fixup" is performed to
+satisfy the target.
+
+If the flag is not provided on the command line, then its default value
+depends on the operating system where rclone runs: "true" on Windows and
+macOS, "false" otherwise. If the flag is provided without a value, then
+it is "true".
+
+VFS Disk Options
+
+This flag allows you to manually set the statistics about the filing
+system. It can be useful when those statistics cannot be read correctly
+automatically.
+
+ --vfs-disk-space-total-size Manually set the total disk space size (example: 256G, default: -1)
+
+Alternate report of used bytes
+
+Some backends, most notably S3, do not report the amount of bytes used.
+If you need this information to be available when running df on the
+filesystem, then pass the flag --vfs-used-is-size to rclone. With this
+flag set, instead of relying on the backend to report this information,
+rclone will scan the whole remote similar to rclone size and compute the
+total used space itself.
+
+WARNING. Contrary to rclone size, this flag ignores filters so that the
+result is accurate. However, this is very inefficient and may cost lots
+of API calls resulting in extra charges. Use it as a last resort and
+only with caching.
+
+Auth Proxy
+
+If you supply the parameter --auth-proxy /path/to/program then rclone
+will use that program to generate backends on the fly which then are
+used to authenticate incoming requests. This uses a simple JSON based
+protocol with input on STDIN and output on STDOUT.
+
+PLEASE NOTE: --auth-proxy and --authorized-keys cannot be used together,
+if --auth-proxy is set the authorized keys option will be ignored.
+
+There is an example program bin/test_proxy.py in the rclone source code.
+
+The program's job is to take a user and pass on the input and turn those
+into the config for a backend on STDOUT in JSON format. This config will
+have any default parameters for the backend added, but it won't use
+configuration from environment variables or command line options - it is
+the job of the proxy program to make a complete config.
+
+This config generated must have this extra parameter - _root - root to
+use for the backend
+
+And it may have this parameter - _obscure - comma separated strings for
+parameters to obscure
+
+If password authentication was used by the client, input to the proxy
+process (on STDIN) would look similar to this:
+
+ {
+ "user": "me",
+ "pass": "mypassword"
+ }
+
+If public-key authentication was used by the client, input to the proxy
+process (on STDIN) would look similar to this:
+
+ {
+ "user": "me",
+ "public_key": "AAAAB3NzaC1yc2EAAAADAQABAAABAQDuwESFdAe14hVS6omeyX7edc...JQdf"
+ }
+
+And as an example return this on STDOUT
+
+ {
+ "type": "sftp",
+ "_root": "",
+ "_obscure": "pass",
+ "user": "me",
+ "pass": "mypassword",
+ "host": "sftp.example.com"
+ }
+
+This would mean that an SFTP backend would be created on the fly for the
+user and pass/public_key returned in the output to the host given. Note
+that since _obscure is set to pass, rclone will obscure the pass
+parameter before creating the backend (which is required for sftp
+backends).
+
+The program can manipulate the supplied user in any way, for example to
+make proxy to many different sftp backends, you could make the user be
+user@example.com and then set the host to example.com in the output and
+the user to user. For security you'd probably want to restrict the host
+to a limited list.
+
+Note that an internal cache is keyed on user so only use that for
+configuration, don't use pass or public_key. This also means that if a
+user's password or public-key is changed the cache will need to expire
+(which takes 5 mins) before it takes effect.
+
+This can be used to build general purpose proxies to any kind of backend
+that rclone supports.
+
+ rclone serve sftp remote:path [flags]
+
+Options
+
+ --addr string IPaddress:Port or :Port to bind server to (default "localhost:2022")
+ --auth-proxy string A program to use to create the backend from the auth
+ --authorized-keys string Authorized keys file (default "~/.ssh/authorized_keys")
+ --dir-cache-time Duration Time to cache directory entries for (default 5m0s)
+ --dir-perms FileMode Directory permissions (default 0777)
+ --file-perms FileMode File permissions (default 0666)
+ --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000)
+ -h, --help help for sftp
+ --key stringArray SSH private host key file (Can be multi-valued, leave blank to auto generate)
+ --no-auth Allow connections with no authentication if set
+ --no-checksum Don't compare checksums on up/download
+ --no-modtime Don't read/write the modification time (can speed things up)
+ --no-seek Don't allow seeking in files
+ --pass string Password for authentication
+ --poll-interval Duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s)
+ --read-only Only allow read-only access
+ --stdio Run an sftp server on stdin/stdout
+ --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000)
+ --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2)
+ --user string User name for authentication
+ --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s)
+ --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off)
+ --vfs-cache-min-free-space SizeSuffix Target minimum free space on the disk containing the cache (default off)
+ --vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off)
+ --vfs-cache-poll-interval Duration Interval to poll the cache for stale objects (default 1m0s)
+ --vfs-case-insensitive If a file name not found, find a case insensitive match
+ --vfs-disk-space-total-size SizeSuffix Specify the total space of disk (default off)
+ --vfs-fast-fingerprint Use fast (less accurate) fingerprints for change detection
+ --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full
+ --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi)
+ --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off)
+ --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms)
+ --vfs-refresh Refreshes the directory cache recursively on start
+ --vfs-used-is-size rclone size Use the rclone size algorithm for Used size
+ --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s)
+ --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s)
Filter Options
@@ -9765,16529 +9909,18246 @@ Flags for filtering directory listings.
--min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off)
--min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off)
-Listing Options
-
-Flags for listing directories.
-
- --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z)
- --fast-list Use recursive list if available; uses more memory but fewer transactions
-
See the global flags page for global options not listed here.
SEE ALSO
-- rclone - Show help for rclone commands, flags and backends.
+- rclone serve - Serve a remote over a protocol.
-rclone tree
+rclone serve webdav
-List the contents of the remote in a tree like fashion.
+Serve remote:path over WebDAV.
Synopsis
-rclone tree lists the contents of a remote in a similar way to the unix
-tree command.
+Run a basic WebDAV server to serve a remote over HTTP via the WebDAV
+protocol. This can be viewed with a WebDAV client, through a web
+browser, or you can make a remote of type WebDAV to read and write it.
-For example
+WebDAV options
- $ rclone tree remote:path
- /
- ├── file1
- ├── file2
- ├── file3
- └── subdir
- ├── file4
- └── file5
-
- 1 directories, 5 files
-
-You can use any of the filtering options with the tree command (e.g.
---include and --exclude. You can also use --fast-list.
+--etag-hash
-The tree command has many options for controlling the listing which are
-compatible with the tree command, for example you can include file sizes
-with --size. Note that not all of them have short options as they
-conflict with rclone's short options.
+This controls the ETag header. Without this flag the ETag will be based
+on the ModTime and Size of the object.
-For a more interactive navigation of the remote see the ncdu command.
+If this flag is set to "auto" then rclone will choose the first
+supported hash on the backend or you can use a named hash such as "MD5"
+or "SHA-1". Use the hashsum command to see the full list.
- rclone tree remote:path [flags]
+Access WebDAV on Windows
-Options
+WebDAV shared folder can be mapped as a drive on Windows, however the
+default settings prevent it. Windows will fail to connect to the server
+using insecure Basic authentication. It will not even display any login
+dialog. Windows requires SSL / HTTPS connection to be used with Basic.
+If you try to connect via Add Network Location Wizard you will get the
+following error: "The folder you entered does not appear to be valid.
+Please choose another". However, you still can connect if you set the
+following registry key on a client machine: HKEY_LOCAL_MACHINEto 2. The
+BasicAuthLevel can be set to the following values: 0 - Basic
+authentication disabled 1 - Basic authentication enabled for SSL
+connections only 2 - Basic authentication enabled for SSL connections
+and for non-SSL connections If required, increase the
+FileSizeLimitInBytes to a higher value. Navigate to the Services
+interface, then restart the WebClient service.
- -a, --all All files are listed (list . files too)
- -d, --dirs-only List directories only
- --dirsfirst List directories before files (-U disables)
- --full-path Print the full path prefix for each file
- -h, --help help for tree
- --level int Descend only level directories deep
- -D, --modtime Print the date of last modification.
- --noindent Don't print indentation lines
- --noreport Turn off file/directory count at end of tree listing
- -o, --output string Output to file instead of stdout
- -p, --protections Print the protections for each file.
- -Q, --quote Quote filenames with double quotes.
- -s, --size Print the size in bytes of each file.
- --sort string Select sort: name,version,size,mtime,ctime
- --sort-ctime Sort files by last status change time
- -t, --sort-modtime Sort files by last modification time
- -r, --sort-reverse Reverse the order of the sort
- -U, --unsorted Leave files unsorted
- --version Sort files alphanumerically by version
+Access Office applications on WebDAV
-Filter Options
+Navigate to following registry HKEY_CURRENT_USER[14.0/15.0/16.0] Create
+a new DWORD BasicAuthLevel with value 2. 0 - Basic authentication
+disabled 1 - Basic authentication enabled for SSL connections only 2 -
+Basic authentication enabled for SSL and for non-SSL connections
-Flags for filtering directory listings.
+https://learn.microsoft.com/en-us/office/troubleshoot/powerpoint/office-opens-blank-from-sharepoint
- --delete-excluded Delete files on dest excluded from sync
- --exclude stringArray Exclude files matching pattern
- --exclude-from stringArray Read file exclude patterns from file (use - to read from stdin)
- --exclude-if-present stringArray Exclude directories if filename is present
- --files-from stringArray Read list of source-file names from file (use - to read from stdin)
- --files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin)
- -f, --filter stringArray Add a file filtering rule
- --filter-from stringArray Read file filtering patterns from a file (use - to read from stdin)
- --ignore-case Ignore case in filters (case insensitive)
- --include stringArray Include files matching pattern
- --include-from stringArray Read file include patterns from file (use - to read from stdin)
- --max-age Duration Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off)
- --max-depth int If set limits the recursion depth to this (default -1)
- --max-size SizeSuffix Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off)
- --metadata-exclude stringArray Exclude metadatas matching pattern
- --metadata-exclude-from stringArray Read metadata exclude patterns from file (use - to read from stdin)
- --metadata-filter stringArray Add a metadata filtering rule
- --metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin)
- --metadata-include stringArray Include metadatas matching pattern
- --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin)
- --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off)
- --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off)
+Server options
-Listing Options
+Use --addr to specify which IP address and port the server should listen
+on, eg --addr 1.2.3.4:8000 or --addr :8080 to listen to all IPs. By
+default it only listens on localhost. You can use port :0 to let the OS
+choose an available port.
-Flags for listing directories.
+If you set --addr to listen on a public or LAN accessible IP address
+then using Authentication is advised - see the next section for info.
- --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z)
- --fast-list Use recursive list if available; uses more memory but fewer transactions
+You can use a unix socket by setting the url to unix:///path/to/socket
+or just by using an absolute path name. Note that unix sockets bypass
+the authentication - this is expected to be done with file system
+permissions.
-See the global flags page for global options not listed here.
+--addr may be repeated to listen on multiple IPs/ports/sockets.
-SEE ALSO
+--server-read-timeout and --server-write-timeout can be used to control
+the timeouts on the server. Note that this is the total time for a
+transfer.
-- rclone - Show help for rclone commands, flags and backends.
+--max-header-bytes controls the maximum number of bytes the server will
+accept in the HTTP header.
-Copying single files
+--baseurl controls the URL prefix that rclone serves from. By default
+rclone will serve from the root. If you used --baseurl "/rclone" then
+rclone would serve from a URL starting with "/rclone/". This is useful
+if you wish to proxy rclone serve. Rclone automatically inserts leading
+and trailing "/" on --baseurl, so --baseurl "rclone",
+--baseurl "/rclone" and --baseurl "/rclone/" are all treated
+identically.
-rclone normally syncs or copies directories. However, if the source
-remote points to a file, rclone will just copy that file. The
-destination remote must point to a directory - rclone will give the
-error
-Failed to create file system for "remote:file": is a file not a directory
-if it isn't.
+TLS (SSL)
-For example, suppose you have a remote with a file in called test.jpg,
-then you could copy just that file like this
+By default this will serve over http. If you want you can serve over
+https. You will need to supply the --cert and --key flags. If you wish
+to do client side certificate validation then you will need to supply
+--client-ca also.
- rclone copy remote:test.jpg /tmp/download
+--cert should be a either a PEM encoded certificate or a concatenation
+of that with the CA certificate. --key should be the PEM encoded private
+key and --client-ca should be the PEM encoded client certificate
+authority certificate.
-The file test.jpg will be placed inside /tmp/download.
+--min-tls-version is minimum TLS version that is acceptable. Valid
+values are "tls1.0", "tls1.1", "tls1.2" and "tls1.3" (default "tls1.0").
-This is equivalent to specifying
+Template
- rclone copy --files-from /tmp/files remote: /tmp/download
+--template allows a user to specify a custom markup template for HTTP
+and WebDAV serve functions. The server exports the following markup to
+be used within the template to server pages:
-Where /tmp/files contains the single line
+ -----------------------------------------------------------------------
+ Parameter Description
+ ----------------------------------- -----------------------------------
+ .Name The full path of a file/directory.
- test.jpg
+ .Title Directory listing of .Name
-It is recommended to use copy when copying individual files, not sync.
-They have pretty much the same effect but copy will use a lot less
-memory.
+ .Sort The current sort used. This is
+ changeable via ?sort= parameter
-Syntax of remote paths
+ Sort Options:
+ namedirfirst,name,size,time
+ (default namedirfirst)
-The syntax of the paths passed to the rclone command are as follows.
+ .Order The current ordering used. This is
+ changeable via ?order= parameter
-/path/to/dir
+ Order Options: asc,desc (default
+ asc)
-This refers to the local file system.
+ .Query Currently unused.
-On Windows \ may be used instead of / in local paths only, non local
-paths must use /. See local filesystem documentation for more about
-Windows-specific paths.
+ .Breadcrumb Allows for creating a relative
+ navigation
-These paths needn't start with a leading / - if they don't then they
-will be relative to the current directory.
+ -- .Link The relative to the root link of
+ the Text.
-remote:path/to/dir
+ -- .Text The Name of the directory.
-This refers to a directory path/to/dir on remote: as defined in the
-config file (configured with rclone config).
+ .Entries Information about a specific
+ file/directory.
-remote:/path/to/dir
+ -- .URL The 'url' of an entry.
-On most backends this is refers to the same directory as
-remote:path/to/dir and that format should be preferred. On a very small
-number of remotes (FTP, SFTP, Dropbox for business) this will refer to a
-different directory. On these, paths without a leading / will refer to
-your "home" directory and paths with a leading / will refer to the root.
+ -- .Leaf Currently same as 'URL' but
+ intended to be 'just' the name.
-:backend:path/to/dir
+ -- .IsDir Boolean for if an entry is a
+ directory or not.
-This is an advanced form for creating remotes on the fly. backend should
-be the name or prefix of a backend (the type in the config file) and all
-the configuration for the backend should be provided on the command line
-(or in environment variables).
+ -- .Size Size in Bytes of the entry.
-Here are some examples:
+ -- .ModTime The UTC timestamp of an entry.
+ -----------------------------------------------------------------------
- rclone lsd --http-url https://pub.rclone.org :http:
+The server also makes the following functions available so that they can
+be used within the template. These functions help extend the options for
+dynamic rendering of HTML. They can be used to render HTML based on
+specific conditions.
-To list all the directories in the root of https://pub.rclone.org/.
+ -----------------------------------------------------------------------
+ Function Description
+ ----------------------------------- -----------------------------------
+ afterEpoch Returns the time since the epoch
+ for the given time.
- rclone lsf --http-url https://example.com :http:path/to/dir
+ contains Checks whether a given substring is
+ present or not in a given string.
-To list files and directories in https://example.com/path/to/dir/
+ hasPrefix Checks whether the given string
+ begins with the specified prefix.
- rclone copy --http-url https://example.com :http:path/to/dir /tmp/dir
+ hasSuffix Checks whether the given string end
+ with the specified suffix.
+ -----------------------------------------------------------------------
-To copy files and directories in https://example.com/path/to/dir to
-/tmp/dir.
+Authentication
- rclone copy --sftp-host example.com :sftp:path/to/dir /tmp/dir
+By default this will serve files without needing a login.
-To copy files and directories from example.com in the relative directory
-path/to/dir to /tmp/dir using sftp.
+You can either use an htpasswd file which can take lots of users, or set
+a single username and password with the --user and --pass flags.
-Connection strings
+If no static users are configured by either of the above methods, and
+client certificates are required by the --client-ca flag passed to the
+server, the client certificate common name will be considered as the
+username.
-The above examples can also be written using a connection string syntax,
-so instead of providing the arguments as command line parameters
---http-url https://pub.rclone.org they are provided as part of the
-remote specification as a kind of connection string.
+Use --htpasswd /path/to/htpasswd to provide an htpasswd file. This is in
+standard apache format and supports MD5, SHA1 and BCrypt for basic
+authentication. Bcrypt is recommended.
- rclone lsd ":http,url='https://pub.rclone.org':"
- rclone lsf ":http,url='https://example.com':path/to/dir"
- rclone copy ":http,url='https://example.com':path/to/dir" /tmp/dir
- rclone copy :sftp,host=example.com:path/to/dir /tmp/dir
+To create an htpasswd file:
-These can apply to modify existing remotes as well as create new remotes
-with the on the fly syntax. This example is equivalent to adding the
---drive-shared-with-me parameter to the remote gdrive:.
+ touch htpasswd
+ htpasswd -B htpasswd user
+ htpasswd -B htpasswd anotherUser
- rclone lsf "gdrive,shared_with_me:path/to/dir"
+The password file can be updated while rclone is running.
-The major advantage to using the connection string style syntax is that
-it only applies to the remote, not to all the remotes of that type of
-the command line. A common confusion is this attempt to copy a file
-shared on google drive to the normal drive which does not work because
-the --drive-shared-with-me flag applies to both the source and the
-destination.
+Use --realm to set the authentication realm.
- rclone copy --drive-shared-with-me gdrive:shared-file.txt gdrive:
+Use --salt to change the password hashing salt from the default. ## VFS
+- Virtual File System
-However using the connection string syntax, this does work.
+This command uses the VFS layer. This adapts the cloud storage objects
+that rclone uses into something which looks much more like a disk filing
+system.
- rclone copy "gdrive,shared_with_me:shared-file.txt" gdrive:
+Cloud storage objects have lots of properties which aren't like disk
+files - you can't extend them or write to the middle of them, so the VFS
+layer has to deal with that. Because there is no one right way of doing
+this there are various options explained below.
-Note that the connection string only affects the options of the
-immediate backend. If for example gdriveCrypt is a crypt based on
-gdrive, then the following command will not work as intended, because
-shared_with_me is ignored by the crypt backend:
+The VFS layer also implements a directory cache - this caches info about
+files and directories (but not the data) in memory.
- rclone copy "gdriveCrypt,shared_with_me:shared-file.txt" gdriveCrypt:
+VFS Directory Cache
-The connection strings have the following syntax
+Using the --dir-cache-time flag, you can control how long a directory
+should be considered up to date and not refreshed from the backend.
+Changes made through the VFS will appear immediately or invalidate the
+cache.
- remote,parameter=value,parameter2=value2:path/to/dir
- :backend,parameter=value,parameter2=value2:path/to/dir
+ --dir-cache-time duration Time to cache directory entries for (default 5m0s)
+ --poll-interval duration Time to wait between polling for changes. Must be smaller than dir-cache-time. Only on supported remotes. Set to 0 to disable (default 1m0s)
-If the parameter has a : or , then it must be placed in quotes " or ',
-so
+However, changes made directly on the cloud storage by the web interface
+or a different copy of rclone will only be picked up once the directory
+cache expires if the backend configured does not support polling for
+changes. If the backend supports polling, changes will be picked up
+within the polling interval.
- remote,parameter="colon:value",parameter2="comma,value":path/to/dir
- :backend,parameter='colon:value',parameter2='comma,value':path/to/dir
+You can send a SIGHUP signal to rclone for it to flush all directory
+caches, regardless of how old they are. Assuming only one rclone
+instance is running, you can reset the cache like this:
-If a quoted value needs to include that quote, then it should be
-doubled, so
+ kill -SIGHUP $(pidof rclone)
- remote,parameter="with""quote",parameter2='with''quote':path/to/dir
+If you configure rclone with a remote control then you can use rclone rc
+to flush the whole directory cache:
-This will make parameter be with"quote and parameter2 be with'quote.
+ rclone rc vfs/forget
-If you leave off the =parameter then rclone will substitute =true which
-works very well with flags. For example, to use s3 configured in the
-environment you could use:
+Or individual files or directories:
- rclone lsd :s3,env_auth:
+ rclone rc vfs/forget file=path/to/file dir=path/to/dir
-Which is equivalent to
+VFS File Buffering
- rclone lsd :s3,env_auth=true:
+The --buffer-size flag determines the amount of memory, that will be
+used to buffer data in advance.
-Note that on the command line you might need to surround these
-connection strings with " or ' to stop the shell interpreting any
-special characters within them.
+Each open file will try to keep the specified amount of data in memory
+at all times. The buffered data is bound to one open file and won't be
+shared.
-If you are a shell master then you'll know which strings are OK and
-which aren't, but if you aren't sure then enclose them in " and use ' as
-the inside quote. This syntax works on all OSes.
+This flag is a upper limit for the used memory per open file. The buffer
+will only use memory for data that is downloaded but not not yet read.
+If the buffer is empty, only a small amount of memory will be used.
- rclone copy ":http,url='https://example.com':path/to/dir" /tmp/dir
+The maximum memory used by rclone for buffering can be up to
+--buffer-size * open files.
-On Linux/macOS some characters are still interpreted inside " strings in
-the shell (notably \ and $ and ") so if your strings contain those you
-can swap the roles of " and ' thus. (This syntax does not work on
-Windows.)
+VFS File Caching
- rclone copy ':http,url="https://example.com":path/to/dir' /tmp/dir
+These flags control the VFS file caching options. File caching is
+necessary to make the VFS layer appear compatible with a normal file
+system. It can be disabled at the cost of some compatibility.
-Connection strings, config and logging
+For example you'll need to enable VFS caching if you want to read and
+write simultaneously to a file. See below for more details.
-If you supply extra configuration to a backend by command line flag,
-environment variable or connection string then rclone will add a suffix
-based on the hash of the config to the name of the remote, eg
+Note that the VFS cache is separate from the cache backend and you may
+find that you need one or the other or both.
- rclone -vv lsf --s3-chunk-size 20M s3:
+ --cache-dir string Directory rclone will use for caching.
+ --vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off)
+ --vfs-cache-max-age duration Max time since last access of objects in the cache (default 1h0m0s)
+ --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off)
+ --vfs-cache-min-free-space SizeSuffix Target minimum free space on the disk containing the cache (default off)
+ --vfs-cache-poll-interval duration Interval to poll the cache for stale objects (default 1m0s)
+ --vfs-write-back duration Time to writeback files after last use when using cache (default 5s)
-Has the log message
+If run with -vv rclone will print the location of the file cache. The
+files are stored in the user cache file area which is OS dependent but
+can be controlled with --cache-dir or setting the appropriate
+environment variable.
- DEBUG : s3: detected overridden config - adding "{Srj1p}" suffix to name
+The cache has 4 different modes selected by --vfs-cache-mode. The higher
+the cache mode the more compatible rclone becomes at the cost of using
+disk space.
-This is so rclone can tell the modified remote apart from the unmodified
-remote when caching the backends.
+Note that files are written back to the remote only when they are closed
+and if they haven't been accessed for --vfs-write-back seconds. If
+rclone is quit or dies with files that haven't been uploaded, these will
+be uploaded next time rclone is run with the same flags.
-This should only be noticeable in the logs.
+If using --vfs-cache-max-size or --vfs-cache-min-free-size note that the
+cache may exceed these quotas for two reasons. Firstly because it is
+only checked every --vfs-cache-poll-interval. Secondly because open
+files cannot be evicted from the cache. When --vfs-cache-max-size or
+--vfs-cache-min-free-size is exceeded, rclone will attempt to evict the
+least accessed files from the cache first. rclone will start with files
+that haven't been accessed for the longest. This cache flushing strategy
+is efficient and more relevant files are likely to remain cached.
-This means that on the fly backends such as
+The --vfs-cache-max-age will evict files from the cache after the set
+time since last access has passed. The default value of 1 hour will
+start evicting files from cache that haven't been accessed for 1 hour.
+When a cached file is accessed the 1 hour timer is reset to 0 and will
+wait for 1 more hour before evicting. Specify the time with standard
+notation, s, m, h, d, w .
- rclone -vv lsf :s3,env_auth:
+You should not run two copies of rclone using the same VFS cache with
+the same or overlapping remotes if using --vfs-cache-mode > off. This
+can potentially cause data corruption if you do. You can work around
+this by giving each rclone its own cache hierarchy with --cache-dir. You
+don't need to worry about this if the remotes in use don't overlap.
-Will get their own names
+--vfs-cache-mode off
- DEBUG : :s3: detected overridden config - adding "{YTu53}" suffix to name
+In this mode (the default) the cache will read directly from the remote
+and write directly to the remote without caching anything on disk.
-Valid remote names
+This will mean some operations are not possible
-Remote names are case sensitive, and must adhere to the following rules:
-- May contain number, letter, _, -, ., +, @ and space. - May not start
-with - or space. - May not end with space.
+- Files can't be opened for both read AND write
+- Files opened for write can't be seeked
+- Existing files opened for write must have O_TRUNC set
+- Files open for read with O_TRUNC will be opened write only
+- Files open for write only will behave as if O_TRUNC was supplied
+- Open modes O_APPEND, O_TRUNC are ignored
+- If an upload fails it can't be retried
-Starting with rclone version 1.61, any Unicode numbers and letters are
-allowed, while in older versions it was limited to plain ASCII (0-9,
-A-Z, a-z). If you use the same rclone configuration from different
-shells, which may be configured with different character encoding, you
-must be cautious to use characters that are possible to write in all of
-them. This is mostly a problem on Windows, where the console
-traditionally uses a non-Unicode character set - defined by the
-so-called "code page".
+--vfs-cache-mode minimal
-Do not use single character names on Windows as it creates ambiguity
-with Windows drives' names, e.g.: remote called C is indistinguishable
-from C drive. Rclone will always assume that single letter name refers
-to a drive.
+This is very similar to "off" except that files opened for read AND
+write will be buffered to disk. This means that files opened for write
+will be a lot more compatible, but uses the minimal disk space.
-Quoting and the shell
+These operations are not possible
-When you are typing commands to your computer you are using something
-called the command line shell. This interprets various characters in an
-OS specific way.
+- Files opened for write only can't be seeked
+- Existing files opened for write must have O_TRUNC set
+- Files opened for write only will ignore O_APPEND, O_TRUNC
+- If an upload fails it can't be retried
-Here are some gotchas which may help users unfamiliar with the shell
-rules
+--vfs-cache-mode writes
-Linux / OSX
+In this mode files opened for read only are still read directly from the
+remote, write only and read/write files are buffered to disk first.
-If your names have spaces or shell metacharacters (e.g. *, ?, $, ', ",
-etc.) then you must quote them. Use single quotes ' by default.
+This mode should support all normal file system operations.
- rclone copy 'Important files?' remote:backup
+If an upload fails it will be retried at exponentially increasing
+intervals up to 1 minute.
-If you want to send a ' you will need to use ", e.g.
+--vfs-cache-mode full
- rclone copy "O'Reilly Reviews" remote:backup
+In this mode all reads and writes are buffered to and from disk. When
+data is read from the remote this is buffered to disk as well.
-The rules for quoting metacharacters are complicated and if you want the
-full details you'll have to consult the manual page for your shell.
+In this mode the files in the cache will be sparse files and rclone will
+keep track of which bits of the files it has downloaded.
-Windows
+So if an application only reads the starts of each file, then rclone
+will only buffer the start of the file. These files will appear to be
+their full size in the cache, but they will be sparse files with only
+the data that has been downloaded present in them.
-If your names have spaces in you need to put them in ", e.g.
+This mode should support all normal file system operations and is
+otherwise identical to --vfs-cache-mode writes.
- rclone copy "E:\folder name\folder name\folder name" remote:backup
+When reading a file rclone will read --buffer-size plus --vfs-read-ahead
+bytes ahead. The --buffer-size is buffered in memory whereas the
+--vfs-read-ahead is buffered on disk.
-If you are using the root directory on its own then don't quote it (see
-#464 for why), e.g.
+When using this mode it is recommended that --buffer-size is not set too
+large and --vfs-read-ahead is set large if required.
- rclone copy E:\ remote:backup
+IMPORTANT not all file systems support sparse files. In particular
+FAT/exFAT do not. Rclone will perform very badly if the cache directory
+is on a filesystem which doesn't support sparse files and it will log an
+ERROR message if one is detected.
-Copying files or directories with : in the names
+Fingerprinting
-rclone uses : to mark a remote name. This is, however, a valid filename
-component in non-Windows OSes. The remote name parser will only search
-for a : up to the first / so if you need to act on a file or directory
-like this then use the full path starting with a /, or use ./ as a
-current directory prefix.
+Various parts of the VFS use fingerprinting to see if a local file copy
+has changed relative to a remote file. Fingerprints are made from:
-So to sync a directory called sync:me to a remote called remote: use
+- size
+- modification time
+- hash
- rclone sync --interactive ./sync:me remote:path
+where available on an object.
-or
+On some backends some of these attributes are slow to read (they take an
+extra API call per object, or extra work per object).
- rclone sync --interactive /full/path/to/sync:me remote:path
+For example hash is slow with the local and sftp backends as they have
+to read the entire file and hash it, and modtime is slow with the s3,
+swift, ftp and qinqstor backends because they need to do an extra API
+call to fetch it.
-Server Side Copy
+If you use the --vfs-fast-fingerprint flag then rclone will not include
+the slow operations in the fingerprint. This makes the fingerprinting
+less accurate but much faster and will improve the opening time of
+cached files.
-Most remotes (but not all - see the overview) support server-side copy.
+If you are running a vfs cache over local, s3 or swift backends then
+using this flag is recommended.
-This means if you want to copy one folder to another then rclone won't
-download all the files and re-upload them; it will instruct the server
-to copy them in place.
+Note that if you change the value of this flag, the fingerprints of the
+files in the cache may be invalidated and the files will need to be
+downloaded again.
-Eg
+VFS Chunked Reading
- rclone copy s3:oldbucket s3:newbucket
-
-Will copy the contents of oldbucket to newbucket without downloading and
-re-uploading.
-
-Remotes which don't support server-side copy will download and re-upload
-in this case.
+When rclone reads files from a remote it reads them in chunks. This
+means that rather than requesting the whole file rclone reads the chunk
+specified. This can reduce the used download quota for some remotes by
+requesting only chunks from the remote that are actually read, at the
+cost of an increased number of requests.
-Server side copies are used with sync and copy and will be identified in
-the log when using the -v flag. The move command may also use them if
-remote doesn't support server-side move directly. This is done by
-issuing a server-side copy then a delete which is much quicker than a
-download and re-upload.
+These flags control the chunking:
-Server side copies will only be attempted if the remote names are the
-same.
+ --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128M)
+ --vfs-read-chunk-size-limit SizeSuffix Max chunk doubling size (default off)
-This can be used when scripting to make aged backups efficiently, e.g.
+Rclone will start reading a chunk of size --vfs-read-chunk-size, and
+then double the size for each read. When --vfs-read-chunk-size-limit is
+specified, and greater than --vfs-read-chunk-size, the chunk size for
+each open file will get doubled only until the specified value is
+reached. If the value is "off", which is the default, the limit is
+disabled and the chunk size will grow indefinitely.
- rclone sync --interactive remote:current-backup remote:previous-backup
- rclone sync --interactive /path/to/files remote:current-backup
+With --vfs-read-chunk-size 100M and --vfs-read-chunk-size-limit 0 the
+following parts will be downloaded: 0-100M, 100M-200M, 200M-300M,
+300M-400M and so on. When --vfs-read-chunk-size-limit 500M is specified,
+the result would be 0-100M, 100M-300M, 300M-700M, 700M-1200M,
+1200M-1700M and so on.
-Metadata support
+Setting --vfs-read-chunk-size to 0 or "off" disables chunked reading.
-Metadata is data about a file which isn't the contents of the file.
-Normally rclone only preserves the modification time and the content
-(MIME) type where possible.
+VFS Performance
-Rclone supports preserving all the available metadata on files (not
-directories) when using the --metadata or -M flag.
+These flags may be used to enable/disable features of the VFS for
+performance or other reasons. See also the chunked reading feature.
-Exactly what metadata is supported and what that support means depends
-on the backend. Backends that support metadata have a metadata section
-in their docs and are listed in the features table (Eg local, s3)
+In particular S3 and Swift benefit hugely from the --no-modtime flag (or
+use --use-server-modtime for a slightly different effect) as each read
+of the modification time takes a transaction.
-Rclone only supports a one-time sync of metadata. This means that
-metadata will be synced from the source object to the destination object
-only when the source object has changed and needs to be re-uploaded. If
-the metadata subsequently changes on the source object without changing
-the object itself then it won't be synced to the destination object.
-This is in line with the way rclone syncs Content-Type without the
---metadata flag.
+ --no-checksum Don't compare checksums on up/download.
+ --no-modtime Don't read/write the modification time (can speed things up).
+ --no-seek Don't allow seeking in files.
+ --read-only Only allow read-only access.
-Using --metadata when syncing from local to local will preserve file
-attributes such as file mode, owner, extended attributes (not Windows).
+Sometimes rclone is delivered reads or writes out of order. Rather than
+seeking rclone will wait a short time for the in sequence read or write
+to come in. These flags only come into effect when not using an on disk
+cache file.
-Note that arbitrary metadata may be added to objects using the
---metadata-set key=value flag when the object is first uploaded. This
-flag can be repeated as many times as necessary.
+ --vfs-read-wait duration Time to wait for in-sequence read before seeking (default 20ms)
+ --vfs-write-wait duration Time to wait for in-sequence write before giving error (default 1s)
-Types of metadata
+When using VFS write caching (--vfs-cache-mode with value writes or
+full), the global flag --transfers can be set to adjust the number of
+parallel uploads of modified files from the cache (the related global
+flag --checkers has no effect on the VFS).
-Metadata is divided into two type. System metadata and User metadata.
+ --transfers int Number of file transfers to run in parallel (default 4)
-Metadata which the backend uses itself is called system metadata. For
-example on the local backend the system metadata uid will store the user
-ID of the file when used on a unix based platform.
+VFS Case Sensitivity
-Arbitrary metadata is called user metadata and this can be set however
-is desired.
+Linux file systems are case-sensitive: two files can differ only by
+case, and the exact case must be used when opening a file.
-When objects are copied from backend to backend, they will attempt to
-interpret system metadata if it is supplied. Metadata may change from
-being user metadata to system metadata as objects are copied between
-different backends. For example copying an object from s3 sets the
-content-type metadata. In a backend which understands this (like
-azureblob) this will become the Content-Type of the object. In a backend
-which doesn't understand this (like the local backend) this will become
-user metadata. However should the local object be copied back to s3, the
-Content-Type will be set correctly.
+File systems in modern Windows are case-insensitive but case-preserving:
+although existing files can be opened using any case, the exact case
+used to create the file is preserved and available for programs to
+query. It is not allowed for two files in the same directory to differ
+only by case.
-Metadata framework
+Usually file systems on macOS are case-insensitive. It is possible to
+make macOS file systems case-sensitive but that is not the default.
-Rclone implements a metadata framework which can read metadata from an
-object and write it to the object when (and only when) it is being
-uploaded.
+The --vfs-case-insensitive VFS flag controls how rclone handles these
+two cases. If its value is "false", rclone passes file names to the
+remote as-is. If the flag is "true" (or appears without a value on the
+command line), rclone may perform a "fixup" as explained below.
-This metadata is stored as a dictionary with string keys and string
-values.
+The user may specify a file name to open/delete/rename/etc with a case
+different than what is stored on the remote. If an argument refers to an
+existing file with exactly the same name, then the case of the existing
+file on the disk will be used. However, if a file name with exactly the
+same name is not found but a name differing only by case exists, rclone
+will transparently fixup the name. This fixup happens only when an
+existing file is requested. Case sensitivity of file names created anew
+by rclone is controlled by the underlying remote.
-There are some limits on the names of the keys (these may be clarified
-further in the future).
+Note that case sensitivity of the operating system running rclone (the
+target) may differ from case sensitivity of a file system presented by
+rclone (the source). The flag controls whether "fixup" is performed to
+satisfy the target.
-- must be lower case
-- may be a-z 0-9 containing . - or _
-- length is backend dependent
+If the flag is not provided on the command line, then its default value
+depends on the operating system where rclone runs: "true" on Windows and
+macOS, "false" otherwise. If the flag is provided without a value, then
+it is "true".
-Each backend can provide system metadata that it understands. Some
-backends can also store arbitrary user metadata.
+VFS Disk Options
-Where possible the key names are standardized, so, for example, it is
-possible to copy object metadata from s3 to azureblob for example and
-metadata will be translated appropriately.
+This flag allows you to manually set the statistics about the filing
+system. It can be useful when those statistics cannot be read correctly
+automatically.
-Some backends have limits on the size of the metadata and rclone will
-give errors on upload if they are exceeded.
+ --vfs-disk-space-total-size Manually set the total disk space size (example: 256G, default: -1)
-Metadata preservation
+Alternate report of used bytes
-The goal of the implementation is to
+Some backends, most notably S3, do not report the amount of bytes used.
+If you need this information to be available when running df on the
+filesystem, then pass the flag --vfs-used-is-size to rclone. With this
+flag set, instead of relying on the backend to report this information,
+rclone will scan the whole remote similar to rclone size and compute the
+total used space itself.
-1. Preserve metadata if at all possible
-2. Interpret metadata if at all possible
+WARNING. Contrary to rclone size, this flag ignores filters so that the
+result is accurate. However, this is very inefficient and may cost lots
+of API calls resulting in extra charges. Use it as a last resort and
+only with caching.
-The consequences of 1 is that you can copy an S3 object to a local disk
-then back to S3 losslessly. Likewise you can copy a local file with file
-attributes and xattrs from local disk to s3 and back again losslessly.
+Auth Proxy
-The consequence of 2 is that you can copy an S3 object with metadata to
-Azureblob (say) and have the metadata appear on the Azureblob object
-also.
+If you supply the parameter --auth-proxy /path/to/program then rclone
+will use that program to generate backends on the fly which then are
+used to authenticate incoming requests. This uses a simple JSON based
+protocol with input on STDIN and output on STDOUT.
-Standard system metadata
+PLEASE NOTE: --auth-proxy and --authorized-keys cannot be used together,
+if --auth-proxy is set the authorized keys option will be ignored.
-Here is a table of standard system metadata which, if appropriate, a
-backend may implement.
+There is an example program bin/test_proxy.py in the rclone source code.
- ----------------------------------------------------------------------------------------------
- key description example
- ---------------------------------- --------------------- -------------------------------------
- mode File type and mode: 0100664
- octal, unix style
+The program's job is to take a user and pass on the input and turn those
+into the config for a backend on STDOUT in JSON format. This config will
+have any default parameters for the backend added, but it won't use
+configuration from environment variables or command line options - it is
+the job of the proxy program to make a complete config.
- uid User ID of owner: 500
- decimal number
+This config generated must have this extra parameter - _root - root to
+use for the backend
- gid Group ID of owner: 500
- decimal number
+And it may have this parameter - _obscure - comma separated strings for
+parameters to obscure
- rdev Device ID (if special 0
- file) => hexadecimal
+If password authentication was used by the client, input to the proxy
+process (on STDIN) would look similar to this:
- atime Time of last access: 2006-01-02T15:04:05.999999999Z07:00
- RFC 3339
+ {
+ "user": "me",
+ "pass": "mypassword"
+ }
- mtime Time of last 2006-01-02T15:04:05.999999999Z07:00
- modification: RFC
- 3339
+If public-key authentication was used by the client, input to the proxy
+process (on STDIN) would look similar to this:
- btime Time of file creation 2006-01-02T15:04:05.999999999Z07:00
- (birth): RFC 3339
+ {
+ "user": "me",
+ "public_key": "AAAAB3NzaC1yc2EAAAADAQABAAABAQDuwESFdAe14hVS6omeyX7edc...JQdf"
+ }
- cache-control Cache-Control header no-cache
+And as an example return this on STDOUT
- content-disposition Content-Disposition inline
- header
+ {
+ "type": "sftp",
+ "_root": "",
+ "_obscure": "pass",
+ "user": "me",
+ "pass": "mypassword",
+ "host": "sftp.example.com"
+ }
- content-encoding Content-Encoding gzip
- header
+This would mean that an SFTP backend would be created on the fly for the
+user and pass/public_key returned in the output to the host given. Note
+that since _obscure is set to pass, rclone will obscure the pass
+parameter before creating the backend (which is required for sftp
+backends).
- content-language Content-Language en-US
- header
+The program can manipulate the supplied user in any way, for example to
+make proxy to many different sftp backends, you could make the user be
+user@example.com and then set the host to example.com in the output and
+the user to user. For security you'd probably want to restrict the host
+to a limited list.
- content-type Content-Type header text/plain
- ----------------------------------------------------------------------------------------------
+Note that an internal cache is keyed on user so only use that for
+configuration, don't use pass or public_key. This also means that if a
+user's password or public-key is changed the cache will need to expire
+(which takes 5 mins) before it takes effect.
-The metadata keys mtime and content-type will take precedence if
-supplied in the metadata over reading the Content-Type or modification
-time of the source object.
+This can be used to build general purpose proxies to any kind of backend
+that rclone supports.
-Hashes are not included in system metadata as there is a well defined
-way of reading those already.
+ rclone serve webdav remote:path [flags]
Options
-Rclone has a number of options to control its behaviour.
+ --addr stringArray IPaddress:Port or :Port to bind server to (default [127.0.0.1:8080])
+ --allow-origin string Origin which cross-domain request (CORS) can be executed from
+ --auth-proxy string A program to use to create the backend from the auth
+ --baseurl string Prefix for URLs - leave blank for root
+ --cert string TLS PEM key (concatenation of certificate and CA certificate)
+ --client-ca string Client certificate authority to verify clients with
+ --dir-cache-time Duration Time to cache directory entries for (default 5m0s)
+ --dir-perms FileMode Directory permissions (default 0777)
+ --disable-dir-list Disable HTML directory list on GET request for a directory
+ --etag-hash string Which hash to use for the ETag, or auto or blank for off
+ --file-perms FileMode File permissions (default 0666)
+ --gid uint32 Override the gid field set by the filesystem (not supported on Windows) (default 1000)
+ -h, --help help for webdav
+ --htpasswd string A htpasswd file - if not provided no authentication is done
+ --key string TLS PEM Private key
+ --max-header-bytes int Maximum size of request header (default 4096)
+ --min-tls-version string Minimum TLS version that is acceptable (default "tls1.0")
+ --no-checksum Don't compare checksums on up/download
+ --no-modtime Don't read/write the modification time (can speed things up)
+ --no-seek Don't allow seeking in files
+ --pass string Password for authentication
+ --poll-interval Duration Time to wait between polling for changes, must be smaller than dir-cache-time and only on supported remotes (set 0 to disable) (default 1m0s)
+ --read-only Only allow read-only access
+ --realm string Realm for authentication
+ --salt string Password hashing salt (default "dlPL2MqE")
+ --server-read-timeout Duration Timeout for server reading data (default 1h0m0s)
+ --server-write-timeout Duration Timeout for server writing data (default 1h0m0s)
+ --template string User-specified template
+ --uid uint32 Override the uid field set by the filesystem (not supported on Windows) (default 1000)
+ --umask int Override the permission bits set by the filesystem (not supported on Windows) (default 2)
+ --user string User name for authentication
+ --vfs-cache-max-age Duration Max time since last access of objects in the cache (default 1h0m0s)
+ --vfs-cache-max-size SizeSuffix Max total size of objects in the cache (default off)
+ --vfs-cache-min-free-space SizeSuffix Target minimum free space on the disk containing the cache (default off)
+ --vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off)
+ --vfs-cache-poll-interval Duration Interval to poll the cache for stale objects (default 1m0s)
+ --vfs-case-insensitive If a file name not found, find a case insensitive match
+ --vfs-disk-space-total-size SizeSuffix Specify the total space of disk (default off)
+ --vfs-fast-fingerprint Use fast (less accurate) fingerprints for change detection
+ --vfs-read-ahead SizeSuffix Extra read ahead over --buffer-size when using cache-mode full
+ --vfs-read-chunk-size SizeSuffix Read the source objects in chunks (default 128Mi)
+ --vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached ('off' is unlimited) (default off)
+ --vfs-read-wait Duration Time to wait for in-sequence read before seeking (default 20ms)
+ --vfs-refresh Refreshes the directory cache recursively on start
+ --vfs-used-is-size rclone size Use the rclone size algorithm for Used size
+ --vfs-write-back Duration Time to writeback files after last use when using cache (default 5s)
+ --vfs-write-wait Duration Time to wait for in-sequence write before giving error (default 1s)
-Options that take parameters can have the values passed in two ways,
---option=value or --option value. However boolean (true/false) options
-behave slightly differently to the other options in that --boolean sets
-the option to true and the absence of the flag sets it to false. It is
-also possible to specify --boolean=false or --boolean=true. Note that
---boolean false is not valid - this is parsed as --boolean and the false
-is parsed as an extra command line argument for rclone.
+Filter Options
-Time or duration options
+Flags for filtering directory listings.
-TIME or DURATION options can be specified as a duration string or a time
-string.
+ --delete-excluded Delete files on dest excluded from sync
+ --exclude stringArray Exclude files matching pattern
+ --exclude-from stringArray Read file exclude patterns from file (use - to read from stdin)
+ --exclude-if-present stringArray Exclude directories if filename is present
+ --files-from stringArray Read list of source-file names from file (use - to read from stdin)
+ --files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin)
+ -f, --filter stringArray Add a file filtering rule
+ --filter-from stringArray Read file filtering patterns from a file (use - to read from stdin)
+ --ignore-case Ignore case in filters (case insensitive)
+ --include stringArray Include files matching pattern
+ --include-from stringArray Read file include patterns from file (use - to read from stdin)
+ --max-age Duration Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off)
+ --max-depth int If set limits the recursion depth to this (default -1)
+ --max-size SizeSuffix Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off)
+ --metadata-exclude stringArray Exclude metadatas matching pattern
+ --metadata-exclude-from stringArray Read metadata exclude patterns from file (use - to read from stdin)
+ --metadata-filter stringArray Add a metadata filtering rule
+ --metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin)
+ --metadata-include stringArray Include metadatas matching pattern
+ --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin)
+ --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off)
+ --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off)
-A duration string is a possibly signed sequence of decimal numbers, each
-with optional fraction and a unit suffix, such as "300ms", "-1.5h" or
-"2h45m". Default units are seconds or the following abbreviations are
-valid:
+See the global flags page for global options not listed here.
-- ms - Milliseconds
-- s - Seconds
-- m - Minutes
-- h - Hours
-- d - Days
-- w - Weeks
-- M - Months
-- y - Years
+SEE ALSO
-These can also be specified as an absolute time in the following
-formats:
+- rclone serve - Serve a remote over a protocol.
-- RFC3339 - e.g. 2006-01-02T15:04:05Z or 2006-01-02T15:04:05+07:00
-- ISO8601 Date and time, local timezone - 2006-01-02T15:04:05
-- ISO8601 Date and time, local timezone - 2006-01-02 15:04:05
-- ISO8601 Date - 2006-01-02 (YYYY-MM-DD)
+rclone settier
-Size options
+Changes storage class/tier of objects in remote.
-Options which use SIZE use KiB (multiples of 1024 bytes) by default.
-However, a suffix of B for Byte, K for KiB, M for MiB, G for GiB, T for
-TiB and P for PiB may be used. These are the binary units, e.g. 1,
-2**10, 2**20, 2**30 respectively.
+Synopsis
---backup-dir=DIR
+rclone settier changes storage tier or class at remote if supported. Few
+cloud storage services provides different storage classes on objects,
+for example AWS S3 and Glacier, Azure Blob storage - Hot, Cool and
+Archive, Google Cloud Storage, Regional Storage, Nearline, Coldline etc.
-When using sync, copy or move any files which would have been
-overwritten or deleted are moved in their original hierarchy into this
-directory.
+Note that, certain tier changes make objects not available to access
+immediately. For example tiering to archive in azure blob storage makes
+objects in frozen state, user can restore by setting tier to Hot/Cool,
+similarly S3 to Glacier makes object inaccessible.true
-If --suffix is set, then the moved files will have the suffix added to
-them. If there is a file with the same path (after the suffix has been
-added) in DIR, then it will be overwritten.
+You can use it to tier single object
-The remote in use must support server-side move or copy and you must use
-the same remote as the destination of the sync. The backup directory
-must not overlap the destination directory without it being excluded by
-a filter rule.
+ rclone settier Cool remote:path/file
-For example
+Or use rclone filters to set tier on only specific files
- rclone sync --interactive /path/to/local remote:current --backup-dir remote:old
+ rclone --include "*.txt" settier Hot remote:path/dir
-will sync /path/to/local to remote:current, but for any files which
-would have been updated or deleted will be stored in remote:old.
+Or just provide remote directory and all files in directory will be
+tiered
-If running rclone from a script you might want to use today's date as
-the directory name passed to --backup-dir to store the old files, or you
-might want to pass --suffix with today's date.
+ rclone settier tier remote:path/dir
-See --compare-dest and --copy-dest.
+ rclone settier tier remote:path [flags]
---bind string
+Options
-Local address to bind to for outgoing connections. This can be an IPv4
-address (1.2.3.4), an IPv6 address (1234::789A) or host name. If the
-host name doesn't resolve or resolves to more than one IP address it
-will give an error.
+ -h, --help help for settier
-You can use --bind 0.0.0.0 to force rclone to use IPv4 addresses and
---bind ::0 to force rclone to use IPv6 addresses.
+See the global flags page for global options not listed here.
---bwlimit=BANDWIDTH_SPEC
+SEE ALSO
-This option controls the bandwidth limit. For example
+- rclone - Show help for rclone commands, flags and backends.
- --bwlimit 10M
+rclone test
-would mean limit the upload and download bandwidth to 10 MiB/s. NB this
-is bytes per second not bits per second. To use a single limit, specify
-the desired bandwidth in KiB/s, or use a suffix B|K|M|G|T|P. The default
-is 0 which means to not limit bandwidth.
+Run a test command
-The upload and download bandwidth can be specified separately, as
---bwlimit UP:DOWN, so
+Synopsis
- --bwlimit 10M:100k
+Rclone test is used to run test commands.
-would mean limit the upload bandwidth to 10 MiB/s and the download
-bandwidth to 100 KiB/s. Either limit can be "off" meaning no limit, so
-to just limit the upload bandwidth you would use
+Select which test command you want with the subcommand, eg
- --bwlimit 10M:off
+ rclone test memory remote:
-this would limit the upload bandwidth to 10 MiB/s but the download
-bandwidth would be unlimited.
+Each subcommand has its own options which you can see in their help.
-When specified as above the bandwidth limits last for the duration of
-run of the rclone binary.
+NB Be careful running these commands, they may do strange things so
+reading their documentation first is recommended.
-It is also possible to specify a "timetable" of limits, which will cause
-certain limits to be applied at certain times. To specify a timetable,
-format your entries as
-WEEKDAY-HH:MM,BANDWIDTH WEEKDAY-HH:MM,BANDWIDTH... where: WEEKDAY is
-optional element.
+Options
-- BANDWIDTH can be a single number, e.g.100k or a pair of numbers for
- upload:download, e.g.10M:1M.
-- WEEKDAY can be written as the whole word or only using the first 3
- characters. It is optional.
-- HH:MM is an hour from 00:00 to 23:59.
+ -h, --help help for test
-An example of a typical timetable to avoid link saturation during
-daytime working hours could be:
+See the global flags page for global options not listed here.
---bwlimit "08:00,512k 12:00,10M 13:00,512k 18:00,30M 23:00,off"
+SEE ALSO
-In this example, the transfer bandwidth will be set to 512 KiB/s at 8am
-every day. At noon, it will rise to 10 MiB/s, and drop back to 512
-KiB/sec at 1pm. At 6pm, the bandwidth limit will be set to 30 MiB/s, and
-at 11pm it will be completely disabled (full speed). Anything between
-11pm and 8am will remain unlimited.
+- rclone - Show help for rclone commands, flags and backends.
+- rclone test changenotify - Log any change notify requests for the
+ remote passed in.
+- rclone test histogram - Makes a histogram of file name characters.
+- rclone test info - Discovers file name or other limitations for
+ paths.
+- rclone test makefile - Make files with random contents of the size
+ given
+- rclone test makefiles - Make a random file hierarchy in a directory
+- rclone test memory - Load all the objects at remote:path into memory
+ and report memory stats.
-An example of timetable with WEEKDAY could be:
+rclone test changenotify
---bwlimit "Mon-00:00,512 Fri-23:59,10M Sat-10:00,1M Sun-20:00,off"
+Log any change notify requests for the remote passed in.
-It means that, the transfer bandwidth will be set to 512 KiB/s on
-Monday. It will rise to 10 MiB/s before the end of Friday. At 10:00 on
-Saturday it will be set to 1 MiB/s. From 20:00 on Sunday it will be
-unlimited.
+ rclone test changenotify remote: [flags]
-Timeslots without WEEKDAY are extended to the whole week. So this
-example:
+Options
---bwlimit "Mon-00:00,512 12:00,1M Sun-20:00,off"
+ -h, --help help for changenotify
+ --poll-interval Duration Time to wait between polling for changes (default 10s)
-Is equivalent to this:
+See the global flags page for global options not listed here.
---bwlimit "Mon-00:00,512Mon-12:00,1M Tue-12:00,1M Wed-12:00,1M Thu-12:00,1M Fri-12:00,1M Sat-12:00,1M Sun-12:00,1M Sun-20:00,off"
+SEE ALSO
-Bandwidth limit apply to the data transfer for all backends. For most
-backends the directory listing bandwidth is also included (exceptions
-being the non HTTP backends, ftp, sftp and storj).
+- rclone test - Run a test command
-Note that the units are Byte/s, not bit/s. Typically connections are
-measured in bit/s - to convert divide by 8. For example, let's say you
-have a 10 Mbit/s connection and you wish rclone to use half of it - 5
-Mbit/s. This is 5/8 = 0.625 MiB/s so you would use a --bwlimit 0.625M
-parameter for rclone.
+rclone test histogram
-On Unix systems (Linux, macOS, …) the bandwidth limiter can be toggled
-by sending a SIGUSR2 signal to rclone. This allows to remove the
-limitations of a long running rclone transfer and to restore it back to
-the value specified with --bwlimit quickly when needed. Assuming there
-is only one rclone instance running, you can toggle the limiter like
-this:
+Makes a histogram of file name characters.
- kill -SIGUSR2 $(pidof rclone)
+Synopsis
-If you configure rclone with a remote control then you can use change
-the bwlimit dynamically:
+This command outputs JSON which shows the histogram of characters used
+in filenames in the remote:path specified.
- rclone rc core/bwlimit rate=1M
+The data doesn't contain any identifying information but is useful for
+the rclone developers when developing filename compression.
---bwlimit-file=BANDWIDTH_SPEC
+ rclone test histogram [remote:path] [flags]
-This option controls per file bandwidth limit. For the options see the
---bwlimit flag.
+Options
-For example use this to allow no transfers to be faster than 1 MiB/s
+ -h, --help help for histogram
- --bwlimit-file 1M
+See the global flags page for global options not listed here.
-This can be used in conjunction with --bwlimit.
+SEE ALSO
-Note that if a schedule is provided the file will use the schedule in
-effect at the start of the transfer.
+- rclone test - Run a test command
---buffer-size=SIZE
+rclone test info
-Use this sized buffer to speed up file transfers. Each --transfer will
-use this much memory for buffering.
+Discovers file name or other limitations for paths.
-When using mount or cmount each open file descriptor will use this much
-memory for buffering. See the mount documentation for more details.
+Synopsis
-Set to 0 to disable the buffering for the minimum memory usage.
+rclone info discovers what filenames and upload methods are possible to
+write to the paths passed in and how long they can be. It can take some
+time. It will write test files into the remote:path passed in. It
+outputs a bit of go code for each one.
-Note that the memory allocation of the buffers is influenced by the
---use-mmap flag.
+NB this can create undeletable files and other hazards - use with care
---cache-dir=DIR
+ rclone test info [remote:path]+ [flags]
-Specify the directory rclone will use for caching, to override the
-default.
+Options
-Default value is depending on operating system: - Windows
-%LocalAppData%\rclone, if LocalAppData is defined. - macOS
-$HOME/Library/Caches/rclone if HOME is defined. - Unix
-$XDG_CACHE_HOME/rclone if XDG_CACHE_HOME is defined, else
-$HOME/.cache/rclone if HOME is defined. - Fallback (on all OS) to
-$TMPDIR/rclone, where TMPDIR is the value from --temp-dir.
+ --all Run all tests
+ --check-base32768 Check can store all possible base32768 characters
+ --check-control Check control characters
+ --check-length Check max filename length
+ --check-normalization Check UTF-8 Normalization
+ --check-streaming Check uploads with indeterminate file size
+ -h, --help help for info
+ --upload-wait Duration Wait after writing a file (default 0s)
+ --write-json string Write results to file
-You can use the config paths command to see the current value.
+See the global flags page for global options not listed here.
-Cache directory is heavily used by the VFS File Caching mount feature,
-but also by serve, GUI and other parts of rclone.
+SEE ALSO
---check-first
+- rclone test - Run a test command
-If this flag is set then in a sync, copy or move, rclone will do all the
-checks to see whether files need to be transferred before doing any of
-the transfers. Normally rclone would start running transfers as soon as
-possible.
+rclone test makefile
-This flag can be useful on IO limited systems where transfers interfere
-with checking.
+Make files with random contents of the size given
-It can also be useful to ensure perfect ordering when using --order-by.
+ rclone test makefile []+ [flags]
-If both --check-first and --order-by are set when doing rclone move then
-rclone will use the transfer thread to delete source files which don't
-need transferring. This will enable perfect ordering of the transfers
-and deletes but will cause the transfer stats to have more items in than
-expected.
+Options
-Using this flag can use more memory as it effectively sets --max-backlog
-to infinite. This means that all the info on the objects to transfer is
-held in memory before the transfers start.
+ --ascii Fill files with random ASCII printable bytes only
+ --chargen Fill files with a ASCII chargen pattern
+ -h, --help help for makefile
+ --pattern Fill files with a periodic pattern
+ --seed int Seed for the random number generator (0 for random) (default 1)
+ --sparse Make the files sparse (appear to be filled with ASCII 0x00)
+ --zero Fill files with ASCII 0x00
---checkers=N
+See the global flags page for global options not listed here.
-Originally controlling just the number of file checkers to run in
-parallel, e.g. by rclone copy. Now a fairly universal parallelism
-control used by rclone in several places.
+SEE ALSO
-Note: checkers do the equality checking of files during a sync. For some
-storage systems (e.g. S3, Swift, Dropbox) this can take a significant
-amount of time so they are run in parallel.
+- rclone test - Run a test command
-The default is to run 8 checkers in parallel. However, in case of
-slow-reacting backends you may need to lower (rather than increase) this
-default by setting --checkers to 4 or less threads. This is especially
-advised if you are experiencing backend server crashes during file
-checking phase (e.g. on subsequent or top-up backups where little or no
-file copying is done and checking takes up most of the time). Increase
-this setting only with utmost care, while monitoring your server health
-and file checking throughput.
+rclone test makefiles
--c, --checksum
+Make a random file hierarchy in a directory
-Normally rclone will look at modification time and size of files to see
-if they are equal. If you set this flag then rclone will check the file
-hash and size to determine if files are equal.
+ rclone test makefiles [flags]
-This is useful when the remote doesn't support setting modified time and
-a more accurate sync is desired than just checking the file size.
+Options
-This is very useful when transferring between remotes which store the
-same hash type on the object, e.g. Drive and Swift. For details of which
-remotes support which hash type see the table in the overview section.
+ --ascii Fill files with random ASCII printable bytes only
+ --chargen Fill files with a ASCII chargen pattern
+ --files int Number of files to create (default 1000)
+ --files-per-directory int Average number of files per directory (default 10)
+ -h, --help help for makefiles
+ --max-depth int Maximum depth of directory hierarchy (default 10)
+ --max-file-size SizeSuffix Maximum size of files to create (default 100)
+ --max-name-length int Maximum size of file names (default 12)
+ --min-file-size SizeSuffix Minimum size of file to create
+ --min-name-length int Minimum size of file names (default 4)
+ --pattern Fill files with a periodic pattern
+ --seed int Seed for the random number generator (0 for random) (default 1)
+ --sparse Make the files sparse (appear to be filled with ASCII 0x00)
+ --zero Fill files with ASCII 0x00
-Eg rclone --checksum sync s3:/bucket swift:/bucket would run much
-quicker than without the --checksum flag.
+See the global flags page for global options not listed here.
-When using this flag, rclone won't update mtimes of remote files if they
-are incorrect as it would normally.
+SEE ALSO
---color WHEN
+- rclone test - Run a test command
-Specify when colors (and other ANSI codes) should be added to the
-output.
+rclone test memory
-AUTO (default) only allows ANSI codes when the output is a terminal
+Load all the objects at remote:path into memory and report memory stats.
-NEVER never allow ANSI codes
+ rclone test memory remote:path [flags]
-ALWAYS always add ANSI codes, regardless of the output format (terminal
-or file)
+Options
---compare-dest=DIR
+ -h, --help help for memory
-When using sync, copy or move DIR is checked in addition to the
-destination for files. If a file identical to the source is found that
-file is NOT copied from source. This is useful to copy just files that
-have changed since the last backup.
+See the global flags page for global options not listed here.
-You must use the same remote as the destination of the sync. The compare
-directory must not overlap the destination directory.
+SEE ALSO
-See --copy-dest and --backup-dir.
+- rclone test - Run a test command
---config=CONFIG_FILE
+rclone touch
-Specify the location of the rclone configuration file, to override the
-default. E.g. rclone config --config="rclone.conf".
+Create new file or change file modification time.
-The exact default is a bit complex to describe, due to changes
-introduced through different versions of rclone while preserving
-backwards compatibility, but in most cases it is as simple as:
+Synopsis
-- %APPDATA%/rclone/rclone.conf on Windows
-- ~/.config/rclone/rclone.conf on other
+Set the modification time on file(s) as specified by remote:path to have
+the current time.
-The complete logic is as follows: Rclone will look for an existing
-configuration file in any of the following locations, in priority order:
+If remote:path does not exist then a zero sized file will be created,
+unless --no-create or --recursive is provided.
-1. rclone.conf (in program directory, where rclone executable is)
-2. %APPDATA%/rclone/rclone.conf (only on Windows)
-3. $XDG_CONFIG_HOME/rclone/rclone.conf (on all systems, including
- Windows)
-4. ~/.config/rclone/rclone.conf (see below for explanation of ~ symbol)
-5. ~/.rclone.conf
+If --recursive is used then recursively sets the modification time on
+all existing files that is found under the path. Filters are supported,
+and you can test with the --dry-run or the --interactive/-i flag.
-If no existing configuration file is found, then a new one will be
-created in the following location:
+If --timestamp is used then sets the modification time to that time
+instead of the current time. Times may be specified as one of:
-- On Windows: Location 2 listed above, except in the unlikely event
- that APPDATA is not defined, then location 4 is used instead.
-- On Unix: Location 3 if XDG_CONFIG_HOME is defined, else location 4.
-- Fallback to location 5 (on all OS), when the rclone directory cannot
- be created, but if also a home directory was not found then path
- .rclone.conf relative to current working directory will be used as a
- final resort.
+- 'YYMMDD' - e.g. 17.10.30
+- 'YYYY-MM-DDTHH:MM:SS' - e.g. 2006-01-02T15:04:05
+- 'YYYY-MM-DDTHH:MM:SS.SSS' - e.g. 2006-01-02T15:04:05.123456789
-The ~ symbol in paths above represent the home directory of the current
-user on any OS, and the value is defined as following:
+Note that value of --timestamp is in UTC. If you want local time then
+add the --localtime flag.
-- On Windows: %HOME% if defined, else %USERPROFILE%, or else
- %HOMEDRIVE%\%HOMEPATH%.
-- On Unix: $HOME if defined, else by looking up current user in
- OS-specific user database (e.g. passwd file), or else use the result
- from shell command cd && pwd.
+ rclone touch remote:path [flags]
-If you run rclone config file you will see where the default location is
-for you.
+Options
-The fact that an existing file rclone.conf in the same directory as the
-rclone executable is always preferred, means that it is easy to run in
-"portable" mode by downloading rclone executable to a writable directory
-and then create an empty file rclone.conf in the same directory.
+ -h, --help help for touch
+ --localtime Use localtime for timestamp, not UTC
+ -C, --no-create Do not create the file if it does not exist (implied with --recursive)
+ -R, --recursive Recursively touch all files
+ -t, --timestamp string Use specified time instead of the current time of day
-If the location is set to empty string "" or path to a file with name
-notfound, or the os null device represented by value NUL on Windows and
-/dev/null on Unix systems, then rclone will keep the config file in
-memory only.
+Important Options
-The file format is basic INI: Sections of text, led by a [section]
-header and followed by key=value entries on separate lines. In rclone
-each remote is represented by its own section, where the section name
-defines the name of the remote. Options are specified as the key=value
-entries, where the key is the option name without the --backend- prefix,
-in lowercase and with _ instead of -. E.g. option --mega-hard-delete
-corresponds to key hard_delete. Only backend options can be specified. A
-special, and required, key type identifies the storage system, where the
-value is the internal lowercase name as returned by command
-rclone help backends. Comments are indicated by ; or # at the beginning
-of a line.
+Important flags useful for most commands.
-Example:
+ -n, --dry-run Do a trial run with no permanent changes
+ -i, --interactive Enable interactive mode
+ -v, --verbose count Print lots more stuff (repeat for more)
- [megaremote]
- type = mega
- user = you@example.com
- pass = PDPcQVVjVtzFY-GTdDFozqBhTdsPg3qH
+Filter Options
-Note that passwords are in obscured form. Also, many storage systems
-uses token-based authentication instead of passwords, and this requires
-additional steps. It is easier, and safer, to use the interactive
-command rclone config instead of manually editing the configuration
-file.
+Flags for filtering directory listings.
-The configuration file will typically contain login information, and
-should therefore have restricted permissions so that only the current
-user can read it. Rclone tries to ensure this when it writes the file.
-You may also choose to encrypt the file.
+ --delete-excluded Delete files on dest excluded from sync
+ --exclude stringArray Exclude files matching pattern
+ --exclude-from stringArray Read file exclude patterns from file (use - to read from stdin)
+ --exclude-if-present stringArray Exclude directories if filename is present
+ --files-from stringArray Read list of source-file names from file (use - to read from stdin)
+ --files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin)
+ -f, --filter stringArray Add a file filtering rule
+ --filter-from stringArray Read file filtering patterns from a file (use - to read from stdin)
+ --ignore-case Ignore case in filters (case insensitive)
+ --include stringArray Include files matching pattern
+ --include-from stringArray Read file include patterns from file (use - to read from stdin)
+ --max-age Duration Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off)
+ --max-depth int If set limits the recursion depth to this (default -1)
+ --max-size SizeSuffix Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off)
+ --metadata-exclude stringArray Exclude metadatas matching pattern
+ --metadata-exclude-from stringArray Read metadata exclude patterns from file (use - to read from stdin)
+ --metadata-filter stringArray Add a metadata filtering rule
+ --metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin)
+ --metadata-include stringArray Include metadatas matching pattern
+ --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin)
+ --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off)
+ --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off)
-When token-based authentication are used, the configuration file must be
-writable, because rclone needs to update the tokens inside it.
+Listing Options
-To reduce risk of corrupting an existing configuration file, rclone will
-not write directly to it when saving changes. Instead it will first
-write to a new, temporary, file. If a configuration file already
-existed, it will (on Unix systems) try to mirror its permissions to the
-new file. Then it will rename the existing file to a temporary name as
-backup. Next, rclone will rename the new file to the correct name,
-before finally cleaning up by deleting the backup file.
+Flags for listing directories.
-If the configuration file path used by rclone is a symbolic link, then
-this will be evaluated and rclone will write to the resolved path,
-instead of overwriting the symbolic link. Temporary files used in the
-process (described above) will be written to the same parent directory
-as that of the resolved configuration file, but if this directory is
-also a symbolic link it will not be resolved and the temporary files
-will be written to the location of the directory symbolic link.
+ --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z)
+ --fast-list Use recursive list if available; uses more memory but fewer transactions
---contimeout=TIME
+See the global flags page for global options not listed here.
-Set the connection timeout. This should be in go time format which looks
-like 5s for 5 seconds, 10m for 10 minutes, or 3h30m.
+SEE ALSO
-The connection timeout is the amount of time rclone will wait for a
-connection to go through to a remote object storage system. It is 1m by
-default.
+- rclone - Show help for rclone commands, flags and backends.
---copy-dest=DIR
+rclone tree
-When using sync, copy or move DIR is checked in addition to the
-destination for files. If a file identical to the source is found that
-file is server-side copied from DIR to the destination. This is useful
-for incremental backup.
+List the contents of the remote in a tree like fashion.
-The remote in use must support server-side copy and you must use the
-same remote as the destination of the sync. The compare directory must
-not overlap the destination directory.
+Synopsis
-See --compare-dest and --backup-dir.
+rclone tree lists the contents of a remote in a similar way to the unix
+tree command.
---dedupe-mode MODE
+For example
-Mode to run dedupe command in. One of interactive, skip, first, newest,
-oldest, rename. The default is interactive.
-See the dedupe command for more information as to what these options
-mean.
+ $ rclone tree remote:path
+ /
+ ├── file1
+ ├── file2
+ ├── file3
+ └── subdir
+ ├── file4
+ └── file5
---default-time TIME
+ 1 directories, 5 files
-If a file or directory does have a modification time rclone can read
-then rclone will display this fixed time instead.
+You can use any of the filtering options with the tree command (e.g.
+--include and --exclude. You can also use --fast-list.
-The default is 2000-01-01 00:00:00 UTC. This can be configured in any of
-the ways shown in the time or duration options.
+The tree command has many options for controlling the listing which are
+compatible with the tree command, for example you can include file sizes
+with --size. Note that not all of them have short options as they
+conflict with rclone's short options.
-For example --default-time 2020-06-01 to set the default time to the 1st
-of June 2020 or --default-time 0s to set the default time to the time
-rclone started up.
+For a more interactive navigation of the remote see the ncdu command.
---disable FEATURE,FEATURE,...
+ rclone tree remote:path [flags]
-This disables a comma separated list of optional features. For example
-to disable server-side move and server-side copy use:
+Options
- --disable move,copy
+ -a, --all All files are listed (list . files too)
+ -d, --dirs-only List directories only
+ --dirsfirst List directories before files (-U disables)
+ --full-path Print the full path prefix for each file
+ -h, --help help for tree
+ --level int Descend only level directories deep
+ -D, --modtime Print the date of last modification.
+ --noindent Don't print indentation lines
+ --noreport Turn off file/directory count at end of tree listing
+ -o, --output string Output to file instead of stdout
+ -p, --protections Print the protections for each file.
+ -Q, --quote Quote filenames with double quotes.
+ -s, --size Print the size in bytes of each file.
+ --sort string Select sort: name,version,size,mtime,ctime
+ --sort-ctime Sort files by last status change time
+ -t, --sort-modtime Sort files by last modification time
+ -r, --sort-reverse Reverse the order of the sort
+ -U, --unsorted Leave files unsorted
+ --version Sort files alphanumerically by version
-The features can be put in any case.
+Filter Options
-To see a list of which features can be disabled use:
+Flags for filtering directory listings.
- --disable help
+ --delete-excluded Delete files on dest excluded from sync
+ --exclude stringArray Exclude files matching pattern
+ --exclude-from stringArray Read file exclude patterns from file (use - to read from stdin)
+ --exclude-if-present stringArray Exclude directories if filename is present
+ --files-from stringArray Read list of source-file names from file (use - to read from stdin)
+ --files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin)
+ -f, --filter stringArray Add a file filtering rule
+ --filter-from stringArray Read file filtering patterns from a file (use - to read from stdin)
+ --ignore-case Ignore case in filters (case insensitive)
+ --include stringArray Include files matching pattern
+ --include-from stringArray Read file include patterns from file (use - to read from stdin)
+ --max-age Duration Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off)
+ --max-depth int If set limits the recursion depth to this (default -1)
+ --max-size SizeSuffix Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off)
+ --metadata-exclude stringArray Exclude metadatas matching pattern
+ --metadata-exclude-from stringArray Read metadata exclude patterns from file (use - to read from stdin)
+ --metadata-filter stringArray Add a metadata filtering rule
+ --metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin)
+ --metadata-include stringArray Include metadatas matching pattern
+ --metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin)
+ --min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off)
+ --min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off)
-The features a remote has can be seen in JSON format with:
+Listing Options
- rclone backend features remote:
+Flags for listing directories.
-See the overview features and optional features to get an idea of which
-feature does what.
+ --default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z)
+ --fast-list Use recursive list if available; uses more memory but fewer transactions
-Note that some features can be set to true if they are true/false
-feature flag features by prefixing them with !. For example the
-CaseInsensitive feature can be forced to false with
---disable CaseInsensitive and forced to true with
---disable '!CaseInsensitive'. In general it isn't a good idea doing this
-but it may be useful in extremis.
+See the global flags page for global options not listed here.
-(Note that ! is a shell command which you will need to escape with
-single quotes or a backslash on unix like platforms.)
+SEE ALSO
-This flag can be useful for debugging and in exceptional circumstances
-(e.g. Google Drive limiting the total volume of Server Side Copies to
-100 GiB/day).
+- rclone - Show help for rclone commands, flags and backends.
---disable-http2
+Copying single files
-This stops rclone from trying to use HTTP/2 if available. This can
-sometimes speed up transfers due to a problem in the Go standard
-library.
+rclone normally syncs or copies directories. However, if the source
+remote points to a file, rclone will just copy that file. The
+destination remote must point to a directory - rclone will give the
+error
+Failed to create file system for "remote:file": is a file not a directory
+if it isn't.
---dscp VALUE
+For example, suppose you have a remote with a file in called test.jpg,
+then you could copy just that file like this
-Specify a DSCP value or name to use in connections. This could help QoS
-system to identify traffic class. BE, EF, DF, LE, CSx and AFxx are
-allowed.
+ rclone copy remote:test.jpg /tmp/download
-See the description of differentiated services to get an idea of this
-field. Setting this to 1 (LE) to identify the flow to SCAVENGER class
-can avoid occupying too much bandwidth in a network with DiffServ
-support (RFC 8622).
+The file test.jpg will be placed inside /tmp/download.
-For example, if you configured QoS on router to handle LE properly.
-Running:
+This is equivalent to specifying
- rclone copy --dscp LE from:/from to:/to
+ rclone copy --files-from /tmp/files remote: /tmp/download
-would make the priority lower than usual internet flows.
+Where /tmp/files contains the single line
-This option has no effect on Windows (see golang/go#42728).
+ test.jpg
--n, --dry-run
+It is recommended to use copy when copying individual files, not sync.
+They have pretty much the same effect but copy will use a lot less
+memory.
-Do a trial run with no permanent changes. Use this to see what rclone
-would do without actually doing it. Useful when setting up the sync
-command which deletes files in the destination.
+Syntax of remote paths
---expect-continue-timeout=TIME
+The syntax of the paths passed to the rclone command are as follows.
-This specifies the amount of time to wait for a server's first response
-headers after fully writing the request headers if the request has an
-"Expect: 100-continue" header. Not all backends support using this.
+/path/to/dir
-Zero means no timeout and causes the body to be sent immediately,
-without waiting for the server to approve. This time does not include
-the time to send the request header.
+This refers to the local file system.
-The default is 1s. Set to 0 to disable.
+On Windows \ may be used instead of / in local paths only, non local
+paths must use /. See local filesystem documentation for more about
+Windows-specific paths.
---error-on-no-transfer
+These paths needn't start with a leading / - if they don't then they
+will be relative to the current directory.
-By default, rclone will exit with return code 0 if there were no errors.
+remote:path/to/dir
-This option allows rclone to return exit code 9 if no files were
-transferred between the source and destination. This allows using rclone
-in scripts, and triggering follow-on actions if data was copied, or
-skipping if not.
+This refers to a directory path/to/dir on remote: as defined in the
+config file (configured with rclone config).
-NB: Enabling this option turns a usually non-fatal error into a
-potentially fatal one - please check and adjust your scripts
-accordingly!
+remote:/path/to/dir
---fs-cache-expire-duration=TIME
+On most backends this is refers to the same directory as
+remote:path/to/dir and that format should be preferred. On a very small
+number of remotes (FTP, SFTP, Dropbox for business) this will refer to a
+different directory. On these, paths without a leading / will refer to
+your "home" directory and paths with a leading / will refer to the root.
-When using rclone via the API rclone caches created remotes for 5
-minutes by default in the "fs cache". This means that if you do repeated
-actions on the same remote then rclone won't have to build it again from
-scratch, which makes it more efficient.
+:backend:path/to/dir
-This flag sets the time that the remotes are cached for. If you set it
-to 0 (or negative) then rclone won't cache the remotes at all.
+This is an advanced form for creating remotes on the fly. backend should
+be the name or prefix of a backend (the type in the config file) and all
+the configuration for the backend should be provided on the command line
+(or in environment variables).
-Note that if you use some flags, eg --backup-dir and if this is set to 0
-rclone may build two remotes (one for the source or destination and one
-for the --backup-dir where it may have only built one before.
+Here are some examples:
---fs-cache-expire-interval=TIME
+ rclone lsd --http-url https://pub.rclone.org :http:
-This controls how often rclone checks for cached remotes to expire. See
-the --fs-cache-expire-duration documentation above for more info. The
-default is 60s, set to 0 to disable expiry.
+To list all the directories in the root of https://pub.rclone.org/.
---header
+ rclone lsf --http-url https://example.com :http:path/to/dir
-Add an HTTP header for all transactions. The flag can be repeated to add
-multiple headers.
+To list files and directories in https://example.com/path/to/dir/
-If you want to add headers only for uploads use --header-upload and if
-you want to add headers only for downloads use --header-download.
+ rclone copy --http-url https://example.com :http:path/to/dir /tmp/dir
-This flag is supported for all HTTP based backends even those not
-supported by --header-upload and --header-download so may be used as a
-workaround for those with care.
+To copy files and directories in https://example.com/path/to/dir to
+/tmp/dir.
- rclone ls remote:test --header "X-Rclone: Foo" --header "X-LetMeIn: Yes"
+ rclone copy --sftp-host example.com :sftp:path/to/dir /tmp/dir
---header-download
+To copy files and directories from example.com in the relative directory
+path/to/dir to /tmp/dir using sftp.
-Add an HTTP header for all download transactions. The flag can be
-repeated to add multiple headers.
+Connection strings
- rclone sync --interactive s3:test/src ~/dst --header-download "X-Amz-Meta-Test: Foo" --header-download "X-Amz-Meta-Test2: Bar"
+The above examples can also be written using a connection string syntax,
+so instead of providing the arguments as command line parameters
+--http-url https://pub.rclone.org they are provided as part of the
+remote specification as a kind of connection string.
-See the GitHub issue here for currently supported backends.
+ rclone lsd ":http,url='https://pub.rclone.org':"
+ rclone lsf ":http,url='https://example.com':path/to/dir"
+ rclone copy ":http,url='https://example.com':path/to/dir" /tmp/dir
+ rclone copy :sftp,host=example.com:path/to/dir /tmp/dir
---header-upload
+These can apply to modify existing remotes as well as create new remotes
+with the on the fly syntax. This example is equivalent to adding the
+--drive-shared-with-me parameter to the remote gdrive:.
-Add an HTTP header for all upload transactions. The flag can be repeated
-to add multiple headers.
+ rclone lsf "gdrive,shared_with_me:path/to/dir"
- rclone sync --interactive ~/src s3:test/dst --header-upload "Content-Disposition: attachment; filename='cool.html'" --header-upload "X-Amz-Meta-Test: FooBar"
+The major advantage to using the connection string style syntax is that
+it only applies to the remote, not to all the remotes of that type of
+the command line. A common confusion is this attempt to copy a file
+shared on google drive to the normal drive which does not work because
+the --drive-shared-with-me flag applies to both the source and the
+destination.
-See the GitHub issue here for currently supported backends.
+ rclone copy --drive-shared-with-me gdrive:shared-file.txt gdrive:
---human-readable
+However using the connection string syntax, this does work.
-Rclone commands output values for sizes (e.g. number of bytes) and
-counts (e.g. number of files) either as raw numbers, or in
-human-readable format.
+ rclone copy "gdrive,shared_with_me:shared-file.txt" gdrive:
-In human-readable format the values are scaled to larger units,
-indicated with a suffix shown after the value, and rounded to three
-decimals. Rclone consistently uses binary units (powers of 2) for sizes
-and decimal units (powers of 10) for counts. The unit prefix for size is
-according to IEC standard notation, e.g. Ki for kibi. Used with byte
-unit, 1 KiB means 1024 Byte. In list type of output, only the unit
-prefix appended to the value (e.g. 9.762Ki), while in more textual
-output the full unit is shown (e.g. 9.762 KiB). For counts the SI
-standard notation is used, e.g. prefix k for kilo. Used with file
-counts, 1k means 1000 files.
+Note that the connection string only affects the options of the
+immediate backend. If for example gdriveCrypt is a crypt based on
+gdrive, then the following command will not work as intended, because
+shared_with_me is ignored by the crypt backend:
-The various list commands output raw numbers by default. Option
---human-readable will make them output values in human-readable format
-instead (with the short unit prefix).
+ rclone copy "gdriveCrypt,shared_with_me:shared-file.txt" gdriveCrypt:
-The about command outputs human-readable by default, with a
-command-specific option --full to output the raw numbers instead.
+The connection strings have the following syntax
-Command size outputs both human-readable and raw numbers in the same
-output.
+ remote,parameter=value,parameter2=value2:path/to/dir
+ :backend,parameter=value,parameter2=value2:path/to/dir
-The tree command also considers --human-readable, but it will not use
-the exact same notation as the other commands: It rounds to one decimal,
-and uses single letter suffix, e.g. K instead of Ki. The reason for this
-is that it relies on an external library.
+If the parameter has a : or , then it must be placed in quotes " or ',
+so
-The interactive command ncdu shows human-readable by default, and
-responds to key u for toggling human-readable format.
+ remote,parameter="colon:value",parameter2="comma,value":path/to/dir
+ :backend,parameter='colon:value',parameter2='comma,value':path/to/dir
---ignore-case-sync
+If a quoted value needs to include that quote, then it should be
+doubled, so
-Using this option will cause rclone to ignore the case of the files when
-synchronizing so files will not be copied/synced when the existing
-filenames are the same, even if the casing is different.
+ remote,parameter="with""quote",parameter2='with''quote':path/to/dir
---ignore-checksum
+This will make parameter be with"quote and parameter2 be with'quote.
-Normally rclone will check that the checksums of transferred files
-match, and give an error "corrupted on transfer" if they don't.
+If you leave off the =parameter then rclone will substitute =true which
+works very well with flags. For example, to use s3 configured in the
+environment you could use:
-You can use this option to skip that check. You should only use it if
-you have had the "corrupted on transfer" error message and you are sure
-you might want to transfer potentially corrupted data.
+ rclone lsd :s3,env_auth:
---ignore-existing
+Which is equivalent to
-Using this option will make rclone unconditionally skip all files that
-exist on the destination, no matter the content of these files.
+ rclone lsd :s3,env_auth=true:
-While this isn't a generally recommended option, it can be useful in
-cases where your files change due to encryption. However, it cannot
-correct partial transfers in case a transfer was interrupted.
+Note that on the command line you might need to surround these
+connection strings with " or ' to stop the shell interpreting any
+special characters within them.
-When performing a move/moveto command, this flag will leave skipped
-files in the source location unchanged when a file with the same name
-exists on the destination.
+If you are a shell master then you'll know which strings are OK and
+which aren't, but if you aren't sure then enclose them in " and use ' as
+the inside quote. This syntax works on all OSes.
---ignore-size
+ rclone copy ":http,url='https://example.com':path/to/dir" /tmp/dir
-Normally rclone will look at modification time and size of files to see
-if they are equal. If you set this flag then rclone will check only the
-modification time. If --checksum is set then it only checks the
-checksum.
+On Linux/macOS some characters are still interpreted inside " strings in
+the shell (notably \ and $ and ") so if your strings contain those you
+can swap the roles of " and ' thus. (This syntax does not work on
+Windows.)
-It will also cause rclone to skip verifying the sizes are the same after
-transfer.
+ rclone copy ':http,url="https://example.com":path/to/dir' /tmp/dir
-This can be useful for transferring files to and from OneDrive which
-occasionally misreports the size of image files (see #399 for more
-info).
+Connection strings, config and logging
--I, --ignore-times
+If you supply extra configuration to a backend by command line flag,
+environment variable or connection string then rclone will add a suffix
+based on the hash of the config to the name of the remote, eg
-Using this option will cause rclone to unconditionally upload all files
-regardless of the state of files on the destination.
+ rclone -vv lsf --s3-chunk-size 20M s3:
-Normally rclone would skip any files that have the same modification
-time and are the same size (or have the same checksum if using
---checksum).
+Has the log message
---immutable
+ DEBUG : s3: detected overridden config - adding "{Srj1p}" suffix to name
-Treat source and destination files as immutable and disallow
-modification.
+This is so rclone can tell the modified remote apart from the unmodified
+remote when caching the backends.
-With this option set, files will be created and deleted as requested,
-but existing files will never be updated. If an existing file does not
-match between the source and destination, rclone will give the error
-Source and destination exist but do not match: immutable file modified.
+This should only be noticeable in the logs.
-Note that only commands which transfer files (e.g. sync, copy, move) are
-affected by this behavior, and only modification is disallowed. Files
-may still be deleted explicitly (e.g. delete, purge) or implicitly (e.g.
-sync, move). Use copy --immutable if it is desired to avoid deletion as
-well as modification.
+This means that on the fly backends such as
-This can be useful as an additional layer of protection for immutable or
-append-only data sets (notably backup archives), where modification
-implies corruption and should not be propagated.
+ rclone -vv lsf :s3,env_auth:
---inplace
+Will get their own names
-The --inplace flag changes the behaviour of rclone when uploading files
-to some backends (backends with the PartialUploads feature flag set)
-such as:
+ DEBUG : :s3: detected overridden config - adding "{YTu53}" suffix to name
-- local
-- ftp
-- sftp
+Valid remote names
-Without --inplace (the default) rclone will first upload to a temporary
-file with an extension like this where XXXXXX represents a random
-string.
+Remote names are case sensitive, and must adhere to the following rules:
+- May contain number, letter, _, -, ., +, @ and space. - May not start
+with - or space. - May not end with space.
- original-file-name.XXXXXX.partial
+Starting with rclone version 1.61, any Unicode numbers and letters are
+allowed, while in older versions it was limited to plain ASCII (0-9,
+A-Z, a-z). If you use the same rclone configuration from different
+shells, which may be configured with different character encoding, you
+must be cautious to use characters that are possible to write in all of
+them. This is mostly a problem on Windows, where the console
+traditionally uses a non-Unicode character set - defined by the
+so-called "code page".
-(rclone will make sure the final name is no longer than 100 characters
-by truncating the original-file-name part if necessary).
+Do not use single character names on Windows as it creates ambiguity
+with Windows drives' names, e.g.: remote called C is indistinguishable
+from C drive. Rclone will always assume that single letter name refers
+to a drive.
-When the upload is complete, rclone will rename the .partial file to the
-correct name, overwriting any existing file at that point. If the upload
-fails then the .partial file will be deleted.
+Quoting and the shell
-This prevents other users of the backend from seeing partially uploaded
-files in their new names and prevents overwriting the old file until the
-new one is completely uploaded.
+When you are typing commands to your computer you are using something
+called the command line shell. This interprets various characters in an
+OS specific way.
-If the --inplace flag is supplied, rclone will upload directly to the
-final name without creating a .partial file.
+Here are some gotchas which may help users unfamiliar with the shell
+rules
-This means that an incomplete file will be visible in the directory
-listings while the upload is in progress and any existing files will be
-overwritten as soon as the upload starts. If the transfer fails then the
-file will be deleted. This can cause data loss of the existing file if
-the transfer fails.
+Linux / OSX
-Note that on the local file system if you don't use --inplace hard links
-(Unix only) will be broken. And if you do use --inplace you won't be
-able to update in use executables.
+If your names have spaces or shell metacharacters (e.g. *, ?, $, ', ",
+etc.) then you must quote them. Use single quotes ' by default.
-Note also that versions of rclone prior to v1.63.0 behave as if the
---inplace flag is always supplied.
+ rclone copy 'Important files?' remote:backup
--i, --interactive
+If you want to send a ' you will need to use ", e.g.
-This flag can be used to tell rclone that you wish a manual confirmation
-before destructive operations.
+ rclone copy "O'Reilly Reviews" remote:backup
-It is recommended that you use this flag while learning rclone
-especially with rclone sync.
+The rules for quoting metacharacters are complicated and if you want the
+full details you'll have to consult the manual page for your shell.
-For example
+Windows
- $ rclone delete --interactive /tmp/dir
- rclone: delete "important-file.txt"?
- y) Yes, this is OK (default)
- n) No, skip this
- s) Skip all delete operations with no more questions
- !) Do all delete operations with no more questions
- q) Exit rclone now.
- y/n/s/!/q> n
+If your names have spaces in you need to put them in ", e.g.
-The options mean
+ rclone copy "E:\folder name\folder name\folder name" remote:backup
-- y: Yes, this operation should go ahead. You can also press Return
- for this to happen. You'll be asked every time unless you choose s
- or !.
-- n: No, do not do this operation. You'll be asked every time unless
- you choose s or !.
-- s: Skip all the following operations of this type with no more
- questions. This takes effect until rclone exits. If there are any
- different kind of operations you'll be prompted for them.
-- !: Do all the following operations with no more questions. Useful if
- you've decided that you don't mind rclone doing that kind of
- operation. This takes effect until rclone exits . If there are any
- different kind of operations you'll be prompted for them.
-- q: Quit rclone now, just in case!
+If you are using the root directory on its own then don't quote it (see
+#464 for why), e.g.
---leave-root
+ rclone copy E:\ remote:backup
-During rmdirs it will not remove root directory, even if it's empty.
+Copying files or directories with : in the names
---log-file=FILE
+rclone uses : to mark a remote name. This is, however, a valid filename
+component in non-Windows OSes. The remote name parser will only search
+for a : up to the first / so if you need to act on a file or directory
+like this then use the full path starting with a /, or use ./ as a
+current directory prefix.
-Log all of rclone's output to FILE. This is not active by default. This
-can be useful for tracking down problems with syncs in combination with
-the -v flag. See the Logging section for more info.
+So to sync a directory called sync:me to a remote called remote: use
-If FILE exists then rclone will append to it.
+ rclone sync --interactive ./sync:me remote:path
-Note that if you are using the logrotate program to manage rclone's
-logs, then you should use the copytruncate option as rclone doesn't have
-a signal to rotate logs.
+or
---log-format LIST
+ rclone sync --interactive /full/path/to/sync:me remote:path
-Comma separated list of log format options. Accepted options are date,
-time, microseconds, pid, longfile, shortfile, UTC. Any other keywords
-will be silently ignored. pid will tag log messages with process
-identifier which useful with rclone mount --daemon. Other accepted
-options are explained in the go documentation. The default log format is
-"date,time".
+Server Side Copy
---log-level LEVEL
+Most remotes (but not all - see the overview) support server-side copy.
-This sets the log level for rclone. The default log level is NOTICE.
+This means if you want to copy one folder to another then rclone won't
+download all the files and re-upload them; it will instruct the server
+to copy them in place.
-DEBUG is equivalent to -vv. It outputs lots of debug info - useful for
-bug reports and really finding out what rclone is doing.
+Eg
-INFO is equivalent to -v. It outputs information about each transfer and
-prints stats once a minute by default.
+ rclone copy s3:oldbucket s3:newbucket
-NOTICE is the default log level if no logging flags are supplied. It
-outputs very little when things are working normally. It outputs
-warnings and significant events.
+Will copy the contents of oldbucket to newbucket without downloading and
+re-uploading.
-ERROR is equivalent to -q. It only outputs error messages.
+Remotes which don't support server-side copy will download and re-upload
+in this case.
---use-json-log
+Server side copies are used with sync and copy and will be identified in
+the log when using the -v flag. The move command may also use them if
+remote doesn't support server-side move directly. This is done by
+issuing a server-side copy then a delete which is much quicker than a
+download and re-upload.
-This switches the log format to JSON for rclone. The fields of json log
-are level, msg, source, time.
+Server side copies will only be attempted if the remote names are the
+same.
---low-level-retries NUMBER
+This can be used when scripting to make aged backups efficiently, e.g.
-This controls the number of low level retries rclone does.
+ rclone sync --interactive remote:current-backup remote:previous-backup
+ rclone sync --interactive /path/to/files remote:current-backup
-A low level retry is used to retry a failing operation - typically one
-HTTP request. This might be uploading a chunk of a big file for example.
-You will see low level retries in the log with the -v flag.
+Metadata support
-This shouldn't need to be changed from the default in normal operations.
-However, if you get a lot of low level retries you may wish to reduce
-the value so rclone moves on to a high level retry (see the --retries
-flag) quicker.
+Metadata is data about a file which isn't the contents of the file.
+Normally rclone only preserves the modification time and the content
+(MIME) type where possible.
-Disable low level retries with --low-level-retries 1.
+Rclone supports preserving all the available metadata on files (not
+directories) when using the --metadata or -M flag.
---max-backlog=N
+Exactly what metadata is supported and what that support means depends
+on the backend. Backends that support metadata have a metadata section
+in their docs and are listed in the features table (Eg local, s3)
-This is the maximum allowable backlog of files in a sync/copy/move
-queued for being checked or transferred.
+Rclone only supports a one-time sync of metadata. This means that
+metadata will be synced from the source object to the destination object
+only when the source object has changed and needs to be re-uploaded. If
+the metadata subsequently changes on the source object without changing
+the object itself then it won't be synced to the destination object.
+This is in line with the way rclone syncs Content-Type without the
+--metadata flag.
-This can be set arbitrarily large. It will only use memory when the
-queue is in use. Note that it will use in the order of N KiB of memory
-when the backlog is in use.
+Using --metadata when syncing from local to local will preserve file
+attributes such as file mode, owner, extended attributes (not Windows).
-Setting this large allows rclone to calculate how many files are pending
-more accurately, give a more accurate estimated finish time and make
---order-by work more accurately.
+Note that arbitrary metadata may be added to objects using the
+--metadata-set key=value flag when the object is first uploaded. This
+flag can be repeated as many times as necessary.
-Setting this small will make rclone more synchronous to the listings of
-the remote which may be desirable.
+The --metadata-mapper flag can be used to pass the name of a program in
+which can transform metadata when it is being copied from source to
+destination.
-Setting this to a negative number will make the backlog as large as
-possible.
+Types of metadata
---max-delete=N
+Metadata is divided into two type. System metadata and User metadata.
-This tells rclone not to delete more than N files. If that limit is
-exceeded then a fatal error will be generated and rclone will stop the
-operation in progress.
+Metadata which the backend uses itself is called system metadata. For
+example on the local backend the system metadata uid will store the user
+ID of the file when used on a unix based platform.
---max-delete-size=SIZE
+Arbitrary metadata is called user metadata and this can be set however
+is desired.
-Rclone will stop deleting files when the total size of deletions has
-reached the size specified. It defaults to off.
+When objects are copied from backend to backend, they will attempt to
+interpret system metadata if it is supplied. Metadata may change from
+being user metadata to system metadata as objects are copied between
+different backends. For example copying an object from s3 sets the
+content-type metadata. In a backend which understands this (like
+azureblob) this will become the Content-Type of the object. In a backend
+which doesn't understand this (like the local backend) this will become
+user metadata. However should the local object be copied back to s3, the
+Content-Type will be set correctly.
-If that limit is exceeded then a fatal error will be generated and
-rclone will stop the operation in progress.
+Metadata framework
---max-depth=N
+Rclone implements a metadata framework which can read metadata from an
+object and write it to the object when (and only when) it is being
+uploaded.
-This modifies the recursion depth for all the commands except purge.
+This metadata is stored as a dictionary with string keys and string
+values.
-So if you do rclone --max-depth 1 ls remote:path you will see only the
-files in the top level directory. Using --max-depth 2 means you will see
-all the files in first two directory levels and so on.
+There are some limits on the names of the keys (these may be clarified
+further in the future).
-For historical reasons the lsd command defaults to using a --max-depth
-of 1 - you can override this with the command line flag.
+- must be lower case
+- may be a-z 0-9 containing . - or _
+- length is backend dependent
-You can use this command to disable recursion (with --max-depth 1).
+Each backend can provide system metadata that it understands. Some
+backends can also store arbitrary user metadata.
-Note that if you use this with sync and --delete-excluded the files not
-recursed through are considered excluded and will be deleted on the
-destination. Test first with --dry-run if you are not sure what will
-happen.
+Where possible the key names are standardized, so, for example, it is
+possible to copy object metadata from s3 to azureblob for example and
+metadata will be translated appropriately.
---max-duration=TIME
+Some backends have limits on the size of the metadata and rclone will
+give errors on upload if they are exceeded.
-Rclone will stop transferring when it has run for the duration
-specified. Defaults to off.
+Metadata preservation
-When the limit is reached all transfers will stop immediately. Use
---cutoff-mode to modify this behaviour.
+The goal of the implementation is to
-Rclone will exit with exit code 10 if the duration limit is reached.
+1. Preserve metadata if at all possible
+2. Interpret metadata if at all possible
---max-transfer=SIZE
+The consequences of 1 is that you can copy an S3 object to a local disk
+then back to S3 losslessly. Likewise you can copy a local file with file
+attributes and xattrs from local disk to s3 and back again losslessly.
-Rclone will stop transferring when it has reached the size specified.
-Defaults to off.
+The consequence of 2 is that you can copy an S3 object with metadata to
+Azureblob (say) and have the metadata appear on the Azureblob object
+also.
-When the limit is reached all transfers will stop immediately. Use
---cutoff-mode to modify this behaviour.
+Standard system metadata
-Rclone will exit with exit code 8 if the transfer limit is reached.
+Here is a table of standard system metadata which, if appropriate, a
+backend may implement.
---cutoff-mode=hard|soft|cautious
+ ----------------------------------------------------------------------------------------------
+ key description example
+ ---------------------------------- --------------------- -------------------------------------
+ mode File type and mode: 0100664
+ octal, unix style
-This modifies the behavior of --max-transfer and --max-duration Defaults
-to --cutoff-mode=hard.
+ uid User ID of owner: 500
+ decimal number
-Specifying --cutoff-mode=hard will stop transferring immediately when
-Rclone reaches the limit.
+ gid Group ID of owner: 500
+ decimal number
-Specifying --cutoff-mode=soft will stop starting new transfers when
-Rclone reaches the limit.
+ rdev Device ID (if special 0
+ file) => hexadecimal
-Specifying --cutoff-mode=cautious will try to prevent Rclone from
-reaching the limit. Only applicable for --max-transfer
+ atime Time of last access: 2006-01-02T15:04:05.999999999Z07:00
+ RFC 3339
--M, --metadata
+ mtime Time of last 2006-01-02T15:04:05.999999999Z07:00
+ modification: RFC
+ 3339
-Setting this flag enables rclone to copy the metadata from the source to
-the destination. For local backends this is ownership, permissions,
-xattr etc. See the #metadata for more info.
+ btime Time of file creation 2006-01-02T15:04:05.999999999Z07:00
+ (birth): RFC 3339
---metadata-set key=value
+ utime Time of file upload: 2006-01-02T15:04:05.999999999Z07:00
+ RFC 3339
-Add metadata key = value when uploading. This can be repeated as many
-times as required. See the #metadata for more info.
+ cache-control Cache-Control header no-cache
---modify-window=TIME
+ content-disposition Content-Disposition inline
+ header
-When checking whether a file has been modified, this is the maximum
-allowed time difference that a file can have and still be considered
-equivalent.
-
-The default is 1ns unless this is overridden by a remote. For example OS
-X only stores modification times to the nearest second so if you are
-reading and writing to an OS X filing system this will be 1s by default.
+ content-encoding Content-Encoding gzip
+ header
-This command line flag allows you to override that computed default.
+ content-language Content-Language en-US
+ header
---multi-thread-write-buffer-size=SIZE
+ content-type Content-Type header text/plain
+ ----------------------------------------------------------------------------------------------
-When transferring with multiple threads, rclone will buffer SIZE bytes
-in memory before writing to disk for each thread.
+The metadata keys mtime and content-type will take precedence if
+supplied in the metadata over reading the Content-Type or modification
+time of the source object.
-This can improve performance if the underlying filesystem does not deal
-well with a lot of small writes in different positions of the file, so
-if you see transfers being limited by disk write speed, you might want
-to experiment with different values. Specially for magnetic drives and
-remote file systems a higher value can be useful.
+Hashes are not included in system metadata as there is a well defined
+way of reading those already.
-Nevertheless, the default of 128k should be fine for almost all use
-cases, so before changing it ensure that network is not really your
-bottleneck.
+Options
-As a final hint, size is not the only factor: block size (or similar
-concept) can have an impact. In one case, we observed that exact
-multiples of 16k performed much better than other values.
+Rclone has a number of options to control its behaviour.
---multi-thread-chunk-size=SizeSuffix
+Options that take parameters can have the values passed in two ways,
+--option=value or --option value. However boolean (true/false) options
+behave slightly differently to the other options in that --boolean sets
+the option to true and the absence of the flag sets it to false. It is
+also possible to specify --boolean=false or --boolean=true. Note that
+--boolean false is not valid - this is parsed as --boolean and the false
+is parsed as an extra command line argument for rclone.
-Normally the chunk size for multi thread transfers is set by the
-backend. However some backends such as local and smb (which implement
-OpenWriterAt but not OpenChunkWriter) don't have a natural chunk size.
+Time or duration options
-In this case the value of this option is used (default 64Mi).
+TIME or DURATION options can be specified as a duration string or a time
+string.
---multi-thread-cutoff=SIZE
+A duration string is a possibly signed sequence of decimal numbers, each
+with optional fraction and a unit suffix, such as "300ms", "-1.5h" or
+"2h45m". Default units are seconds or the following abbreviations are
+valid:
-When transferring files above SIZE to capable backends, rclone will use
-multiple threads to transfer the file (default 256M).
+- ms - Milliseconds
+- s - Seconds
+- m - Minutes
+- h - Hours
+- d - Days
+- w - Weeks
+- M - Months
+- y - Years
-Capable backends are marked in the overview as MultithreadUpload. (They
-need to implement either the OpenWriterAt or OpenChunkedWriter internal
-interfaces). These include include, local, s3, azureblob, b2,
-oracleobjectstorage and smb at the time of writing.
+These can also be specified as an absolute time in the following
+formats:
-On the local disk, rclone preallocates the file (using
-fallocate(FALLOC_FL_KEEP_SIZE) on unix or NTSetInformationFile on
-Windows both of which takes no time) then each thread writes directly
-into the file at the correct place. This means that rclone won't create
-fragmented or sparse files and there won't be any assembly time at the
-end of the transfer.
+- RFC3339 - e.g. 2006-01-02T15:04:05Z or 2006-01-02T15:04:05+07:00
+- ISO8601 Date and time, local timezone - 2006-01-02T15:04:05
+- ISO8601 Date and time, local timezone - 2006-01-02 15:04:05
+- ISO8601 Date - 2006-01-02 (YYYY-MM-DD)
-The number of threads used to transfer is controlled by
---multi-thread-streams.
+Size options
-Use -vv if you wish to see info about the threads.
+Options which use SIZE use KiB (multiples of 1024 bytes) by default.
+However, a suffix of B for Byte, K for KiB, M for MiB, G for GiB, T for
+TiB and P for PiB may be used. These are the binary units, e.g. 1,
+2**10, 2**20, 2**30 respectively.
-This will work with the sync/copy/move commands and friends
-copyto/moveto. Multi thread transfers will be used with rclone mount and
-rclone serve if --vfs-cache-mode is set to writes or above.
+--backup-dir=DIR
-NB that this only works with supported backends as the destination but
-will work with any backend as the source.
+When using sync, copy or move any files which would have been
+overwritten or deleted are moved in their original hierarchy into this
+directory.
-NB that multi-thread copies are disabled for local to local copies as
-they are faster without unless --multi-thread-streams is set explicitly.
+If --suffix is set, then the moved files will have the suffix added to
+them. If there is a file with the same path (after the suffix has been
+added) in DIR, then it will be overwritten.
-NB on Windows using multi-thread transfers to the local disk will cause
-the resulting files to be sparse. Use --local-no-sparse to disable
-sparse files (which may cause long delays at the start of transfers) or
-disable multi-thread transfers with --multi-thread-streams 0
+The remote in use must support server-side move or copy and you must use
+the same remote as the destination of the sync. The backup directory
+must not overlap the destination directory without it being excluded by
+a filter rule.
---multi-thread-streams=N
+For example
-When using multi thread transfers (see above --multi-thread-cutoff) this
-sets the number of streams to use. Set to 0 to disable multi thread
-transfers (Default 4).
+ rclone sync --interactive /path/to/local remote:current --backup-dir remote:old
-If the backend has a --backend-upload-concurrency setting (eg
---s3-upload-concurrency) then this setting will be used as the number of
-transfers instead if it is larger than the value of
---multi-thread-streams or --multi-thread-streams isn't set.
+will sync /path/to/local to remote:current, but for any files which
+would have been updated or deleted will be stored in remote:old.
---no-check-dest
+If running rclone from a script you might want to use today's date as
+the directory name passed to --backup-dir to store the old files, or you
+might want to pass --suffix with today's date.
-The --no-check-dest can be used with move or copy and it causes rclone
-not to check the destination at all when copying files.
+See --compare-dest and --copy-dest.
-This means that:
+--bind string
-- the destination is not listed minimising the API calls
-- files are always transferred
-- this can cause duplicates on remotes which allow it (e.g. Google
- Drive)
-- --retries 1 is recommended otherwise you'll transfer everything
- again on a retry
+Local address to bind to for outgoing connections. This can be an IPv4
+address (1.2.3.4), an IPv6 address (1234::789A) or host name. If the
+host name doesn't resolve or resolves to more than one IP address it
+will give an error.
-This flag is useful to minimise the transactions if you know that none
-of the files are on the destination.
+You can use --bind 0.0.0.0 to force rclone to use IPv4 addresses and
+--bind ::0 to force rclone to use IPv6 addresses.
-This is a specialized flag which should be ignored by most users!
+--bwlimit=BANDWIDTH_SPEC
---no-gzip-encoding
+This option controls the bandwidth limit. For example
-Don't set Accept-Encoding: gzip. This means that rclone won't ask the
-server for compressed files automatically. Useful if you've set the
-server to return files with Content-Encoding: gzip but you uploaded
-compressed files.
+ --bwlimit 10M
-There is no need to set this in normal operation, and doing so will
-decrease the network transfer efficiency of rclone.
+would mean limit the upload and download bandwidth to 10 MiB/s. NB this
+is bytes per second not bits per second. To use a single limit, specify
+the desired bandwidth in KiB/s, or use a suffix B|K|M|G|T|P. The default
+is 0 which means to not limit bandwidth.
---no-traverse
+The upload and download bandwidth can be specified separately, as
+--bwlimit UP:DOWN, so
-The --no-traverse flag controls whether the destination file system is
-traversed when using the copy or move commands. --no-traverse is not
-compatible with sync and will be ignored if you supply it with sync.
+ --bwlimit 10M:100k
-If you are only copying a small number of files (or are filtering most
-of the files) and/or have a large number of files on the destination
-then --no-traverse will stop rclone listing the destination and save
-time.
+would mean limit the upload bandwidth to 10 MiB/s and the download
+bandwidth to 100 KiB/s. Either limit can be "off" meaning no limit, so
+to just limit the upload bandwidth you would use
-However, if you are copying a large number of files, especially if you
-are doing a copy where lots of the files under consideration haven't
-changed and won't need copying then you shouldn't use --no-traverse.
+ --bwlimit 10M:off
-See rclone copy for an example of how to use it.
+this would limit the upload bandwidth to 10 MiB/s but the download
+bandwidth would be unlimited.
---no-unicode-normalization
+When specified as above the bandwidth limits last for the duration of
+run of the rclone binary.
-Don't normalize unicode characters in filenames during the sync routine.
+It is also possible to specify a "timetable" of limits, which will cause
+certain limits to be applied at certain times. To specify a timetable,
+format your entries as
+WEEKDAY-HH:MM,BANDWIDTH WEEKDAY-HH:MM,BANDWIDTH... where: WEEKDAY is
+optional element.
-Sometimes, an operating system will store filenames containing unicode
-parts in their decomposed form (particularly macOS). Some cloud storage
-systems will then recompose the unicode, resulting in duplicate files if
-the data is ever copied back to a local filesystem.
+- BANDWIDTH can be a single number, e.g.100k or a pair of numbers for
+ upload:download, e.g.10M:1M.
+- WEEKDAY can be written as the whole word or only using the first 3
+ characters. It is optional.
+- HH:MM is an hour from 00:00 to 23:59.
-Using this flag will disable that functionality, treating each unicode
-character as unique. For example, by default eÌ and é will be normalized
-into the same character. With --no-unicode-normalization they will be
-treated as unique characters.
+An example of a typical timetable to avoid link saturation during
+daytime working hours could be:
---no-update-modtime
+--bwlimit "08:00,512k 12:00,10M 13:00,512k 18:00,30M 23:00,off"
-When using this flag, rclone won't update modification times of remote
-files if they are incorrect as it would normally.
+In this example, the transfer bandwidth will be set to 512 KiB/s at 8am
+every day. At noon, it will rise to 10 MiB/s, and drop back to 512
+KiB/sec at 1pm. At 6pm, the bandwidth limit will be set to 30 MiB/s, and
+at 11pm it will be completely disabled (full speed). Anything between
+11pm and 8am will remain unlimited.
-This can be used if the remote is being synced with another tool also
-(e.g. the Google Drive client).
+An example of timetable with WEEKDAY could be:
---order-by string
+--bwlimit "Mon-00:00,512 Fri-23:59,10M Sat-10:00,1M Sun-20:00,off"
-The --order-by flag controls the order in which files in the backlog are
-processed in rclone sync, rclone copy and rclone move.
+It means that, the transfer bandwidth will be set to 512 KiB/s on
+Monday. It will rise to 10 MiB/s before the end of Friday. At 10:00 on
+Saturday it will be set to 1 MiB/s. From 20:00 on Sunday it will be
+unlimited.
-The order by string is constructed like this. The first part describes
-what aspect is being measured:
+Timeslots without WEEKDAY are extended to the whole week. So this
+example:
-- size - order by the size of the files
-- name - order by the full path of the files
-- modtime - order by the modification date of the files
+--bwlimit "Mon-00:00,512 12:00,1M Sun-20:00,off"
-This can have a modifier appended with a comma:
+Is equivalent to this:
-- ascending or asc - order so that the smallest (or oldest) is
- processed first
-- descending or desc - order so that the largest (or newest) is
- processed first
-- mixed - order so that the smallest is processed first for some
- threads and the largest for others
+--bwlimit "Mon-00:00,512Mon-12:00,1M Tue-12:00,1M Wed-12:00,1M Thu-12:00,1M Fri-12:00,1M Sat-12:00,1M Sun-12:00,1M Sun-20:00,off"
-If the modifier is mixed then it can have an optional percentage (which
-defaults to 50), e.g. size,mixed,25 which means that 25% of the threads
-should be taking the smallest items and 75% the largest. The threads
-which take the smallest first will always take the smallest first and
-likewise the largest first threads. The mixed mode can be useful to
-minimise the transfer time when you are transferring a mixture of large
-and small files - the large files are guaranteed upload threads and
-bandwidth and the small files will be processed continuously.
+Bandwidth limit apply to the data transfer for all backends. For most
+backends the directory listing bandwidth is also included (exceptions
+being the non HTTP backends, ftp, sftp and storj).
-If no modifier is supplied then the order is ascending.
+Note that the units are Byte/s, not bit/s. Typically connections are
+measured in bit/s - to convert divide by 8. For example, let's say you
+have a 10 Mbit/s connection and you wish rclone to use half of it - 5
+Mbit/s. This is 5/8 = 0.625 MiB/s so you would use a --bwlimit 0.625M
+parameter for rclone.
-For example
+On Unix systems (Linux, macOS, …) the bandwidth limiter can be toggled
+by sending a SIGUSR2 signal to rclone. This allows to remove the
+limitations of a long running rclone transfer and to restore it back to
+the value specified with --bwlimit quickly when needed. Assuming there
+is only one rclone instance running, you can toggle the limiter like
+this:
-- --order-by size,desc - send the largest files first
-- --order-by modtime,ascending - send the oldest files first
-- --order-by name - send the files with alphabetically by path first
+ kill -SIGUSR2 $(pidof rclone)
-If the --order-by flag is not supplied or it is supplied with an empty
-string then the default ordering will be used which is as scanned. With
---checkers 1 this is mostly alphabetical, however with the default
---checkers 8 it is somewhat random.
+If you configure rclone with a remote control then you can use change
+the bwlimit dynamically:
-Limitations
+ rclone rc core/bwlimit rate=1M
-The --order-by flag does not do a separate pass over the data. This
-means that it may transfer some files out of the order specified if
+--bwlimit-file=BANDWIDTH_SPEC
-- there are no files in the backlog or the source has not been fully
- scanned yet
-- there are more than --max-backlog files in the backlog
+This option controls per file bandwidth limit. For the options see the
+--bwlimit flag.
-Rclone will do its best to transfer the best file it has so in practice
-this should not cause a problem. Think of --order-by as being more of a
-best efforts flag rather than a perfect ordering.
+For example use this to allow no transfers to be faster than 1 MiB/s
-If you want perfect ordering then you will need to specify --check-first
-which will find all the files which need transferring first before
-transferring any.
+ --bwlimit-file 1M
---password-command SpaceSepList
+This can be used in conjunction with --bwlimit.
-This flag supplies a program which should supply the config password
-when run. This is an alternative to rclone prompting for the password or
-setting the RCLONE_CONFIG_PASS variable.
+Note that if a schedule is provided the file will use the schedule in
+effect at the start of the transfer.
-The argument to this should be a command with a space separated list of
-arguments. If one of the arguments has a space in then enclose it in ",
-if you want a literal " in an argument then enclose the argument in "
-and double the ". See CSV encoding for more info.
+--buffer-size=SIZE
-Eg
+Use this sized buffer to speed up file transfers. Each --transfer will
+use this much memory for buffering.
- --password-command echo hello
- --password-command echo "hello with space"
- --password-command echo "hello with ""quotes"" and space"
+When using mount or cmount each open file descriptor will use this much
+memory for buffering. See the mount documentation for more details.
-See the Configuration Encryption for more info.
+Set to 0 to disable the buffering for the minimum memory usage.
-See a Windows PowerShell example on the Wiki.
+Note that the memory allocation of the buffers is influenced by the
+--use-mmap flag.
--P, --progress
+--cache-dir=DIR
-This flag makes rclone update the stats in a static block in the
-terminal providing a realtime overview of the transfer.
+Specify the directory rclone will use for caching, to override the
+default.
-Any log messages will scroll above the static block. Log messages will
-push the static block down to the bottom of the terminal where it will
-stay.
+Default value is depending on operating system: - Windows
+%LocalAppData%\rclone, if LocalAppData is defined. - macOS
+$HOME/Library/Caches/rclone if HOME is defined. - Unix
+$XDG_CACHE_HOME/rclone if XDG_CACHE_HOME is defined, else
+$HOME/.cache/rclone if HOME is defined. - Fallback (on all OS) to
+$TMPDIR/rclone, where TMPDIR is the value from --temp-dir.
-Normally this is updated every 500mS but this period can be overridden
-with the --stats flag.
+You can use the config paths command to see the current value.
-This can be used with the --stats-one-line flag for a simpler display.
+Cache directory is heavily used by the VFS File Caching mount feature,
+but also by serve, GUI and other parts of rclone.
-Note: On Windows until this bug is fixed all non-ASCII characters will
-be replaced with . when --progress is in use.
+--check-first
---progress-terminal-title
+If this flag is set then in a sync, copy or move, rclone will do all the
+checks to see whether files need to be transferred before doing any of
+the transfers. Normally rclone would start running transfers as soon as
+possible.
-This flag, when used with -P/--progress, will print the string ETA: %s
-to the terminal title.
+This flag can be useful on IO limited systems where transfers interfere
+with checking.
--q, --quiet
+It can also be useful to ensure perfect ordering when using --order-by.
-This flag will limit rclone's output to error messages only.
+If both --check-first and --order-by are set when doing rclone move then
+rclone will use the transfer thread to delete source files which don't
+need transferring. This will enable perfect ordering of the transfers
+and deletes but will cause the transfer stats to have more items in than
+expected.
---refresh-times
+Using this flag can use more memory as it effectively sets --max-backlog
+to infinite. This means that all the info on the objects to transfer is
+held in memory before the transfers start.
-The --refresh-times flag can be used to update modification times of
-existing files when they are out of sync on backends which don't support
-hashes.
+--checkers=N
-This is useful if you uploaded files with the incorrect timestamps and
-you now wish to correct them.
+Originally controlling just the number of file checkers to run in
+parallel, e.g. by rclone copy. Now a fairly universal parallelism
+control used by rclone in several places.
-This flag is only useful for destinations which don't support hashes
-(e.g. crypt).
+Note: checkers do the equality checking of files during a sync. For some
+storage systems (e.g. S3, Swift, Dropbox) this can take a significant
+amount of time so they are run in parallel.
-This can be used any of the sync commands sync, copy or move.
+The default is to run 8 checkers in parallel. However, in case of
+slow-reacting backends you may need to lower (rather than increase) this
+default by setting --checkers to 4 or less threads. This is especially
+advised if you are experiencing backend server crashes during file
+checking phase (e.g. on subsequent or top-up backups where little or no
+file copying is done and checking takes up most of the time). Increase
+this setting only with utmost care, while monitoring your server health
+and file checking throughput.
-To use this flag you will need to be doing a modification time sync (so
-not using --size-only or --checksum). The flag will have no effect when
-using --size-only or --checksum.
+-c, --checksum
-If this flag is used when rclone comes to upload a file it will check to
-see if there is an existing file on the destination. If this file
-matches the source with size (and checksum if available) but has a
-differing timestamp then instead of re-uploading it, rclone will update
-the timestamp on the destination file. If the checksum does not match
-rclone will upload the new file. If the checksum is absent (e.g. on a
-crypt backend) then rclone will update the timestamp.
+Normally rclone will look at modification time and size of files to see
+if they are equal. If you set this flag then rclone will check the file
+hash and size to determine if files are equal.
-Note that some remotes can't set the modification time without
-re-uploading the file so this flag is less useful on them.
+This is useful when the remote doesn't support setting modified time and
+a more accurate sync is desired than just checking the file size.
-Normally if you are doing a modification time sync rclone will update
-modification times without --refresh-times provided that the remote
-supports checksums and the checksums match on the file. However if the
-checksums are absent then rclone will upload the file rather than
-setting the timestamp as this is the safe behaviour.
+This is very useful when transferring between remotes which store the
+same hash type on the object, e.g. Drive and Swift. For details of which
+remotes support which hash type see the table in the overview section.
---retries int
+Eg rclone --checksum sync s3:/bucket swift:/bucket would run much
+quicker than without the --checksum flag.
-Retry the entire sync if it fails this many times it fails (default 3).
+When using this flag, rclone won't update mtimes of remote files if they
+are incorrect as it would normally.
-Some remotes can be unreliable and a few retries help pick up the files
-which didn't get transferred because of errors.
+--color WHEN
-Disable retries with --retries 1.
+Specify when colors (and other ANSI codes) should be added to the
+output.
---retries-sleep=TIME
+AUTO (default) only allows ANSI codes when the output is a terminal
-This sets the interval between each retry specified by --retries
+NEVER never allow ANSI codes
-The default is 0. Use 0 to disable.
+ALWAYS always add ANSI codes, regardless of the output format (terminal
+or file)
---server-side-across-configs
+--compare-dest=DIR
-Allow server-side operations (e.g. copy or move) to work across
-different configurations.
+When using sync, copy or move DIR is checked in addition to the
+destination for files. If a file identical to the source is found that
+file is NOT copied from source. This is useful to copy just files that
+have changed since the last backup.
-This can be useful if you wish to do a server-side copy or move between
-two remotes which use the same backend but are configured differently.
+You must use the same remote as the destination of the sync. The compare
+directory must not overlap the destination directory.
-Note that this isn't enabled by default because it isn't easy for rclone
-to tell if it will work between any two configurations.
+See --copy-dest and --backup-dir.
---size-only
+--config=CONFIG_FILE
-Normally rclone will look at modification time and size of files to see
-if they are equal. If you set this flag then rclone will check only the
-size.
+Specify the location of the rclone configuration file, to override the
+default. E.g. rclone config --config="rclone.conf".
-This can be useful transferring files from Dropbox which have been
-modified by the desktop sync client which doesn't set checksums of
-modification times in the same way as rclone.
+The exact default is a bit complex to describe, due to changes
+introduced through different versions of rclone while preserving
+backwards compatibility, but in most cases it is as simple as:
---stats=TIME
+- %APPDATA%/rclone/rclone.conf on Windows
+- ~/.config/rclone/rclone.conf on other
-Commands which transfer data (sync, copy, copyto, move, moveto) will
-print data transfer stats at regular intervals to show their progress.
+The complete logic is as follows: Rclone will look for an existing
+configuration file in any of the following locations, in priority order:
-This sets the interval.
+1. rclone.conf (in program directory, where rclone executable is)
+2. %APPDATA%/rclone/rclone.conf (only on Windows)
+3. $XDG_CONFIG_HOME/rclone/rclone.conf (on all systems, including
+ Windows)
+4. ~/.config/rclone/rclone.conf (see below for explanation of ~ symbol)
+5. ~/.rclone.conf
-The default is 1m. Use 0 to disable.
+If no existing configuration file is found, then a new one will be
+created in the following location:
-If you set the stats interval then all commands can show stats. This can
-be useful when running other commands, check or mount for example.
+- On Windows: Location 2 listed above, except in the unlikely event
+ that APPDATA is not defined, then location 4 is used instead.
+- On Unix: Location 3 if XDG_CONFIG_HOME is defined, else location 4.
+- Fallback to location 5 (on all OS), when the rclone directory cannot
+ be created, but if also a home directory was not found then path
+ .rclone.conf relative to current working directory will be used as a
+ final resort.
-Stats are logged at INFO level by default which means they won't show at
-default log level NOTICE. Use --stats-log-level NOTICE or -v to make
-them show. See the Logging section for more info on log levels.
+The ~ symbol in paths above represent the home directory of the current
+user on any OS, and the value is defined as following:
-Note that on macOS you can send a SIGINFO (which is normally ctrl-T in
-the terminal) to make the stats print immediately.
+- On Windows: %HOME% if defined, else %USERPROFILE%, or else
+ %HOMEDRIVE%\%HOMEPATH%.
+- On Unix: $HOME if defined, else by looking up current user in
+ OS-specific user database (e.g. passwd file), or else use the result
+ from shell command cd && pwd.
---stats-file-name-length integer
+If you run rclone config file you will see where the default location is
+for you.
-By default, the --stats output will truncate file names and paths longer
-than 40 characters. This is equivalent to providing
---stats-file-name-length 40. Use --stats-file-name-length 0 to disable
-any truncation of file names printed by stats.
+The fact that an existing file rclone.conf in the same directory as the
+rclone executable is always preferred, means that it is easy to run in
+"portable" mode by downloading rclone executable to a writable directory
+and then create an empty file rclone.conf in the same directory.
---stats-log-level string
+If the location is set to empty string "" or path to a file with name
+notfound, or the os null device represented by value NUL on Windows and
+/dev/null on Unix systems, then rclone will keep the config file in
+memory only.
-Log level to show --stats output at. This can be DEBUG, INFO, NOTICE, or
-ERROR. The default is INFO. This means at the default level of logging
-which is NOTICE the stats won't show - if you want them to then use
---stats-log-level NOTICE. See the Logging section for more info on log
-levels.
-
---stats-one-line
+The file format is basic INI: Sections of text, led by a [section]
+header and followed by key=value entries on separate lines. In rclone
+each remote is represented by its own section, where the section name
+defines the name of the remote. Options are specified as the key=value
+entries, where the key is the option name without the --backend- prefix,
+in lowercase and with _ instead of -. E.g. option --mega-hard-delete
+corresponds to key hard_delete. Only backend options can be specified. A
+special, and required, key type identifies the storage system, where the
+value is the internal lowercase name as returned by command
+rclone help backends. Comments are indicated by ; or # at the beginning
+of a line.
-When this is specified, rclone condenses the stats into a single line
-showing the most important stats only.
+Example:
---stats-one-line-date
+ [megaremote]
+ type = mega
+ user = you@example.com
+ pass = PDPcQVVjVtzFY-GTdDFozqBhTdsPg3qH
-When this is specified, rclone enables the single-line stats and
-prepends the display with a date string. The default is
-2006/01/02 15:04:05 -
+Note that passwords are in obscured form. Also, many storage systems
+uses token-based authentication instead of passwords, and this requires
+additional steps. It is easier, and safer, to use the interactive
+command rclone config instead of manually editing the configuration
+file.
---stats-one-line-date-format
+The configuration file will typically contain login information, and
+should therefore have restricted permissions so that only the current
+user can read it. Rclone tries to ensure this when it writes the file.
+You may also choose to encrypt the file.
-When this is specified, rclone enables the single-line stats and
-prepends the display with a user-supplied date string. The date string
-MUST be enclosed in quotes. Follow golang specs for date formatting
-syntax.
+When token-based authentication are used, the configuration file must be
+writable, because rclone needs to update the tokens inside it.
---stats-unit=bits|bytes
+To reduce risk of corrupting an existing configuration file, rclone will
+not write directly to it when saving changes. Instead it will first
+write to a new, temporary, file. If a configuration file already
+existed, it will (on Unix systems) try to mirror its permissions to the
+new file. Then it will rename the existing file to a temporary name as
+backup. Next, rclone will rename the new file to the correct name,
+before finally cleaning up by deleting the backup file.
-By default, data transfer rates will be printed in bytes per second.
+If the configuration file path used by rclone is a symbolic link, then
+this will be evaluated and rclone will write to the resolved path,
+instead of overwriting the symbolic link. Temporary files used in the
+process (described above) will be written to the same parent directory
+as that of the resolved configuration file, but if this directory is
+also a symbolic link it will not be resolved and the temporary files
+will be written to the location of the directory symbolic link.
-This option allows the data rate to be printed in bits per second.
+--contimeout=TIME
-Data transfer volume will still be reported in bytes.
+Set the connection timeout. This should be in go time format which looks
+like 5s for 5 seconds, 10m for 10 minutes, or 3h30m.
-The rate is reported as a binary unit, not SI unit. So 1 Mbit/s equals
-1,048,576 bit/s and not 1,000,000 bit/s.
+The connection timeout is the amount of time rclone will wait for a
+connection to go through to a remote object storage system. It is 1m by
+default.
-The default is bytes.
+--copy-dest=DIR
---suffix=SUFFIX
+When using sync, copy or move DIR is checked in addition to the
+destination for files. If a file identical to the source is found that
+file is server-side copied from DIR to the destination. This is useful
+for incremental backup.
-When using sync, copy or move any files which would have been
-overwritten or deleted will have the suffix added to them. If there is a
-file with the same path (after the suffix has been added), then it will
-be overwritten.
+The remote in use must support server-side copy and you must use the
+same remote as the destination of the sync. The compare directory must
+not overlap the destination directory.
-The remote in use must support server-side move or copy and you must use
-the same remote as the destination of the sync.
+See --compare-dest and --backup-dir.
-This is for use with files to add the suffix in the current directory or
-with --backup-dir. See --backup-dir for more info.
+--dedupe-mode MODE
-For example
+Mode to run dedupe command in. One of interactive, skip, first, newest,
+oldest, rename. The default is interactive.
+See the dedupe command for more information as to what these options
+mean.
- rclone copy --interactive /path/to/local/file remote:current --suffix .bak
+--default-time TIME
-will copy /path/to/local to remote:current, but for any files which
-would have been updated or deleted have .bak added.
+If a file or directory does have a modification time rclone can read
+then rclone will display this fixed time instead.
-If using rclone sync with --suffix and without --backup-dir then it is
-recommended to put a filter rule in excluding the suffix otherwise the
-sync will delete the backup files.
+The default is 2000-01-01 00:00:00 UTC. This can be configured in any of
+the ways shown in the time or duration options.
- rclone sync --interactive /path/to/local/file remote:current --suffix .bak --exclude "*.bak"
+For example --default-time 2020-06-01 to set the default time to the 1st
+of June 2020 or --default-time 0s to set the default time to the time
+rclone started up.
---suffix-keep-extension
+--disable FEATURE,FEATURE,...
-When using --suffix, setting this causes rclone put the SUFFIX before
-the extension of the files that it backs up rather than after.
+This disables a comma separated list of optional features. For example
+to disable server-side move and server-side copy use:
-So let's say we had --suffix -2019-01-01, without the flag file.txt
-would be backed up to file.txt-2019-01-01 and with the flag it would be
-backed up to file-2019-01-01.txt. This can be helpful to make sure the
-suffixed files can still be opened.
+ --disable move,copy
-If a file has two (or more) extensions and the second (or subsequent)
-extension is recognised as a valid mime type, then the suffix will go
-before that extension. So file.tar.gz would be backed up to
-file-2019-01-01.tar.gz whereas file.badextension.gz would be backed up
-to file.badextension-2019-01-01.gz.
+The features can be put in any case.
---syslog
+To see a list of which features can be disabled use:
-On capable OSes (not Windows or Plan9) send all log output to syslog.
+ --disable help
-This can be useful for running rclone in a script or rclone mount.
+The features a remote has can be seen in JSON format with:
---syslog-facility string
+ rclone backend features remote:
-If using --syslog this sets the syslog facility (e.g. KERN, USER). See
-man syslog for a list of possible facilities. The default facility is
-DAEMON.
+See the overview features and optional features to get an idea of which
+feature does what.
---temp-dir=DIR
+Note that some features can be set to true if they are true/false
+feature flag features by prefixing them with !. For example the
+CaseInsensitive feature can be forced to false with
+--disable CaseInsensitive and forced to true with
+--disable '!CaseInsensitive'. In general it isn't a good idea doing this
+but it may be useful in extremis.
-Specify the directory rclone will use for temporary files, to override
-the default. Make sure the directory exists and have accessible
-permissions.
+(Note that ! is a shell command which you will need to escape with
+single quotes or a backslash on unix like platforms.)
-By default the operating system's temp directory will be used: - On Unix
-systems, $TMPDIR if non-empty, else /tmp. - On Windows, the first
-non-empty value from %TMP%, %TEMP%, %USERPROFILE%, or the Windows
-directory.
+This flag can be useful for debugging and in exceptional circumstances
+(e.g. Google Drive limiting the total volume of Server Side Copies to
+100 GiB/day).
-When overriding the default with this option, the specified path will be
-set as value of environment variable TMPDIR on Unix systems and TMP and
-TEMP on Windows.
+--disable-http2
-You can use the config paths command to see the current value.
+This stops rclone from trying to use HTTP/2 if available. This can
+sometimes speed up transfers due to a problem in the Go standard
+library.
---tpslimit float
+--dscp VALUE
-Limit transactions per second to this number. Default is 0 which is used
-to mean unlimited transactions per second.
+Specify a DSCP value or name to use in connections. This could help QoS
+system to identify traffic class. BE, EF, DF, LE, CSx and AFxx are
+allowed.
-A transaction is roughly defined as an API call; its exact meaning will
-depend on the backend. For HTTP based backends it is an HTTP
-PUT/GET/POST/etc and its response. For FTP/SFTP it is a round trip
-transaction over TCP.
+See the description of differentiated services to get an idea of this
+field. Setting this to 1 (LE) to identify the flow to SCAVENGER class
+can avoid occupying too much bandwidth in a network with DiffServ
+support (RFC 8622).
-For example, to limit rclone to 10 transactions per second use
---tpslimit 10, or to 1 transaction every 2 seconds use --tpslimit 0.5.
+For example, if you configured QoS on router to handle LE properly.
+Running:
-Use this when the number of transactions per second from rclone is
-causing a problem with the cloud storage provider (e.g. getting you
-banned or rate limited).
+ rclone copy --dscp LE from:/from to:/to
-This can be very useful for rclone mount to control the behaviour of
-applications using it.
+would make the priority lower than usual internet flows.
-This limit applies to all HTTP based backends and to the FTP and SFTP
-backends. It does not apply to the local backend or the Storj backend.
+This option has no effect on Windows (see golang/go#42728).
-See also --tpslimit-burst.
+-n, --dry-run
---tpslimit-burst int
+Do a trial run with no permanent changes. Use this to see what rclone
+would do without actually doing it. Useful when setting up the sync
+command which deletes files in the destination.
-Max burst of transactions for --tpslimit (default 1).
+--expect-continue-timeout=TIME
-Normally --tpslimit will do exactly the number of transaction per second
-specified. However if you supply --tps-burst then rclone can save up
-some transactions from when it was idle giving a burst of up to the
-parameter supplied.
+This specifies the amount of time to wait for a server's first response
+headers after fully writing the request headers if the request has an
+"Expect: 100-continue" header. Not all backends support using this.
-For example if you provide --tpslimit-burst 10 then if rclone has been
-idle for more than 10*--tpslimit then it can do 10 transactions very
-quickly before they are limited again.
+Zero means no timeout and causes the body to be sent immediately,
+without waiting for the server to approve. This time does not include
+the time to send the request header.
-This may be used to increase performance of --tpslimit without changing
-the long term average number of transactions per second.
+The default is 1s. Set to 0 to disable.
---track-renames
+--error-on-no-transfer
-By default, rclone doesn't keep track of renamed files, so if you rename
-a file locally then sync it to a remote, rclone will delete the old file
-on the remote and upload a new copy.
+By default, rclone will exit with return code 0 if there were no errors.
-An rclone sync with --track-renames runs like a normal sync, but keeps
-track of objects which exist in the destination but not in the source
-(which would normally be deleted), and which objects exist in the source
-but not the destination (which would normally be transferred). These
-objects are then candidates for renaming.
+This option allows rclone to return exit code 9 if no files were
+transferred between the source and destination. This allows using rclone
+in scripts, and triggering follow-on actions if data was copied, or
+skipping if not.
-After the sync, rclone matches up the source only and destination only
-objects using the --track-renames-strategy specified and either renames
-the destination object or transfers the source and deletes the
-destination object. --track-renames is stateless like all of rclone's
-syncs.
+NB: Enabling this option turns a usually non-fatal error into a
+potentially fatal one - please check and adjust your scripts
+accordingly!
-To use this flag the destination must support server-side copy or
-server-side move, and to use a hash based --track-renames-strategy (the
-default) the source and the destination must have a compatible hash.
+--fs-cache-expire-duration=TIME
-If the destination does not support server-side copy or move, rclone
-will fall back to the default behaviour and log an error level message
-to the console.
+When using rclone via the API rclone caches created remotes for 5
+minutes by default in the "fs cache". This means that if you do repeated
+actions on the same remote then rclone won't have to build it again from
+scratch, which makes it more efficient.
-Encrypted destinations are not currently supported by --track-renames if
---track-renames-strategy includes hash.
+This flag sets the time that the remotes are cached for. If you set it
+to 0 (or negative) then rclone won't cache the remotes at all.
-Note that --track-renames is incompatible with --no-traverse and that it
-uses extra memory to keep track of all the rename candidates.
+Note that if you use some flags, eg --backup-dir and if this is set to 0
+rclone may build two remotes (one for the source or destination and one
+for the --backup-dir where it may have only built one before.
-Note also that --track-renames is incompatible with --delete-before and
-will select --delete-after instead of --delete-during.
+--fs-cache-expire-interval=TIME
---track-renames-strategy (hash,modtime,leaf,size)
+This controls how often rclone checks for cached remotes to expire. See
+the --fs-cache-expire-duration documentation above for more info. The
+default is 60s, set to 0 to disable expiry.
-This option changes the file matching criteria for --track-renames.
+--header
-The matching is controlled by a comma separated selection of these
-tokens:
+Add an HTTP header for all transactions. The flag can be repeated to add
+multiple headers.
-- modtime - the modification time of the file - not supported on all
- backends
-- hash - the hash of the file contents - not supported on all backends
-- leaf - the name of the file not including its directory name
-- size - the size of the file (this is always enabled)
+If you want to add headers only for uploads use --header-upload and if
+you want to add headers only for downloads use --header-download.
-The default option is hash.
+This flag is supported for all HTTP based backends even those not
+supported by --header-upload and --header-download so may be used as a
+workaround for those with care.
-Using --track-renames-strategy modtime,leaf would match files based on
-modification time, the leaf of the file name and the size only.
+ rclone ls remote:test --header "X-Rclone: Foo" --header "X-LetMeIn: Yes"
-Using --track-renames-strategy modtime or leaf can enable
---track-renames support for encrypted destinations.
+--header-download
-Note that the hash strategy is not supported with encrypted
-destinations.
+Add an HTTP header for all download transactions. The flag can be
+repeated to add multiple headers.
---delete-(before,during,after)
+ rclone sync --interactive s3:test/src ~/dst --header-download "X-Amz-Meta-Test: Foo" --header-download "X-Amz-Meta-Test2: Bar"
-This option allows you to specify when files on your destination are
-deleted when you sync folders.
+See the GitHub issue here for currently supported backends.
-Specifying the value --delete-before will delete all files present on
-the destination, but not on the source before starting the transfer of
-any new or updated files. This uses two passes through the file systems,
-one for the deletions and one for the copies.
+--header-upload
-Specifying --delete-during will delete files while checking and
-uploading files. This is the fastest option and uses the least memory.
+Add an HTTP header for all upload transactions. The flag can be repeated
+to add multiple headers.
-Specifying --delete-after (the default value) will delay deletion of
-files until all new/updated files have been successfully transferred.
-The files to be deleted are collected in the copy pass then deleted
-after the copy pass has completed successfully. The files to be deleted
-are held in memory so this mode may use more memory. This is the safest
-mode as it will only delete files if there have been no errors
-subsequent to that. If there have been errors before the deletions start
-then you will get the message
-not deleting files as there were IO errors.
+ rclone sync --interactive ~/src s3:test/dst --header-upload "Content-Disposition: attachment; filename='cool.html'" --header-upload "X-Amz-Meta-Test: FooBar"
---fast-list
+See the GitHub issue here for currently supported backends.
-When doing anything which involves a directory listing (e.g. sync, copy,
-ls - in fact nearly every command), rclone normally lists a directory
-and processes it before using more directory lists to process any
-subdirectories. This can be parallelised and works very quickly using
-the least amount of memory.
+--human-readable
-However, some remotes have a way of listing all files beneath a
-directory in one (or a small number) of transactions. These tend to be
-the bucket-based remotes (e.g. S3, B2, GCS, Swift).
+Rclone commands output values for sizes (e.g. number of bytes) and
+counts (e.g. number of files) either as raw numbers, or in
+human-readable format.
-If you use the --fast-list flag then rclone will use this method for
-listing directories. This will have the following consequences for the
-listing:
+In human-readable format the values are scaled to larger units,
+indicated with a suffix shown after the value, and rounded to three
+decimals. Rclone consistently uses binary units (powers of 2) for sizes
+and decimal units (powers of 10) for counts. The unit prefix for size is
+according to IEC standard notation, e.g. Ki for kibi. Used with byte
+unit, 1 KiB means 1024 Byte. In list type of output, only the unit
+prefix appended to the value (e.g. 9.762Ki), while in more textual
+output the full unit is shown (e.g. 9.762 KiB). For counts the SI
+standard notation is used, e.g. prefix k for kilo. Used with file
+counts, 1k means 1000 files.
-- It will use fewer transactions (important if you pay for them)
-- It will use more memory. Rclone has to load the whole listing into
- memory.
-- It may be faster because it uses fewer transactions
-- It may be slower because it can't be parallelized
+The various list commands output raw numbers by default. Option
+--human-readable will make them output values in human-readable format
+instead (with the short unit prefix).
-rclone should always give identical results with and without
---fast-list.
+The about command outputs human-readable by default, with a
+command-specific option --full to output the raw numbers instead.
-If you pay for transactions and can fit your entire sync listing into
-memory then --fast-list is recommended. If you have a very big sync to
-do then don't use --fast-list otherwise you will run out of memory.
+Command size outputs both human-readable and raw numbers in the same
+output.
-If you use --fast-list on a remote which doesn't support it, then rclone
-will just ignore it.
+The tree command also considers --human-readable, but it will not use
+the exact same notation as the other commands: It rounds to one decimal,
+and uses single letter suffix, e.g. K instead of Ki. The reason for this
+is that it relies on an external library.
---timeout=TIME
+The interactive command ncdu shows human-readable by default, and
+responds to key u for toggling human-readable format.
-This sets the IO idle timeout. If a transfer has started but then
-becomes idle for this long it is considered broken and disconnected.
+--ignore-case-sync
-The default is 5m. Set to 0 to disable.
+Using this option will cause rclone to ignore the case of the files when
+synchronizing so files will not be copied/synced when the existing
+filenames are the same, even if the casing is different.
---transfers=N
+--ignore-checksum
-The number of file transfers to run in parallel. It can sometimes be
-useful to set this to a smaller number if the remote is giving a lot of
-timeouts or bigger if you have lots of bandwidth and a fast remote.
+Normally rclone will check that the checksums of transferred files
+match, and give an error "corrupted on transfer" if they don't.
-The default is to run 4 file transfers in parallel.
+You can use this option to skip that check. You should only use it if
+you have had the "corrupted on transfer" error message and you are sure
+you might want to transfer potentially corrupted data.
-Look at --multi-thread-streams if you would like to control single file
-transfers.
+--ignore-existing
--u, --update
+Using this option will make rclone unconditionally skip all files that
+exist on the destination, no matter the content of these files.
-This forces rclone to skip any files which exist on the destination and
-have a modified time that is newer than the source file.
+While this isn't a generally recommended option, it can be useful in
+cases where your files change due to encryption. However, it cannot
+correct partial transfers in case a transfer was interrupted.
-This can be useful in avoiding needless transfers when transferring to a
-remote which doesn't support modification times directly (or when using
---use-server-modtime to avoid extra API calls) as it is more accurate
-than a --size-only check and faster than using --checksum. On such
-remotes (or when using --use-server-modtime) the time checked will be
-the uploaded time.
+When performing a move/moveto command, this flag will leave skipped
+files in the source location unchanged when a file with the same name
+exists on the destination.
-If an existing destination file has a modification time older than the
-source file's, it will be updated if the sizes are different. If the
-sizes are the same, it will be updated if the checksum is different or
-not available.
+--ignore-size
-If an existing destination file has a modification time equal (within
-the computed modify window) to the source file's, it will be updated if
-the sizes are different. The checksum will not be checked in this case
-unless the --checksum flag is provided.
+Normally rclone will look at modification time and size of files to see
+if they are equal. If you set this flag then rclone will check only the
+modification time. If --checksum is set then it only checks the
+checksum.
-In all other cases the file will not be updated.
+It will also cause rclone to skip verifying the sizes are the same after
+transfer.
-Consider using the --modify-window flag to compensate for time skews
-between the source and the backend, for backends that do not support mod
-times, and instead use uploaded times. However, if the backend does not
-support checksums, note that syncing or copying within the time skew
-window may still result in additional transfers for safety.
+This can be useful for transferring files to and from OneDrive which
+occasionally misreports the size of image files (see #399 for more
+info).
---use-mmap
+-I, --ignore-times
-If this flag is set then rclone will use anonymous memory allocated by
-mmap on Unix based platforms and VirtualAlloc on Windows for its
-transfer buffers (size controlled by --buffer-size). Memory allocated
-like this does not go on the Go heap and can be returned to the OS
-immediately when it is finished with.
+Using this option will cause rclone to unconditionally upload all files
+regardless of the state of files on the destination.
-If this flag is not set then rclone will allocate and free the buffers
-using the Go memory allocator which may use more memory as memory pages
-are returned less aggressively to the OS.
+Normally rclone would skip any files that have the same modification
+time and are the same size (or have the same checksum if using
+--checksum).
-It is possible this does not work well on all platforms so it is
-disabled by default; in the future it may be enabled by default.
+--immutable
---use-server-modtime
+Treat source and destination files as immutable and disallow
+modification.
-Some object-store backends (e.g, Swift, S3) do not preserve file
-modification times (modtime). On these backends, rclone stores the
-original modtime as additional metadata on the object. By default it
-will make an API call to retrieve the metadata when the modtime is
-needed by an operation.
+With this option set, files will be created and deleted as requested,
+but existing files will never be updated. If an existing file does not
+match between the source and destination, rclone will give the error
+Source and destination exist but do not match: immutable file modified.
-Use this flag to disable the extra API call and rely instead on the
-server's modified time. In cases such as a local to remote sync using
---update, knowing the local file is newer than the time it was last
-uploaded to the remote is sufficient. In those cases, this flag can
-speed up the process and reduce the number of API calls necessary.
+Note that only commands which transfer files (e.g. sync, copy, move) are
+affected by this behavior, and only modification is disallowed. Files
+may still be deleted explicitly (e.g. delete, purge) or implicitly (e.g.
+sync, move). Use copy --immutable if it is desired to avoid deletion as
+well as modification.
-Using this flag on a sync operation without also using --update would
-cause all files modified at any time other than the last upload time to
-be uploaded again, which is probably not what you want.
+This can be useful as an additional layer of protection for immutable or
+append-only data sets (notably backup archives), where modification
+implies corruption and should not be propagated.
--v, -vv, --verbose
+--inplace
-With -v rclone will tell you about each file that is transferred and a
-small number of significant events.
+The --inplace flag changes the behaviour of rclone when uploading files
+to some backends (backends with the PartialUploads feature flag set)
+such as:
-With -vv rclone will become very verbose telling you about every file it
-considers and transfers. Please send bug reports with a log with this
-setting.
+- local
+- ftp
+- sftp
-When setting verbosity as an environment variable, use RCLONE_VERBOSE=1
-or RCLONE_VERBOSE=2 for -v and -vv respectively.
+Without --inplace (the default) rclone will first upload to a temporary
+file with an extension like this, where XXXXXX represents a random
+string and .partial is --partial-suffix value (.partial by default).
--V, --version
+ original-file-name.XXXXXX.partial
-Prints the version number
+(rclone will make sure the final name is no longer than 100 characters
+by truncating the original-file-name part if necessary).
-SSL/TLS options
+When the upload is complete, rclone will rename the .partial file to the
+correct name, overwriting any existing file at that point. If the upload
+fails then the .partial file will be deleted.
-The outgoing SSL/TLS connections rclone makes can be controlled with
-these options. For example this can be very useful with the HTTP or
-WebDAV backends. Rclone HTTP servers have their own set of configuration
-for SSL/TLS which you can find in their documentation.
+This prevents other users of the backend from seeing partially uploaded
+files in their new names and prevents overwriting the old file until the
+new one is completely uploaded.
---ca-cert stringArray
+If the --inplace flag is supplied, rclone will upload directly to the
+final name without creating a .partial file.
-This loads the PEM encoded certificate authority certificates and uses
-it to verify the certificates of the servers rclone connects to.
+This means that an incomplete file will be visible in the directory
+listings while the upload is in progress and any existing files will be
+overwritten as soon as the upload starts. If the transfer fails then the
+file will be deleted. This can cause data loss of the existing file if
+the transfer fails.
-If you have generated certificates signed with a local CA then you will
-need this flag to connect to servers using those certificates.
+Note that on the local file system if you don't use --inplace hard links
+(Unix only) will be broken. And if you do use --inplace you won't be
+able to update in use executables.
---client-cert string
+Note also that versions of rclone prior to v1.63.0 behave as if the
+--inplace flag is always supplied.
-This loads the PEM encoded client side certificate.
+-i, --interactive
-This is used for mutual TLS authentication.
+This flag can be used to tell rclone that you wish a manual confirmation
+before destructive operations.
-The --client-key flag is required too when using this.
+It is recommended that you use this flag while learning rclone
+especially with rclone sync.
---client-key string
+For example
-This loads the PEM encoded client side private key used for mutual TLS
-authentication. Used in conjunction with --client-cert.
+ $ rclone delete --interactive /tmp/dir
+ rclone: delete "important-file.txt"?
+ y) Yes, this is OK (default)
+ n) No, skip this
+ s) Skip all delete operations with no more questions
+ !) Do all delete operations with no more questions
+ q) Exit rclone now.
+ y/n/s/!/q> n
---no-check-certificate=true/false
+The options mean
---no-check-certificate controls whether a client verifies the server's
-certificate chain and host name. If --no-check-certificate is true, TLS
-accepts any certificate presented by the server and any host name in
-that certificate. In this mode, TLS is susceptible to man-in-the-middle
-attacks.
+- y: Yes, this operation should go ahead. You can also press Return
+ for this to happen. You'll be asked every time unless you choose s
+ or !.
+- n: No, do not do this operation. You'll be asked every time unless
+ you choose s or !.
+- s: Skip all the following operations of this type with no more
+ questions. This takes effect until rclone exits. If there are any
+ different kind of operations you'll be prompted for them.
+- !: Do all the following operations with no more questions. Useful if
+ you've decided that you don't mind rclone doing that kind of
+ operation. This takes effect until rclone exits . If there are any
+ different kind of operations you'll be prompted for them.
+- q: Quit rclone now, just in case!
-This option defaults to false.
+--leave-root
-This should be used only for testing.
+During rmdirs it will not remove root directory, even if it's empty.
-Configuration Encryption
+--log-file=FILE
-Your configuration file contains information for logging in to your
-cloud services. This means that you should keep your rclone.conf file in
-a secure location.
+Log all of rclone's output to FILE. This is not active by default. This
+can be useful for tracking down problems with syncs in combination with
+the -v flag. See the Logging section for more info.
-If you are in an environment where that isn't possible, you can add a
-password to your configuration. This means that you will have to supply
-the password every time you start rclone.
+If FILE exists then rclone will append to it.
-To add a password to your rclone configuration, execute rclone config.
+Note that if you are using the logrotate program to manage rclone's
+logs, then you should use the copytruncate option as rclone doesn't have
+a signal to rotate logs.
- >rclone config
- Current remotes:
+--log-format LIST
- e) Edit existing remote
- n) New remote
- d) Delete remote
- s) Set configuration password
- q) Quit config
- e/n/d/s/q>
+Comma separated list of log format options. Accepted options are date,
+time, microseconds, pid, longfile, shortfile, UTC. Any other keywords
+will be silently ignored. pid will tag log messages with process
+identifier which useful with rclone mount --daemon. Other accepted
+options are explained in the go documentation. The default log format is
+"date,time".
-Go into s, Set configuration password:
+--log-level LEVEL
- e/n/d/s/q> s
- Your configuration is not encrypted.
- If you add a password, you will protect your login information to cloud services.
- a) Add Password
- q) Quit to main menu
- a/q> a
- Enter NEW configuration password:
- password:
- Confirm NEW password:
- password:
- Password set
- Your configuration is encrypted.
- c) Change Password
- u) Unencrypt configuration
- q) Quit to main menu
- c/u/q>
+This sets the log level for rclone. The default log level is NOTICE.
-Your configuration is now encrypted, and every time you start rclone you
-will have to supply the password. See below for details. In the same
-menu, you can change the password or completely remove encryption from
-your configuration.
+DEBUG is equivalent to -vv. It outputs lots of debug info - useful for
+bug reports and really finding out what rclone is doing.
-There is no way to recover the configuration if you lose your password.
+INFO is equivalent to -v. It outputs information about each transfer and
+prints stats once a minute by default.
-rclone uses nacl secretbox which in turn uses XSalsa20 and Poly1305 to
-encrypt and authenticate your configuration with secret-key
-cryptography. The password is SHA-256 hashed, which produces the key for
-secretbox. The hashed password is not stored.
+NOTICE is the default log level if no logging flags are supplied. It
+outputs very little when things are working normally. It outputs
+warnings and significant events.
-While this provides very good security, we do not recommend storing your
-encrypted rclone configuration in public if it contains sensitive
-information, maybe except if you use a very strong password.
+ERROR is equivalent to -q. It only outputs error messages.
-If it is safe in your environment, you can set the RCLONE_CONFIG_PASS
-environment variable to contain your password, in which case it will be
-used for decrypting the configuration.
+--use-json-log
-You can set this for a session from a script. For unix like systems save
-this to a file called set-rclone-password:
+This switches the log format to JSON for rclone. The fields of json log
+are level, msg, source, time.
- #!/bin/echo Source this file don't run it
+--low-level-retries NUMBER
- read -s RCLONE_CONFIG_PASS
- export RCLONE_CONFIG_PASS
+This controls the number of low level retries rclone does.
-Then source the file when you want to use it. From the shell you would
-do source set-rclone-password. It will then ask you for the password and
-set it in the environment variable.
+A low level retry is used to retry a failing operation - typically one
+HTTP request. This might be uploading a chunk of a big file for example.
+You will see low level retries in the log with the -v flag.
-An alternate means of supplying the password is to provide a script
-which will retrieve the password and print on standard output. This
-script should have a fully specified path name and not rely on any
-environment variables. The script is supplied either via
---password-command="..." command line argument or via the
-RCLONE_PASSWORD_COMMAND environment variable.
+This shouldn't need to be changed from the default in normal operations.
+However, if you get a lot of low level retries you may wish to reduce
+the value so rclone moves on to a high level retry (see the --retries
+flag) quicker.
-One useful example of this is using the passwordstore application to
-retrieve the password:
+Disable low level retries with --low-level-retries 1.
- export RCLONE_PASSWORD_COMMAND="pass rclone/config"
+--max-backlog=N
-If the passwordstore password manager holds the password for the rclone
-configuration, using the script method means the password is primarily
-protected by the passwordstore system, and is never embedded in the
-clear in scripts, nor available for examination using the standard
-commands available. It is quite possible with long running rclone
-sessions for copies of passwords to be innocently captured in log files
-or terminal scroll buffers, etc. Using the script method of supplying
-the password enhances the security of the config password considerably.
+This is the maximum allowable backlog of files in a sync/copy/move
+queued for being checked or transferred.
-If you are running rclone inside a script, unless you are using the
---password-command method, you might want to disable password prompts.
-To do that, pass the parameter --ask-password=false to rclone. This will
-make rclone fail instead of asking for a password if RCLONE_CONFIG_PASS
-doesn't contain a valid password, and --password-command has not been
-supplied.
+This can be set arbitrarily large. It will only use memory when the
+queue is in use. Note that it will use in the order of N KiB of memory
+when the backlog is in use.
-Whenever running commands that may be affected by options in a
-configuration file, rclone will look for an existing file according to
-the rules described above, and load any it finds. If an encrypted file
-is found, this includes decrypting it, with the possible consequence of
-a password prompt. When executing a command line that you know are not
-actually using anything from such a configuration file, you can avoid it
-being loaded by overriding the location, e.g. with one of the documented
-special values for memory-only configuration. Since only backend options
-can be stored in configuration files, this is normally unnecessary for
-commands that do not operate on backends, e.g. genautocomplete. However,
-it will be relevant for commands that do operate on backends in general,
-but are used without referencing a stored remote, e.g. listing local
-filesystem paths, or connection strings: rclone --config="" ls .
+Setting this large allows rclone to calculate how many files are pending
+more accurately, give a more accurate estimated finish time and make
+--order-by work more accurately.
-Developer options
+Setting this small will make rclone more synchronous to the listings of
+the remote which may be desirable.
-These options are useful when developing or debugging rclone. There are
-also some more remote specific options which aren't documented here
-which are used for testing. These start with remote name e.g.
---drive-test-option - see the docs for the remote in question.
+Setting this to a negative number will make the backlog as large as
+possible.
---cpuprofile=FILE
+--max-delete=N
-Write CPU profile to file. This can be analysed with go tool pprof.
+This tells rclone not to delete more than N files. If that limit is
+exceeded then a fatal error will be generated and rclone will stop the
+operation in progress.
---dump flag,flag,flag
+--max-delete-size=SIZE
-The --dump flag takes a comma separated list of flags to dump info
-about.
+Rclone will stop deleting files when the total size of deletions has
+reached the size specified. It defaults to off.
-Note that some headers including Accept-Encoding as shown may not be
-correct in the request and the response may not show Content-Encoding if
-the go standard libraries auto gzip encoding was in effect. In this case
-the body of the request will be gunzipped before showing it.
+If that limit is exceeded then a fatal error will be generated and
+rclone will stop the operation in progress.
-The available flags are:
+--max-depth=N
---dump headers
+This modifies the recursion depth for all the commands except purge.
-Dump HTTP headers with Authorization: lines removed. May still contain
-sensitive info. Can be very verbose. Useful for debugging only.
+So if you do rclone --max-depth 1 ls remote:path you will see only the
+files in the top level directory. Using --max-depth 2 means you will see
+all the files in first two directory levels and so on.
-Use --dump auth if you do want the Authorization: headers.
+For historical reasons the lsd command defaults to using a --max-depth
+of 1 - you can override this with the command line flag.
---dump bodies
+You can use this command to disable recursion (with --max-depth 1).
-Dump HTTP headers and bodies - may contain sensitive info. Can be very
-verbose. Useful for debugging only.
+Note that if you use this with sync and --delete-excluded the files not
+recursed through are considered excluded and will be deleted on the
+destination. Test first with --dry-run if you are not sure what will
+happen.
-Note that the bodies are buffered in memory so don't use this for
-enormous files.
+--max-duration=TIME
---dump requests
+Rclone will stop transferring when it has run for the duration
+specified. Defaults to off.
-Like --dump bodies but dumps the request bodies and the response
-headers. Useful for debugging download problems.
+When the limit is reached all transfers will stop immediately. Use
+--cutoff-mode to modify this behaviour.
---dump responses
+Rclone will exit with exit code 10 if the duration limit is reached.
-Like --dump bodies but dumps the response bodies and the request
-headers. Useful for debugging upload problems.
+--max-transfer=SIZE
---dump auth
+Rclone will stop transferring when it has reached the size specified.
+Defaults to off.
-Dump HTTP headers - will contain sensitive info such as Authorization:
-headers - use --dump headers to dump without Authorization: headers. Can
-be very verbose. Useful for debugging only.
+When the limit is reached all transfers will stop immediately. Use
+--cutoff-mode to modify this behaviour.
---dump filters
+Rclone will exit with exit code 8 if the transfer limit is reached.
-Dump the filters to the output. Useful to see exactly what include and
-exclude options are filtering on.
+--cutoff-mode=hard|soft|cautious
---dump goroutines
+This modifies the behavior of --max-transfer and --max-duration Defaults
+to --cutoff-mode=hard.
-This dumps a list of the running go-routines at the end of the command
-to standard output.
+Specifying --cutoff-mode=hard will stop transferring immediately when
+Rclone reaches the limit.
---dump openfiles
+Specifying --cutoff-mode=soft will stop starting new transfers when
+Rclone reaches the limit.
-This dumps a list of the open files at the end of the command. It uses
-the lsof command to do that so you'll need that installed to use it.
+Specifying --cutoff-mode=cautious will try to prevent Rclone from
+reaching the limit. Only applicable for --max-transfer
---memprofile=FILE
+-M, --metadata
-Write memory profile to file. This can be analysed with go tool pprof.
+Setting this flag enables rclone to copy the metadata from the source to
+the destination. For local backends this is ownership, permissions,
+xattr etc. See the metadata section for more info.
+
+--metadata-mapper SpaceSepList
+
+If you supply the parameter --metadata-mapper /path/to/program then
+rclone will use that program to map metadata from source object to
+destination object.
+
+The argument to this flag should be a command with an optional space
+separated list of arguments. If one of the arguments has a space in then
+enclose it in ", if you want a literal " in an argument then enclose the
+argument in " and double the ". See CSV encoding for more info.
+
+ --metadata-mapper "python bin/test_metadata_mapper.py"
+ --metadata-mapper 'python bin/test_metadata_mapper.py "argument with a space"'
+ --metadata-mapper 'python bin/test_metadata_mapper.py "argument with ""two"" quotes"'
+
+This uses a simple JSON based protocol with input on STDIN and output on
+STDOUT. This will be called for every file and directory copied and may
+be called concurrently.
+
+The program's job is to take a metadata blob on the input and turn it
+into a metadata blob on the output suitable for the destination backend.
+
+Input to the program (via STDIN) might look like this. This provides
+some context for the Metadata which may be important.
+
+- SrcFs is the config string for the remote that the object is
+ currently on.
+- SrcFsType is the name of the source backend.
+- DstFs is the config string for the remote that the object is being
+ copied to
+- DstFsType is the name of the destination backend.
+- Remote is the path of the file relative to the root.
+- Size, MimeType, ModTime are attributes of the file.
+- IsDir is true if this is a directory (not yet implemented).
+- ID is the source ID of the file if known.
+- Metadata is the backend specific metadata as described in the
+ backend docs.
-Filtering
+ {
+ "SrcFs": "gdrive:",
+ "SrcFsType": "drive",
+ "DstFs": "newdrive:user",
+ "DstFsType": "onedrive",
+ "Remote": "test.txt",
+ "Size": 6,
+ "MimeType": "text/plain; charset=utf-8",
+ "ModTime": "2022-10-11T17:53:10.286745272+01:00",
+ "IsDir": false,
+ "ID": "xyz",
+ "Metadata": {
+ "btime": "2022-10-11T16:53:11Z",
+ "content-type": "text/plain; charset=utf-8",
+ "mtime": "2022-10-11T17:53:10.286745272+01:00",
+ "owner": "user1@domain1.com",
+ "permissions": "...",
+ "description": "my nice file",
+ "starred": "false"
+ }
+ }
-For the filtering options
+The program should then modify the input as desired and send it to
+STDOUT. The returned Metadata field will be used in its entirety for the
+destination object. Any other fields will be ignored. Note in this
+example we translate user names and permissions and add something to the
+description:
-- --delete-excluded
-- --filter
-- --filter-from
-- --exclude
-- --exclude-from
-- --exclude-if-present
-- --include
-- --include-from
-- --files-from
-- --files-from-raw
-- --min-size
-- --max-size
-- --min-age
-- --max-age
-- --dump filters
-- --metadata-include
-- --metadata-include-from
-- --metadata-exclude
-- --metadata-exclude-from
-- --metadata-filter
-- --metadata-filter-from
+ {
+ "Metadata": {
+ "btime": "2022-10-11T16:53:11Z",
+ "content-type": "text/plain; charset=utf-8",
+ "mtime": "2022-10-11T17:53:10.286745272+01:00",
+ "owner": "user1@domain2.com",
+ "permissions": "...",
+ "description": "my nice file [migrated from domain1]",
+ "starred": "false"
+ }
+ }
-See the filtering section.
+Metadata can be removed here too.
-Remote control
+An example python program might look something like this to implement
+the above transformations.
-For the remote control options and for instructions on how to remote
-control rclone
+ import sys, json
-- --rc
-- and anything starting with --rc-
+ i = json.load(sys.stdin)
+ metadata = i["Metadata"]
+ # Add tag to description
+ if "description" in metadata:
+ metadata["description"] += " [migrated from domain1]"
+ else:
+ metadata["description"] = "[migrated from domain1]"
+ # Modify owner
+ if "owner" in metadata:
+ metadata["owner"] = metadata["owner"].replace("domain1.com", "domain2.com")
+ o = { "Metadata": metadata }
+ json.dump(o, sys.stdout, indent="\t")
-See the remote control section.
+You can find this example (slightly expanded) in the rclone source code
+at bin/test_metadata_mapper.py.
-Logging
+If you want to see the input to the metadata mapper and the output
+returned from it in the log you can use -vv --dump mapper.
-rclone has 4 levels of logging, ERROR, NOTICE, INFO and DEBUG.
+See the metadata section for more info.
-By default, rclone logs to standard error. This means you can redirect
-standard error and still see the normal output of rclone commands (e.g.
-rclone ls).
+--metadata-set key=value
-By default, rclone will produce Error and Notice level messages.
+Add metadata key = value when uploading. This can be repeated as many
+times as required. See the metadata section for more info.
-If you use the -q flag, rclone will only produce Error messages.
+--modify-window=TIME
-If you use the -v flag, rclone will produce Error, Notice and Info
-messages.
+When checking whether a file has been modified, this is the maximum
+allowed time difference that a file can have and still be considered
+equivalent.
-If you use the -vv flag, rclone will produce Error, Notice, Info and
-Debug messages.
+The default is 1ns unless this is overridden by a remote. For example OS
+X only stores modification times to the nearest second so if you are
+reading and writing to an OS X filing system this will be 1s by default.
-You can also control the log levels with the --log-level flag.
+This command line flag allows you to override that computed default.
-If you use the --log-file=FILE option, rclone will redirect Error, Info
-and Debug messages along with standard error to FILE.
+--multi-thread-write-buffer-size=SIZE
-If you use the --syslog flag then rclone will log to syslog and the
---syslog-facility control which facility it uses.
+When transferring with multiple threads, rclone will buffer SIZE bytes
+in memory before writing to disk for each thread.
-Rclone prefixes all log messages with their level in capitals, e.g. INFO
-which makes it easy to grep the log file for different kinds of
-information.
+This can improve performance if the underlying filesystem does not deal
+well with a lot of small writes in different positions of the file, so
+if you see transfers being limited by disk write speed, you might want
+to experiment with different values. Specially for magnetic drives and
+remote file systems a higher value can be useful.
-Exit Code
+Nevertheless, the default of 128k should be fine for almost all use
+cases, so before changing it ensure that network is not really your
+bottleneck.
-If any errors occur during the command execution, rclone will exit with
-a non-zero exit code. This allows scripts to detect when rclone
-operations have failed.
+As a final hint, size is not the only factor: block size (or similar
+concept) can have an impact. In one case, we observed that exact
+multiples of 16k performed much better than other values.
-During the startup phase, rclone will exit immediately if an error is
-detected in the configuration. There will always be a log message
-immediately before exiting.
+--multi-thread-chunk-size=SizeSuffix
-When rclone is running it will accumulate errors as it goes along, and
-only exit with a non-zero exit code if (after retries) there were still
-failed transfers. For every error counted there will be a high priority
-log message (visible with -q) showing the message and which file caused
-the problem. A high priority message is also shown when starting a retry
-so the user can see that any previous error messages may not be valid
-after the retry. If rclone has done a retry it will log a high priority
-message if the retry was successful.
+Normally the chunk size for multi thread transfers is set by the
+backend. However some backends such as local and smb (which implement
+OpenWriterAt but not OpenChunkWriter) don't have a natural chunk size.
-List of exit codes
+In this case the value of this option is used (default 64Mi).
-- 0 - success
-- 1 - Syntax or usage error
-- 2 - Error not otherwise categorised
-- 3 - Directory not found
-- 4 - File not found
-- 5 - Temporary error (one that more retries might fix) (Retry errors)
-- 6 - Less serious errors (like 461 errors from dropbox) (NoRetry
- errors)
-- 7 - Fatal error (one that more retries won't fix, like account
- suspended) (Fatal errors)
-- 8 - Transfer exceeded - limit set by --max-transfer reached
-- 9 - Operation successful, but no files transferred
-- 10 - Duration exceeded - limit set by --max-duration reached
+--multi-thread-cutoff=SIZE
-Environment Variables
+When transferring files above SIZE to capable backends, rclone will use
+multiple threads to transfer the file (default 256M).
-Rclone can be configured entirely using environment variables. These can
-be used to set defaults for options or config file entries.
+Capable backends are marked in the overview as MultithreadUpload. (They
+need to implement either the OpenWriterAt or OpenChunkedWriter internal
+interfaces). These include include, local, s3, azureblob, b2,
+oracleobjectstorage and smb at the time of writing.
-Options
+On the local disk, rclone preallocates the file (using
+fallocate(FALLOC_FL_KEEP_SIZE) on unix or NTSetInformationFile on
+Windows both of which takes no time) then each thread writes directly
+into the file at the correct place. This means that rclone won't create
+fragmented or sparse files and there won't be any assembly time at the
+end of the transfer.
-Every option in rclone can have its default set by environment variable.
+The number of threads used to transfer is controlled by
+--multi-thread-streams.
-To find the name of the environment variable, first, take the long
-option name, strip the leading --, change - to _, make upper case and
-prepend RCLONE_.
+Use -vv if you wish to see info about the threads.
-For example, to always set --stats 5s, set the environment variable
-RCLONE_STATS=5s. If you set stats on the command line this will override
-the environment variable setting.
+This will work with the sync/copy/move commands and friends
+copyto/moveto. Multi thread transfers will be used with rclone mount and
+rclone serve if --vfs-cache-mode is set to writes or above.
-Or to always use the trash in drive --drive-use-trash, set
-RCLONE_DRIVE_USE_TRASH=true.
+NB that this only works with supported backends as the destination but
+will work with any backend as the source.
-Verbosity is slightly different, the environment variable equivalent of
---verbose or -v is RCLONE_VERBOSE=1, or for -vv, RCLONE_VERBOSE=2.
+NB that multi-thread copies are disabled for local to local copies as
+they are faster without unless --multi-thread-streams is set explicitly.
-The same parser is used for the options and the environment variables so
-they take exactly the same form.
+NB on Windows using multi-thread transfers to the local disk will cause
+the resulting files to be sparse. Use --local-no-sparse to disable
+sparse files (which may cause long delays at the start of transfers) or
+disable multi-thread transfers with --multi-thread-streams 0
-The options set by environment variables can be seen with the -vv flag,
-e.g. rclone version -vv.
+--multi-thread-streams=N
-Config file
+When using multi thread transfers (see above --multi-thread-cutoff) this
+sets the number of streams to use. Set to 0 to disable multi thread
+transfers (Default 4).
-You can set defaults for values in the config file on an individual
-remote basis. The names of the config items are documented in the page
-for each backend.
+If the backend has a --backend-upload-concurrency setting (eg
+--s3-upload-concurrency) then this setting will be used as the number of
+transfers instead if it is larger than the value of
+--multi-thread-streams or --multi-thread-streams isn't set.
-To find the name of the environment variable, you need to set, take
-RCLONE_CONFIG_ + name of remote + _ + name of config file option and
-make it all uppercase. Note one implication here is the remote's name
-must be convertible into a valid environment variable name, so it can
-only contain letters, digits, or the _ (underscore) character.
+--no-check-dest
-For example, to configure an S3 remote named mys3: without a config file
-(using unix ways of setting environment variables):
+The --no-check-dest can be used with move or copy and it causes rclone
+not to check the destination at all when copying files.
- $ export RCLONE_CONFIG_MYS3_TYPE=s3
- $ export RCLONE_CONFIG_MYS3_ACCESS_KEY_ID=XXX
- $ export RCLONE_CONFIG_MYS3_SECRET_ACCESS_KEY=XXX
- $ rclone lsd mys3:
- -1 2016-09-21 12:54:21 -1 my-bucket
- $ rclone listremotes | grep mys3
- mys3:
+This means that:
-Note that if you want to create a remote using environment variables you
-must create the ..._TYPE variable as above.
+- the destination is not listed minimising the API calls
+- files are always transferred
+- this can cause duplicates on remotes which allow it (e.g. Google
+ Drive)
+- --retries 1 is recommended otherwise you'll transfer everything
+ again on a retry
-Note that the name of a remote created using environment variable is
-case insensitive, in contrast to regular remotes stored in config file
-as documented above. You must write the name in uppercase in the
-environment variable, but as seen from example above it will be listed
-and can be accessed in lowercase, while you can also refer to the same
-remote in uppercase:
+This flag is useful to minimise the transactions if you know that none
+of the files are on the destination.
- $ rclone lsd mys3:
- -1 2016-09-21 12:54:21 -1 my-bucket
- $ rclone lsd MYS3:
- -1 2016-09-21 12:54:21 -1 my-bucket
+This is a specialized flag which should be ignored by most users!
-Note that you can only set the options of the immediate backend, so
-RCLONE_CONFIG_MYS3CRYPT_ACCESS_KEY_ID has no effect, if myS3Crypt is a
-crypt remote based on an S3 remote. However RCLONE_S3_ACCESS_KEY_ID will
-set the access key of all remotes using S3, including myS3Crypt.
+--no-gzip-encoding
-Note also that now rclone has connection strings, it is probably easier
-to use those instead which makes the above example
+Don't set Accept-Encoding: gzip. This means that rclone won't ask the
+server for compressed files automatically. Useful if you've set the
+server to return files with Content-Encoding: gzip but you uploaded
+compressed files.
- rclone lsd :s3,access_key_id=XXX,secret_access_key=XXX:
+There is no need to set this in normal operation, and doing so will
+decrease the network transfer efficiency of rclone.
-Precedence
+--no-traverse
-The various different methods of backend configuration are read in this
-order and the first one with a value is used.
+The --no-traverse flag controls whether the destination file system is
+traversed when using the copy or move commands. --no-traverse is not
+compatible with sync and will be ignored if you supply it with sync.
-- Parameters in connection strings, e.g. myRemote,skip_links:
-- Flag values as supplied on the command line, e.g. --skip-links
-- Remote specific environment vars, e.g.
- RCLONE_CONFIG_MYREMOTE_SKIP_LINKS (see above).
-- Backend-specific environment vars, e.g. RCLONE_LOCAL_SKIP_LINKS.
-- Backend generic environment vars, e.g. RCLONE_SKIP_LINKS.
-- Config file, e.g. skip_links = true.
-- Default values, e.g. false - these can't be changed.
+If you are only copying a small number of files (or are filtering most
+of the files) and/or have a large number of files on the destination
+then --no-traverse will stop rclone listing the destination and save
+time.
-So if both --skip-links is supplied on the command line and an
-environment variable RCLONE_LOCAL_SKIP_LINKS is set, the command line
-flag will take preference.
+However, if you are copying a large number of files, especially if you
+are doing a copy where lots of the files under consideration haven't
+changed and won't need copying then you shouldn't use --no-traverse.
-The backend configurations set by environment variables can be seen with
-the -vv flag, e.g. rclone about myRemote: -vv.
+See rclone copy for an example of how to use it.
-For non backend configuration the order is as follows:
+--no-unicode-normalization
-- Flag values as supplied on the command line, e.g. --stats 5s.
-- Environment vars, e.g. RCLONE_STATS=5s.
-- Default values, e.g. 1m - these can't be changed.
+Don't normalize unicode characters in filenames during the sync routine.
-Other environment variables
+Sometimes, an operating system will store filenames containing unicode
+parts in their decomposed form (particularly macOS). Some cloud storage
+systems will then recompose the unicode, resulting in duplicate files if
+the data is ever copied back to a local filesystem.
-- RCLONE_CONFIG_PASS set to contain your config file password (see
- Configuration Encryption section)
-- HTTP_PROXY, HTTPS_PROXY and NO_PROXY (or the lowercase versions
- thereof).
- - HTTPS_PROXY takes precedence over HTTP_PROXY for https requests.
- - The environment values may be either a complete URL or a
- "host[:port]" for, in which case the "http" scheme is assumed.
-- USER and LOGNAME values are used as fallbacks for current username.
- The primary method for looking up username is OS-specific: Windows
- API on Windows, real user ID in /etc/passwd on Unix systems. In the
- documentation the current username is simply referred to as $USER.
-- RCLONE_CONFIG_DIR - rclone sets this variable for use in config
- files and sub processes to point to the directory holding the config
- file.
+Using this flag will disable that functionality, treating each unicode
+character as unique. For example, by default eÌ and é will be normalized
+into the same character. With --no-unicode-normalization they will be
+treated as unique characters.
-The options set by environment variables can be seen with the -vv and
---log-level=DEBUG flags, e.g. rclone version -vv.
+--no-update-modtime
-Configuring rclone on a remote / headless machine
+When using this flag, rclone won't update modification times of remote
+files if they are incorrect as it would normally.
-Some of the configurations (those involving oauth2) require an Internet
-connected web browser.
+This can be used if the remote is being synced with another tool also
+(e.g. the Google Drive client).
-If you are trying to set rclone up on a remote or headless box with no
-browser available on it (e.g. a NAS or a server in a datacenter) then
-you will need to use an alternative means of configuration. There are
-two ways of doing it, described below.
+--order-by string
-Configuring using rclone authorize
+The --order-by flag controls the order in which files in the backlog are
+processed in rclone sync, rclone copy and rclone move.
-On the headless box run rclone config but answer N to the
-Use web browser to automatically authenticate? question.
+The order by string is constructed like this. The first part describes
+what aspect is being measured:
- ...
- Remote config
- Use web browser to automatically authenticate rclone with remote?
- * Say Y if the machine running rclone has a web browser you can use
- * Say N if running rclone on a (remote) machine without web browser access
- If not sure try Y. If Y failed, try N.
- y) Yes (default)
- n) No
- y/n> n
- For this to work, you will need rclone available on a machine that has
- a web browser available.
+- size - order by the size of the files
+- name - order by the full path of the files
+- modtime - order by the modification date of the files
- For more help and alternate methods see: https://rclone.org/remote_setup/
+This can have a modifier appended with a comma:
- Execute the following on the machine with the web browser (same rclone
- version recommended):
+- ascending or asc - order so that the smallest (or oldest) is
+ processed first
+- descending or desc - order so that the largest (or newest) is
+ processed first
+- mixed - order so that the smallest is processed first for some
+ threads and the largest for others
- rclone authorize "amazon cloud drive"
+If the modifier is mixed then it can have an optional percentage (which
+defaults to 50), e.g. size,mixed,25 which means that 25% of the threads
+should be taking the smallest items and 75% the largest. The threads
+which take the smallest first will always take the smallest first and
+likewise the largest first threads. The mixed mode can be useful to
+minimise the transfer time when you are transferring a mixture of large
+and small files - the large files are guaranteed upload threads and
+bandwidth and the small files will be processed continuously.
- Then paste the result below:
- result>
+If no modifier is supplied then the order is ascending.
-Then on your main desktop machine
+For example
- rclone authorize "amazon cloud drive"
- If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth
- Log in and authorize rclone for access
- Waiting for code...
- Got code
- Paste the following into your remote machine --->
- SECRET_TOKEN
- <---End paste
+- --order-by size,desc - send the largest files first
+- --order-by modtime,ascending - send the oldest files first
+- --order-by name - send the files with alphabetically by path first
-Then back to the headless box, paste in the code
+If the --order-by flag is not supplied or it is supplied with an empty
+string then the default ordering will be used which is as scanned. With
+--checkers 1 this is mostly alphabetical, however with the default
+--checkers 8 it is somewhat random.
- result> SECRET_TOKEN
- --------------------
- [acd12]
- client_id =
- client_secret =
- token = SECRET_TOKEN
- --------------------
- y) Yes this is OK
- e) Edit this remote
- d) Delete this remote
- y/e/d>
+Limitations
-Configuring by copying the config file
+The --order-by flag does not do a separate pass over the data. This
+means that it may transfer some files out of the order specified if
-Rclone stores all of its config in a single configuration file. This can
-easily be copied to configure a remote rclone.
+- there are no files in the backlog or the source has not been fully
+ scanned yet
+- there are more than --max-backlog files in the backlog
-So first configure rclone on your desktop machine with
+Rclone will do its best to transfer the best file it has so in practice
+this should not cause a problem. Think of --order-by as being more of a
+best efforts flag rather than a perfect ordering.
- rclone config
+If you want perfect ordering then you will need to specify --check-first
+which will find all the files which need transferring first before
+transferring any.
-to set up the config file.
+--partial-suffix
-Find the config file by running rclone config file, for example
+When --inplace is not used, it causes rclone to use the --partial-suffix
+as suffix for temporary files.
- $ rclone config file
- Configuration file is stored at:
- /home/user/.rclone.conf
+Suffix length limit is 16 characters.
-Now transfer it to the remote box (scp, cut paste, ftp, sftp, etc.) and
-place it in the correct place (use rclone config file on the remote box
-to find out where).
+The default is .partial.
-Configuring using SSH Tunnel
+--password-command SpaceSepList
-Linux and MacOS users can utilize SSH Tunnel to redirect the headless
-box port 53682 to local machine by using the following command:
+This flag supplies a program which should supply the config password
+when run. This is an alternative to rclone prompting for the password or
+setting the RCLONE_CONFIG_PASS variable.
- ssh -L localhost:53682:localhost:53682 username@remote_server
+The argument to this should be a command with a space separated list of
+arguments. If one of the arguments has a space in then enclose it in ",
+if you want a literal " in an argument then enclose the argument in "
+and double the ". See CSV encoding for more info.
-Then on the headless box run rclone config and answer Y to the
-Use web browser to automatically authenticate? question.
+Eg
- ...
- Remote config
- Use web browser to automatically authenticate rclone with remote?
- * Say Y if the machine running rclone has a web browser you can use
- * Say N if running rclone on a (remote) machine without web browser access
- If not sure try Y. If Y failed, try N.
- y) Yes (default)
- n) No
- y/n> y
+ --password-command "echo hello"
+ --password-command 'echo "hello with space"'
+ --password-command 'echo "hello with ""quotes"" and space"'
-Then copy and paste the auth url
-http://127.0.0.1:53682/auth?state=xxxxxxxxxxxx to the browser on your
-local machine, complete the auth and it is done.
+See the Configuration Encryption for more info.
-Filtering, includes and excludes
+See a Windows PowerShell example on the Wiki.
-Filter flags determine which files rclone sync, move, ls, lsl, md5sum,
-sha1sum, size, delete, check and similar commands apply to.
+-P, --progress
-They are specified in terms of path/file name patterns; path/file lists;
-file age and size, or presence of a file in a directory. Bucket based
-remotes without the concept of directory apply filters to object key,
-age and size in an analogous way.
+This flag makes rclone update the stats in a static block in the
+terminal providing a realtime overview of the transfer.
-Rclone purge does not obey filters.
+Any log messages will scroll above the static block. Log messages will
+push the static block down to the bottom of the terminal where it will
+stay.
-To test filters without risk of damage to data, apply them to rclone ls,
-or with the --dry-run and -vv flags.
+Normally this is updated every 500mS but this period can be overridden
+with the --stats flag.
-Rclone filter patterns can only be used in filter command line options,
-not in the specification of a remote.
+This can be used with the --stats-one-line flag for a simpler display.
-E.g. rclone copy "remote:dir*.jpg" /path/to/dir does not have a filter
-effect. rclone copy remote:dir /path/to/dir --include "*.jpg" does.
+Note: On Windows until this bug is fixed all non-ASCII characters will
+be replaced with . when --progress is in use.
-Important Avoid mixing any two of --include..., --exclude... or
---filter... flags in an rclone command. The results might not be what
-you expect. Instead use a --filter... flag.
+--progress-terminal-title
-Patterns for matching path/file names
+This flag, when used with -P/--progress, will print the string ETA: %s
+to the terminal title.
-Pattern syntax
+-q, --quiet
-Here is a formal definition of the pattern syntax, examples are below.
+This flag will limit rclone's output to error messages only.
-Rclone matching rules follow a glob style:
+--refresh-times
- * matches any sequence of non-separator (/) characters
- ** matches any sequence of characters including / separators
- ? matches any single non-separator (/) character
- [ [ ! ] { character-range } ]
- character class (must be non-empty)
- { pattern-list }
- pattern alternatives
- {{ regexp }}
- regular expression to match
- c matches character c (c != *, **, ?, \, [, {, })
- \c matches reserved character c (c = *, **, ?, \, [, {, }) or character class
+The --refresh-times flag can be used to update modification times of
+existing files when they are out of sync on backends which don't support
+hashes.
-character-range:
+This is useful if you uploaded files with the incorrect timestamps and
+you now wish to correct them.
- c matches character c (c != \, -, ])
- \c matches reserved character c (c = \, -, ])
- lo - hi matches character c for lo <= c <= hi
+This flag is only useful for destinations which don't support hashes
+(e.g. crypt).
-pattern-list:
+This can be used any of the sync commands sync, copy or move.
- pattern { , pattern }
- comma-separated (without spaces) patterns
+To use this flag you will need to be doing a modification time sync (so
+not using --size-only or --checksum). The flag will have no effect when
+using --size-only or --checksum.
-character classes (see Go regular expression reference) include:
+If this flag is used when rclone comes to upload a file it will check to
+see if there is an existing file on the destination. If this file
+matches the source with size (and checksum if available) but has a
+differing timestamp then instead of re-uploading it, rclone will update
+the timestamp on the destination file. If the checksum does not match
+rclone will upload the new file. If the checksum is absent (e.g. on a
+crypt backend) then rclone will update the timestamp.
- Named character classes (e.g. [\d], [^\d], [\D], [^\D])
- Perl character classes (e.g. \s, \S, \w, \W)
- ASCII character classes (e.g. [[:alnum:]], [[:alpha:]], [[:punct:]], [[:xdigit:]])
+Note that some remotes can't set the modification time without
+re-uploading the file so this flag is less useful on them.
-regexp for advanced users to insert a regular expression - see below for
-more info:
+Normally if you are doing a modification time sync rclone will update
+modification times without --refresh-times provided that the remote
+supports checksums and the checksums match on the file. However if the
+checksums are absent then rclone will upload the file rather than
+setting the timestamp as this is the safe behaviour.
- Any re2 regular expression not containing `}}`
+--retries int
-If the filter pattern starts with a / then it only matches at the top
-level of the directory tree, relative to the root of the remote (not
-necessarily the root of the drive). If it does not start with / then it
-is matched starting at the end of the path/file name but it only matches
-a complete path element - it must match from a / separator or the
-beginning of the path/file.
+Retry the entire sync if it fails this many times it fails (default 3).
- file.jpg - matches "file.jpg"
- - matches "directory/file.jpg"
- - doesn't match "afile.jpg"
- - doesn't match "directory/afile.jpg"
- /file.jpg - matches "file.jpg" in the root directory of the remote
- - doesn't match "afile.jpg"
- - doesn't match "directory/file.jpg"
+Some remotes can be unreliable and a few retries help pick up the files
+which didn't get transferred because of errors.
-The top level of the remote might not be the top level of the drive.
+Disable retries with --retries 1.
-E.g. for a Microsoft Windows local directory structure
+--retries-sleep=TIME
- F:
- ├── bkp
- ├── data
- │ ├── excl
- │ │ ├── 123.jpg
- │ │ └── 456.jpg
- │ ├── incl
- │ │ └── document.pdf
+This sets the interval between each retry specified by --retries
-To copy the contents of folder data into folder bkp excluding the
-contents of subfolder exclthe following command treats F:\data and
-F:\bkp as top level for filtering.
+The default is 0. Use 0 to disable.
-rclone copy F:\data\ F:\bkp\ --exclude=/excl/**
+--server-side-across-configs
-Important Use / in path/file name patterns and not \ even if running on
-Microsoft Windows.
+Allow server-side operations (e.g. copy or move) to work across
+different configurations.
-Simple patterns are case sensitive unless the --ignore-case flag is
-used.
+This can be useful if you wish to do a server-side copy or move between
+two remotes which use the same backend but are configured differently.
-Without --ignore-case (default)
+Note that this isn't enabled by default because it isn't easy for rclone
+to tell if it will work between any two configurations.
- potato - matches "potato"
- - doesn't match "POTATO"
+--size-only
-With --ignore-case
+Normally rclone will look at modification time and size of files to see
+if they are equal. If you set this flag then rclone will check only the
+size.
- potato - matches "potato"
- - matches "POTATO"
+This can be useful transferring files from Dropbox which have been
+modified by the desktop sync client which doesn't set checksums of
+modification times in the same way as rclone.
-Using regular expressions in filter patterns
+--stats=TIME
-The syntax of filter patterns is glob style matching (like bash uses) to
-make things easy for users. However this does not provide absolute
-control over the matching, so for advanced users rclone also provides a
-regular expression syntax.
+Commands which transfer data (sync, copy, copyto, move, moveto) will
+print data transfer stats at regular intervals to show their progress.
-The regular expressions used are as defined in the Go regular expression
-reference. Regular expressions should be enclosed in {{ }}. They will
-match only the last path segment if the glob doesn't start with / or the
-whole path name if it does. Note that rclone does not attempt to parse
-the supplied regular expression, meaning that using any regular
-expression filter will prevent rclone from using directory filter rules,
-as it will instead check every path against the supplied regular
-expression(s).
+This sets the interval.
-Here is how the {{regexp}} is transformed into an full regular
-expression to match the entire path:
+The default is 1m. Use 0 to disable.
- {{regexp}} becomes (^|/)(regexp)$
- /{{regexp}} becomes ^(regexp)$
+If you set the stats interval then all commands can show stats. This can
+be useful when running other commands, check or mount for example.
-Regexp syntax can be mixed with glob syntax, for example
+Stats are logged at INFO level by default which means they won't show at
+default log level NOTICE. Use --stats-log-level NOTICE or -v to make
+them show. See the Logging section for more info on log levels.
- *.{{jpe?g}} to match file.jpg, file.jpeg but not file.png
+Note that on macOS you can send a SIGINFO (which is normally ctrl-T in
+the terminal) to make the stats print immediately.
-You can also use regexp flags - to set case insensitive, for example
+--stats-file-name-length integer
- *.{{(?i)jpg}} to match file.jpg, file.JPG but not file.png
+By default, the --stats output will truncate file names and paths longer
+than 40 characters. This is equivalent to providing
+--stats-file-name-length 40. Use --stats-file-name-length 0 to disable
+any truncation of file names printed by stats.
-Be careful with wildcards in regular expressions - you don't want them
-to match path separators normally. To match any file name starting with
-start and ending with end write
+--stats-log-level string
- {{start[^/]*end\.jpg}}
+Log level to show --stats output at. This can be DEBUG, INFO, NOTICE, or
+ERROR. The default is INFO. This means at the default level of logging
+which is NOTICE the stats won't show - if you want them to then use
+--stats-log-level NOTICE. See the Logging section for more info on log
+levels.
-Not
+--stats-one-line
- {{start.*end\.jpg}}
+When this is specified, rclone condenses the stats into a single line
+showing the most important stats only.
-Which will match a directory called start with a file called end.jpg in
-it as the .* will match / characters.
+--stats-one-line-date
-Note that you can use -vv --dump filters to show the filter patterns in
-regexp format - rclone implements the glob patterns by transforming them
-into regular expressions.
+When this is specified, rclone enables the single-line stats and
+prepends the display with a date string. The default is
+2006/01/02 15:04:05 -
-Filter pattern examples
+--stats-one-line-date-format
- Description Pattern Matches Does not match
- --------------- ---------------- ------------------------------- ------------------
- Wildcard *.jpg /file.jpg /file.png
- /dir/file.jpg /dir/file.png
- Rooted /*.jpg /file.jpg /file.png
- /file2.jpg /dir/file.jpg
- Alternates *.{jpg,png} /file.jpg /file.gif
- /dir/file.png /dir/file.gif
- Path Wildcard dir/** /dir/anyfile file.png
- /subdir/dir/subsubdir/anyfile /subdir/file.png
- Any Char *.t?t /file.txt /file.qxt
- /dir/file.tzt /dir/file.png
- Range *.[a-z] /file.a /file.0
- /dir/file.b /dir/file.1
- Escape *.\?\?\? /file.??? /file.abc
- /dir/file.??? /dir/file.def
- Class *.\d\d\d /file.012 /file.abc
- /dir/file.345 /dir/file.def
- Regexp *.{{jpe?g}} /file.jpeg /file.png
- /dir/file.jpg /dir/file.jpeeg
- Rooted Regexp /{{.*\.jpe?g}} /file.jpeg /file.png
- /file.jpg /dir/file.jpg
+When this is specified, rclone enables the single-line stats and
+prepends the display with a user-supplied date string. The date string
+MUST be enclosed in quotes. Follow golang specs for date formatting
+syntax.
-How filter rules are applied to files
+--stats-unit=bits|bytes
-Rclone path/file name filters are made up of one or more of the
-following flags:
+By default, data transfer rates will be printed in bytes per second.
-- --include
-- --include-from
-- --exclude
-- --exclude-from
-- --filter
-- --filter-from
+This option allows the data rate to be printed in bits per second.
-There can be more than one instance of individual flags.
+Data transfer volume will still be reported in bytes.
-Rclone internally uses a combined list of all the include and exclude
-rules. The order in which rules are processed can influence the result
-of the filter.
+The rate is reported as a binary unit, not SI unit. So 1 Mbit/s equals
+1,048,576 bit/s and not 1,000,000 bit/s.
-All flags of the same type are processed together in the order above,
-regardless of what order the different types of flags are included on
-the command line.
+The default is bytes.
-Multiple instances of the same flag are processed from left to right
-according to their position in the command line.
+--suffix=SUFFIX
-To mix up the order of processing includes and excludes use --filter...
-flags.
+When using sync, copy or move any files which would have been
+overwritten or deleted will have the suffix added to them. If there is a
+file with the same path (after the suffix has been added), then it will
+be overwritten.
-Within --include-from, --exclude-from and --filter-from flags rules are
-processed from top to bottom of the referenced file.
+The remote in use must support server-side move or copy and you must use
+the same remote as the destination of the sync.
-If there is an --include or --include-from flag specified, rclone
-implies a - ** rule which it adds to the bottom of the internal rule
-list. Specifying a + rule with a --filter... flag does not imply that
-rule.
+This is for use with files to add the suffix in the current directory or
+with --backup-dir. See --backup-dir for more info.
-Each path/file name passed through rclone is matched against the
-combined filter list. At first match to a rule the path/file name is
-included or excluded and no further filter rules are processed for that
-path/file.
+For example
-If rclone does not find a match, after testing against all rules
-(including the implied rule if appropriate), the path/file name is
-included.
+ rclone copy --interactive /path/to/local/file remote:current --suffix .bak
-Any path/file included at that stage is processed by the rclone command.
+will copy /path/to/local to remote:current, but for any files which
+would have been updated or deleted have .bak added.
---files-from and --files-from-raw flags over-ride and cannot be combined
-with other filter options.
+If using rclone sync with --suffix and without --backup-dir then it is
+recommended to put a filter rule in excluding the suffix otherwise the
+sync will delete the backup files.
-To see the internal combined rule list, in regular expression form, for
-a command add the --dump filters flag. Running an rclone command with
---dump filters and -vv flags lists the internal filter elements and
-shows how they are applied to each source path/file. There is not
-currently a means provided to pass regular expression filter options
-into rclone directly though character class filter rules contain
-character classes. Go regular expression reference
+ rclone sync --interactive /path/to/local/file remote:current --suffix .bak --exclude "*.bak"
-How filter rules are applied to directories
+--suffix-keep-extension
-Rclone commands are applied to path/file names not directories. The
-entire contents of a directory can be matched to a filter by the pattern
-directory/* or recursively by directory/**.
+When using --suffix, setting this causes rclone put the SUFFIX before
+the extension of the files that it backs up rather than after.
-Directory filter rules are defined with a closing / separator.
+So let's say we had --suffix -2019-01-01, without the flag file.txt
+would be backed up to file.txt-2019-01-01 and with the flag it would be
+backed up to file-2019-01-01.txt. This can be helpful to make sure the
+suffixed files can still be opened.
-E.g. /directory/subdirectory/ is an rclone directory filter rule.
+If a file has two (or more) extensions and the second (or subsequent)
+extension is recognised as a valid mime type, then the suffix will go
+before that extension. So file.tar.gz would be backed up to
+file-2019-01-01.tar.gz whereas file.badextension.gz would be backed up
+to file.badextension-2019-01-01.gz.
-Rclone commands can use directory filter rules to determine whether they
-recurse into subdirectories. This potentially optimises access to a
-remote by avoiding listing unnecessary directories. Whether optimisation
-is desirable depends on the specific filter rules and source remote
-content.
+--syslog
-If any regular expression filters are in use, then no directory
-recursion optimisation is possible, as rclone must check every path
-against the supplied regular expression(s).
+On capable OSes (not Windows or Plan9) send all log output to syslog.
-Directory recursion optimisation occurs if either:
+This can be useful for running rclone in a script or rclone mount.
-- A source remote does not support the rclone ListR primitive. local,
- sftp, Microsoft OneDrive and WebDAV do not support ListR. Google
- Drive and most bucket type storage do. Full list
+--syslog-facility string
-- On other remotes (those that support ListR), if the rclone command
- is not naturally recursive, and provided it is not run with the
- --fast-list flag. ls, lsf -R and size are naturally recursive but
- sync, copy and move are not.
+If using --syslog this sets the syslog facility (e.g. KERN, USER). See
+man syslog for a list of possible facilities. The default facility is
+DAEMON.
-- Whenever the --disable ListR flag is applied to an rclone command.
+--temp-dir=DIR
-Rclone commands imply directory filter rules from path/file filter
-rules. To view the directory filter rules rclone has implied for a
-command specify the --dump filters flag.
+Specify the directory rclone will use for temporary files, to override
+the default. Make sure the directory exists and have accessible
+permissions.
-E.g. for an include rule
+By default the operating system's temp directory will be used: - On Unix
+systems, $TMPDIR if non-empty, else /tmp. - On Windows, the first
+non-empty value from %TMP%, %TEMP%, %USERPROFILE%, or the Windows
+directory.
- /a/*.jpg
+When overriding the default with this option, the specified path will be
+set as value of environment variable TMPDIR on Unix systems and TMP and
+TEMP on Windows.
-Rclone implies the directory include rule
+You can use the config paths command to see the current value.
- /a/
+--tpslimit float
-Directory filter rules specified in an rclone command can limit the
-scope of an rclone command but path/file filters still have to be
-specified.
+Limit transactions per second to this number. Default is 0 which is used
+to mean unlimited transactions per second.
-E.g. rclone ls remote: --include /directory/ will not match any files.
-Because it is an --include option the --exclude ** rule is implied, and
-the /directory/ pattern serves only to optimise access to the remote by
-ignoring everything outside of that directory.
+A transaction is roughly defined as an API call; its exact meaning will
+depend on the backend. For HTTP based backends it is an HTTP
+PUT/GET/POST/etc and its response. For FTP/SFTP it is a round trip
+transaction over TCP.
-E.g. rclone ls remote: --filter-from filter-list.txt with a file
-filter-list.txt:
+For example, to limit rclone to 10 transactions per second use
+--tpslimit 10, or to 1 transaction every 2 seconds use --tpslimit 0.5.
- - /dir1/
- - /dir2/
- + *.pdf
- - **
+Use this when the number of transactions per second from rclone is
+causing a problem with the cloud storage provider (e.g. getting you
+banned or rate limited).
-All files in directories dir1 or dir2 or their subdirectories are
-completely excluded from the listing. Only files of suffix pdf in the
-root of remote: or its subdirectories are listed. The - ** rule prevents
-listing of any path/files not previously matched by the rules above.
+This can be very useful for rclone mount to control the behaviour of
+applications using it.
-Option exclude-if-present creates a directory exclude rule based on the
-presence of a file in a directory and takes precedence over other rclone
-directory filter rules.
+This limit applies to all HTTP based backends and to the FTP and SFTP
+backends. It does not apply to the local backend or the Storj backend.
-When using pattern list syntax, if a pattern item contains either / or
-**, then rclone will not able to imply a directory filter rule from this
-pattern list.
+See also --tpslimit-burst.
-E.g. for an include rule
+--tpslimit-burst int
- {dir1/**,dir2/**}
+Max burst of transactions for --tpslimit (default 1).
-Rclone will match files below directories dir1 or dir2 only, but will
-not be able to use this filter to exclude a directory dir3 from being
-traversed.
+Normally --tpslimit will do exactly the number of transaction per second
+specified. However if you supply --tps-burst then rclone can save up
+some transactions from when it was idle giving a burst of up to the
+parameter supplied.
-Directory recursion optimisation may affect performance, but normally
-not the result. One exception to this is sync operations with option
---create-empty-src-dirs, where any traversed empty directories will be
-created. With the pattern list example {dir1/**,dir2/**} above, this
-would create an empty directory dir3 on destination (when it exists on
-source). Changing the filter to {dir1,dir2}/**, or splitting it into two
-include rules --include dir1/** --include dir2/**, will match the same
-files while also filtering directories, with the result that an empty
-directory dir3 will no longer be created.
+For example if you provide --tpslimit-burst 10 then if rclone has been
+idle for more than 10*--tpslimit then it can do 10 transactions very
+quickly before they are limited again.
---exclude - Exclude files matching pattern
+This may be used to increase performance of --tpslimit without changing
+the long term average number of transactions per second.
-Excludes path/file names from an rclone command based on a single
-exclude rule.
+--track-renames
-This flag can be repeated. See above for the order filter flags are
-processed in.
+By default, rclone doesn't keep track of renamed files, so if you rename
+a file locally then sync it to a remote, rclone will delete the old file
+on the remote and upload a new copy.
---exclude should not be used with --include, --include-from, --filter or
---filter-from flags.
+An rclone sync with --track-renames runs like a normal sync, but keeps
+track of objects which exist in the destination but not in the source
+(which would normally be deleted), and which objects exist in the source
+but not the destination (which would normally be transferred). These
+objects are then candidates for renaming.
---exclude has no effect when combined with --files-from or
---files-from-raw flags.
+After the sync, rclone matches up the source only and destination only
+objects using the --track-renames-strategy specified and either renames
+the destination object or transfers the source and deletes the
+destination object. --track-renames is stateless like all of rclone's
+syncs.
-E.g. rclone ls remote: --exclude *.bak excludes all .bak files from
-listing.
+To use this flag the destination must support server-side copy or
+server-side move, and to use a hash based --track-renames-strategy (the
+default) the source and the destination must have a compatible hash.
-E.g. rclone size remote: "--exclude /dir/**" returns the total size of
-all files on remote: excluding those in root directory dir and sub
-directories.
+If the destination does not support server-side copy or move, rclone
+will fall back to the default behaviour and log an error level message
+to the console.
-E.g. on Microsoft Windows rclone ls remote: --exclude "*\[{JP,KR,HK}\]*"
-lists the files in remote: without [JP] or [KR] or [HK] in their name.
-Quotes prevent the shell from interpreting the \ characters.\ characters
-escape the [ and ] so an rclone filter treats them literally rather than
-as a character-range. The { and } define an rclone pattern list. For
-other operating systems single quotes are required ie
-rclone ls remote: --exclude '*\[{JP,KR,HK}\]*'
+Encrypted destinations are not currently supported by --track-renames if
+--track-renames-strategy includes hash.
---exclude-from - Read exclude patterns from file
+Note that --track-renames is incompatible with --no-traverse and that it
+uses extra memory to keep track of all the rename candidates.
-Excludes path/file names from an rclone command based on rules in a
-named file. The file contains a list of remarks and pattern rules.
+Note also that --track-renames is incompatible with --delete-before and
+will select --delete-after instead of --delete-during.
-For an example exclude-file.txt:
+--track-renames-strategy (hash,modtime,leaf,size)
- # a sample exclude rule file
- *.bak
- file2.jpg
+This option changes the file matching criteria for --track-renames.
-rclone ls remote: --exclude-from exclude-file.txt lists the files on
-remote: except those named file2.jpg or with a suffix .bak. That is
-equivalent to rclone ls remote: --exclude file2.jpg --exclude "*.bak".
+The matching is controlled by a comma separated selection of these
+tokens:
-This flag can be repeated. See above for the order filter flags are
-processed in.
+- modtime - the modification time of the file - not supported on all
+ backends
+- hash - the hash of the file contents - not supported on all backends
+- leaf - the name of the file not including its directory name
+- size - the size of the file (this is always enabled)
-The --exclude-from flag is useful where multiple exclude filter rules
-are applied to an rclone command.
+The default option is hash.
---exclude-from should not be used with --include, --include-from,
---filter or --filter-from flags.
+Using --track-renames-strategy modtime,leaf would match files based on
+modification time, the leaf of the file name and the size only.
---exclude-from has no effect when combined with --files-from or
---files-from-raw flags.
+Using --track-renames-strategy modtime or leaf can enable
+--track-renames support for encrypted destinations.
---exclude-from followed by - reads filter rules from standard input.
+Note that the hash strategy is not supported with encrypted
+destinations.
---include - Include files matching pattern
+--delete-(before,during,after)
-Adds a single include rule based on path/file names to an rclone
-command.
+This option allows you to specify when files on your destination are
+deleted when you sync folders.
-This flag can be repeated. See above for the order filter flags are
-processed in.
+Specifying the value --delete-before will delete all files present on
+the destination, but not on the source before starting the transfer of
+any new or updated files. This uses two passes through the file systems,
+one for the deletions and one for the copies.
---include has no effect when combined with --files-from or
---files-from-raw flags.
+Specifying --delete-during will delete files while checking and
+uploading files. This is the fastest option and uses the least memory.
---include implies --exclude ** at the end of an rclone internal filter
-list. Therefore if you mix --include and --include-from flags with
---exclude, --exclude-from, --filter or --filter-from, you must use
-include rules for all the files you want in the include statement. For
-more flexibility use the --filter-from flag.
+Specifying --delete-after (the default value) will delay deletion of
+files until all new/updated files have been successfully transferred.
+The files to be deleted are collected in the copy pass then deleted
+after the copy pass has completed successfully. The files to be deleted
+are held in memory so this mode may use more memory. This is the safest
+mode as it will only delete files if there have been no errors
+subsequent to that. If there have been errors before the deletions start
+then you will get the message
+not deleting files as there were IO errors.
-E.g. rclone ls remote: --include "*.{png,jpg}" lists the files on
-remote: with suffix .png and .jpg. All other files are excluded.
+--fast-list
-E.g. multiple rclone copy commands can be combined with --include and a
-pattern-list.
+When doing anything which involves a directory listing (e.g. sync, copy,
+ls - in fact nearly every command), rclone has different strategies to
+choose from.
+
+The basic strategy is to list one directory and processes it before
+using more directory lists to process any subdirectories. This is a
+mandatory backend feature, called List, which means it is supported by
+all backends. This strategy uses small amount of memory, and because it
+can be parallelised it is fast for operations involving processing of
+the list results.
+
+Some backends provide the support for an alternative strategy, where all
+files beneath a directory can be listed in one (or a small number) of
+transactions. Rclone supports this alternative strategy through an
+optional backend feature called ListR. You can see in the storage system
+overview documentation's optional features section which backends it is
+enabled for (these tend to be the bucket-based ones, e.g. S3, B2, GCS,
+Swift). This strategy requires fewer transactions for highly recursive
+operations, which is important on backends where this is charged or
+heavily rate limited. It may be faster (due to fewer transactions) or
+slower (because it can't be parallelized) depending on different
+parameters, and may require more memory if rclone has to keep the whole
+listing in memory.
+
+Which listing strategy rclone picks for a given operation is
+complicated, but in general it tries to choose the best possible. It
+will prefer ListR in situations where it doesn't need to store the
+listed files in memory, e.g. for unlimited recursive ls command
+variants. In other situations it will prefer List, e.g. for sync and
+copy, where it needs to keep the listed files in memory, and is
+performing operations on them where parallelization may be a huge
+advantage.
+
+Rclone is not able to take all relevant parameters into account for
+deciding the best strategy, and therefore allows you to influence the
+choice in two ways: You can stop rclone from using ListR by disabling
+the feature, using the --disable option (--disable ListR), or you can
+allow rclone to use ListR where it would normally choose not to do so
+due to higher memory usage, using the --fast-list option. Rclone should
+always produce identical results either way. Using --disable ListR or
+--fast-list on a remote which doesn't support ListR does nothing, rclone
+will just ignore it.
- rclone copy /vol1/A remote:A
- rclone copy /vol1/B remote:B
+A rule of thumb is that if you pay for transactions and can fit your
+entire sync listing into memory, then --fast-list is recommended. If you
+have a very big sync to do, then don't use --fast-list, otherwise you
+will run out of memory. Run some tests and compare before you decide,
+and if in doubt then just leave the default, let rclone decide, i.e. not
+use --fast-list.
-is equivalent to:
+--timeout=TIME
- rclone copy /vol1 remote: --include "{A,B}/**"
+This sets the IO idle timeout. If a transfer has started but then
+becomes idle for this long it is considered broken and disconnected.
-E.g. rclone ls remote:/wheat --include "??[^[:punct:]]*" lists the files
-remote: directory wheat (and subdirectories) whose third character is
-not punctuation. This example uses an ASCII character class.
+The default is 5m. Set to 0 to disable.
---include-from - Read include patterns from file
+--transfers=N
-Adds path/file names to an rclone command based on rules in a named
-file. The file contains a list of remarks and pattern rules.
+The number of file transfers to run in parallel. It can sometimes be
+useful to set this to a smaller number if the remote is giving a lot of
+timeouts or bigger if you have lots of bandwidth and a fast remote.
-For an example include-file.txt:
+The default is to run 4 file transfers in parallel.
- # a sample include rule file
- *.jpg
- file2.avi
+Look at --multi-thread-streams if you would like to control single file
+transfers.
-rclone ls remote: --include-from include-file.txt lists the files on
-remote: with name file2.avi or suffix .jpg. That is equivalent to
-rclone ls remote: --include file2.avi --include "*.jpg".
+-u, --update
-This flag can be repeated. See above for the order filter flags are
-processed in.
+This forces rclone to skip any files which exist on the destination and
+have a modified time that is newer than the source file.
-The --include-from flag is useful where multiple include filter rules
-are applied to an rclone command.
+This can be useful in avoiding needless transfers when transferring to a
+remote which doesn't support modification times directly (or when using
+--use-server-modtime to avoid extra API calls) as it is more accurate
+than a --size-only check and faster than using --checksum. On such
+remotes (or when using --use-server-modtime) the time checked will be
+the uploaded time.
---include-from implies --exclude ** at the end of an rclone internal
-filter list. Therefore if you mix --include and --include-from flags
-with --exclude, --exclude-from, --filter or --filter-from, you must use
-include rules for all the files you want in the include statement. For
-more flexibility use the --filter-from flag.
+If an existing destination file has a modification time older than the
+source file's, it will be updated if the sizes are different. If the
+sizes are the same, it will be updated if the checksum is different or
+not available.
---exclude-from has no effect when combined with --files-from or
---files-from-raw flags.
+If an existing destination file has a modification time equal (within
+the computed modify window) to the source file's, it will be updated if
+the sizes are different. The checksum will not be checked in this case
+unless the --checksum flag is provided.
---exclude-from followed by - reads filter rules from standard input.
+In all other cases the file will not be updated.
---filter - Add a file-filtering rule
+Consider using the --modify-window flag to compensate for time skews
+between the source and the backend, for backends that do not support mod
+times, and instead use uploaded times. However, if the backend does not
+support checksums, note that syncing or copying within the time skew
+window may still result in additional transfers for safety.
-Specifies path/file names to an rclone command, based on a single
-include or exclude rule, in + or - format.
+--use-mmap
-This flag can be repeated. See above for the order filter flags are
-processed in.
+If this flag is set then rclone will use anonymous memory allocated by
+mmap on Unix based platforms and VirtualAlloc on Windows for its
+transfer buffers (size controlled by --buffer-size). Memory allocated
+like this does not go on the Go heap and can be returned to the OS
+immediately when it is finished with.
---filter + differs from --include. In the case of --include rclone
-implies an --exclude * rule which it adds to the bottom of the internal
-rule list. --filter...+ does not imply that rule.
+If this flag is not set then rclone will allocate and free the buffers
+using the Go memory allocator which may use more memory as memory pages
+are returned less aggressively to the OS.
---filter has no effect when combined with --files-from or
---files-from-raw flags.
+It is possible this does not work well on all platforms so it is
+disabled by default; in the future it may be enabled by default.
---filter should not be used with --include, --include-from, --exclude or
---exclude-from flags.
+--use-server-modtime
-E.g. rclone ls remote: --filter "- *.bak" excludes all .bak files from a
-list of remote:.
+Some object-store backends (e.g, Swift, S3) do not preserve file
+modification times (modtime). On these backends, rclone stores the
+original modtime as additional metadata on the object. By default it
+will make an API call to retrieve the metadata when the modtime is
+needed by an operation.
---filter-from - Read filtering patterns from a file
+Use this flag to disable the extra API call and rely instead on the
+server's modified time. In cases such as a local to remote sync using
+--update, knowing the local file is newer than the time it was last
+uploaded to the remote is sufficient. In those cases, this flag can
+speed up the process and reduce the number of API calls necessary.
-Adds path/file names to an rclone command based on rules in a named
-file. The file contains a list of remarks and pattern rules. Include
-rules start with + and exclude rules with -. ! clears existing rules.
-Rules are processed in the order they are defined.
+Using this flag on a sync operation without also using --update would
+cause all files modified at any time other than the last upload time to
+be uploaded again, which is probably not what you want.
-This flag can be repeated. See above for the order filter flags are
-processed in.
+-v, -vv, --verbose
-Arrange the order of filter rules with the most restrictive first and
-work down.
+With -v rclone will tell you about each file that is transferred and a
+small number of significant events.
-E.g. for filter-file.txt:
+With -vv rclone will become very verbose telling you about every file it
+considers and transfers. Please send bug reports with a log with this
+setting.
- # a sample filter rule file
- - secret*.jpg
- + *.jpg
- + *.png
- + file2.avi
- - /dir/Trash/**
- + /dir/**
- # exclude everything else
- - *
+When setting verbosity as an environment variable, use RCLONE_VERBOSE=1
+or RCLONE_VERBOSE=2 for -v and -vv respectively.
-rclone ls remote: --filter-from filter-file.txt lists the path/files on
-remote: including all jpg and png files, excluding any matching
-secret*.jpg and including file2.avi. It also includes everything in the
-directory dir at the root of remote, except remote:dir/Trash which it
-excludes. Everything else is excluded.
+-V, --version
-E.g. for an alternative filter-file.txt:
+Prints the version number
- - secret*.jpg
- + *.jpg
- + *.png
- + file2.avi
- - *
+SSL/TLS options
-Files file1.jpg, file3.png and file2.avi are listed whilst secret17.jpg
-and files without the suffix .jpgor.png` are excluded.
+The outgoing SSL/TLS connections rclone makes can be controlled with
+these options. For example this can be very useful with the HTTP or
+WebDAV backends. Rclone HTTP servers have their own set of configuration
+for SSL/TLS which you can find in their documentation.
-E.g. for an alternative filter-file.txt:
+--ca-cert stringArray
- + *.jpg
- + *.gif
- !
- + 42.doc
- - *
+This loads the PEM encoded certificate authority certificates and uses
+it to verify the certificates of the servers rclone connects to.
-Only file 42.doc is listed. Prior rules are cleared by the !.
+If you have generated certificates signed with a local CA then you will
+need this flag to connect to servers using those certificates.
---files-from - Read list of source-file names
+--client-cert string
-Adds path/files to an rclone command from a list in a named file. Rclone
-processes the path/file names in the order of the list, and no others.
+This loads the PEM encoded client side certificate.
-Other filter flags (--include, --include-from, --exclude,
---exclude-from, --filter and --filter-from) are ignored when
---files-from is used.
+This is used for mutual TLS authentication.
---files-from expects a list of files as its input. Leading or trailing
-whitespace is stripped from the input lines. Lines starting with # or ;
-are ignored.
+The --client-key flag is required too when using this.
-Rclone commands with a --files-from flag traverse the remote, treating
-the names in --files-from as a set of filters.
+--client-key string
-If the --no-traverse and --files-from flags are used together an rclone
-command does not traverse the remote. Instead it addresses each
-path/file named in the file individually. For each path/file name, that
-requires typically 1 API call. This can be efficient for a short
---files-from list and a remote containing many files.
+This loads the PEM encoded client side private key used for mutual TLS
+authentication. Used in conjunction with --client-cert.
-Rclone commands do not error if any names in the --files-from file are
-missing from the source remote.
+--no-check-certificate=true/false
-The --files-from flag can be repeated in a single rclone command to read
-path/file names from more than one file. The files are read from left to
-right along the command line.
+--no-check-certificate controls whether a client verifies the server's
+certificate chain and host name. If --no-check-certificate is true, TLS
+accepts any certificate presented by the server and any host name in
+that certificate. In this mode, TLS is susceptible to man-in-the-middle
+attacks.
-Paths within the --files-from file are interpreted as starting with the
-root specified in the rclone command. Leading / separators are ignored.
-See --files-from-raw if you need the input to be processed in a raw
-manner.
+This option defaults to false.
-E.g. for a file files-from.txt:
+This should be used only for testing.
- # comment
- file1.jpg
- subdir/file2.jpg
+Configuration Encryption
-rclone copy --files-from files-from.txt /home/me/pics remote:pics copies
-the following, if they exist, and only those files.
+Your configuration file contains information for logging in to your
+cloud services. This means that you should keep your rclone.conf file in
+a secure location.
- /home/me/pics/file1.jpg → remote:pics/file1.jpg
- /home/me/pics/subdir/file2.jpg → remote:pics/subdir/file2.jpg
+If you are in an environment where that isn't possible, you can add a
+password to your configuration. This means that you will have to supply
+the password every time you start rclone.
-E.g. to copy the following files referenced by their absolute paths:
+To add a password to your rclone configuration, execute rclone config.
- /home/user1/42
- /home/user1/dir/ford
- /home/user2/prefect
+ >rclone config
+ Current remotes:
-First find a common subdirectory - in this case /home and put the
-remaining files in files-from.txt with or without leading /, e.g.
+ e) Edit existing remote
+ n) New remote
+ d) Delete remote
+ s) Set configuration password
+ q) Quit config
+ e/n/d/s/q>
- user1/42
- user1/dir/ford
- user2/prefect
+Go into s, Set configuration password:
-Then copy these to a remote:
+ e/n/d/s/q> s
+ Your configuration is not encrypted.
+ If you add a password, you will protect your login information to cloud services.
+ a) Add Password
+ q) Quit to main menu
+ a/q> a
+ Enter NEW configuration password:
+ password:
+ Confirm NEW password:
+ password:
+ Password set
+ Your configuration is encrypted.
+ c) Change Password
+ u) Unencrypt configuration
+ q) Quit to main menu
+ c/u/q>
- rclone copy --files-from files-from.txt /home remote:backup
+Your configuration is now encrypted, and every time you start rclone you
+will have to supply the password. See below for details. In the same
+menu, you can change the password or completely remove encryption from
+your configuration.
-The three files are transferred as follows:
+There is no way to recover the configuration if you lose your password.
- /home/user1/42 → remote:backup/user1/important
- /home/user1/dir/ford → remote:backup/user1/dir/file
- /home/user2/prefect → remote:backup/user2/stuff
+rclone uses nacl secretbox which in turn uses XSalsa20 and Poly1305 to
+encrypt and authenticate your configuration with secret-key
+cryptography. The password is SHA-256 hashed, which produces the key for
+secretbox. The hashed password is not stored.
-Alternatively if / is chosen as root files-from.txt will be:
+While this provides very good security, we do not recommend storing your
+encrypted rclone configuration in public if it contains sensitive
+information, maybe except if you use a very strong password.
- /home/user1/42
- /home/user1/dir/ford
- /home/user2/prefect
+If it is safe in your environment, you can set the RCLONE_CONFIG_PASS
+environment variable to contain your password, in which case it will be
+used for decrypting the configuration.
-The copy command will be:
+You can set this for a session from a script. For unix like systems save
+this to a file called set-rclone-password:
- rclone copy --files-from files-from.txt / remote:backup
+ #!/bin/echo Source this file don't run it
-Then there will be an extra home directory on the remote:
+ read -s RCLONE_CONFIG_PASS
+ export RCLONE_CONFIG_PASS
- /home/user1/42 → remote:backup/home/user1/42
- /home/user1/dir/ford → remote:backup/home/user1/dir/ford
- /home/user2/prefect → remote:backup/home/user2/prefect
+Then source the file when you want to use it. From the shell you would
+do source set-rclone-password. It will then ask you for the password and
+set it in the environment variable.
---files-from-raw - Read list of source-file names without any processing
+An alternate means of supplying the password is to provide a script
+which will retrieve the password and print on standard output. This
+script should have a fully specified path name and not rely on any
+environment variables. The script is supplied either via
+--password-command="..." command line argument or via the
+RCLONE_PASSWORD_COMMAND environment variable.
-This flag is the same as --files-from except that input is read in a raw
-manner. Lines with leading / trailing whitespace, and lines starting
-with ; or # are read without any processing. rclone lsf has a compatible
-format that can be used to export file lists from remotes for input to
---files-from-raw.
+One useful example of this is using the passwordstore application to
+retrieve the password:
---ignore-case - make searches case insensitive
+ export RCLONE_PASSWORD_COMMAND="pass rclone/config"
-By default, rclone filter patterns are case sensitive. The --ignore-case
-flag makes all of the filters patterns on the command line case
-insensitive.
+If the passwordstore password manager holds the password for the rclone
+configuration, using the script method means the password is primarily
+protected by the passwordstore system, and is never embedded in the
+clear in scripts, nor available for examination using the standard
+commands available. It is quite possible with long running rclone
+sessions for copies of passwords to be innocently captured in log files
+or terminal scroll buffers, etc. Using the script method of supplying
+the password enhances the security of the config password considerably.
-E.g. --include "zaphod.txt" does not match a file Zaphod.txt. With
---ignore-case a match is made.
+If you are running rclone inside a script, unless you are using the
+--password-command method, you might want to disable password prompts.
+To do that, pass the parameter --ask-password=false to rclone. This will
+make rclone fail instead of asking for a password if RCLONE_CONFIG_PASS
+doesn't contain a valid password, and --password-command has not been
+supplied.
-Quoting shell metacharacters
+Whenever running commands that may be affected by options in a
+configuration file, rclone will look for an existing file according to
+the rules described above, and load any it finds. If an encrypted file
+is found, this includes decrypting it, with the possible consequence of
+a password prompt. When executing a command line that you know are not
+actually using anything from such a configuration file, you can avoid it
+being loaded by overriding the location, e.g. with one of the documented
+special values for memory-only configuration. Since only backend options
+can be stored in configuration files, this is normally unnecessary for
+commands that do not operate on backends, e.g. genautocomplete. However,
+it will be relevant for commands that do operate on backends in general,
+but are used without referencing a stored remote, e.g. listing local
+filesystem paths, or connection strings: rclone --config="" ls .
-Rclone commands with filter patterns containing shell metacharacters may
-not as work as expected in your shell and may require quoting.
+Developer options
-E.g. linux, OSX (* metacharacter)
+These options are useful when developing or debugging rclone. There are
+also some more remote specific options which aren't documented here
+which are used for testing. These start with remote name e.g.
+--drive-test-option - see the docs for the remote in question.
-- --include \*.jpg
-- --include '*.jpg'
-- --include='*.jpg'
+--cpuprofile=FILE
-Microsoft Windows expansion is done by the command, not shell, so
---include *.jpg does not require quoting.
+Write CPU profile to file. This can be analysed with go tool pprof.
-If the rclone error
-Command .... needs .... arguments maximum: you provided .... non flag arguments:
-is encountered, the cause is commonly spaces within the name of a remote
-or flag value. The fix then is to quote values containing spaces.
+--dump flag,flag,flag
-Other filters
+The --dump flag takes a comma separated list of flags to dump info
+about.
---min-size - Don't transfer any file smaller than this
+Note that some headers including Accept-Encoding as shown may not be
+correct in the request and the response may not show Content-Encoding if
+the go standard libraries auto gzip encoding was in effect. In this case
+the body of the request will be gunzipped before showing it.
-Controls the minimum size file within the scope of an rclone command.
-Default units are KiB but abbreviations K, M, G, T or P are valid.
+The available flags are:
-E.g. rclone ls remote: --min-size 50k lists files on remote: of 50 KiB
-size or larger.
+--dump headers
-See the size option docs for more info.
+Dump HTTP headers with Authorization: lines removed. May still contain
+sensitive info. Can be very verbose. Useful for debugging only.
---max-size - Don't transfer any file larger than this
+Use --dump auth if you do want the Authorization: headers.
-Controls the maximum size file within the scope of an rclone command.
-Default units are KiB but abbreviations K, M, G, T or P are valid.
+--dump bodies
-E.g. rclone ls remote: --max-size 1G lists files on remote: of 1 GiB
-size or smaller.
+Dump HTTP headers and bodies - may contain sensitive info. Can be very
+verbose. Useful for debugging only.
-See the size option docs for more info.
+Note that the bodies are buffered in memory so don't use this for
+enormous files.
---max-age - Don't transfer any file older than this
+--dump requests
-Controls the maximum age of files within the scope of an rclone command.
+Like --dump bodies but dumps the request bodies and the response
+headers. Useful for debugging download problems.
---max-age applies only to files and not to directories.
+--dump responses
-E.g. rclone ls remote: --max-age 2d lists files on remote: of 2 days old
-or less.
+Like --dump bodies but dumps the response bodies and the request
+headers. Useful for debugging upload problems.
-See the time option docs for valid formats.
+--dump auth
---min-age - Don't transfer any file younger than this
+Dump HTTP headers - will contain sensitive info such as Authorization:
+headers - use --dump headers to dump without Authorization: headers. Can
+be very verbose. Useful for debugging only.
-Controls the minimum age of files within the scope of an rclone command.
-(see --max-age for valid formats)
+--dump filters
---min-age applies only to files and not to directories.
+Dump the filters to the output. Useful to see exactly what include and
+exclude options are filtering on.
-E.g. rclone ls remote: --min-age 2d lists files on remote: of 2 days old
-or more.
+--dump goroutines
-See the time option docs for valid formats.
+This dumps a list of the running go-routines at the end of the command
+to standard output.
-Other flags
+--dump openfiles
---delete-excluded - Delete files on dest excluded from sync
+This dumps a list of the open files at the end of the command. It uses
+the lsof command to do that so you'll need that installed to use it.
-Important this flag is dangerous to your data - use with --dry-run and
--v first.
+--dump mapper
-In conjunction with rclone sync, --delete-excluded deletes any files on
-the destination which are excluded from the command.
+This shows the JSON blobs being sent to the program supplied with
+--metadata-mapper and received from it. It can be useful for debugging
+the metadata mapper interface.
-E.g. the scope of rclone sync --interactive A: B: can be restricted:
+--memprofile=FILE
- rclone --min-size 50k --delete-excluded sync A: B:
+Write memory profile to file. This can be analysed with go tool pprof.
-All files on B: which are less than 50 KiB are deleted because they are
-excluded from the rclone sync command.
+Filtering
---dump filters - dump the filters to the output
+For the filtering options
-Dumps the defined filters to standard output in regular expression
-format.
+- --delete-excluded
+- --filter
+- --filter-from
+- --exclude
+- --exclude-from
+- --exclude-if-present
+- --include
+- --include-from
+- --files-from
+- --files-from-raw
+- --min-size
+- --max-size
+- --min-age
+- --max-age
+- --dump filters
+- --metadata-include
+- --metadata-include-from
+- --metadata-exclude
+- --metadata-exclude-from
+- --metadata-filter
+- --metadata-filter-from
-Useful for debugging.
+See the filtering section.
-Exclude directory based on a file
+Remote control
-The --exclude-if-present flag controls whether a directory is within the
-scope of an rclone command based on the presence of a named file within
-it. The flag can be repeated to check for multiple file names, presence
-of any of them will exclude the directory.
+For the remote control options and for instructions on how to remote
+control rclone
-This flag has a priority over other filter flags.
+- --rc
+- and anything starting with --rc-
-E.g. for the following directory structure:
+See the remote control section.
- dir1/file1
- dir1/dir2/file2
- dir1/dir2/dir3/file3
- dir1/dir2/dir3/.ignore
+Logging
-The command rclone ls --exclude-if-present .ignore dir1 does not list
-dir3, file3 or .ignore.
+rclone has 4 levels of logging, ERROR, NOTICE, INFO and DEBUG.
-Metadata filters
+By default, rclone logs to standard error. This means you can redirect
+standard error and still see the normal output of rclone commands (e.g.
+rclone ls).
-The metadata filters work in a very similar way to the normal file name
-filters, except they match metadata on the object.
+By default, rclone will produce Error and Notice level messages.
-The metadata should be specified as key=value patterns. This may be
-wildcarded using the normal filter patterns or regular expressions.
+If you use the -q flag, rclone will only produce Error messages.
-For example if you wished to list only local files with a mode of 100664
-you could do that with:
+If you use the -v flag, rclone will produce Error, Notice and Info
+messages.
- rclone lsf -M --files-only --metadata-include "mode=100664" .
+If you use the -vv flag, rclone will produce Error, Notice, Info and
+Debug messages.
-Or if you wished to show files with an atime, mtime or btime at a given
-date:
+You can also control the log levels with the --log-level flag.
- rclone lsf -M --files-only --metadata-include "[abm]time=2022-12-16*" .
+If you use the --log-file=FILE option, rclone will redirect Error, Info
+and Debug messages along with standard error to FILE.
-Like file filtering, metadata filtering only applies to files not to
-directories.
+If you use the --syslog flag then rclone will log to syslog and the
+--syslog-facility control which facility it uses.
-The filters can be applied using these flags.
+Rclone prefixes all log messages with their level in capitals, e.g. INFO
+which makes it easy to grep the log file for different kinds of
+information.
-- --metadata-include - Include metadatas matching pattern
-- --metadata-include-from - Read metadata include patterns from file
- (use - to read from stdin)
-- --metadata-exclude - Exclude metadatas matching pattern
-- --metadata-exclude-from - Read metadata exclude patterns from file
- (use - to read from stdin)
-- --metadata-filter - Add a metadata filtering rule
-- --metadata-filter-from - Read metadata filtering patterns from a
- file (use - to read from stdin)
+Exit Code
-Each flag can be repeated. See the section on how filter rules are
-applied for more details - these flags work in an identical way to the
-file name filtering flags, but instead of file name patterns have
-metadata patterns.
+If any errors occur during the command execution, rclone will exit with
+a non-zero exit code. This allows scripts to detect when rclone
+operations have failed.
-Common pitfalls
+During the startup phase, rclone will exit immediately if an error is
+detected in the configuration. There will always be a log message
+immediately before exiting.
-The most frequent filter support issues on the rclone forum are:
+When rclone is running it will accumulate errors as it goes along, and
+only exit with a non-zero exit code if (after retries) there were still
+failed transfers. For every error counted there will be a high priority
+log message (visible with -q) showing the message and which file caused
+the problem. A high priority message is also shown when starting a retry
+so the user can see that any previous error messages may not be valid
+after the retry. If rclone has done a retry it will log a high priority
+message if the retry was successful.
-- Not using paths relative to the root of the remote
-- Not using / to match from the root of a remote
-- Not using ** to match the contents of a directory
+List of exit codes
-GUI (Experimental)
+- 0 - success
+- 1 - Syntax or usage error
+- 2 - Error not otherwise categorised
+- 3 - Directory not found
+- 4 - File not found
+- 5 - Temporary error (one that more retries might fix) (Retry errors)
+- 6 - Less serious errors (like 461 errors from dropbox) (NoRetry
+ errors)
+- 7 - Fatal error (one that more retries won't fix, like account
+ suspended) (Fatal errors)
+- 8 - Transfer exceeded - limit set by --max-transfer reached
+- 9 - Operation successful, but no files transferred
+- 10 - Duration exceeded - limit set by --max-duration reached
-Rclone can serve a web based GUI (graphical user interface). This is
-somewhat experimental at the moment so things may be subject to change.
+Environment Variables
-Run this command in a terminal and rclone will download and then display
-the GUI in a web browser.
+Rclone can be configured entirely using environment variables. These can
+be used to set defaults for options or config file entries.
- rclone rcd --rc-web-gui
+Options
-This will produce logs like this and rclone needs to continue to run to
-serve the GUI:
+Every option in rclone can have its default set by environment variable.
- 2019/08/25 11:40:14 NOTICE: A new release for gui is present at https://github.com/rclone/rclone-webui-react/releases/download/v0.0.6/currentbuild.zip
- 2019/08/25 11:40:14 NOTICE: Downloading webgui binary. Please wait. [Size: 3813937, Path : /home/USER/.cache/rclone/webgui/v0.0.6.zip]
- 2019/08/25 11:40:16 NOTICE: Unzipping
- 2019/08/25 11:40:16 NOTICE: Serving remote control on http://127.0.0.1:5572/
+To find the name of the environment variable, first, take the long
+option name, strip the leading --, change - to _, make upper case and
+prepend RCLONE_.
-This assumes you are running rclone locally on your machine. It is
-possible to separate the rclone and the GUI - see below for details.
+For example, to always set --stats 5s, set the environment variable
+RCLONE_STATS=5s. If you set stats on the command line this will override
+the environment variable setting.
-If you wish to check for updates then you can add --rc-web-gui-update to
-the command line.
+Or to always use the trash in drive --drive-use-trash, set
+RCLONE_DRIVE_USE_TRASH=true.
-If you find your GUI broken, you may force it to update by add
---rc-web-gui-force-update.
+Verbosity is slightly different, the environment variable equivalent of
+--verbose or -v is RCLONE_VERBOSE=1, or for -vv, RCLONE_VERBOSE=2.
-By default, rclone will open your browser. Add
---rc-web-gui-no-open-browser to disable this feature.
+The same parser is used for the options and the environment variables so
+they take exactly the same form.
-Using the GUI
+The options set by environment variables can be seen with the -vv flag,
+e.g. rclone version -vv.
-Once the GUI opens, you will be looking at the dashboard which has an
-overall overview.
+Config file
-On the left hand side you will see a series of view buttons you can
-click on:
+You can set defaults for values in the config file on an individual
+remote basis. The names of the config items are documented in the page
+for each backend.
-- Dashboard - main overview
-- Configs - examine and create new configurations
-- Explorer - view, download and upload files to the cloud storage
- systems
-- Backend - view or alter the backend config
-- Log out
+To find the name of the environment variable, you need to set, take
+RCLONE_CONFIG_ + name of remote + _ + name of config file option and
+make it all uppercase. Note one implication here is the remote's name
+must be convertible into a valid environment variable name, so it can
+only contain letters, digits, or the _ (underscore) character.
-(More docs and walkthrough video to come!)
+For example, to configure an S3 remote named mys3: without a config file
+(using unix ways of setting environment variables):
-How it works
+ $ export RCLONE_CONFIG_MYS3_TYPE=s3
+ $ export RCLONE_CONFIG_MYS3_ACCESS_KEY_ID=XXX
+ $ export RCLONE_CONFIG_MYS3_SECRET_ACCESS_KEY=XXX
+ $ rclone lsd mys3:
+ -1 2016-09-21 12:54:21 -1 my-bucket
+ $ rclone listremotes | grep mys3
+ mys3:
-When you run the rclone rcd --rc-web-gui this is what happens
+Note that if you want to create a remote using environment variables you
+must create the ..._TYPE variable as above.
-- Rclone starts but only runs the remote control API ("rc").
-- The API is bound to localhost with an auto-generated username and
- password.
-- If the API bundle is missing then rclone will download it.
-- rclone will start serving the files from the API bundle over the
- same port as the API
-- rclone will open the browser with a login_token so it can log
- straight in.
+Note that the name of a remote created using environment variable is
+case insensitive, in contrast to regular remotes stored in config file
+as documented above. You must write the name in uppercase in the
+environment variable, but as seen from example above it will be listed
+and can be accessed in lowercase, while you can also refer to the same
+remote in uppercase:
-Advanced use
+ $ rclone lsd mys3:
+ -1 2016-09-21 12:54:21 -1 my-bucket
+ $ rclone lsd MYS3:
+ -1 2016-09-21 12:54:21 -1 my-bucket
-The rclone rcd may use any of the flags documented on the rc page.
+Note that you can only set the options of the immediate backend, so
+RCLONE_CONFIG_MYS3CRYPT_ACCESS_KEY_ID has no effect, if myS3Crypt is a
+crypt remote based on an S3 remote. However RCLONE_S3_ACCESS_KEY_ID will
+set the access key of all remotes using S3, including myS3Crypt.
-The flag --rc-web-gui is shorthand for
+Note also that now rclone has connection strings, it is probably easier
+to use those instead which makes the above example
-- Download the web GUI if necessary
-- Check we are using some authentication
-- --rc-user gui
-- --rc-pass
-- --rc-serve
+ rclone lsd :s3,access_key_id=XXX,secret_access_key=XXX:
-These flags can be overridden as desired.
+Precedence
-See also the rclone rcd documentation.
+The various different methods of backend configuration are read in this
+order and the first one with a value is used.
-Example: Running a public GUI
+- Parameters in connection strings, e.g. myRemote,skip_links:
+- Flag values as supplied on the command line, e.g. --skip-links
+- Remote specific environment vars, e.g.
+ RCLONE_CONFIG_MYREMOTE_SKIP_LINKS (see above).
+- Backend-specific environment vars, e.g. RCLONE_LOCAL_SKIP_LINKS.
+- Backend generic environment vars, e.g. RCLONE_SKIP_LINKS.
+- Config file, e.g. skip_links = true.
+- Default values, e.g. false - these can't be changed.
-For example the GUI could be served on a public port over SSL using an
-htpasswd file using the following flags:
+So if both --skip-links is supplied on the command line and an
+environment variable RCLONE_LOCAL_SKIP_LINKS is set, the command line
+flag will take preference.
-- --rc-web-gui
-- --rc-addr :443
-- --rc-htpasswd /path/to/htpasswd
-- --rc-cert /path/to/ssl.crt
-- --rc-key /path/to/ssl.key
+The backend configurations set by environment variables can be seen with
+the -vv flag, e.g. rclone about myRemote: -vv.
-Example: Running a GUI behind a proxy
+For non backend configuration the order is as follows:
-If you want to run the GUI behind a proxy at /rclone you could use these
-flags:
+- Flag values as supplied on the command line, e.g. --stats 5s.
+- Environment vars, e.g. RCLONE_STATS=5s.
+- Default values, e.g. 1m - these can't be changed.
-- --rc-web-gui
-- --rc-baseurl rclone
-- --rc-htpasswd /path/to/htpasswd
+Other environment variables
-Or instead of htpasswd if you just want a single user and password:
+- RCLONE_CONFIG_PASS set to contain your config file password (see
+ Configuration Encryption section)
+- HTTP_PROXY, HTTPS_PROXY and NO_PROXY (or the lowercase versions
+ thereof).
+ - HTTPS_PROXY takes precedence over HTTP_PROXY for https requests.
+ - The environment values may be either a complete URL or a
+ "host[:port]" for, in which case the "http" scheme is assumed.
+- USER and LOGNAME values are used as fallbacks for current username.
+ The primary method for looking up username is OS-specific: Windows
+ API on Windows, real user ID in /etc/passwd on Unix systems. In the
+ documentation the current username is simply referred to as $USER.
+- RCLONE_CONFIG_DIR - rclone sets this variable for use in config
+ files and sub processes to point to the directory holding the config
+ file.
-- --rc-user me
-- --rc-pass mypassword
+The options set by environment variables can be seen with the -vv and
+--log-level=DEBUG flags, e.g. rclone version -vv.
-Project
+Configuring rclone on a remote / headless machine
-The GUI is being developed in the: rclone/rclone-webui-react repository.
+Some of the configurations (those involving oauth2) require an Internet
+connected web browser.
-Bug reports and contributions are very welcome :-)
+If you are trying to set rclone up on a remote or headless box with no
+browser available on it (e.g. a NAS or a server in a datacenter) then
+you will need to use an alternative means of configuration. There are
+two ways of doing it, described below.
-If you have questions then please ask them on the rclone forum.
+Configuring using rclone authorize
-Remote controlling rclone with its API
+On the headless box run rclone config but answer N to the
+Use web browser to automatically authenticate? question.
-If rclone is run with the --rc flag then it starts an HTTP server which
-can be used to remote control rclone using its API.
+ ...
+ Remote config
+ Use web browser to automatically authenticate rclone with remote?
+ * Say Y if the machine running rclone has a web browser you can use
+ * Say N if running rclone on a (remote) machine without web browser access
+ If not sure try Y. If Y failed, try N.
+ y) Yes (default)
+ n) No
+ y/n> n
+ For this to work, you will need rclone available on a machine that has
+ a web browser available.
-You can either use the rc command to access the API or use HTTP
-directly.
+ For more help and alternate methods see: https://rclone.org/remote_setup/
-If you just want to run a remote control then see the rcd command.
+ Execute the following on the machine with the web browser (same rclone
+ version recommended):
-Supported parameters
+ rclone authorize "amazon cloud drive"
---rc
+ Then paste the result below:
+ result>
-Flag to start the http server listen on remote requests
+Then on your main desktop machine
---rc-addr=IP
+ rclone authorize "amazon cloud drive"
+ If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth
+ Log in and authorize rclone for access
+ Waiting for code...
+ Got code
+ Paste the following into your remote machine --->
+ SECRET_TOKEN
+ <---End paste
-IPaddress:Port or :Port to bind server to. (default "localhost:5572")
+Then back to the headless box, paste in the code
---rc-cert=KEY
+ result> SECRET_TOKEN
+ --------------------
+ [acd12]
+ client_id =
+ client_secret =
+ token = SECRET_TOKEN
+ --------------------
+ y) Yes this is OK
+ e) Edit this remote
+ d) Delete this remote
+ y/e/d>
-SSL PEM key (concatenation of certificate and CA certificate)
+Configuring by copying the config file
---rc-client-ca=PATH
+Rclone stores all of its config in a single configuration file. This can
+easily be copied to configure a remote rclone.
-Client certificate authority to verify clients with
+So first configure rclone on your desktop machine with
---rc-htpasswd=PATH
+ rclone config
-htpasswd file - if not provided no authentication is done
+to set up the config file.
---rc-key=PATH
+Find the config file by running rclone config file, for example
-SSL PEM Private key
+ $ rclone config file
+ Configuration file is stored at:
+ /home/user/.rclone.conf
---rc-max-header-bytes=VALUE
+Now transfer it to the remote box (scp, cut paste, ftp, sftp, etc.) and
+place it in the correct place (use rclone config file on the remote box
+to find out where).
-Maximum size of request header (default 4096)
+Configuring using SSH Tunnel
---rc-min-tls-version=VALUE
+Linux and MacOS users can utilize SSH Tunnel to redirect the headless
+box port 53682 to local machine by using the following command:
-The minimum TLS version that is acceptable. Valid values are "tls1.0",
-"tls1.1", "tls1.2" and "tls1.3" (default "tls1.0").
+ ssh -L localhost:53682:localhost:53682 username@remote_server
---rc-user=VALUE
+Then on the headless box run rclone config and answer Y to the
+Use web browser to automatically authenticate? question.
-User name for authentication.
+ ...
+ Remote config
+ Use web browser to automatically authenticate rclone with remote?
+ * Say Y if the machine running rclone has a web browser you can use
+ * Say N if running rclone on a (remote) machine without web browser access
+ If not sure try Y. If Y failed, try N.
+ y) Yes (default)
+ n) No
+ y/n> y
---rc-pass=VALUE
+Then copy and paste the auth url
+http://127.0.0.1:53682/auth?state=xxxxxxxxxxxx to the browser on your
+local machine, complete the auth and it is done.
-Password for authentication.
+Filtering, includes and excludes
---rc-realm=VALUE
+Filter flags determine which files rclone sync, move, ls, lsl, md5sum,
+sha1sum, size, delete, check and similar commands apply to.
-Realm for authentication (default "rclone")
+They are specified in terms of path/file name patterns; path/file lists;
+file age and size, or presence of a file in a directory. Bucket based
+remotes without the concept of directory apply filters to object key,
+age and size in an analogous way.
---rc-server-read-timeout=DURATION
+Rclone purge does not obey filters.
-Timeout for server reading data (default 1h0m0s)
+To test filters without risk of damage to data, apply them to rclone ls,
+or with the --dry-run and -vv flags.
---rc-server-write-timeout=DURATION
+Rclone filter patterns can only be used in filter command line options,
+not in the specification of a remote.
-Timeout for server writing data (default 1h0m0s)
+E.g. rclone copy "remote:dir*.jpg" /path/to/dir does not have a filter
+effect. rclone copy remote:dir /path/to/dir --include "*.jpg" does.
---rc-serve
+Important Avoid mixing any two of --include..., --exclude... or
+--filter... flags in an rclone command. The results might not be what
+you expect. Instead use a --filter... flag.
-Enable the serving of remote objects via the HTTP interface. This means
-objects will be accessible at http://127.0.0.1:5572/ by default, so you
-can browse to http://127.0.0.1:5572/ or http://127.0.0.1:5572/* to see a
-listing of the remotes. Objects may be requested from remotes using this
-syntax http://127.0.0.1:5572/[remote:path]/path/to/object
+Patterns for matching path/file names
-Default Off.
+Pattern syntax
---rc-files /path/to/directory
+Here is a formal definition of the pattern syntax, examples are below.
-Path to local files to serve on the HTTP server.
+Rclone matching rules follow a glob style:
-If this is set then rclone will serve the files in that directory. It
-will also open the root in the web browser if specified. This is for
-implementing browser based GUIs for rclone functions.
+ * matches any sequence of non-separator (/) characters
+ ** matches any sequence of characters including / separators
+ ? matches any single non-separator (/) character
+ [ [ ! ] { character-range } ]
+ character class (must be non-empty)
+ { pattern-list }
+ pattern alternatives
+ {{ regexp }}
+ regular expression to match
+ c matches character c (c != *, **, ?, \, [, {, })
+ \c matches reserved character c (c = *, **, ?, \, [, {, }) or character class
-If --rc-user or --rc-pass is set then the URL that is opened will have
-the authorization in the URL in the http://user:pass@localhost/ style.
+character-range:
-Default Off.
+ c matches character c (c != \, -, ])
+ \c matches reserved character c (c = \, -, ])
+ lo - hi matches character c for lo <= c <= hi
---rc-enable-metrics
+pattern-list:
-Enable OpenMetrics/Prometheus compatible endpoint at /metrics.
+ pattern { , pattern }
+ comma-separated (without spaces) patterns
-Default Off.
+character classes (see Go regular expression reference) include:
---rc-web-gui
+ Named character classes (e.g. [\d], [^\d], [\D], [^\D])
+ Perl character classes (e.g. \s, \S, \w, \W)
+ ASCII character classes (e.g. [[:alnum:]], [[:alpha:]], [[:punct:]], [[:xdigit:]])
-Set this flag to serve the default web gui on the same port as rclone.
+regexp for advanced users to insert a regular expression - see below for
+more info:
-Default Off.
+ Any re2 regular expression not containing `}}`
---rc-allow-origin
+If the filter pattern starts with a / then it only matches at the top
+level of the directory tree, relative to the root of the remote (not
+necessarily the root of the drive). If it does not start with / then it
+is matched starting at the end of the path/file name but it only matches
+a complete path element - it must match from a / separator or the
+beginning of the path/file.
-Set the allowed Access-Control-Allow-Origin for rc requests.
+ file.jpg - matches "file.jpg"
+ - matches "directory/file.jpg"
+ - doesn't match "afile.jpg"
+ - doesn't match "directory/afile.jpg"
+ /file.jpg - matches "file.jpg" in the root directory of the remote
+ - doesn't match "afile.jpg"
+ - doesn't match "directory/file.jpg"
-Can be used with --rc-web-gui if the rclone is running on different IP
-than the web-gui.
+The top level of the remote might not be the top level of the drive.
-Default is IP address on which rc is running.
+E.g. for a Microsoft Windows local directory structure
---rc-web-fetch-url
+ F:
+ ├── bkp
+ ├── data
+ │ ├── excl
+ │ │ ├── 123.jpg
+ │ │ └── 456.jpg
+ │ ├── incl
+ │ │ └── document.pdf
-Set the URL to fetch the rclone-web-gui files from.
+To copy the contents of folder data into folder bkp excluding the
+contents of subfolder exclthe following command treats F:\data and
+F:\bkp as top level for filtering.
-Default
-https://api.github.com/repos/rclone/rclone-webui-react/releases/latest.
+rclone copy F:\data\ F:\bkp\ --exclude=/excl/**
---rc-web-gui-update
+Important Use / in path/file name patterns and not \ even if running on
+Microsoft Windows.
-Set this flag to check and update rclone-webui-react from the
-rc-web-fetch-url.
+Simple patterns are case sensitive unless the --ignore-case flag is
+used.
-Default Off.
+Without --ignore-case (default)
---rc-web-gui-force-update
+ potato - matches "potato"
+ - doesn't match "POTATO"
-Set this flag to force update rclone-webui-react from the
-rc-web-fetch-url.
+With --ignore-case
-Default Off.
+ potato - matches "potato"
+ - matches "POTATO"
---rc-web-gui-no-open-browser
+Using regular expressions in filter patterns
-Set this flag to disable opening browser automatically when using
-web-gui.
+The syntax of filter patterns is glob style matching (like bash uses) to
+make things easy for users. However this does not provide absolute
+control over the matching, so for advanced users rclone also provides a
+regular expression syntax.
-Default Off.
+The regular expressions used are as defined in the Go regular expression
+reference. Regular expressions should be enclosed in {{ }}. They will
+match only the last path segment if the glob doesn't start with / or the
+whole path name if it does. Note that rclone does not attempt to parse
+the supplied regular expression, meaning that using any regular
+expression filter will prevent rclone from using directory filter rules,
+as it will instead check every path against the supplied regular
+expression(s).
---rc-job-expire-duration=DURATION
+Here is how the {{regexp}} is transformed into an full regular
+expression to match the entire path:
-Expire finished async jobs older than DURATION (default 60s).
+ {{regexp}} becomes (^|/)(regexp)$
+ /{{regexp}} becomes ^(regexp)$
---rc-job-expire-interval=DURATION
+Regexp syntax can be mixed with glob syntax, for example
-Interval duration to check for expired async jobs (default 10s).
+ *.{{jpe?g}} to match file.jpg, file.jpeg but not file.png
---rc-no-auth
+You can also use regexp flags - to set case insensitive, for example
-By default rclone will require authorisation to have been set up on the
-rc interface in order to use any methods which access any rclone
-remotes. Eg operations/list is denied as it involved creating a remote
-as is sync/copy.
+ *.{{(?i)jpg}} to match file.jpg, file.JPG but not file.png
-If this is set then no authorisation will be required on the server to
-use these methods. The alternative is to use --rc-user and --rc-pass and
-use these credentials in the request.
+Be careful with wildcards in regular expressions - you don't want them
+to match path separators normally. To match any file name starting with
+start and ending with end write
-Default Off.
+ {{start[^/]*end\.jpg}}
---rc-baseurl
+Not
-Prefix for URLs.
+ {{start.*end\.jpg}}
-Default is root
+Which will match a directory called start with a file called end.jpg in
+it as the .* will match / characters.
---rc-template
+Note that you can use -vv --dump filters to show the filter patterns in
+regexp format - rclone implements the glob patterns by transforming them
+into regular expressions.
-User-specified template.
+Filter pattern examples
-Accessing the remote control via the rclone rc command
+ Description Pattern Matches Does not match
+ --------------- ---------------- ------------------------------- ------------------
+ Wildcard *.jpg /file.jpg /file.png
+ /dir/file.jpg /dir/file.png
+ Rooted /*.jpg /file.jpg /file.png
+ /file2.jpg /dir/file.jpg
+ Alternates *.{jpg,png} /file.jpg /file.gif
+ /dir/file.png /dir/file.gif
+ Path Wildcard dir/** /dir/anyfile file.png
+ /subdir/dir/subsubdir/anyfile /subdir/file.png
+ Any Char *.t?t /file.txt /file.qxt
+ /dir/file.tzt /dir/file.png
+ Range *.[a-z] /file.a /file.0
+ /dir/file.b /dir/file.1
+ Escape *.\?\?\? /file.??? /file.abc
+ /dir/file.??? /dir/file.def
+ Class *.\d\d\d /file.012 /file.abc
+ /dir/file.345 /dir/file.def
+ Regexp *.{{jpe?g}} /file.jpeg /file.png
+ /dir/file.jpg /dir/file.jpeeg
+ Rooted Regexp /{{.*\.jpe?g}} /file.jpeg /file.png
+ /file.jpg /dir/file.jpg
-Rclone itself implements the remote control protocol in its rclone rc
-command.
+How filter rules are applied to files
-You can use it like this
+Rclone path/file name filters are made up of one or more of the
+following flags:
- $ rclone rc rc/noop param1=one param2=two
- {
- "param1": "one",
- "param2": "two"
- }
+- --include
+- --include-from
+- --exclude
+- --exclude-from
+- --filter
+- --filter-from
-Run rclone rc on its own to see the help for the installed remote
-control commands.
+There can be more than one instance of individual flags.
-JSON input
+Rclone internally uses a combined list of all the include and exclude
+rules. The order in which rules are processed can influence the result
+of the filter.
-rclone rc also supports a --json flag which can be used to send more
-complicated input parameters.
+All flags of the same type are processed together in the order above,
+regardless of what order the different types of flags are included on
+the command line.
- $ rclone rc --json '{ "p1": [1,"2",null,4], "p2": { "a":1, "b":2 } }' rc/noop
- {
- "p1": [
- 1,
- "2",
- null,
- 4
- ],
- "p2": {
- "a": 1,
- "b": 2
- }
- }
+Multiple instances of the same flag are processed from left to right
+according to their position in the command line.
-If the parameter being passed is an object then it can be passed as a
-JSON string rather than using the --json flag which simplifies the
-command line.
+To mix up the order of processing includes and excludes use --filter...
+flags.
- rclone rc operations/list fs=/tmp remote=test opt='{"showHash": true}'
+Within --include-from, --exclude-from and --filter-from flags rules are
+processed from top to bottom of the referenced file.
-Rather than
+If there is an --include or --include-from flag specified, rclone
+implies a - ** rule which it adds to the bottom of the internal rule
+list. Specifying a + rule with a --filter... flag does not imply that
+rule.
- rclone rc operations/list --json '{"fs": "/tmp", "remote": "test", "opt": {"showHash": true}}'
+Each path/file name passed through rclone is matched against the
+combined filter list. At first match to a rule the path/file name is
+included or excluded and no further filter rules are processed for that
+path/file.
-Special parameters
+If rclone does not find a match, after testing against all rules
+(including the implied rule if appropriate), the path/file name is
+included.
-The rc interface supports some special parameters which apply to all
-commands. These start with _ to show they are different.
+Any path/file included at that stage is processed by the rclone command.
-Running asynchronous jobs with _async = true
+--files-from and --files-from-raw flags over-ride and cannot be combined
+with other filter options.
-Each rc call is classified as a job and it is assigned its own id. By
-default jobs are executed immediately as they are created or
-synchronously.
+To see the internal combined rule list, in regular expression form, for
+a command add the --dump filters flag. Running an rclone command with
+--dump filters and -vv flags lists the internal filter elements and
+shows how they are applied to each source path/file. There is not
+currently a means provided to pass regular expression filter options
+into rclone directly though character class filter rules contain
+character classes. Go regular expression reference
-If _async has a true value when supplied to an rc call then it will
-return immediately with a job id and the task will be run in the
-background. The job/status call can be used to get information of the
-background job. The job can be queried for up to 1 minute after it has
-finished.
+How filter rules are applied to directories
-It is recommended that potentially long running jobs, e.g. sync/sync,
-sync/copy, sync/move, operations/purge are run with the _async flag to
-avoid any potential problems with the HTTP request and response timing
-out.
+Rclone commands are applied to path/file names not directories. The
+entire contents of a directory can be matched to a filter by the pattern
+directory/* or recursively by directory/**.
-Starting a job with the _async flag:
+Directory filter rules are defined with a closing / separator.
- $ rclone rc --json '{ "p1": [1,"2",null,4], "p2": { "a":1, "b":2 }, "_async": true }' rc/noop
- {
- "jobid": 2
- }
+E.g. /directory/subdirectory/ is an rclone directory filter rule.
-Query the status to see if the job has finished. For more information on
-the meaning of these return parameters see the job/status call.
+Rclone commands can use directory filter rules to determine whether they
+recurse into subdirectories. This potentially optimises access to a
+remote by avoiding listing unnecessary directories. Whether optimisation
+is desirable depends on the specific filter rules and source remote
+content.
- $ rclone rc --json '{ "jobid":2 }' job/status
- {
- "duration": 0.000124163,
- "endTime": "2018-10-27T11:38:07.911245881+01:00",
- "error": "",
- "finished": true,
- "id": 2,
- "output": {
- "_async": true,
- "p1": [
- 1,
- "2",
- null,
- 4
- ],
- "p2": {
- "a": 1,
- "b": 2
- }
- },
- "startTime": "2018-10-27T11:38:07.911121728+01:00",
- "success": true
- }
+If any regular expression filters are in use, then no directory
+recursion optimisation is possible, as rclone must check every path
+against the supplied regular expression(s).
-job/list can be used to show the running or recently completed jobs
+Directory recursion optimisation occurs if either:
- $ rclone rc job/list
- {
- "jobids": [
- 2
- ]
- }
+- A source remote does not support the rclone ListR primitive. local,
+ sftp, Microsoft OneDrive and WebDAV do not support ListR. Google
+ Drive and most bucket type storage do. Full list
-Setting config flags with _config
+- On other remotes (those that support ListR), if the rclone command
+ is not naturally recursive, and provided it is not run with the
+ --fast-list flag. ls, lsf -R and size are naturally recursive but
+ sync, copy and move are not.
-If you wish to set config (the equivalent of the global flags) for the
-duration of an rc call only then pass in the _config parameter.
+- Whenever the --disable ListR flag is applied to an rclone command.
-This should be in the same format as the config key returned by
-options/get.
+Rclone commands imply directory filter rules from path/file filter
+rules. To view the directory filter rules rclone has implied for a
+command specify the --dump filters flag.
-For example, if you wished to run a sync with the --checksum parameter,
-you would pass this parameter in your JSON blob.
+E.g. for an include rule
- "_config":{"CheckSum": true}
+ /a/*.jpg
-If using rclone rc this could be passed as
+Rclone implies the directory include rule
- rclone rc sync/sync ... _config='{"CheckSum": true}'
+ /a/
-Any config parameters you don't set will inherit the global defaults
-which were set with command line flags or environment variables.
+Directory filter rules specified in an rclone command can limit the
+scope of an rclone command but path/file filters still have to be
+specified.
-Note that it is possible to set some values as strings or integers - see
-data types for more info. Here is an example setting the equivalent of
---buffer-size in string or integer format.
+E.g. rclone ls remote: --include /directory/ will not match any files.
+Because it is an --include option the --exclude ** rule is implied, and
+the /directory/ pattern serves only to optimise access to the remote by
+ignoring everything outside of that directory.
- "_config":{"BufferSize": "42M"}
- "_config":{"BufferSize": 44040192}
+E.g. rclone ls remote: --filter-from filter-list.txt with a file
+filter-list.txt:
-If you wish to check the _config assignment has worked properly then
-calling options/local will show what the value got set to.
+ - /dir1/
+ - /dir2/
+ + *.pdf
+ - **
-Setting filter flags with _filter
+All files in directories dir1 or dir2 or their subdirectories are
+completely excluded from the listing. Only files of suffix pdf in the
+root of remote: or its subdirectories are listed. The - ** rule prevents
+listing of any path/files not previously matched by the rules above.
-If you wish to set filters for the duration of an rc call only then pass
-in the _filter parameter.
+Option exclude-if-present creates a directory exclude rule based on the
+presence of a file in a directory and takes precedence over other rclone
+directory filter rules.
-This should be in the same format as the filter key returned by
-options/get.
+When using pattern list syntax, if a pattern item contains either / or
+**, then rclone will not able to imply a directory filter rule from this
+pattern list.
-For example, if you wished to run a sync with these flags
+E.g. for an include rule
- --max-size 1M --max-age 42s --include "a" --include "b"
+ {dir1/**,dir2/**}
-you would pass this parameter in your JSON blob.
+Rclone will match files below directories dir1 or dir2 only, but will
+not be able to use this filter to exclude a directory dir3 from being
+traversed.
- "_filter":{"MaxSize":"1M", "IncludeRule":["a","b"], "MaxAge":"42s"}
+Directory recursion optimisation may affect performance, but normally
+not the result. One exception to this is sync operations with option
+--create-empty-src-dirs, where any traversed empty directories will be
+created. With the pattern list example {dir1/**,dir2/**} above, this
+would create an empty directory dir3 on destination (when it exists on
+source). Changing the filter to {dir1,dir2}/**, or splitting it into two
+include rules --include dir1/** --include dir2/**, will match the same
+files while also filtering directories, with the result that an empty
+directory dir3 will no longer be created.
-If using rclone rc this could be passed as
+--exclude - Exclude files matching pattern
- rclone rc ... _filter='{"MaxSize":"1M", "IncludeRule":["a","b"], "MaxAge":"42s"}'
+Excludes path/file names from an rclone command based on a single
+exclude rule.
-Any filter parameters you don't set will inherit the global defaults
-which were set with command line flags or environment variables.
+This flag can be repeated. See above for the order filter flags are
+processed in.
-Note that it is possible to set some values as strings or integers - see
-data types for more info. Here is an example setting the equivalent of
---buffer-size in string or integer format.
+--exclude should not be used with --include, --include-from, --filter or
+--filter-from flags.
- "_filter":{"MinSize": "42M"}
- "_filter":{"MinSize": 44040192}
+--exclude has no effect when combined with --files-from or
+--files-from-raw flags.
-If you wish to check the _filter assignment has worked properly then
-calling options/local will show what the value got set to.
+E.g. rclone ls remote: --exclude *.bak excludes all .bak files from
+listing.
-Assigning operations to groups with _group = value
+E.g. rclone size remote: "--exclude /dir/**" returns the total size of
+all files on remote: excluding those in root directory dir and sub
+directories.
-Each rc call has its own stats group for tracking its metrics. By
-default grouping is done by the composite group name from prefix job/
-and id of the job like so job/1.
+E.g. on Microsoft Windows rclone ls remote: --exclude "*\[{JP,KR,HK}\]*"
+lists the files in remote: without [JP] or [KR] or [HK] in their name.
+Quotes prevent the shell from interpreting the \ characters.\ characters
+escape the [ and ] so an rclone filter treats them literally rather than
+as a character-range. The { and } define an rclone pattern list. For
+other operating systems single quotes are required ie
+rclone ls remote: --exclude '*\[{JP,KR,HK}\]*'
-If _group has a value then stats for that request will be grouped under
-that value. This allows caller to group stats under their own name.
+--exclude-from - Read exclude patterns from file
-Stats for specific group can be accessed by passing group to core/stats:
+Excludes path/file names from an rclone command based on rules in a
+named file. The file contains a list of remarks and pattern rules.
- $ rclone rc --json '{ "group": "job/1" }' core/stats
- {
- "speed": 12345
- ...
- }
+For an example exclude-file.txt:
-Data types
+ # a sample exclude rule file
+ *.bak
+ file2.jpg
-When the API returns types, these will mostly be straight forward
-integer, string or boolean types.
+rclone ls remote: --exclude-from exclude-file.txt lists the files on
+remote: except those named file2.jpg or with a suffix .bak. That is
+equivalent to rclone ls remote: --exclude file2.jpg --exclude "*.bak".
-However some of the types returned by the options/get call and taken by
-the options/set calls as well as the vfsOpt, mountOpt and the _config
-parameters.
+This flag can be repeated. See above for the order filter flags are
+processed in.
-- Duration - these are returned as an integer duration in nanoseconds.
- They may be set as an integer, or they may be set with time string,
- eg "5s". See the options section for more info.
-- Size - these are returned as an integer number of bytes. They may be
- set as an integer or they may be set with a size suffix string, eg
- "10M". See the options section for more info.
-- Enumerated type (such as CutoffMode, DumpFlags, LogLevel,
- VfsCacheMode - these will be returned as an integer and may be set
- as an integer but more conveniently they can be set as a string, eg
- "HARD" for CutoffMode or DEBUG for LogLevel.
-- BandwidthSpec - this will be set and returned as a string, eg "1M".
+The --exclude-from flag is useful where multiple exclude filter rules
+are applied to an rclone command.
-Specifying remotes to work on
+--exclude-from should not be used with --include, --include-from,
+--filter or --filter-from flags.
-Remotes are specified with the fs=, srcFs=, dstFs= parameters depending
-on the command being used.
+--exclude-from has no effect when combined with --files-from or
+--files-from-raw flags.
-The parameters can be a string as per the rest of rclone, eg
-s3:bucket/path or :sftp:/my/dir. They can also be specified as JSON
-blobs.
+--exclude-from followed by - reads filter rules from standard input.
-If specifying a JSON blob it should be a object mapping strings to
-strings. These values will be used to configure the remote. There are 3
-special values which may be set:
+--include - Include files matching pattern
-- type - set to type to specify a remote called :type:
-- _name - set to name to specify a remote called name:
-- _root - sets the root of the remote - may be empty
+Adds a single include rule based on path/file names to an rclone
+command.
-One of _name or type should normally be set. If the local backend is
-desired then type should be set to local. If _root isn't specified then
-it defaults to the root of the remote.
+This flag can be repeated. See above for the order filter flags are
+processed in.
-For example this JSON is equivalent to remote:/tmp
+--include has no effect when combined with --files-from or
+--files-from-raw flags.
- {
- "_name": "remote",
- "_path": "/tmp"
- }
+--include implies --exclude ** at the end of an rclone internal filter
+list. Therefore if you mix --include and --include-from flags with
+--exclude, --exclude-from, --filter or --filter-from, you must use
+include rules for all the files you want in the include statement. For
+more flexibility use the --filter-from flag.
-And this is equivalent to :sftp,host='example.com':/tmp
+E.g. rclone ls remote: --include "*.{png,jpg}" lists the files on
+remote: with suffix .png and .jpg. All other files are excluded.
- {
- "type": "sftp",
- "host": "example.com",
- "_path": "/tmp"
- }
+E.g. multiple rclone copy commands can be combined with --include and a
+pattern-list.
-And this is equivalent to /tmp/dir
+ rclone copy /vol1/A remote:A
+ rclone copy /vol1/B remote:B
- {
- type = "local",
- _ path = "/tmp/dir"
- }
+is equivalent to:
-Supported commands
+ rclone copy /vol1 remote: --include "{A,B}/**"
-backend/command: Runs a backend command.
+E.g. rclone ls remote:/wheat --include "??[^[:punct:]]*" lists the files
+remote: directory wheat (and subdirectories) whose third character is
+not punctuation. This example uses an ASCII character class.
-This takes the following parameters:
+--include-from - Read include patterns from file
-- command - a string with the command name
-- fs - a remote name string e.g. "drive:"
-- arg - a list of arguments for the backend command
-- opt - a map of string to string of options
+Adds path/file names to an rclone command based on rules in a named
+file. The file contains a list of remarks and pattern rules.
-Returns:
+For an example include-file.txt:
-- result - result from the backend command
+ # a sample include rule file
+ *.jpg
+ file2.avi
-Example:
+rclone ls remote: --include-from include-file.txt lists the files on
+remote: with name file2.avi or suffix .jpg. That is equivalent to
+rclone ls remote: --include file2.avi --include "*.jpg".
- rclone rc backend/command command=noop fs=. -o echo=yes -o blue -a path1 -a path2
+This flag can be repeated. See above for the order filter flags are
+processed in.
-Returns
+The --include-from flag is useful where multiple include filter rules
+are applied to an rclone command.
- {
- "result": {
- "arg": [
- "path1",
- "path2"
- ],
- "name": "noop",
- "opt": {
- "blue": "",
- "echo": "yes"
- }
- }
- }
+--include-from implies --exclude ** at the end of an rclone internal
+filter list. Therefore if you mix --include and --include-from flags
+with --exclude, --exclude-from, --filter or --filter-from, you must use
+include rules for all the files you want in the include statement. For
+more flexibility use the --filter-from flag.
-Note that this is the direct equivalent of using this "backend" command:
+--exclude-from has no effect when combined with --files-from or
+--files-from-raw flags.
- rclone backend noop . -o echo=yes -o blue path1 path2
+--exclude-from followed by - reads filter rules from standard input.
-Note that arguments must be preceded by the "-a" flag
+--filter - Add a file-filtering rule
-See the backend command for more information.
+Specifies path/file names to an rclone command, based on a single
+include or exclude rule, in + or - format.
-Authentication is required for this call.
+This flag can be repeated. See above for the order filter flags are
+processed in.
-cache/expire: Purge a remote from cache
+--filter + differs from --include. In the case of --include rclone
+implies an --exclude * rule which it adds to the bottom of the internal
+rule list. --filter...+ does not imply that rule.
-Purge a remote from the cache backend. Supports either a directory or a
-file. Params: - remote = path to remote (required) - withData =
-true/false to delete cached data (chunks) as well (optional)
+--filter has no effect when combined with --files-from or
+--files-from-raw flags.
-Eg
+--filter should not be used with --include, --include-from, --exclude or
+--exclude-from flags.
- rclone rc cache/expire remote=path/to/sub/folder/
- rclone rc cache/expire remote=/ withData=true
+E.g. rclone ls remote: --filter "- *.bak" excludes all .bak files from a
+list of remote:.
-cache/fetch: Fetch file chunks
+--filter-from - Read filtering patterns from a file
-Ensure the specified file chunks are cached on disk.
+Adds path/file names to an rclone command based on rules in a named
+file. The file contains a list of remarks and pattern rules. Include
+rules start with + and exclude rules with -. ! clears existing rules.
+Rules are processed in the order they are defined.
-The chunks= parameter specifies the file chunks to check. It takes a
-comma separated list of array slice indices. The slice indices are
-similar to Python slices: start[:end]
+This flag can be repeated. See above for the order filter flags are
+processed in.
-start is the 0 based chunk number from the beginning of the file to
-fetch inclusive. end is 0 based chunk number from the beginning of the
-file to fetch exclusive. Both values can be negative, in which case they
-count from the back of the file. The value "-5:" represents the last 5
-chunks of a file.
+Arrange the order of filter rules with the most restrictive first and
+work down.
-Some valid examples are: ":5,-5:" -> the first and last five chunks
-"0,-2" -> the first and the second last chunk "0:10" -> the first ten
-chunks
+E.g. for filter-file.txt:
-Any parameter with a key that starts with "file" can be used to specify
-files to fetch, e.g.
+ # a sample filter rule file
+ - secret*.jpg
+ + *.jpg
+ + *.png
+ + file2.avi
+ - /dir/Trash/**
+ + /dir/**
+ # exclude everything else
+ - *
- rclone rc cache/fetch chunks=0 file=hello file2=home/goodbye
+rclone ls remote: --filter-from filter-file.txt lists the path/files on
+remote: including all jpg and png files, excluding any matching
+secret*.jpg and including file2.avi. It also includes everything in the
+directory dir at the root of remote, except remote:dir/Trash which it
+excludes. Everything else is excluded.
-File names will automatically be encrypted when the a crypt remote is
-used on top of the cache.
+E.g. for an alternative filter-file.txt:
-cache/stats: Get cache stats
+ - secret*.jpg
+ + *.jpg
+ + *.png
+ + file2.avi
+ - *
-Show statistics for the cache remote.
+Files file1.jpg, file3.png and file2.avi are listed whilst secret17.jpg
+and files without the suffix .jpgor.png` are excluded.
-config/create: create the config for a remote.
+E.g. for an alternative filter-file.txt:
-This takes the following parameters:
+ + *.jpg
+ + *.gif
+ !
+ + 42.doc
+ - *
-- name - name of remote
-- parameters - a map of { "key": "value" } pairs
-- type - type of the new remote
-- opt - a dictionary of options to control the configuration
- - obscure - declare passwords are plain and need obscuring
- - noObscure - declare passwords are already obscured and don't
- need obscuring
- - nonInteractive - don't interact with a user, return questions
- - continue - continue the config process with an answer
- - all - ask all the config questions not just the post config ones
- - state - state to restart with - used with continue
- - result - result to restart with - used with continue
+Only file 42.doc is listed. Prior rules are cleared by the !.
-See the config create command for more information on the above.
+--files-from - Read list of source-file names
-Authentication is required for this call.
+Adds path/files to an rclone command from a list in a named file. Rclone
+processes the path/file names in the order of the list, and no others.
-config/delete: Delete a remote in the config file.
+Other filter flags (--include, --include-from, --exclude,
+--exclude-from, --filter and --filter-from) are ignored when
+--files-from is used.
-Parameters:
+--files-from expects a list of files as its input. Leading or trailing
+whitespace is stripped from the input lines. Lines starting with # or ;
+are ignored.
-- name - name of remote to delete
+Rclone commands with a --files-from flag traverse the remote, treating
+the names in --files-from as a set of filters.
-See the config delete command for more information on the above.
+If the --no-traverse and --files-from flags are used together an rclone
+command does not traverse the remote. Instead it addresses each
+path/file named in the file individually. For each path/file name, that
+requires typically 1 API call. This can be efficient for a short
+--files-from list and a remote containing many files.
-Authentication is required for this call.
+Rclone commands do not error if any names in the --files-from file are
+missing from the source remote.
-config/dump: Dumps the config file.
+The --files-from flag can be repeated in a single rclone command to read
+path/file names from more than one file. The files are read from left to
+right along the command line.
-Returns a JSON object: - key: value
+Paths within the --files-from file are interpreted as starting with the
+root specified in the rclone command. Leading / separators are ignored.
+See --files-from-raw if you need the input to be processed in a raw
+manner.
-Where keys are remote names and values are the config parameters.
+E.g. for a file files-from.txt:
-See the config dump command for more information on the above.
+ # comment
+ file1.jpg
+ subdir/file2.jpg
-Authentication is required for this call.
+rclone copy --files-from files-from.txt /home/me/pics remote:pics copies
+the following, if they exist, and only those files.
-config/get: Get a remote in the config file.
+ /home/me/pics/file1.jpg → remote:pics/file1.jpg
+ /home/me/pics/subdir/file2.jpg → remote:pics/subdir/file2.jpg
-Parameters:
+E.g. to copy the following files referenced by their absolute paths:
-- name - name of remote to get
+ /home/user1/42
+ /home/user1/dir/ford
+ /home/user2/prefect
-See the config dump command for more information on the above.
+First find a common subdirectory - in this case /home and put the
+remaining files in files-from.txt with or without leading /, e.g.
-Authentication is required for this call.
+ user1/42
+ user1/dir/ford
+ user2/prefect
-config/listremotes: Lists the remotes in the config file and defined in environment variables.
+Then copy these to a remote:
-Returns - remotes - array of remote names
+ rclone copy --files-from files-from.txt /home remote:backup
-See the listremotes command for more information on the above.
+The three files are transferred as follows:
-Authentication is required for this call.
+ /home/user1/42 → remote:backup/user1/important
+ /home/user1/dir/ford → remote:backup/user1/dir/file
+ /home/user2/prefect → remote:backup/user2/stuff
-config/password: password the config for a remote.
+Alternatively if / is chosen as root files-from.txt will be:
-This takes the following parameters:
+ /home/user1/42
+ /home/user1/dir/ford
+ /home/user2/prefect
-- name - name of remote
-- parameters - a map of { "key": "value" } pairs
+The copy command will be:
-See the config password command for more information on the above.
+ rclone copy --files-from files-from.txt / remote:backup
-Authentication is required for this call.
+Then there will be an extra home directory on the remote:
-config/providers: Shows how providers are configured in the config file.
+ /home/user1/42 → remote:backup/home/user1/42
+ /home/user1/dir/ford → remote:backup/home/user1/dir/ford
+ /home/user2/prefect → remote:backup/home/user2/prefect
-Returns a JSON object: - providers - array of objects
+--files-from-raw - Read list of source-file names without any processing
-See the config providers command for more information on the above.
+This flag is the same as --files-from except that input is read in a raw
+manner. Lines with leading / trailing whitespace, and lines starting
+with ; or # are read without any processing. rclone lsf has a compatible
+format that can be used to export file lists from remotes for input to
+--files-from-raw.
-Authentication is required for this call.
+--ignore-case - make searches case insensitive
-config/setpath: Set the path of the config file
+By default, rclone filter patterns are case sensitive. The --ignore-case
+flag makes all of the filters patterns on the command line case
+insensitive.
-Parameters:
+E.g. --include "zaphod.txt" does not match a file Zaphod.txt. With
+--ignore-case a match is made.
-- path - path to the config file to use
+Quoting shell metacharacters
-Authentication is required for this call.
+Rclone commands with filter patterns containing shell metacharacters may
+not as work as expected in your shell and may require quoting.
-config/update: update the config for a remote.
+E.g. linux, OSX (* metacharacter)
-This takes the following parameters:
+- --include \*.jpg
+- --include '*.jpg'
+- --include='*.jpg'
-- name - name of remote
-- parameters - a map of { "key": "value" } pairs
-- opt - a dictionary of options to control the configuration
- - obscure - declare passwords are plain and need obscuring
- - noObscure - declare passwords are already obscured and don't
- need obscuring
- - nonInteractive - don't interact with a user, return questions
- - continue - continue the config process with an answer
- - all - ask all the config questions not just the post config ones
- - state - state to restart with - used with continue
- - result - result to restart with - used with continue
+Microsoft Windows expansion is done by the command, not shell, so
+--include *.jpg does not require quoting.
-See the config update command for more information on the above.
+If the rclone error
+Command .... needs .... arguments maximum: you provided .... non flag arguments:
+is encountered, the cause is commonly spaces within the name of a remote
+or flag value. The fix then is to quote values containing spaces.
-Authentication is required for this call.
+Other filters
-core/bwlimit: Set the bandwidth limit.
+--min-size - Don't transfer any file smaller than this
-This sets the bandwidth limit to the string passed in. This should be a
-single bandwidth limit entry or a pair of upload:download bandwidth.
+Controls the minimum size file within the scope of an rclone command.
+Default units are KiB but abbreviations K, M, G, T or P are valid.
-Eg
+E.g. rclone ls remote: --min-size 50k lists files on remote: of 50 KiB
+size or larger.
- rclone rc core/bwlimit rate=off
- {
- "bytesPerSecond": -1,
- "bytesPerSecondTx": -1,
- "bytesPerSecondRx": -1,
- "rate": "off"
- }
- rclone rc core/bwlimit rate=1M
- {
- "bytesPerSecond": 1048576,
- "bytesPerSecondTx": 1048576,
- "bytesPerSecondRx": 1048576,
- "rate": "1M"
- }
- rclone rc core/bwlimit rate=1M:100k
- {
- "bytesPerSecond": 1048576,
- "bytesPerSecondTx": 1048576,
- "bytesPerSecondRx": 131072,
- "rate": "1M"
- }
+See the size option docs for more info.
-If the rate parameter is not supplied then the bandwidth is queried
+--max-size - Don't transfer any file larger than this
- rclone rc core/bwlimit
- {
- "bytesPerSecond": 1048576,
- "bytesPerSecondTx": 1048576,
- "bytesPerSecondRx": 1048576,
- "rate": "1M"
- }
+Controls the maximum size file within the scope of an rclone command.
+Default units are KiB but abbreviations K, M, G, T or P are valid.
-The format of the parameter is exactly the same as passed to --bwlimit
-except only one bandwidth may be specified.
+E.g. rclone ls remote: --max-size 1G lists files on remote: of 1 GiB
+size or smaller.
-In either case "rate" is returned as a human-readable string, and
-"bytesPerSecond" is returned as a number.
+See the size option docs for more info.
-core/command: Run a rclone terminal command over rc.
+--max-age - Don't transfer any file older than this
-This takes the following parameters:
+Controls the maximum age of files within the scope of an rclone command.
-- command - a string with the command name.
-- arg - a list of arguments for the backend command.
-- opt - a map of string to string of options.
-- returnType - one of ("COMBINED_OUTPUT", "STREAM",
- "STREAM_ONLY_STDOUT", "STREAM_ONLY_STDERR").
- - Defaults to "COMBINED_OUTPUT" if not set.
- - The STREAM returnTypes will write the output to the body of the
- HTTP message.
- - The COMBINED_OUTPUT will write the output to the "result"
- parameter.
+--max-age applies only to files and not to directories.
-Returns:
+E.g. rclone ls remote: --max-age 2d lists files on remote: of 2 days old
+or less.
-- result - result from the backend command.
- - Only set when using returnType "COMBINED_OUTPUT".
-- error - set if rclone exits with an error code.
-- returnType - one of ("COMBINED_OUTPUT", "STREAM",
- "STREAM_ONLY_STDOUT", "STREAM_ONLY_STDERR").
+See the time option docs for valid formats.
-Example:
+--min-age - Don't transfer any file younger than this
- rclone rc core/command command=ls -a mydrive:/ -o max-depth=1
- rclone rc core/command -a ls -a mydrive:/ -o max-depth=1
+Controls the minimum age of files within the scope of an rclone command.
+(see --max-age for valid formats)
-Returns:
+--min-age applies only to files and not to directories.
- {
- "error": false,
- "result": ""
- }
+E.g. rclone ls remote: --min-age 2d lists files on remote: of 2 days old
+or more.
- OR
- {
- "error": true,
- "result": ""
- }
+See the time option docs for valid formats.
-Authentication is required for this call.
+Other flags
-core/du: Returns disk usage of a locally attached disk.
+--delete-excluded - Delete files on dest excluded from sync
-This returns the disk usage for the local directory passed in as dir.
+Important this flag is dangerous to your data - use with --dry-run and
+-v first.
-If the directory is not passed in, it defaults to the directory pointed
-to by --cache-dir.
+In conjunction with rclone sync, --delete-excluded deletes any files on
+the destination which are excluded from the command.
-- dir - string (optional)
+E.g. the scope of rclone sync --interactive A: B: can be restricted:
-Returns:
+ rclone --min-size 50k --delete-excluded sync A: B:
- {
- "dir": "/",
- "info": {
- "Available": 361769115648,
- "Free": 361785892864,
- "Total": 982141468672
- }
- }
+All files on B: which are less than 50 KiB are deleted because they are
+excluded from the rclone sync command.
-core/gc: Runs a garbage collection.
+--dump filters - dump the filters to the output
-This tells the go runtime to do a garbage collection run. It isn't
-necessary to call this normally, but it can be useful for debugging
-memory problems.
+Dumps the defined filters to standard output in regular expression
+format.
-core/group-list: Returns list of stats.
+Useful for debugging.
-This returns list of stats groups currently in memory.
+Exclude directory based on a file
-Returns the following values:
+The --exclude-if-present flag controls whether a directory is within the
+scope of an rclone command based on the presence of a named file within
+it. The flag can be repeated to check for multiple file names, presence
+of any of them will exclude the directory.
- {
- "groups": an array of group names:
- [
- "group1",
- "group2",
- ...
- ]
- }
+This flag has a priority over other filter flags.
-core/memstats: Returns the memory statistics
+E.g. for the following directory structure:
-This returns the memory statistics of the running program. What the
-values mean are explained in the go docs:
-https://golang.org/pkg/runtime/#MemStats
+ dir1/file1
+ dir1/dir2/file2
+ dir1/dir2/dir3/file3
+ dir1/dir2/dir3/.ignore
-The most interesting values for most people are:
-
-- HeapAlloc - this is the amount of memory rclone is actually using
-- HeapSys - this is the amount of memory rclone has obtained from the
- OS
-- Sys - this is the total amount of memory requested from the OS
- - It is virtual memory so may include unused memory
+The command rclone ls --exclude-if-present .ignore dir1 does not list
+dir3, file3 or .ignore.
-core/obscure: Obscures a string passed in.
+Metadata filters
-Pass a clear string and rclone will obscure it for the config file: -
-clear - string
+The metadata filters work in a very similar way to the normal file name
+filters, except they match metadata on the object.
-Returns: - obscured - string
+The metadata should be specified as key=value patterns. This may be
+wildcarded using the normal filter patterns or regular expressions.
-core/pid: Return PID of current process
+For example if you wished to list only local files with a mode of 100664
+you could do that with:
-This returns PID of current process. Useful for stopping rclone process.
+ rclone lsf -M --files-only --metadata-include "mode=100664" .
-core/quit: Terminates the app.
+Or if you wished to show files with an atime, mtime or btime at a given
+date:
-(Optional) Pass an exit code to be used for terminating the app: -
-exitCode - int
+ rclone lsf -M --files-only --metadata-include "[abm]time=2022-12-16*" .
-core/stats: Returns stats about current transfers.
+Like file filtering, metadata filtering only applies to files not to
+directories.
-This returns all available stats:
+The filters can be applied using these flags.
- rclone rc core/stats
+- --metadata-include - Include metadatas matching pattern
+- --metadata-include-from - Read metadata include patterns from file
+ (use - to read from stdin)
+- --metadata-exclude - Exclude metadatas matching pattern
+- --metadata-exclude-from - Read metadata exclude patterns from file
+ (use - to read from stdin)
+- --metadata-filter - Add a metadata filtering rule
+- --metadata-filter-from - Read metadata filtering patterns from a
+ file (use - to read from stdin)
-If group is not provided then summed up stats for all groups will be
-returned.
+Each flag can be repeated. See the section on how filter rules are
+applied for more details - these flags work in an identical way to the
+file name filtering flags, but instead of file name patterns have
+metadata patterns.
-Parameters
+Common pitfalls
-- group - name of the stats group (string)
+The most frequent filter support issues on the rclone forum are:
-Returns the following values:
+- Not using paths relative to the root of the remote
+- Not using / to match from the root of a remote
+- Not using ** to match the contents of a directory
- {
- "bytes": total transferred bytes since the start of the group,
- "checks": number of files checked,
- "deletes" : number of files deleted,
- "elapsedTime": time in floating point seconds since rclone was started,
- "errors": number of errors,
- "eta": estimated time in seconds until the group completes,
- "fatalError": boolean whether there has been at least one fatal error,
- "lastError": last error string,
- "renames" : number of files renamed,
- "retryError": boolean showing whether there has been at least one non-NoRetryError,
- "serverSideCopies": number of server side copies done,
- "serverSideCopyBytes": number bytes server side copied,
- "serverSideMoves": number of server side moves done,
- "serverSideMoveBytes": number bytes server side moved,
- "speed": average speed in bytes per second since start of the group,
- "totalBytes": total number of bytes in the group,
- "totalChecks": total number of checks in the group,
- "totalTransfers": total number of transfers in the group,
- "transferTime" : total time spent on running jobs,
- "transfers": number of transferred files,
- "transferring": an array of currently active file transfers:
- [
- {
- "bytes": total transferred bytes for this file,
- "eta": estimated time in seconds until file transfer completion
- "name": name of the file,
- "percentage": progress of the file transfer in percent,
- "speed": average speed over the whole transfer in bytes per second,
- "speedAvg": current speed in bytes per second as an exponentially weighted moving average,
- "size": size of the file in bytes
- }
- ],
- "checking": an array of names of currently active file checks
- []
- }
+GUI (Experimental)
-Values for "transferring", "checking" and "lastError" are only assigned
-if data is available. The value for "eta" is null if an eta cannot be
-determined.
+Rclone can serve a web based GUI (graphical user interface). This is
+somewhat experimental at the moment so things may be subject to change.
-core/stats-delete: Delete stats group.
+Run this command in a terminal and rclone will download and then display
+the GUI in a web browser.
-This deletes entire stats group.
+ rclone rcd --rc-web-gui
-Parameters
+This will produce logs like this and rclone needs to continue to run to
+serve the GUI:
-- group - name of the stats group (string)
+ 2019/08/25 11:40:14 NOTICE: A new release for gui is present at https://github.com/rclone/rclone-webui-react/releases/download/v0.0.6/currentbuild.zip
+ 2019/08/25 11:40:14 NOTICE: Downloading webgui binary. Please wait. [Size: 3813937, Path : /home/USER/.cache/rclone/webgui/v0.0.6.zip]
+ 2019/08/25 11:40:16 NOTICE: Unzipping
+ 2019/08/25 11:40:16 NOTICE: Serving remote control on http://127.0.0.1:5572/
-core/stats-reset: Reset stats.
+This assumes you are running rclone locally on your machine. It is
+possible to separate the rclone and the GUI - see below for details.
-This clears counters, errors and finished transfers for all stats or
-specific stats group if group is provided.
+If you wish to check for updates then you can add --rc-web-gui-update to
+the command line.
-Parameters
+If you find your GUI broken, you may force it to update by add
+--rc-web-gui-force-update.
-- group - name of the stats group (string)
+By default, rclone will open your browser. Add
+--rc-web-gui-no-open-browser to disable this feature.
-core/transferred: Returns stats about completed transfers.
+Using the GUI
-This returns stats about completed transfers:
+Once the GUI opens, you will be looking at the dashboard which has an
+overall overview.
- rclone rc core/transferred
+On the left hand side you will see a series of view buttons you can
+click on:
-If group is not provided then completed transfers for all groups will be
-returned.
+- Dashboard - main overview
+- Configs - examine and create new configurations
+- Explorer - view, download and upload files to the cloud storage
+ systems
+- Backend - view or alter the backend config
+- Log out
-Note only the last 100 completed transfers are returned.
+(More docs and walkthrough video to come!)
-Parameters
+How it works
-- group - name of the stats group (string)
+When you run the rclone rcd --rc-web-gui this is what happens
-Returns the following values:
+- Rclone starts but only runs the remote control API ("rc").
+- The API is bound to localhost with an auto-generated username and
+ password.
+- If the API bundle is missing then rclone will download it.
+- rclone will start serving the files from the API bundle over the
+ same port as the API
+- rclone will open the browser with a login_token so it can log
+ straight in.
- {
- "transferred": an array of completed transfers (including failed ones):
- [
- {
- "name": name of the file,
- "size": size of the file in bytes,
- "bytes": total transferred bytes for this file,
- "checked": if the transfer is only checked (skipped, deleted),
- "timestamp": integer representing millisecond unix epoch,
- "error": string description of the error (empty if successful),
- "jobid": id of the job that this transfer belongs to
- }
- ]
- }
+Advanced use
-core/version: Shows the current version of rclone and the go runtime.
+The rclone rcd may use any of the flags documented on the rc page.
-This shows the current version of go and the go runtime:
+The flag --rc-web-gui is shorthand for
-- version - rclone version, e.g. "v1.53.0"
-- decomposed - version number as [major, minor, patch]
-- isGit - boolean - true if this was compiled from the git version
-- isBeta - boolean - true if this is a beta version
-- os - OS in use as according to Go
-- arch - cpu architecture in use according to Go
-- goVersion - version of Go runtime in use
-- linking - type of rclone executable (static or dynamic)
-- goTags - space separated build tags or "none"
+- Download the web GUI if necessary
+- Check we are using some authentication
+- --rc-user gui
+- --rc-pass