Sync command

Copies multiple files from source to destination. Optionally deletes or hides destination files that the source does not have.

The synchronizer can copy files:

  • From a B2 bucket to a local destination.

  • From a local source to a B2 bucket.

  • From one B2 bucket to another.

  • Between different folders in the same B2 bucket.

Use b2://<bucketName>/<prefix> for B2 paths, e.g. b2://my-bucket-name/a/path/prefix/.

Progress is displayed on the console unless --no-progress is specified. A list of actions taken is always printed.

Specify --dry-run to simulate the actions that would be taken.

To allow sync to run when the source directory is empty, potentially deleting all files in a bucket, specify --allow-empty-source. The default is to fail when the specified source directory doesn’t exist or is empty. (This check only applies to version 1.0 and later.)

Use –threads to manually adjust the number of threads used in the operation. Otherwise, the number of threads will be automatically chosen.

You can alternatively control number of threads per each operation. The number of files processed in parallel is set by --sync-threads, the number of files/file parts downloaded in parallel is set by``–download-threads``, and the number of files/file parts uploaded in parallel is set by –upload-threads`. All the three parameters can be set to the same value by --threads. Experiment with parameters if the defaults are not working well.

Users with low-performance networks may benefit from reducing the number of threads. Using just one thread will minimize the impact on other users of the network.

Note

Note that using multiple threads could be detrimental to the other users on your network.

You can specify --exclude-regex to selectively ignore files that match the given pattern. Ignored files will not copy during the sync operation. The pattern is a regular expression that is tested against the full path of each file.

You can specify --include-regex to selectively override ignoring files that match the given --exclude-regex pattern by an --include-regex pattern. Similarly to --exclude-regex, the pattern is a regular expression that is tested against the full path of each file.

Note

Note that --include-regex cannot be used without --exclude-regex.

You can specify --exclude-all-symlinks to skip symlinks when syncing from a local source.

When a directory is excluded by using --exclude-dir-regex, all of the files within it are excluded, even if they match an --include-regex pattern. This means that there is no need to look inside excluded directories, and you can exclude directories containing files for which you don’t have read permission and avoid getting errors.

The --exclude-dir-regex is a regular expression that is tested against the full path of each directory. The path being matched does not have a trailing /, so don’t include on in your regular expression.

Multiple regex rules can be applied by supplying them as pipe delimited instructions. Note that the regex for this command is Python regex. Reference: https://docs.python.org/3/library/re.html

Regular expressions are considered a match if they match a substring starting at the first character. .*e will match hello. This is not ideal, but we will maintain this behavior for compatibility. If you want to match the entire path, put a $ at the end of the regex, such as .*llo$.

You can specify --exclude-if-modified-after to selectively ignore file versions (including hide markers) which were synced after given time (for local source) or ignore only specific file versions (for b2 source). Ignored files or file versions will not be taken for consideration during sync. The time should be given as a seconds timestamp (e.g. “1367900664”) If you need milliseconds precision, put it after the comma (e.g. “1367900664.152”)

Files are considered to be the same if they have the same name and modification time. This behaviour can be changed using the --compare-versions option. Possible values are:

  • none: Comparison using the file name only

  • modTime: Comparison using the modification time (default)

  • size: Comparison using the file size

A future enhancement may add the ability to compare the SHA1 checksum of the files.

Fuzzy comparison of files based on modTime or size can be enabled by specifying the --compare-threshold option. This will treat modTimes (in milliseconds) or sizes (in bytes) as the same if they are within the comparison threshold. Files that match, within the threshold, will not be synced. Specifying --verbose and --dry-run can be useful to determine comparison value differences.

When a destination file is present that is not in the source, the default is to leave it there. Specifying --delete means to delete destination files that are not in the source.

When the destination is B2, you have the option of leaving older versions in place. Specifying --keep-days will delete any older versions more than the given number of days old, based on the modification time of the file. This option is not available when the destination is a local folder.

Files at the source that have a newer modification time are always copied to the destination. If the destination file is newer, the default is to report an error and stop. But with --skip-newer set, those files will just be skipped. With --replace-newer set, the old file from the source will replace the newer one in the destination.

To make the destination exactly match the source, use:

b2 sync --delete --replace-newer ... ...

Warning

Using --delete deletes files! We recommend not using it. If you use --keep-days instead, you will have some time to recover your files if you discover they are missing on the source end.

To make the destination match the source, but retain previous versions for 30 days:

b2 sync --keep-days 30 --replace-newer ... b2://...

Example of sync being used with --exclude-regex. This will ignore .DS_Store files and .Spotlight-V100 folders:

b2 sync --exclude-regex '(.*\.DS_Store)|(.*\.Spotlight-V100)' ... b2://...

To request SSE-B2 or SSE-C encryption for destination files, please set --destination-server-side-encryption=SSE-B2/SSE-C. The default algorithm is set to AES256 which can be changed with --destination-server-side-encryption-algorithm parameter. Using SSE-C requires providing B2_DESTINATION_SSE_C_KEY_B64 environment variable, containing the base64 encoded encryption key. If B2_DESTINATION_SSE_C_KEY_ID environment variable is provided, it’s value will be saved as sse_c_key_id in the uploaded file’s fileInfo.

To access SSE-C encrypted files, please set --source-server-side-encryption=SSE-C. The default algorithm is set to AES256 which can by changed with --source-server-side-encryption-algorithm parameter. Using SSE-C requires providing B2_SOURCE_SSE_C_KEY_B64 environment variable, containing the base64 encoded encryption key.

Use –write-buffer-size to set the size (in bytes) of the buffer used to write files.

Use –skip-hash-verification to disable hash check on downloaded files.

Use –max-download-streams-per-file to set max num of streams for parallel downloader.

Use –incremental-mode to allow for incremental file uploads to safe bandwidth. This will only affect files, which have been appended to since last upload.

Requires capabilities:

  • listFiles

  • readFiles (for downloading)

  • writeFiles (for uploading)

b2 sync [-h] [--no-progress] [--dry-run] [--allow-empty-source]
        [--exclude-all-symlinks] [--sync-threads SYNC_THREADS]
        [--download-threads DOWNLOAD_THREADS]
        [--upload-threads UPLOAD_THREADS]
        [--compare-versions {none,modTime,size}] [--compare-threshold MILLIS]
        [--exclude-regex REGEX] [--include-regex REGEX]
        [--exclude-dir-regex REGEX] [--exclude-if-modified-after TIMESTAMP]
        [--threads THREADS]
        [--destination-server-side-encryption {SSE-B2,SSE-C}]
        [--destination-server-side-encryption-algorithm {AES256}]
        [--source-server-side-encryption {SSE-C}]
        [--source-server-side-encryption-algorithm {AES256}]
        [--write-buffer-size BYTES] [--skip-hash-verification]
        [--max-download-streams-per-file MAX_DOWNLOAD_STREAMS_PER_FILE]
        [--incremental-mode] [--skip-newer | --replace-newer]
        [--delete | --keep-days DAYS]
        source destination

Positional Arguments

source
destination

Named Arguments

--no-progress

Default: False

--dry-run

Default: False

--allow-empty-source

Default: False

--exclude-all-symlinks

Default: False

--sync-threads

Default: 10

--download-threads

Default: 10

--upload-threads

Default: 10

--compare-versions

Possible choices: none, modTime, size

Default: “modTime”

--compare-threshold
--exclude-regex

Default: []

--include-regex

Default: []

--exclude-dir-regex

Default: []

--exclude-if-modified-after
--threads
--destination-server-side-encryption

Possible choices: SSE-B2, SSE-C

--destination-server-side-encryption-algorithm

Possible choices: AES256

Default: “AES256”

--source-server-side-encryption

Possible choices: SSE-C

--source-server-side-encryption-algorithm

Possible choices: AES256

Default: “AES256”

--write-buffer-size
--skip-hash-verification

Default: False

--max-download-streams-per-file
--incremental-mode

Default: False

--skip-newer

Default: False

--replace-newer

Default: False

--delete

Default: False

--keep-days