SYNOPSIS

saldl [OPTIONS] URL

DESCRIPTION

saldl is a CLI downloader based on libcurl.

By default, it splits a download into fixed-sized chunks and download them in-order with multiple concurrent connections.

Many OPTIONS are available to change the default behavior.

Files and Dirs

<filename>.part.sal

The incomplete/partial file that will eventually be renamed to <filename>.

<filename>.ctrl.sal

This file contains the minimal information required about the download progress to allow resuming.

<filename>.tmp.sal/

By default, saldl uses temporary files to store chunks before they are merged. Those files are created in this dir.

Chunk Progress States

4

Chunk already merged in <filename>.part.sal

3

Chunk downloaded, but still not merged.

This state should be transient. If it’s not, your system probably has load problems (CPU/IO/memory).

2

Chunk download is in progress.

1

Chunk is queued for download.

This is a transient state you shouldn’t see. Unless you have serious load/congestion problems in your system.

0

Chunk still not queued for download.

Note

When resuming from a previous session, an incomplete chunk would initially have state 0. That doesn’t mean it will be emptied and downloaded from scratch. Sub-chunk resume will work as expected.

OPTIONS

General

-d, --dry-run

Do not download, only display information.

--get-info

Do not download, only print specific informational to stdout.
The option can be passed multiple times with different arguments.

(Valid arguments are "file-name", "file-size", and "effective-url".)

--mirror-url

If the mirror is valid, Assign half the connections to it.

A mirror URL is considered valid when compared to URL if:
  1. Both do not have the same effective URL.

  2. Both have the same content size.

  3. Both support ranges.

  4. Both are not FTP.

  5. Both are not using content encoding. Or both do, but we will not decode/decompress.

--fatal-if-invalid-mirror

Fatally fail if the mirror is invalid. Only a warning is displayed by default.

--stdout

Write/Pipe output to stdout.

Everything else will behave in the same way except:
  1. The pseudo output filename "STDOUT" will be used.

  2. Chunks will only be merged (written to stdout) in order.

  3. Resume is disabled.

  4. No state is saved in a <filename>.ctrl.sal file.

  5. If single mode is used, timeouts and other non-fatal response errors will become fatal.

If -m/--memory-buffers are used, saldl will not create or write to any files directly.

-m, --memory-buffers

Use memory buffers instead of temp files for downloading chunks.
This mode increases memory overhead. And we lose sub-chunk resumability. But it might help performance in some situations, especially when IO is the bottleneck.

--read-only

Don’t create files or write anything to disk. This should be only used to test network performance.

--no-mmap (default mode only)

Read chunk files into buffers instead of using mmap().
This option hurts performance and should only be used for debugging.

UI Options

-i interval, --status-refresh-interval=interval

Minimum refresh interval between status updates in seconds. Increasing the default should help if terminal emulation is slow. (default: 0.3)

-C, --no-color

Disable colors in output.

Pass it twice (-CC) to disable all other formattings and escape codes (auto-set if not tty).

-V , --verbosity

Increase verbosity level.
(See Verbosity Levels for details.)

--verbose-libcurl

Enable libcurl’s verbose output. Regardless of verbosity level.

--no-status

Disable progress status output. (For debugging purposes only)

Resume Options

-r, --resume

resume download.
(requires a <filename>.ctrl.sal to exist with a matching filesize)

-f, --force

If not resuming, and <filename>.part.sal exists, truncate the file and start over.

Chunk sizes and connections

-S, --single

a single chunk single connection mode (wget-like).

-s size, --chunk-size=size

Set size for chunks in bytes. A unit suffix can be used. (default: 1m)

-c num, --connections=num

number of concurrent connections. (default: 6)

-l num, --last-chunks-first=num

the number of last chunks that should be downloaded first. (default: 0)

-L size, --last-size-first=size

The minimum size that should be downloaded first at the end of the file. A unit suffix can be used.
This should be useful when chunk-size is not known beforehand. e.g. when -a/--auto-size is used.
This option overrides -l/--last-chunks-first. (default: 0)

-F, --allow-ftp-segments

Due to a certain limitation in the information provided by libcurl, single mode is force-enabled by default with FTP links. This option overrides that behavior.

--random-order

Download chunks in random order.

--merge-in-order

Only merge chunks in-order.

This works as if we were piping to the output file, which could be useful with some storage devices.

Warning
  1. It does not make sense to use --merge-in-order with --last-chunks-first or --last-size-first. As merging those chunks will be delayed. Even if they were downloaded early.

  2. If memory buffers --merge-in-order are both used, completely downloaded chunks could be lost due to delayed merges if the download was interrupted.

-a num, --auto-size=num

increase chunk size so that chunk progress can fit in num lines.

-w, --whole-file

work like typical download accelerators (no. of chunks = no. of connections).

Filename Options

-o filename, --output-filename=filename

Use this filename instead of the detected name.

-n, --no-path

assume path is relative, replace / and : with _.

-D dirname, --root-dir=dirname

Prepend filename with this path.

-g, --filename-from-redirect

Get filename from redirected URL instead of the original URL.

-G, --keep-GET-attrs

keep GET attributes at the end of a filename. (This option has no effect if -o/--output-filename is used)

-A, --no-attachment-detection

Do not use Content-Disposition attachment filename if present.

-t, --auto-trunc

truncate filename if name or path is too long.

-T, --smart-trunc

same as -t/--auto-trunc but tries to keep the file extension. (overrides -t/--auto-trunc)

Network Options

-6, --resolve-ipv6

Only resolve to IPv6 addresses.

-4, --resolve-ipv4

Only resolve to IPv4 addresses.

Note

If both -6 and -4 are passed, what’s passed last will take precedence.

-R bandwidth, --connection-max-rate=bandwidth

maximum rate per connection in bytes/s. A unit suffix can be used. (default: 0 [unlimited])

-O, --no-timeouts

disable all timeouts.

-x proxy, --proxy=proxy

set proxy.

-X proxy, --tunnel-proxy=proxy

similar to --proxy but tunneling all traffic through it.

-N, --no-proxy

disable all proxies even if set in the environment.

--skip-TLS-verification

Skip TLS/SSL verification. Use at your own risk.

Custom Request Options

-E, --auto-referer

auto set referer in case of a redirect.

-e referer, --referer=referer

set referer.

-M local-file, --since-file-mtime=local-file

Sets the If-Modified-Since header with the last modification time of the passed local file.
This option has no effect if -I/--no-remote-info was passed.

-Y date-expression, --date-cond=date-expression

Sets If-Modified-Since header according to the date expression passed. Consult curl_getdate(3) for details.
If the expression starts with '-', then If-Unmodified-Since is set.
This option overrides -M/--since-file-mtime.
This option has no effect if -I/--no-remote-info was passed.

-z, --compress

Request a compressed response. Single mode would be forced if the server supports compression. Unless -Z/--no-decompress was passed.

-Z, --no-decompress

Disable decompression, whether compression was requested or forced by the server.

-K cookie-file, --cookie-file=cookie-file

File to read cookies from .

-k cookies, --inline-cookies=cookies

Set cookies.

-p post-data, --post=post-data

Send post-data in a simple POST request (no multipart).

-P raw-post-data, --raw-post=raw-post-data

Send raw-post-data as-is including headers (supports multipart).

Warning

--post and --raw-post will double-post by default. Unless -I/--no-remote-info is also passed.

-U, --no-user-agent

Don’t set user agent (disables default agent).

-u agent, --user-agent=agent

set user agent.

The following settings should not be generally used, but might help in rare cases:

-H, --custom-headers=header(s)

Add one or more custom headers to the request. Or suppress an auto one. Those headers would be sent to the remote server. If you need to send them to the proxy, use --proxy-custom-headers.

This option can be passed multiple times.
If only one header is passed. It must not end with \r\n. libcurl will add this for us. Multiple headers must be separated by \r\n. And the last on must also not end with an \r\n.

Three Header formats are acceptable:
1- Header with info

Name: info

2- Header without info

Name;

3- Suppress default header

Name:

--proxy-custom-headers=header(s)

Works like --custom-headers. But the headers are passed to the proxy instead of the remote server.

--use-HEAD

Use HEAD instead of GET to retrieve download information from servers.

-I, --no-remote-info

Just GET. Don’t do anything else. (-S/--signle is forced, -r/--resume is disabled)

--no-http2

Don’t try HTTP/2 requests.

--http2-upgrade

Try HTTP/2 requests with plain HTTP.

--no-tcp-keep-alive

Don’t send TCP keep-alive probes.

--assume-range-support

Don’t use this. The wrong data will probably be downloaded.

ENVIRONMENT

In addition to all environment variables affecting libcurl(3). The following variable(s) can affect saldl:

SALDL_EXTRA_ARGS

Append these arguments to argv.

Unit Suffixes

OPTIONS that take an argument representing a byte value can be suffixed with one of the following letters:

K or k

multiply size by 1024 (KiB).

M or m

multiply size by 1024*1024 (MiB).

G or g

multiply size by 1024*1024*1024 (GiB).

Verbosity Levels

saldl only shows essential information with the default verbosity level. You can pass -V/--verbosity multiple times to increase verbosity.

Note

Verbosity levels are incremental. Each level includes all messages from lower levels.

-V

Show non-fatal errors.

-VV

Show warnings.

-VVV

Show informational messages.

Warning

The following levels are noisy. They should only be used for debugging/tracing purposes.

-VVVV

Show debug messages.

-VVVVV

Enable verbose output in libcurl.

Note

This can be enabled independently with the --verbose-libcurl option.

-VVVVVV

Show debug messages from event loops.

EXIT STATUS

0

Success.

1

A fatal error happened. A helpful message should explain the error.

WWW

BUGS

AUTHOR

Mohammad AlSaleh (MoSal@Github).

COPYING

Copyright (C) 2014-2015 Mohammad AlSaleh. Free use of this software is granted under the terms of the GNU Affero General Public License (AGPL).