Please suggest me the FASTEST way to transfer 35GB compressed (using 7zip) data from my laptop to remote server. Please post the command as well. I've used SCP but it's terribly slow, rsync is better but with its speed of roughly 1MB/min, it might take a very very long time. Is there any better way?
73 Answers
ftp and http are the fastest , ftp is faster for a smaller number of large files and http faster for a larger number of small files (no surprise considering the nature of http traffic).
See
Additional benchmarks -
Note: Your internet provider may limit the upload speed depending on your provider. If your provider is imposing a speed or data cap it may not matter.
You may be able to use other methods as well, see
735GiB will take around 25 days at 1MB/minute (17kB/s). As it will take a long time, I'd focus on a way that lets you resume easily. This rules out scp as far as I am aware, and I'd look into the following candidates:
- http(s)
- rsync
- Sneakernet
Setting up a webserver for serving a single file is fairly easy; a standard Ubuntu install can do it without any configuration, as long as the machine is reachable from the Internet. Simply place the file in /var/www/html/ (or link it there).
Use wget -c to resume download if interrupted. This works quite reliably. Since it's a one time thing, don't bother with ssl certificates - encrypt and sign the data if you're worried about security and integrity.
Rsync is transfered over SSH, so it's secure. It supports seamless continuation. rsync -P localfile :remotefile should support resume, and transfer the file without any worries.
But 25 days is a long time. You can probably copy the drive to a USB drive, and send it in the mail faster. 64GiB USB drives can be had for around 20USD. If you DHL it, it'll make it to almost anywhere within 3-4 days.
In short - the quickest option with this amount of data, and the connection speed you're seeing, the best option is probably just to stuff it on an external drive, and mail it.
Or go somewhere with a decent connection - 35GiB at a 100Mb/s connection will take under an hour.
For encrypting and signing the file, you can use gpg. Set up a key pair, encrypt the file and sign it. On the remote end you verify signature, and decrypt it. This ensures security and integrity when transporting trough an untrusted channel like mail or http. If you use rsync, ssh will take care of this for you.
As Panther has stated, unencrypted options such as FTP or HTTP would be the fastest. However, if there is anything you prefer to have not public on the internet, I recommend you use an encrypted method. You could try something like creating a simple PHP login page(using an if statement), install and configure nginx or apache for HTTPS, and have a download link only you can access. HTTPS should be fast as it uses compression, and you can use a program such as axel to install that file using multiple connections.
The simplest answer, assuming your internet connection is the bottleneck, and not the CPU, would be to use SFTP with the -C flag to enable compression, although since you already compressed it, the amount that can be further compressed is rather small. The easiest thing to do is to compress the files again with a more efficient method.
