From my experience, it is quite slow to use scp to transfer data between my laptop and EC2. In this article, I’ll show you how to upload and download files to/from Amazon EC2 with Dropbox, which is much faster.
Table of Contents
1. Transfer data with scp
1.1 Upload files to EC2
scp -i wp_sparkandshine.pem sparkandshine.net.tar.gz email@example.com:/var/www/.
1.2 Download files from EC2
Use scp (secure copy) to download data from EC2.
# Step 1: Compress data tar czf sparkandshine.net.tar.gz sparkandshine.net/ # OR tar czf $(date +%Y%m%d-%H%M%S).tar.gz sparkandshine.net/ # filename with the current date and time, e.g., 20160425-190528.tar.gz # Step 2: Download data $ scp -i wp_sparkandshine.pem firstname.lastname@example.org:/var/www/sparkandshine.net.tar.gz ~/ sparkandshine.net.tar.gz 100% 903MB 339.0KB/s 45:29
As you can see, the data rate is so slow and it took 45 minutes to transfer 903M. I am figuring out how to download data by using Dropbox API.
2. Transfer data with Dropbox
2.1 Upload files to EC2
The most common way to transfer data between clients and servers is to use FTP applications like FileZilla. It is very slow from my experience.
I come up with another way to speed up the transfer rate with the help of Dropbox. Compress data, upload to Dropbox, get the shared link and use
wget to download data on the server side. In my case,
$ wget https://www.dropbox.com/s/***/public_html.tar.gz 100%[=======>] 219,742,208 7.75MB/s in 37s 2015-04-27 09:21:08 (5.71 MB/s) – ‘public_html.tar’ saved [219742208/219742208]
2.2 Download files from EC2
- Install the Python module Dropbox by
sudo pip install dropbox
- Obtain the access token from Dropbox Developers by creating an app
The following source code is given along with API v2 (simpler, more consistent, and more comprehensive).
#!/usr/bin/env python # -*- coding: utf-8 -*- import dropbox class TransferData: def __init__(self, access_token): self.access_token = access_token def upload_file(self, file_from=None, file_to=None): """upload a file to Dropbox using API v2 """ dbx = dropbox.Dropbox(self.access_token) #files_upload(f, path, mode=WriteMode('add', None), autorename=False, client_modified=None, mute=False) with open(file_from, 'rb') as f: dbx.files_upload(f, file_to) def main(): access_token = '******' transferData = TransferData(access_token) file_from = 'test.txt' file_to = '/test_dropbox/test.txt' # The full path to upload the file to, including the file name # API v2 transferData.upload_file(file_from=file_from, file_to=file_to) if __name__ == '__main__': main()
The source code with API v1 and v2 is hosted on my GitHub, here.