From my experience, it is quite slow to use scp to transfer data between my laptop and EC2. In this article, I’ll show you how to upload and download files to/from Amazon EC2 with Dropbox, which is much faster.

1. Transfer data with scp

1.1 Upload files to EC2

scp -i wp_sparkandshine.pem sparkandshine.net.tar.gz ubuntu@xx.xx.xx.xx:/var/www/.

1.2 Download files from EC2

Use scp (secure copy) to download data from EC2.

# Step 1: Compress data
tar czf sparkandshine.net.tar.gz sparkandshine.net/
# OR
tar czf $(date +%Y%m%d-%H%M%S).tar.gz sparkandshine.net/    # filename with the current date and time, e.g., 20160425-190528.tar.gz 

# Step 2: Download data
$ scp -i wp_sparkandshine.pem ubuntu@xx.xx.xx.xx:/var/www/sparkandshine.net.tar.gz ~/
sparkandshine.net.tar.gz                       100%  903MB 339.0KB/s   45:29

As you can see, the data rate is so slow and it took 45 minutes to transfer 903M. I am figuring out how to download data by using Dropbox API.

2. Transfer data with Dropbox

2.1 Upload files to EC2

The most common way to transfer data between clients and servers is to use FTP applications like FileZilla. It is very slow from my experience.

I come up with another way to speed up the transfer rate with the help of Dropbox. Compress data, upload to Dropbox, get the shared link and use wget to download data on the server side. In my case,

$ wget https://www.dropbox.com/s/***/public_html.tar.gz
100%[=======>] 219,742,208 7.75MB/s   in 37s 
2015-04-27 09:21:08 (5.71 MB/s) – ‘public_html.tar’ saved [219742208/219742208]

2.2 Download files from EC2

The first step is to upload files to Dropbox and then download the files from Dropbox. Both Dropbox API v1 and v2 allow to upload data. Before moving forward,

• Install the Python module Dropbox by sudo pip install dropbox
• Obtain the access token from Dropbox Developers by creating an app

The following source code is given along with API v2 (simpler, more consistent, and more comprehensive).

#!/usr/bin/env python
# -*- coding: utf-8 -*-
import dropbox

class TransferData:
    def __init__(self, access_token):
        self.access_token = access_token

    def upload_file(self, file_from=None, file_to=None):
        """upload a file to Dropbox using API v2
        """
        dbx = dropbox.Dropbox(self.access_token)
        #files_upload(f, path, mode=WriteMode('add', None), autorename=False, client_modified=None, mute=False)

        with open(file_from, 'rb') as f:
            dbx.files_upload(f, file_to)

def main():
    access_token = '******'
    transferData = TransferData(access_token)

    file_from = 'test.txt'
    file_to = '/test_dropbox/test.txt'  # The full path to upload the file to, including the file name

    # API v2
    transferData.upload_file(file_from=file_from, file_to=file_to)

if __name__ == '__main__':
    main()

The source code with API v1 and v2 is hosted on my GitHub, here.

References:
[1] How Can I Download a File from EC2
[2] upload file to my dropbox from python script
[3] Dropbox for Python Developers
[4] Dropbox APP v1: Core API for Python Documentation

赞赏

微信赞赏支付宝赞赏