Here is how I have launched a simple but effective data pipeline on AWS just with an EC2 micro instance and one tiny S3 bucket (but behind the scene there are a lot of codes and bash files 😎):

Create a bucket and launch one ec2

0- Go to s3 panel and create a bucket, for example “files-1234”.

1- Launch a t2.micro EC2 instance with Ubuntu OS and name it ETLProject

2- Select instance and then click on Actions > Security> Modify IAM Role. Create a role with admin or read-only access to s3. If you want to transfer files later from ec2 to s3 it’s better to grant admin permission or read-write permission.

Operations on our PC

3- Connect to it from personal computer cli by running a command like this. First, cd to the directory of ETLProject.pem and then

$ chmod 400 ETLProject.pem

$ ssh -i "ETLProject.pem" ubuntu@ec2-3-71-182-555.eu-east-1.compute.amazonaws.com

Note that “ETLProject.pem” file should be created at the time when we’re launching our instance.

Install required tools

4- Now run these commands to install aws-cli and python

$ sudo apt update

$ sudo apt install python3 pandas

$ sudo snap install aws-cli --classic

Copy files from our PC to aws

5- From our computer’s cli which has been configured by the AWS cli credentials run this command to copy our desired files and scripts to s3

$ aws s3 cp <Fully Qualified Local filename> s3://files-1234

Or

$ for f in *.py; do aws s3 cp $f s3://files-1234; done

Setup cronjobs

6- Since I’m going to schedule a task, I go for cron jobs. So in the instance cli I run

$ sudo usermod -a -G crontab $(whoami)

Copy files from our aws to ec2

7- Then I copy some scripts that are in a s3 bucket to this instance by running:

$ aws s3 cp s3://files-1234/script.py .

$ aws s3 cp s3://files-1234/script.sh .

The above command will be copying the script.py and script.sh to the current directory of my instance

Schedule your job

8- Now schedule your job like this:

$ crontab -e

and then

*/5 * * * * /home/ubuntu/SOME-DIRECTORY/script.sh

That’s it. Now your script will do its job. After one day or one week you can see the results and copy the output again to s3 by

$ aws s3 cp /home/ubuntu/SOME-DIRECTORY s3://files-1234

Or you can write a bash script again to copy the final result for you.