Using WinSCP to upload files to AWS S3
Background
This article was written on 16-Oct-24.
I already use AWS Glacier for my company files archive, but then it seems AWS wants us to move to S3 (and use archive storage class).
The application FastGlacier developer also states that they no longer maintain the app, so I have no choice but to move my archive, use AWS S3 and select an app to upload my (archive) files to AWS S3.
Why use winscp?
WinSCP has been around for a long time, which we mainly use to interface between Microsoft Windows and Linux.
So it should be there for another longer time and the best part is, for version 6.3.5 it supports AWS S3 natively.
Precautions
Do not delete any of your files until you are sure that files are uploaded properly with the correct size.
It is recommended to do a trial restore (and open the file) to ensure that your archived files are safe and sound.
Prerequisites
You will need an AWS admin account.
Enabling your AWS Region
AWS account will be created as a global account.
But there maybe a government requirement that your data must be kept in your country.
And it can be that network speed is faster if you use an AWS region which is nearest to where you live.
You can enable/disable AWS region from account menu.
Then scroll to region and select which region you want to enable/disable.
Once you have enabled your desired region, do not forget to switch to that region.
This way, your newly created AWS resources will be in your desired region.
Create AWS backup usergroup and user
Yes, it is recommended to create a special user credential to perform your file copy (and archive) to AWS.
And in general, you should always assign access policies to a user group.
Go to IAM module and create a new user group.
And most importantly, assign a permission policy.
Since we are going to manage S3, then assign S3FullAccess Policy.
Create user
Once user group and permission policy is assigned, create a user and assign it into the user group.
As this user is only a worker, then no need to allow it to login to console.
Create user access key
Press “create user” and open it.
Note down the AWS ARN (Access Resource Name) as we will use it to provide access to our bucket.
Then press “create access key”.
Since we will be using winscp, then create access key for third party service
Save your access key and secret access key — which we will use in winscp.
Create S3 bucket
Now that you have your usergroup and user created, we can proceed with creating an S3 bucket.
The term bucket is more or less similar to folder.
Go to S3 module and create a bucket.
Make sure that you are in the correct AWS region.
Since we will be accessing our S3 using winscp (from public), then we need to allow public access using policies and disable ACL access.
I did not change any other setting and just press “create bucket” to create your bucket.
Open your bucket and go to properties to take note of its ARN.
We will use bucket ARN to create access policy.
Assigning permission to your bucket
Open your bucket and go to “permissions”.
Make sure that public access is open using policies
Create access policy
This step might be daunting for those who are not familiar with creating script, but need not to worry, just imitate what I did.
Do not forget to select “All Actions” and press “Add Statements” and then “Generate Policy”.
Paste into bucket policy and press save changes.
That’s it, your bucket is ready for access.
Setup winscp
Assuming you have winscp installed, go to new connection.
Please note to change host name to your bucket region.
In my case it is in HK, so it is s3.ap-east-1.amazonaws.com.
Then put in your AWS user Access key ID and Secret access key.
Then press advanced to configure your region.
Save your connection configuration and press login.
You should be able to see your created bucket.
To upload files, just select from left pane and select upload.
S3 storage class
Wait, we are not finished yet!
Our original objective is to store archive.
What I meant by archive is something which I would not touch (read: restore) for a long time, let’s say, maybe once every two years.
For such, and to reduce AWS S3 cost, we need to change storage class.
Please read below article on AWS S3 storage classes.
https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html
Changing S3 storage class
Assuming you already finished uploading your archive, to change your bucket storage class, open your bucket from AWS console and go to objects.
Select all your objects, press “Actions” button and select “edit storage class”
For my archive, I choose “Glacier Deep Archive” and press “save changes”.
Please note that this action may incur in cost (per object).
When successful, your object storage class will change to “Glacier Deep Archive”.
But what about those orange ones?
Well, those are new uploaded files and their storage classes do not change automatically.
I already set a transition lifecycle, but it does not seem to work.
So for now, do not forget to check your object storage classes and change them if necessary.
I hope this article helps you in working with AWS S3, archives and winscp!
