Upload a Huge File to AWS S3 with Node

The Standard Way

Using AWS Node.js SDK we want to upload a huge file to AWS S3. Using putObject will probably timeout for a huge file. So we need streams.

Stream the Upload

Create the read stream:

var readStream = fs.createReadStream(fileName);  
var params = {Bucket: bucket, Key: key, Body: readStream};

Define how it is divided into parts (5KB), and the degree of concurrency (10):

var options = { partSize: 5 * 1024 * 1024, queueSize: 10 };  

var s3 = new AWS.S3({ httpOptions: { timeout: 10 * 60 * 1000 }});
s3.upload(params, options)

Track progress:

.on('httpUploadProgress', function(evt) {  
console.log('Completed ' +
(evt.loaded * 100 / evt.total).toFixed() +
'% of upload');

On completion:

.send(function(err, data) {  
console.log('Upload done');

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store