Upload to S3 with cURL and AIR NativeProcess

This application demonstrates how to upload files to Amazon S3 (part of Amazon Web Services) with cURL from an Adobe AIR application using the AIR 2 NativeProcess API. This application uses the AIR 2.0 or higher SDK, Flash Builder 4, and the cURL native application for uploading files.

To use this application also depends on the as3corelib, as3awss3lib, and as3crypto libraries. You will also need an Amazon S3 developer account and have cURL installed on your system.

Why upload with cURL instead of AIR? Part of the problem with uploading through an AIR application, or any Flash application, is the size limit of the uploads (I think something like 100MB is the recommended size). What happens when you want to upload gigabytes of data? Offloading the uploading of large files to a native process solves this problem. You can also spawn multiple instances of the native process to do multiple uploads. This makes sure your AIR application continues to be responsive and performs well while the uploads happen in the background. Using something like cURL means you have a cross-platform native process that can be installed along with your application.

The main guts of the application revolve around assembling the correct cURL arguments to upload the files to the S3 service. This also requires properly creating an policy expected by S3 to complete the upload process. Here is the core code for creating the S3 policy file and the arguments for cURL:

/**
* Uploads a file to S3 using cURL using the AIR NativeProcess API.
*
* @param file File object
* */
protected function saveFile(folderid:String, file:File):void
{
createProgressPanel(); // add our progress bar

var cURL:File = File.applicationDirectory;

if (Capabilities.os.toLowerCase().indexOf("win") > -1) {
cURL = cURL.resolvePath("bin/curl.exe");
} else if (Capabilities.os.toLowerCase().indexOf("mac") > -1) {
cURL = cURL.resolvePath("/usr/bin/curl");
}

var contentType:String = "multipart/form-data";
var arguments:Vector. = getArguments("PUT", folderid, file, contentType);

var nativeProcessStartupInfo:NativeProcessStartupInfo = new NativeProcessStartupInfo();
nativeProcessStartupInfo.arguments = arguments;
nativeProcessStartupInfo.executable = cURL;

process = new NativeProcess();
process.addEventListener(ProgressEvent.STANDARD_INPUT_PROGRESS, onInputProgress);
process.addEventListener(ProgressEvent.STANDARD_OUTPUT_DATA, onStandardOutputData);
process.addEventListener(ProgressEvent.STANDARD_ERROR_DATA, onStandardErrorData);
process.addEventListener(NativeProcessExitEvent.EXIT, onStandardOutputExit);

process.addEventListener(IOErrorEvent.STANDARD_OUTPUT_IO_ERROR, onOutputIOError);
process.addEventListener(IOErrorEvent.STANDARD_ERROR_IO_ERROR, onStandardIOError);

process.start(nativeProcessStartupInfo);
}

/**
* Create the native process starupt info arguments. Be sure to check the cURL documentation
* for more functions on upload: http://curl.haxx.se/docs/manual.html
*
* @param method POST or GET arguements
* @param bucketname The S3 bucket name for upload target
* @param file File to upload
* @param contentType The mime type for the upload
* @param secure Boolean value for usting https or http
* @return Vector.
* */
protected function getArguments(method:String, bucketname:String, file:File, contentType:String = "multipart/form-data", secure:Boolean = false):Vector.
{
var protocol:String = (secure)? "https":"http";
var path:String = protocol + "://" + bucketname + "." + AMAZON_ENDPOINT
var policy:String = getPolicy(bucketname, contentType);

var arguments:Vector. = new Vector.();
arguments.push("-#"); // gives us a ### % ouput for progress from cURL
arguments.push("-F key=" + file.name );
arguments.push("-F AWSAccessKeyId=" + this.accessKey );
arguments.push("-F policy=" + policy );
arguments.push("-F signature=" + getSignature( policy) );
arguments.push("-F Content-Type=" + contentType);
arguments.push("-F file=@" + file.nativePath);
arguments.push(path);

return arguments;
}

/**
* Creates the policy file for the S3 upload. For more information on AWS policy files:
* http://aws.amazon.com/articles/1434. The paramater content-length-range restricts
* the file upload size. Remove it if you want to have no restrictions on upload size.
* */
protected function getPolicy(bucketname:String, contentType:String):String
{
// date has to be some time in the future so uploads don't expire in progress
var obj:Object = {"expiration": "2015-06-15T12:00:00.000Z",
"conditions": [
{"bucket": bucketname},
["starts-with", "$key", ""],
["starts-with", "$Content-Type", ""],
["content-length-range", 0, 1048576]
]
}

var json:String = JSON.encode(obj);
var encoded: String = Base64.encode(json);

return encoded;
}

/**
* Craete the signature for S3. For more information on S3 signatures:
* http://aws.amazon.com/articles/1434
* */
protected function getSignature(policy:String):String
{
var policyBytes:ByteArray = new ByteArray();
policyBytes.writeUTFBytes(policy);

var secretAccessKeyBytes:ByteArray = new ByteArray();
secretAccessKeyBytes.writeUTFBytes(this.secretAccessKey);

var hmacBytes:ByteArray = hmac.compute(secretAccessKeyBytes, policyBytes);

return Base64.encodeByteArray(hmacBytes);
}

The tricky part was getting upload feedback from cURL in a format that we could use to display the upload progress. You might notice in the code above that we add an argument to cURL to output a % for uploads in progress. We need to parse this information when it comes back in from the ProgressEvent.STANDARD_ERROR_DATA event of the NativeProcess:

/**
* Handles writing within the process such as percent complete.
* */
protected function onStandardErrorData(event:ProgressEvent):void
{
var output:String = ( process.standardError.readUTFBytes(process.standardError.bytesAvailable) );
var regex:RegExp = /([0-9\.]+)/;
var exec:String = regex.exec(output);

if(exec) {
var arry:Array = exec.split(",");
var percent:Number = Math.round(Number(arry[0])); // save percent complete

if(percent > percentComplete) {
percentComplete = percent;
progressBar.setProgress(percentComplete, progressBar.maximum);
}
}
}

The AIR application installer file and the full code for this example can be found at the GitHub repository:

AWSS3cURL

Additional Resources
Amazon Dev Article on Form Post Upload
Amazon S3 Forum discussion on cURL
cURL
Amazon S3 Manager

– Mister

3 Comments

  1. Do you have to have cURL installed on the device using the AIR app or can you embed cURL into the swf?

    1. If you have a Mac, you probably already have cURL installed. If not, then you will need to package cURL up with your AIR application and distribute/install them together on the clients machine.

Comments are closed.