Musings of a PC

Thoughts about Windows, TV and technology in general

Calling AWS APIs from xMatters

In Automated AWS EBS expansion with xMatters and Automated AWS EBS expansion with xMatters – part 2, I discussed a complete workflow in xMatters that reacts to a disc low free space alarm sent from AWS CloudWatch via SNS and then takes the appropriate steps to resize the affected volume, both in EBS and the (Linux) operating system.

xMatters is a really powerful and flexible platform, but it does have some limitations and restrictions. One of the big challenges to solve when creating the workflow was that the AWS SDK could not be used. It isn’t provided as part of the xMatters platform and it cannot be installed as a library for use by the Javascript step code.

The job of the SDK, ultimately, is to make it easy to use the underlying AWS APIs which, themselves, are accessed via REST calls. Amazon document the APIs, their endpoints and their payloads, so it should be quite straightforward to write some Javascript to call the APIs directly.

Right?

Well, not quite.

The biggest challenge is correctly signing the request so that it can then be processed by AWS. Again, Amazon has documented this process, including a step-by-step process. Some Internet searches later and I’ve even found that someone has already written the steps in Javascript. Unfortunately, there is another gotcha.

The Crypto library used uses setTimeout and this isn’t available on the xMatters platform. So … back to Internet searches … and ultimately I find this: jsSHA – SHA Hashes and HMAC in JavaScript (coursesweb.net). An implementation that has no external requirements. Yay!

There are still some gotchas around calling the APIs, both in the Javascript and when building a workflow in xMatters so let’s dig into the code a bit deeper to better understand what is required.

const jsSHA = require('jsSHA');

const hmacSha256 = (signingKey, stringToSign, type="HEX") => {
    var sha_ob = new jsSHA("SHA-256", "TEXT");
    sha_ob.setHMACKey(signingKey, type);
    sha_ob.update(stringToSign);
    return sha_ob.getHMAC("HEX");
};

function getSignatureKey(key, dateStamp, regionName, serviceName) {
    var kDate = hmacSha256(AWS4${key}, dateStamp, "TEXT");
    var kRegion = hmacSha256(kDate, regionName);
    var kService = hmacSha256(kRegion, serviceName);
    var kSigning = hmacSha256(kService, "aws4_request");
    return kSigning;
}

By adding jsSHA as a library to a xMatters workflow, the above code implements the steps required to create a signature for a given AWS secret key, date stamp, region and service name.

function prependLeadingZeroes(n) {
     if (n <= 9) {
         return "0" + n;
     }
     return n.toString();
}

function hashSha256(stringToHash) {
     var sha_ob = new jsSHA('SHA-256', "TEXT");
     sha_ob.update(stringToHash);
     return sha_ob.getHash("HEX");
}

function buildHeader(access_key, secret_key, region, request_parameters) {
     const method = "GET";
     const service = "ec2";
     const host = service+"."+region+".amazonaws.com";
     const t = new Date();
     const datestamp = ${t.getFullYear()}${prependLeadingZeroes(t.getMonth()+1)}${prependLeadingZeroes(t.getDate())};
     // 4-digit year, 2-digit month, 2-digit date, T, 2-digit hour, 2-digit minutes, 2-digit seconds, Z
     const amzdate = datestamp+"T"+prependLeadingZeroes(t.getHours())+prependLeadingZeroes(t.getMinutes())+prependLeadingZeroes(t.getSeconds())+"Z";
     const canonical_uri = "/";
     const canonical_querystring = request_parameters;
     const canonical_headers = `host:${host}\nx-amz-date:${amzdate}\n`;
     const signed_headers = 'host;x-amz-date';
     // Calculate the hash of the payload which, for GET, is empty
     const payload_hash = hashSha256("");
     const canonical_request = method + '\n' + canonical_uri + '\n' + canonical_querystring + '\n' + canonical_headers + '\n' + signed_headers + '\n' + payload_hash;
     const algorithm = 'AWS4-HMAC-SHA256';
     const credential_scope = datestamp + '/' + region + '/' + service + '/' + 'aws4_request';
     const string_to_sign = algorithm + '\n' +  amzdate + '\n' +  credential_scope + '\n' +  hashSha256(canonical_request);
     const signing_key = getSignatureKey(secret_key, datestamp, region, service);
     const signature = hmacSha256(signing_key, string_to_sign).toString('hex');
     const authorization_header = algorithm + ' ' + 'Credential=' + access_key + '/' + credential_scope + ', ' +  'SignedHeaders=' + signed_headers + ', ' + 'Signature=' + signature;
     var requestHeaders = {
         "host": host,
         "x-amz-date": amzdate,
         "Content-type": "application/json",
         "Authorization": authorization_header };
     return [
         requestHeaders,
         host
     ];
 }

This code should be fairly self-explanatory but, in summary, it takes the access key & secret key, region and request parameters and returns the appropriate request headers and the required host (endpoint) for the request.

AWS has different endpoints not just for each service but also for each region. So, for example, making an EC2 call for us-east-1 requires using ec2.us-east-1.amazonaws.com, while making a SSM call for eu-west-2 requires using ssm.eu-west-2.amazonaws.com. xMatters doesn’t allow scripts to dynamically reference endpoints. Instead the endpoints must be separate configured as part of the workflow and the script then dynamically changes which xMatters endpoint is referenced:

function executeEc2Action(access_key, secret_key, region, request_parameters) {
     const blob = buildHeader(access_key, secret_key, region, request_parameters);
     var requestHeaders = blob[0];
     const host = blob[1];
     if (input.AWSSessionToken) {
         requestHeaders["X-Amz-Security-Token"] = input.AWSSessionToken;
     }
     var ec2Request = http.request({
         endpoint: host,
         path: "/?"+request_parameters,
         method: 'GET',
         headers: requestHeaders
     });
     return ec2Request.write();
 }

So, we’re almost there. We can now execute an EC2 call like this:

var ec2Response = executeEc2Action(access_key, secret_key, region, "Action=ModifyVolume&Size="+volume_size+"&Version=2016-11-15&VolumeId="+volume_id);

A really important point to make here: the different keys used in request_parameters function parameter must, repeat MUST, be in alphabetical order. In other words: Action, Size, Version, VolumeId. If they are not in alphabetical order, the call will fail with “AuthFailure – AWS was not able to validate the provided access credentials”.

In trying to troubleshoot that particular problem, I came across AWS Signature v4 Calculator (com.s3-website-us-west-2.amazonaws.com) which shows the result of the signature calculations at each step, thus making it easier to pinpoint where the code might be wrong. If you find yourself debugging/troubleshooting in this area, just remember to keep the date & time the same in both the website and the code as the signature calculations do rely on them so the slightly variance will give different results.

So, we now have the ability to call any AWS API so long as we present the parameters correctly. The final piece of the puzzle is decoding what comes back from AWS. If you’ve ever used boto3, you’ll know that it returns JSON. Curiously, the AWS APIs do not … they return XML! I’m not strong at parsing XML paths but, thankfully, xMatters includes a number of libraries for XML manipulation, including JXON, a library to convert XML to JSON.

var json_response = JXON.parse(ec2Response.body);

xMatters doesn’t allow one library to reference another library, unfortunately, which means that all of the AWS code needs to be duplicated in each script. Apart from that, though, it should now be quite straightforward to call any AWS API from within a xMatters workflow.

All of the scripts written for the resize workflow can be found at https://github.com/linaro-its/xmatters-ebs-automation

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: