Musings of a PC

Thoughts about Windows, TV and technology in general

Automated AWS EBS expansion with xMatters – part 2

In part 1, I wrote about a workflow created for xMatters that reacted to CloudWatch alarms delivered via SNS when the free space on a server was running low.

Since writing that, a bug was discovered (now fixed) that prevented the filing system associated with the volume from being resized if the volume was not stored on a NVMe device. That bug resulted in the realisation that there was a “gap” in the workflow:

  • CloudWatch alarm goes off, triggering the workflow
  • Workflow expands the volume but doesn’t resize the filing system
  • CloudWatch alarm clears due to free space increasing
  • Time passes …
  • CloudWatch alarm goes off, triggering the workflow
  • … and around we go again

In other words, there was the risk that the workflow would continue to grow the volume without a corresponding resizing of the filing system, thereby never stopping the alarm loop.

To correct that behaviour, a new step has been added to the workflow:

CheckFS step added between CheckVolume and ModifyVolume

The new step – SSM-CheckFS – takes the following actions:

  • Runs some commands (see below) on the host to determine the size of the filing system.
  • Compares that size with the size of the volume.
  • Sets an output to indicate whether or not to proceed with the workflow.

The commands run on the host are as follows:

BLOCKCOUNT=$(sudo dumpe2fs -h /dev/<device> | grep 'Block count' | awk '{print $3}')
BLOCKSIZE=$(sudo dumpe2fs -h /dev/<device> | grep 'Block size' | awk '{print $3}')
echo "$(($BLOCKCOUNT * $BLOCKSIZE))"

By using dumpe2fs, we can determine the block count and size for the underlying filing system. That is then returned to the calling script.

The sources for each of the steps in the workflow can be obtained from https://github.com/linaro-its/xmatters-ebs-automation.

The entire workflow can be obtained from https://github.com/linaro-its/xm-labs-aws-ebs-resize.

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: