Three time-saving AWS command-line tricks

One of the best things about AWS, compared to other cloud service providers, are their command line tools. Because the command line tools use the same REST API as programming language SDK packages, you can make the same calls from the command line as from any other supported language. This means that absolutely all AWS API functionality works great from the command line. Everything you can do from the AWS web site, you can also achieve in the command line. This is great for ad-hoc tasks and inspecting your AWS assets.

Because the AWS command line tools follow the universal REST API, most operations also return a lot of data, typically in the JSON format. This makes them slightly difficult to chain for scripting more complex operations. Standard UNIX tools aren’t that great for processing JSON, so people often struggle to post-process command results.

However, the AWS command line tools also have a few hidden features that can save you a ton of time if you want to scripting common administrative tasks. In fact, pretty much all the post-processing you’d ever need to chain commands together is already build into the tools, just not that easy to find.

Extract just the data you really want

One quite common task is to pull out just a single piece of information you really need from the output. For example, to create an API Gateway and add resources to it, we need first to create a new gateway, get the ID, then get the automatically created root resource ID, and add another resource path to it. Creating a new API Gateway instance returns the ID we need to add resources to it, but it also returns other information we don’t really need:

$ aws apigateway create-rest-api --name test-api
{
    "name": "test-api", 
    "id": "vacw3n52ng", 
    "createdDate": 1482183599
}

You can extract just the bits you need by passing --query to any AWS command line and pass the name of the field you want. For example:

$ aws apigateway create-rest-api --name test-api --query id
"vacw3n52ng"

Convert to plain text

JSON strings are always under quotes, so the API ID printed by the previous command isn’t that easy to directly pipe into other tools. Don’t jump into sed just to delete those quotes. Use --output text, and the results will be plain text, not JSON.

$ aws apigateway create-rest-api --name test-api --query id --output text
vacw3n52ng

This is now ready for using in other commands. You can store the result directly into a shell variable:

$ API_ID=$(aws apigateway create-rest-api --name test-api --query id --output text)
$ aws apigateway get-resources --rest-api-id $API_ID --profile
{
    "items": [
        {
            "path": "/", 
            "id": "85axcuqmt6"
        }
    ]
} 

Of course, we can now use --output and --query to get just the ID of the root resource out – that’s the only piece of information we really need. Use [] to index arrays. So, really useful version of the second command would be something like this:

$ aws apigateway get-resources --rest-api-id $API_ID --query 'items[0].id' --output text
85axcuqmt6

You can also use --output text without specifying --query. This will flatten the JSON structures into tabular text, which is easy to process with standard UNIX tools.

Search through collections

The --query argument is actually a JMES Path expression, so you can also filter and search collections. JMES Path is mostly logical for anyone used to JSON, apart from strings. Use the backtick (`) to enclose strings. You can also specify a condition starting with a question mark, instead of a numerical index. For example, here’s how to find the REST API we previously created by name:

$ aws apigateway get-rest-apis --query 'items[?name==`test-api`]'
[
    {
        "createdDate": 1482184421, 
        "id": "y2un07jpyf", 
        "name": "test-api"
    }
]

You can also specify more complex conditions, such as a search by substring. For example, here’s how to find all the APIs in your account that start with the word test:

$ aws apigateway get-rest-apis --query 'items[?starts_with(name,`test`)]'
[ 
    {
        "createdDate": 1480787161, 
        "id": "pn8ifj74m4", 
        "name": "test1480787148579"
    }, 
    {
        "createdDate": 1482184421, 
        "id": "y2un07jpyf", 
        "name": "test-api"
    }
]

You can filter the results further by adding a field name. For example, here’s how to get just the IDs out:

$ aws apigateway get-rest-apis --query 'items[?starts_with(name,`test`)].id'
[ 
    "pn8ifj74m4", 
    "y2un07jpyf"
]

Finally, use --output text to convert this into a set of plain-text values that your shell can easily iterate on.

A useful example

Let’s put all that together now into a convenient function to delete AWS IAM roles. I often have to clean up IAM roles after experimenting, but AWS refuses to delete a role if it has any attached policies. So we first look for all the test roles, then remove all the policies inside them, and then finally remove the roles themselves. Here’s a nice little shell script that does all that:

roles=$(aws iam list-roles --query 'Roles[?starts_with(RoleName, `test`)].RoleName' --output text)

for role in $roles; do
  echo deleting policies for role $role
  policies=$(aws iam list-role-policies --role-name=$role --query PolicyNames --output text)
  for policy in $policies; do 
    echo deleting policy $policy for role $role
    aws iam delete-role-policy --policy-name $policy --role-name $role
  done
  echo deleting role $role
  aws iam delete-role --role-name $role
done