CloudTrail Alarm Insights
When you’re managing an AWS environment, it is good to know when your infrastructure configuration changes. Even in a highly dynamic AWS environment there are some things that we expect to be fairly static. For example, if a VPC network gateway configuration changes and you didn’t expect that change then it could be a sign that your account has been compromised.
Typical things that we expect to be pretty stable over time include:
- S3 Bucket Changes - changes to bucket policies, lifecycle rules, or replication configuration
- Security Groups - changes to any security group rules or creation or deletion of security groups
- Network ACLs - creation, deletion or changes to network ACLs.
- Network Gateways - changes to customer gateways or internet gateways
- EC2 instances - starting, stopping, rebooting or terminating instances
- CloudTrail - creation, updates or deletion of a trail or changes to the logging destination
- Console Signin - failures of users to sign in
- Authorization Failures - AWS API access failures
- Root Account - usage of the root account
- Non-MFA Access - usage of accounts without MFA enabled or in use
Of course, every AWS account environment is different and in some of the above configuration may be very volatile in your account. For example if you’re using any kind of autoscaling, then you’ll see EC2 instances starting and terminating all the time. But for most AWS admins, at least some of the above events should be fairly rare and it would be good to know when they occur, so we can investigate.
Assuming you’re logging your CloudTrail to CloudWatch (which I strongly encourage), AWS helpfully supplies example CloudWatch metrics which you can use to raise alarms when one of the above events occurs. They even provide a CloudFormation template that configures these metrics and alarms for you. There is a link to set this template up from the CloudTrail page in the AWS Console.
However, one of the things that bugs me about CloudWatch metrics for this particular purpose is that the alert emails are very terse and they provide no context for what raised the alarm. If you get more than the occasional alert, tracing the event back to the logs by digging through the CloudWatch logs to find the matching log lines is pretty painful.
For example, here is what an typical alarm looks like in the AWS Console:
Here is what the matching email notification looks like:
Now these might be useful to you if you’re monitoring a metric like CPU utilisation, but I’d argue they’re basically useless for tracking configuration change events.
Whilst there are a range of commercial SIEM tools available which you could forward these logs to, if you want a lightweight solution which enables you to investigate these events directly from within your AWS console, you can leverage CloudWatch Insights to extract the relevant events to summarise them for you.
At the bottom of this post is the JSON code for a CloudWatch Dashboard that uses CloudWatch Insights to do similar queries to the above CloudWatch Metrics/Alarms. Once you’ve set this up, you can quickly drill down into the events and see exactly what’s caused the alarm.
To set this up in your own account just import the below JSON by going into CloudWatch > Dashboards, and creating a new Dashboard (e.g. ‘CloudTrail_API_Activity’) then click on Actions > View/edit source and paste in the below.
The below assumes that your CloudTrail log group is called “CloudTrail/DefaultLogGroup”. You’ll need to search and replace if its called something different.
{
"start": "-P3D",
"widgets": [
{
"type": "log",
"x": 0,
"y": 0,
"width": 24,
"height": 6,
"properties": {
"query": "SOURCE 'CloudTrail/DefaultLogGroup' | filter eventName == 'AuthorizeSecurityGroupIngress' or eventName == 'AuthorizeSecurityGroupEgress' or eventName == 'RevokeSecurityGroupIngress' or eventName == 'RevokeSecurityGroupEgress' or eventName == 'CreateSecurityGroup' or eventName == 'DeleteSecurityGroup'",
"region": "ap-southeast-2",
"title": "SecurityGroupChanges",
"view": "table"
}
},
{
"type": "log",
"x": 0,
"y": 6,
"width": 24,
"height": 6,
"properties": {
"query": "SOURCE 'CloudTrail/DefaultLogGroup' | filter eventName == 'CreateNetworkAcl' or eventName == 'CreateNetworkAclEntry' or eventName == 'DeleteNetworkAcl' or eventName == 'DeleteNetworkAclEntry' or eventName == 'ReplaceNetworkAclEntry' or eventName == 'ReplaceNetworkAclAssociation'",
"region": "ap-southeast-2",
"stacked": false,
"view": "table",
"title": "NetworkAclChanges"
}
},
{
"type": "log",
"x": 0,
"y": 12,
"width": 24,
"height": 6,
"properties": {
"query": "SOURCE 'CloudTrail/DefaultLogGroup' | filter eventName == 'CreateCustomerGateway' or eventName == 'DeleteCustomerGateway' or eventName == 'AttachInternetGateway' or eventName == 'CreateInternetGateway' or eventName == 'DeleteInternetGateway' or eventName == 'DetachInternetGateway'",
"region": "ap-southeast-2",
"stacked": false,
"view": "table",
"title": "GatewayChanges"
}
},
{
"type": "log",
"x": 0,
"y": 18,
"width": 24,
"height": 6,
"properties": {
"query": "SOURCE 'CloudTrail/DefaultLogGroup' | filter eventName like /^(CreateVpc|DeleteVpc|ModifyVpcAttribute|AcceptVpcPeeringConnection|CreateVpcPeeringConnection|DeleteVpcPeeringConnection|RejectVpcPeeringConnection|AttachClassicLinkVpc|DetachClassicLinkVpc|DisableVpcClassicLink|EnableVpcClassicLink)$/",
"region": "ap-southeast-2",
"stacked": false,
"view": "table",
"title": "VpcChanges"
}
},
{
"type": "log",
"x": 0,
"y": 24,
"width": 24,
"height": 6,
"properties": {
"query": "SOURCE 'CloudTrail/DefaultLogGroup' | filter eventName like /^(RunInstances|RebootInstances|StartInstances|StopInstances|TerminateInstances)$/",
"region": "ap-southeast-2",
"stacked": false,
"view": "table",
"title": "EC2InstanceChanges"
}
},
{
"type": "log",
"x": 0,
"y": 30,
"width": 24,
"height": 6,
"properties": {
"query": "SOURCE 'CloudTrail/DefaultLogGroup' | filter eventName like /^(CreateTrail|UpdateTrail|DeleteTrail|StartLogging|StopLogging)$/\n| fields @timestamp, eventName, userIdentity.userName, errorMessage, @message",
"region": "ap-southeast-2",
"stacked": false,
"view": "table",
"title": "CloudTrailChanges"
}
},
{
"type": "log",
"x": 0,
"y": 36,
"width": 24,
"height": 6,
"properties": {
"query": "SOURCE 'CloudTrail/DefaultLogGroup' | filter eventName == 'ConsoleLogin' and errorMessage == 'Failed authentication'",
"region": "ap-southeast-2",
"stacked": false,
"view": "table",
"title": "ConsoleSignInFailures"
}
},
{
"type": "log",
"x": 0,
"y": 42,
"width": 24,
"height": 6,
"properties": {
"query": "SOURCE 'CloudTrail/DefaultLogGroup' | filter errorCode like /UnauthorizedOperation$/ or errorCode like /^AccessDenied/",
"region": "ap-southeast-2",
"stacked": false,
"view": "table",
"title": "AuthorizationFailures"
}
},
{
"type": "log",
"x": 0,
"y": 48,
"width": 24,
"height": 6,
"properties": {
"query": "SOURCE 'CloudTrail/DefaultLogGroup' | filter eventName like /^(DeleteGroupPolicy|DeleteRolePolicy|DeleteUserPolicy|PutGroupPolicy|PutRolePolicy|PutUserPolicy|CreatePolicy|DeletePolicy|CreatePolicyVersion|DeletePolicyVersion|AttachRolePolicy|DetachRolePolicy|AttachUserPolicy|DetachUserPolicy|AttachGroupPolicy|DetachGroupPolicy)$/\n| fields @timestamp, eventName, userIdentity.userName, requestParameters.policyName, @message",
"region": "ap-southeast-2",
"stacked": false,
"view": "table",
"title": "IAMPolicyChanges"
}
}
]
}