Get started with AWS Unified CloudWatch agent

Amazon Web Services (AWS) offers a monitoring service called Amazon CloudWatch. To simplify the collection of custom metrics and logs from EC2 instances and on-premises servers, AWS introduced the Unified CloudWatch Agent.

Before the Unified CloudWatch Agent, there were separate agents for collecting system-level metrics and logs – namely the CloudWatch Logs agent and CloudWatch Monitoring scripts. The Unified CloudWatch Agent combines the capabilities of both into a single agent, simplifying setup and maintenance.

The unified CloudWatch agent enables you to do the following:

  • Collect internal system-level metrics from Amazon EC2 instances across operating systems. The metrics can include in-guest metrics, in addition to the metrics for EC2 instances. The additional metrics that can be collected are listed in Metrics collected by the CloudWatch agent.

  • Collect system-level metrics from on-premises servers. These can include servers in a hybrid environment as well as servers not managed by AWS.

  • Retrieve custom metrics from your applications or services using the StatsD and collectd protocols. StatsD is supported on both Linux servers and servers running Windows Server. collectd is supported only on Linux servers.

  • Collect logs from Amazon EC2 instances and on-premises servers, running either Linux or Windows Server.

  • Versions 1.300025.0 and later can collect traces from the X-Ray Open Telemetry auto-instrumention SDKs and send them to the X-Ray backend.

    Using the CloudWatch agent instead of the X-Ray daemon to collect traces can help you reduce the number of agents that you manage.

You can store and view the metrics that you collect with the CloudWatch agent in CloudWatch just as you can with any other CloudWatch metrics. The default namespace for metrics collected by the CloudWatch agent is CWAgent, although you can specify a different namespace when you configure the agent.

The logs collected by the unified CloudWatch agent are processed and stored in Amazon CloudWatch Logs, just like logs collected by the older CloudWatch Logs agent. For information about CloudWatch Logs pricing, see Amazon CloudWatch Pricing.

Metrics collected by the CloudWatch agent are billed as custom metrics. For more information about CloudWatch metrics pricing, see Amazon CloudWatch Pricing.

The CloudWatch agent is open-source under the MIT license, and is hosted on GitHub

Agent installation process overview

You can download and install the CloudWatch agent manually using the command line, or you can integrate it with SSM. The general flow of installing the CloudWatch agent using either method is as follows:

  1. Create IAM roles or users that enable the agent to collect metrics from the server and optionally to integrate with AWS Systems Manager.

  2. Download the agent package.

  3. Modify the CloudWatch agent configuration file and specify the metrics that you want to collect.

  4. Install and start the agent on your servers. As you install the agent on an EC2 instance, you attach the IAM role that you created in step 1. As you install the agent on an on-premises server, you specify a named profile that contains the credentials of the IAM user that you created in step 1.

Create IAM roles to use with the CloudWatch agent on Amazon EC2 instances

The first procedure creates the IAM role that you must attach to each Amazon EC2 instance that runs the CloudWatch agent. This role provides permissions for reading information from the instance and writing it to CloudWatch.

The second procedure creates the IAM role that you must attach to the Amazon EC2 instance being used to create the CloudWatch agent configuration file. This step is necessary if you're going to store this file in Systems Manager Parameter Store so that other servers can use it. This role provides permissions for writing to Parameter Store, in addition to the permissions for reading information from the instance and writing it to CloudWatch. This role includes permissions sufficient to run the CloudWatch agent as well as to write to Parameter Store.

To create the IAM role necessary for each server to run the CloudWatch agent

  1. Sign in to the AWS Management Console and open the IAM console at https://console.aws.amazon.com/iam/.

  2. In the navigation pane, choose Roles and then choose Create role.

  3. Under Select type of trusted entity, choose AWS service.

  4. Immediately under Common use cases, choose EC2, and then choose Next: Permissions.

  5. In the list of policies, select the check box next to CloudWatchAgentServerPolicy. If necessary, use the search box to find the policy.

  6. To use Systems Manager to install or configure the CloudWatch agent, select the box next to AmazonSSMManagedInstanceCore. This AWS managed policy enables an instance to use Systems Manager service core functionality. If necessary, use the search box to find the policy. This policy isn't necessary if you start and configure the agent only through the command line.

  7. Choose Next: Tags.

  8. (Optional) Add one or more tag-key value pairs to organize, track, or control access for this role, and then choose Next: Review.

  9. For Role name, enter a name for your new role, such as CloudWatchAgentServerRole or another name that you prefer.

  10. (Optional) For Role description, enter a description.

  11. Confirm that CloudWatchAgentServerPolicy and optionally AmazonSSMManagedInstanceCore appear next to Policies.

  12. Choose Create role.

    The role CloudWatchAgentServerRole is now created.

To create the IAM role for an administrator to write to Parameter Store

The following procedure creates the IAM role that can also write to Parameter Store. You can use this role to store the agent configuration file in Parameter Store so that other servers can retrieve it.

💡
The permissions for writing to Parameter Store provide broad access. This role shouldn't be attached to all your servers, and only administrators should use it. After you create the agent configuration file and copy it to Parameter Store, you should detach this role from the instance and use CloudWatchAgentServerRole instead.
  1. Sign in to the AWS Management Console and open the IAM console at https://console.aws.amazon.com/iam/.

  2. In the navigation pane, choose Roles and then choose Create role.

  3. Under Select type of trusted entity, choose AWS service.

  4. Immediately under Choose the service that will use this role, choose EC2, and then choose Next: Permissions.

  5. In the list of policies, select the check box next to CloudWatchAgentAdminPolicy. If necessary, use the search box to find the policy.

  6. To use Systems Manager to install or configure the CloudWatch agent, select the box next to AmazonSSMManagedInstanceCore. This AWS managed policy enables an instance to use Systems Manager service core functionality. If necessary, use the search box to find the policy. This policy isn't necessary if you start and configure the agent only through the command line.

  7. Choose Next: Tags.

  8. (Optional) Add one or more tag-key value pairs to organize, track, or control access for this role, and then choose Next: Review.

  9. For Role name, enter a name for your new role, such as CloudWatchAgentAdminRole or another name that you prefer.

  10. (Optional) For Role description, enter a description.

  11. Confirm that CloudWatchAgentAdminPolicy and optionally AmazonSSMManagedInstanceCore appear next to Policies.

  12. Choose Create role.

    The role CloudWatchAgentAdminRole is now created.

Bash script to generate logs

💡
To launch EC2 use the following guide: How to launch a single EC2 instance via AWS CLI

Here's a bash script that generates a log message every 60 seconds and stores it in the /var/log/ directory with the filename simple_log.log:

#!/bin/bash

LOG_FILE="/var/log/simple_log.log"

# Ensure the user has sufficient privileges to write to /var/
if [[ $EUID -ne 0 ]]; then
   echo "This script must be run as root (or with sudo)" 1>&2
   exit 1
fi

# Create or truncate the log file
> $LOG_FILE

while true; do
    echo "Log entry at $(date)" >> $LOG_FILE
    sleep 60
done

Steps to use the script:

  1. Save the above code to a file, say generate_logs.sh.

  2. Give the script execute permissions: chmod +x generate_logs.sh

  3. Run the script with root privileges (since we're writing to the /var/log/ directory): sudo ./generate_logs.sh

This script will then generate a log entry in /var/log/simple_log.log every 60 seconds.

To stop it, you can simply press Ctrl+C if you're running it in the foreground or find its process ID with ps and use kill if it's running in the background.

Installing the CloudWatch agent

The CloudWatch agent is available as a package in Amazon Linux 2. If you are using this operating system, you can install the package by entering the following command. You must also make sure that the IAM role attached to the instance has the CloudWatchAgentServerPolicy attached.

For more information, see Create IAM roles to use with the CloudWatch agent on Amazon EC2 instances.

sudo yum install amazon-cloudwatch-agent

On all supported operating systems including Linux and Windows Server, you can download and install the CloudWatch agent using either the command line with an Amazon S3 download link, using Amazon EC2 Systems Manager, or using an AWS CloudFormation template.

Create the CloudWatch agent configuration file

Before running the CloudWatch agent on any servers, you must create a CloudWatch agent configuration file.

The agent configuration file is a JSON file that specifies the metrics, logs, and traces that the agent is to collect, including custom metrics. You can create it by using the wizard or by creating it yourself from scratch. You could also use the wizard to initially create the configuration file and then modify it manually. If you create or modify the file manually, the process is more complex, but you have more control over the metrics collected and can specify metrics not available through the wizard.

Any time you change the agent configuration file, you must then restart the agent to have the changes take effect. To restart the agent, follow the instructions in Start the CloudWatch agent.

After you have created a configuration file, you can save it manually as a JSON file and then use this file when installing the agent on your servers. Alternatively, you can store it in Systems Manager Parameter Store if you're going to use Systems Manager when you install the agent on servers.

Create the CloudWatch agent configuration file with the wizard

The agent configuration file wizard, amazon-cloudwatch-agent-config-wizard, asks a series of questions, including the following:

  • Are you installing the agent on an Amazon EC2 instance or an on-premises server?

  • Is the server running Linux or Windows Server?

  • Do you want the agent to also send log files to CloudWatch Logs? If so, do you have an existing CloudWatch Logs agent configuration file? If yes, the CloudWatch agent can use this file to determine the logs to collect from the server.

  • If the agent will send log files to CloudWatch Logs, what retention period do you want for those log files? The default value of -1 sets the log items to never expire.

  • If you're going to collect metrics from the server, do you want to monitor one of the default sets of metrics or customize the list of metrics that you collect?

  • Do you want to collect custom metrics from your applications or services, using StatsD or collectd?

  • Are you migrating from an existing SSM Agent?

The wizard can autodetect the credentials and AWS Region to use if you have the AWS credentials and configuration files in place before you start the wizard. For more information about these files, see Configuration and Credential Files in the AWS Systems Manager User Guide.

In the AWS credentials file, the wizard checks for default credentials and also looks for an AmazonCloudWatchAgent section such as the following:

[AmazonCloudWatchAgent]
aws_access_key_id = my_access_key
aws_secret_access_key = my_secret_key

The wizard displays the default credentials, the credentials from the AmazonCloudWatchAgent, and an Others option. You can select which credentials to use. If you choose Others, you can input credentials.

For my_access_key and my_secret_key, use the keys from the IAM user that has the permissions to write to Systems Manager Parameter Store. For more information about the IAM users needed for the CloudWatch agent, see Create IAM users to use with the CloudWatch agent on on-premises servers.

In the AWS configuration file, you can specify the Region that the agent sends metrics to if it's different than the [default] section. The default is to publish the metrics to the Region where the Amazon EC2 instance is located. If the metrics should be published to a different Region, specify the Region here. In the following example, the metrics are published to the us-west-1 Region.

[AmazonCloudWatchAgent]
region = us-west-1

Run the CloudWatch agent configuration wizard

To create the CloudWatch agent configuration file
  1. Start the CloudWatch agent configuration wizard by entering the following:

     sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard
    
  2. Answer the questions to customize the configuration file for your server.

  • Current configuration file for the agent config.json

{
        "agent": {
                "metrics_collection_interval": 60,
                "run_as_user": "root"
        },
        "metrics": {
                "aggregation_dimensions": [
                        [
                                "InstanceId"
                        ]
                ],
                "append_dimensions": {
                        "AutoScalingGroupName": "${aws:AutoScalingGroupName}",
                        "ImageId": "${aws:ImageId}",
                        "InstanceId": "${aws:InstanceId}",
                        "InstanceType": "${aws:InstanceType}"
                },
                "metrics_collected": {
                        "disk": {
                                "measurement": [
                                        "used_percent"
                                ],
                                "metrics_collection_interval": 60,
                                "resources": [
                                        "*"
                                ]
                        },
                        "mem": {
                                "measurement": [
                                        "mem_used_percent"
                                ],
                                "metrics_collection_interval": 60
                        }
                }
        }
}

💡
If you want to store the agent configuration file in the SSM parameter store, you have first to deattach CloudWatchAgentServerRole then attach CloudWatchAgentAdminRole. After configuration file will be pushed to SSM you can reattach CloudWatchAgentServerRole to EC2 instance

If you're storing the configuration file locally, the configuration file config.json is stored in /opt/aws/amazon-cloudwatch-agent/bin/ on Linux servers

💡
Don't forget to configure access and secret keys on your EC2 instance with aws configure command

To activate the configuration file for the CloudWatch agent run the following command:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/opt/aws/ama
zon-cloudwatch-agent/bin/config.json -s

Verify logs in CloudWatch console

To check if logs are pushed by unified agent to the CloudWatch console, first run the Bash script to generate log files in simple_log.log file:

sudo ./generate_logs.sh

To check the logs in CloudWatch console, go to Log Groups:

From Log Streams we can see the actual log file events:

References

  1. Use the unified CloudWatch agent to get started with CloudWatch Logs

  2. Create the CloudWatch agent configuration file with the wizard

  3. Configuration and credential file settings

  4. Installing the CloudWatch agent

  5. https://github.com/aws/amazon-cloudwatch-agent/

  6. Create IAM roles and users for use with the CloudWatch agent

  7. How do I install and configure the unified CloudWatch agent to push metrics and logs from my EC2 instance to CloudWatch?