Logging and analysis is a huge field, where entire product stacks are built around it to make it easier. AWS’s CloudWatch service collects usage mathematics automatically, but it can also be configured to collect logs from your EC2 instances.
Why do logs aggregate in the first place?
Say you run a web server like nginx. Each time someone joins your site, a new line is created in a log file that contains details about the visit. This information can be quite useful; for example, nginx records the following data for each request:
- IP address of the connecting user
- Username, if you use basic authentication (empty most of the time)
- Time for request
- The actual request (for example “
- Status code is returned
- Sent bytes, excluding HTTP headers (useful for tracking the actual size of traffic)
- HTTP reference (that is, the website the user came from)
- User agent for the user̵7;s browser
Although an analytics suite like Google Analytics contains a lot of this information, log files are also created automatically and updated in real time. If you wanted to know how much traffic you get from a certain IP range, or what your main referral sources are, the question of your log files can return results very quickly. (Elasticsearch is good for this; AWS offers it as a managed service that works well with CloudWatch Logs.)
Now say that you have many web servers – suddenly the problem becomes a little more complicated than just searching in a single log file. Even with just two servers, you will not get exact results unless the logs are gathered in one place. This is where CloudWatch’s log flow feature comes in handy.
How to set up CloudWatch logs
To get an EC2 instance linked to CloudWatch logs, you must install the logging agent that handles sending the logs to CloudWatchFirst, and you must configure a new IAM role for the agent to act as.
This role must be linked to your instance, so from the EC2 Management Console right-click on your instance and select “Instance Settings”> “Attach / Replace IAM Role”:
When done, return to the role creation tab and select the newly created permission.
Give the role a name, and you should be good to go. Go back to the EC2 console and hit update on the roll dropdown. You should see the agent role of the logs.
Once the permissions are out of the way, you can install the agent. If you are on Amazon Linux, select the package that is available on
sudo yum install -y awslogs
If you are on Debian / Ubuntu you need to download the installer instead:
curl https://s3.amazonaws.com//aws-cloudwatch/downloads/latest/awslogs-agent-setup.py -O
Then run the installer and enter the region:
sudo python ./awslogs-agent-setup.py --region us-east-1
Here you configure which log files the log agent processes. By default, it sends
/var/log/syslog, which logs many system actions. You can add more log files here. Each log file is collected in a group (whose name is the default for the log site) and is given a timestamp.
Logs from individual instances are separated by instance IDs, but you can see a total stream for each log group consisting of all instances that send logs to that group. Once you have configured the agent, the logs start appearing in CloudWatch immediately (give or take about five seconds).
From here, you can use the search bar in the log viewer to perform simple searches and use CloudWatch’s built-in Insights tool to query your logs.
If you want more search power, or want to visualize things with Kibana, you can use AWS’s hosted Elasticsearch service, which integrates well with CloudWatch Logs.