قالب وردپرس درنا توس
Home / Tips and Tricks / How to set up CloudWatch’s log aggregation service – CloudSavvy IT

How to set up CloudWatch’s log aggregation service – CloudSavvy IT



AWS logo

Logging and analysis is a huge field, where entire product stacks are built around it to make it easier. AWS’s CloudWatch service collects usage mathematics automatically, but it can also be configured to collect logs from your EC2 instances.

Why do logs aggregate in the first place?

Say you run a web server like nginx. Each time someone joins your site, a new line is created in a log file that contains details about the visit. This information can be quite useful; for example, nginx records the following data for each request:

  • IP address of the connecting user
  • Username, if you use basic authentication (empty most of the time)
  • Time for request
  • The actual request (for example “GET /index.php?url=abc“)
  • Status code is returned
  • Sent bytes, excluding HTTP headers (useful for tracking the actual size of traffic)
  • HTTP reference (that is, the website the user came from)
  • User agent for the user̵
    7;s browser

Although an analytics suite like Google Analytics contains a lot of this information, log files are also created automatically and updated in real time. If you wanted to know how much traffic you get from a certain IP range, or what your main referral sources are, the question of your log files can return results very quickly. (Elasticsearch is good for this; AWS offers it as a managed service that works well with CloudWatch Logs.)

Now say that you have many web servers – suddenly the problem becomes a little more complicated than just searching in a single log file. Even with just two servers, you will not get exact results unless the logs are gathered in one place. This is where CloudWatch’s log flow feature comes in handy.

How to set up CloudWatch logs

To get an EC2 instance linked to CloudWatch logs, you must install the logging agent that handles sending the logs to CloudWatchFirst, and you must configure a new IAM role for the agent to act as.

This role must be linked to your instance, so from the EC2 Management Console right-click on your instance and select “Instance Settings”> “Attach / Replace IAM Role”:

instance settings> attach / replace IAM role” width=”700″ height=”279″ onload=”pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);” onerror=”this.onerror=null;pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon(this);”/></p>
<p>If this is the first time you are doing this, choose to create a new role on the IAM console.  Create a new role and then select “EC2” as the service that uses the role.</p>
<p>Then add permissions to the role.  Create a new permission and paste it into the following JSON:</p>
<pre>{

When done, return to the role creation tab and select the newly created permission.

The role creates the tab where a newly created permission can be deleted.

Give the role a name, and you should be good to go. Go back to the EC2 console and hit update on the roll dropdown. You should see the agent role of the logs.

Once the permissions are out of the way, you can install the agent. If you are on Amazon Linux, select the package that is available on yum:

sudo yum install -y awslogs

If you are on Debian / Ubuntu you need to download the installer instead:

curl https://s3.amazonaws.com//aws-cloudwatch/downloads/latest/awslogs-agent-setup.py -O

Then run the installer and enter the region:

sudo python ./awslogs-agent-setup.py --region us-east-1

Here you configure which log files the log agent processes. By default, it sends /var/log/syslog, which logs many system actions. You can add more log files here. Each log file is collected in a group (whose name is the default for the log site) and is given a timestamp.

logman translation

Logs from individual instances are separated by instance IDs, but you can see a total stream for each log group consisting of all instances that send logs to that group. Once you have configured the agent, the logs start appearing in CloudWatch immediately (give or take about five seconds).

cloud log viewer

From here, you can use the search bar in the log viewer to perform simple searches and use CloudWatch’s built-in Insights tool to query your logs.

cloud view

If you want more search power, or want to visualize things with Kibana, you can use AWS’s hosted Elasticsearch service, which integrates well with CloudWatch Logs.


Source link