قالب وردپرس درنا توس
Home / Tips and Tricks / How to set up an Elasticsearch analysis and monitoring panel for your business – CloudSavvy IT

How to set up an Elasticsearch analysis and monitoring panel for your business – CloudSavvy IT



Analysis window.

Analytics is important for any business that handles a lot of data. Elasticsearch is a log and index management tool that can be used to monitor the health of your server distributions and to retrieve useful insights from customer access logs.

Why is data collection useful?

Data is big business – most of the internet is free to access because companies make money from data collected from users, which is often used by marketing companies to tailor more targeted ads.

But even if you do not collect and sell user data for profit, data of some kind can be used to make valuable business insights. For example, if you run a website, it is useful to log traffic information so that you get an idea of ​​who is using your service and where they come from.

If you have many servers, you can log system metrics such as CPU and memory usage over time, which can be used to identify bottlenecks in your infrastructure and better provide your future resources.

You can log all types of data, not just traffic or system information. If you have a complicated application, it can be useful to log on to keystrokes and clicks and which elements your users interact with, so that you can get a sense of how users use your app. You can then use that information to design a better experience for them.

Ultimately, it̵

7;s up to you what you decide to log in based on your specific business needs, but no matter what your sector is, you can benefit from understanding the data you produce.

What is Elasticsearch?

Elasticsearch is a search and analysis engine. In short, it stores data with timestamps and keeps track of the indexes and key keywords to make it easy to search through that information. It’s the heart of Elastic stack, an important tool for running DIY analytics settings. Even very large companies run huge Elasticsearch clusters to analyze terabyte data.

Although you can also use ready-made analytics suites such as Google Analytics, Elasticsearch gives you the flexibility to design your own dashboards and visualizations based on all types of data. The schedule is agnostic; you just send a few logs to store it and it is indexed for search.

Kibana is a visualization panel for Elasticsearch and also serves as a general web-based GUI to manage your instance. It is used to create dashboards and graphs of data, something you can use to understand the often millions of log entries.

Kibana is a visualization panel for Elasticsearch.

You can log in to Elasticsearch via two main methods – logging in file-based logs or logging directly through the API or SDK. To make the former easier, Elastic Beats offers lightweight data transmitters that you can install on your server to send data to Elasticsearch. If you need additional processing, there is also Logstash, a data collection and transformation pipeline for modifying logs before sending them to Elasticsearch.

A good start would be to retrieve your existing logs, such as an NGINX web server’s access logs or file logs created by your application, with a log provider on the server. If you want to customize data taken, you can also log JSON documents directly into the Elasticsearch API. We discuss how to set both below.

If you instead mainly run a generic website, you may also want to look at Google Analytics, a free analytics package tailored for website owners. You can read our website analysis tool guide to learn more.

RELATED: Do you need analysis for your website? Here are four tools you can use

Installing Elasticsearch

The first step is to run Elasticsearch on your server. We show steps for Debian-based Linux distributions like Ubuntu, but if you do not have it apt-get, you can follow Elastic instructions for your operating system.

To get started, you need to add the elastic repository to yours apt-get installation and install some prerequisites:

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
sudo apt-get install apt-transport-https
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-7.x.list

And finally, install Elasticsearch yourself:

sudo apt-get update && sudo apt-get install elasticsearch

By default, Elasticsearch runs on port 9200 and is insecure. If you do not set up additional user authentication and authorization, you will want to keep this port closed on the server.

Whatever you do, you want to make sure that it is not just open to the internet. This is actually a common problem with Elasticsearch; because it has no security features by default and if port 9200 or Kibana web panel is open to the entire internet, anyone can read your logs. Microsoft made this mistake with Bing’s Elasticsearch server and exposed 6.5 TB web search logs.

The easiest way to secure Elasticsearch is to keep the 9200 closed and set up basic authentication for the Kibana web panel using an NGINX proxy, as we show below. For simple distributions, it works well. However, if you need to manage multiple users and set permission levels for each of them, you will want to explore how to configure user authentication and user permissions.

Set up and secure Kibana

Kibana is a visualization panel:

sudo apt-get update && sudo apt-get install kibana

You want to activate the service so that it starts at startup:

sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable kibana.service

No additional installation is required. Kibana will now run on port 5601. If you want to change this, you can edit /etc/kibana/kibana.yml.

You should definitely keep this port closed to the public, as there is no authentication by default. However, you can whitelist your IP address to access it:

sudo ufw allow from x.x.x.x to any port 5601

A better solution is to create an NGINX reverse proxy. You can secure this with basic authentication so that anyone trying to access it must enter a password. This keeps it open from the internet without whitelisted IP addresses, but protects it from random hackers.

Even if you have NGINX installed, you will need to install apache2-utils, and create a password file with htpasswd:

sudo apt-get install apache2-utils
sudo htpasswd -c /etc/nginx/.htpasswd admin

Then you can create a new configuration file for Kibana:

sudo nano /etc/nginx/sites-enabled/kibana

And paste in the following configuration:

  upstream elasticsearch {
    server 127.0.0.1:9200;
    keepalive 15;
  }

  upstream kibana {
    server 127.0.0.1:5601;
    keepalive 15;
  }

  server {
    listen 9201;
    server_name elastic.example.com;

    location / {
      auth_basic "Restricted Access";
      auth_basic_user_file /etc/nginx/.htpasswd;


      proxy_pass http://elasticsearch;
      proxy_redirect off;
      proxy_buffering off;

      proxy_http_version 1.1;
      proxy_set_header Connection "Keep-Alive";
      proxy_set_header Proxy-Connection "Keep-Alive";
    }

  }

  server {
    listen 80;
    server_name elastic.example.com;

    location / {
      auth_basic "Restricted Access";
      auth_basic_user_file /etc/nginx/.htpasswd;

      proxy_pass http://kibana;
      proxy_redirect off;
      proxy_buffering off;

      proxy_http_version 1.1;
      proxy_set_header Connection "Keep-Alive";
      proxy_set_header Proxy-Connection "Keep-Alive";
    }
  }

This configuration sets Kibana to listen to port 80 using the password file you generated earlier. You have to change elastic.example.com to match your site name. Restart NGINX:

sudo service nginx restart

And you should now see the Kibana Dashboard after entering your password.

You can get started with some of the sample data, but if you want to get something meaningful out of this, you need to get started sending your own logs.

Connect log unloader

To retrieve logs in Elasticsearch, you must send them from the source server to your Elasticsearch server. To do this, Elastic provides lightweight stock unloaders called Beats. There are a lot of types for different use cases; Metricbeat collects system metrics as CPU usage. Packetbeat is a network packet analyzer that tracks traffic data. Heartbeat tracks uptime for URLs.

The simplest for most basic logs is called Filebeat and can be easily configured to send events from system log files.

Install Filebeat from apt. Alternatively, you can download the binary for your distribution:

sudo apt-get install filebeat

To set it, you need to edit the configuration file:

sudo nano /etc/filebeat/filebeat.yml

Here are two main things to edit. During filebeat.inputsyou must change “enabled” to true, then add any log paths that Filebeat should search for and send.

Edit the configuration file.

Then, under “Elasticsearch Output”:

Add a username and password.

If you do not use localhostyou must add a username and password in this section:

username: "filebeat_writer" 
password: "YOUR_PASSWORD"

Then launch Filebeat. Keep in mind that once started, it will immediately send all previous logs to Elasticsearch, which can be a lot of data if you do not rotate your log files:

sudo service filebeat start

Using Kibana (Making Sense of the Noise)

Elasticsearch sorts data into indexes used for organizational purposes. Kibana uses “Index Patterns” to actually use data, so you need to create one under Stack Management> Index Patterns.

Create and index under Stack Management> Index Patterns.  “Width =” 580 “height =” 328 “onload =” pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon (this);  “onerror =” this.onerror = null;  pagespeed.lazyLoadImages.loadIfVisibleAndMaybe);  “/></p>
<p>An index <em>pattern</em> can match multiple indices with wildcards.  For example, by default, Filebeat logs with daily time-based indexes, which can be easily rotated after a few months, if you want to save space:</p>
<pre>filebeat-*</pre>
<p>You can change this index name in the Filebeat configuration.  It may make sense to divide it by hostname or by the type of logs sent.  By default, everything is sent to the same filbeat index.</p><div><script async src=

You can browse the logs under the “Discover” tab in the sidebar. Filebeat indexes documents with a timestamp based on when it was sent to Elasticsearch, so if you’ve been running your server for a while, you’ll probably see a lot of log entries.

If you have never searched your logs before, you will immediately see why it is a bad thing to have an open SSH port with password authentication. Searching for “failed password” shows that this regular Linux server without password login disabled has over 22,000 logs entries from automatic bots that try random root passwords for a few months.

Looking for

Under the tab “Visualize” you can create graphs and visualizations of data in indexes. Each index will have field, which will have a data type such as number and string.

Visualizations have two components: measurement values ​​and buckets. The Measured Values ​​section calculates values ​​based on fields. On an area plot, this represents the Y-axis. This includes, for example, taking an average of all elements or calculating the sum of all items. Min / Max are also useful for capturing outliers in data. Percentile rows can be useful for visualizing the uniformity of data.

Buckets basically organize data into groups. On an area plot, this is the X-axis. The simplest form of this is a date histogram, which shows data over time, but it can also be grouped by significant terms and other factors. You can also divide the entire chart or series by specific terms.

Divide the whole chart or series with specific terms.

When you are done with your visualization, you can add it to a dashboard for quick access.

Add a visualization to a quick access dashboard.

One of the most important useful functions in dashboards is to be able to search and change time intervals for all visualizations on the dashboard. For example, you can filter results to show only data from a specific server or set all graphs to show the last 24 hours.

Direct API logging

Logging in with Beats is nice for connecting Elasticsearch to existing services, but if you run your own application, it may make more sense to cut out the middleman and log documents directly.

Direct logging is quite simple. Elasticsearch provides an API for that, so all you need to do is submit a JSON-formatted document to the following URL and replace indexname with the index you send to:

http://example.com:9200/indexname/_doc

Of course, you can do this programmatically with the language and HTTP library you choose.

Send a JSON-formatted document and replace the index name with the index you are sending to.

But if you send multiple logs per second, you may want to implement a queue and send them in bulk to the following URL:

http://example.com:9200/_bulk

However, it expects a rather strange formatting: separated list pairs of objects with a new line. The first sets the index to be used, and the second is the actual JSON document.

{ "index" : { "_index" : "test"} }
{ "field1" : "value1" }
{ "index" : { "_index" : "test2"} }
{ "field1" : "value1" }
{ "index" : { "_index" : "test3"} }
{ "field1" : "value1" }

You may not have an out-of-the-box way of dealing with this, so you may need to deal with it yourself. For example, in IC #, you can use StringBuilder as an execution way to add the desired formatting around the serialized object:

private string GetESBulkString(List list, string index)
{
      var builder = new StringBuilder(40 * list.Count);

      foreach (var item in list)
      {
           builder.Append(@"{""index"":{""_index"":""");
           builder.Append(index);
           builder.Append(@"""}}");

           builder.Append("n");

           builder.Append(JsonConvert.SerializeObject(item));
           builder.Append("n");
       } 

       return builder.ToString();
}

Source link