Elasticsearch Logstash Kibana (ELK) for Tuleap downloads

At Enalean, we noticed information is lacking about Tuleap usage: number of new downloads, number of updates, repartition between operating systems and so on.

Using ELK

We already know that this kind of information can be found in the Apache access_log file. So we had to parse this log, extract the information and analyze it. We first thought of a custom solution with a data extractor to be build. Finally, we did not select this solution because better standard solutions already exist. We focused on the Elasticsearch Logstash Kibana (ELK) stack. We already heard about it but we had not yet the opportunity to experience it. So, I gave a brown bag to my colleagues to make them aware about it.


What are the components of ELK?

The acronym ELK means Elasticsearch Logstash Kibana. They are three softwares that can interact together to provide useful information from logs.



Elasticsearch Index formatted data
Logstash Parse log and send formatted data to ES
John Doe
Kibana Provide data from ES server with search queries



How ELK works for us?


When a user downloads Tuleap from rpm or apt repositories, an entry is added in access_log file. For security aspects, we’ve isolated the ELK stack from the server which hosts the yum and apt repositories. To have data from the access_log file, we ‘ve created a secure copy of this file through an SSH tunnel to the ELK server. Logstash “listens” this copied file, knows that new entries are added and compared to the old copied file, parse and format it with a template. These formatted data are sent to Elasticsearch which store them. Kibana send queries (defined by the user in the web UI) to Elasticsearch, get results and display them in graphs and tabs in the web UI.

What kind of data we want to analyze?

Our main goal of using this stack is to have information about downloads and updates of Tuleap:

  1. Number of downloads
  2. Number of updates
  3. Repartition of downloads/updates per OS
  4. Trends

These are some examples of results we will get:

With these data, we will be able to know the trends of the downloads/updates following an event we sponsored or press releases we publish for instance. On the technical side, we will be able to know if we should improve Tuleap compatibility on Debian OS for instance, regarding number of downloads and trends for this OS.

About the Author

How great is the challenge of creating economic value for a company with a libre software. I enjoy this! It encourages me to think business and communication in a disruptive way. I believe in the core value of FLOSS and agile spirit: open minded listening, transparency and co-creation. I'm Marketing Manager at Enalean.

Write Your Comment

twenty − nineteen =

You may use these HTML tags and attributes:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Websites located at enalean.com and other enalean.com subdomains need to store and access cookies on your device. We need your acceptance. Get more information.

Yes, I agree No, I disagree