Part 2 here Basic Logging & Debugging in Microservices - Part 2
ElasticSearch for storing the log data
At this time, you may have a good logging library that you built on your own. Your application
probably produced a bunch of log entries and you can debug your application using cat
and tail
command. You can also open the log file in your favorite text editor and search for the log entries
that you want. However, there should be a better solution for that. Imagine when your log file grows
to thousands or millions or even billions of lines, that’s not feasible to debug manually like that.
Fortunately, ElasticSearch is a good choice for organising log entries. Setting up ElasticSearch is a quite straight forward task. There is even pre-built docker image for it. Our team at AR doesn’t even care about optimising it because we rarely face the problem with it. Of course, our ElasticSearch instance goes down sometimes, but since it’s not a critical module of the system, doesn’t affect main application’s status, we don’t need to invest much time in optimising it. What we do is to configure restart policy so that Docker can recover the process after OOM. Another thing we need to do is to set up a daily script for deleting the old log entries to save the disk space. You can find installation instructions for ElasticSearch on its official Docker image page
fluentd for pushing the log data to ElasticSearch
Now you have your ElasticSearch server up and running. The next thing you need to do is to push
your log data to it. fluentd, an open source data collector for unified logging layer, allows you
to unify data collection and consumption for a better use and understanding of data. If your application
writes all the log entries into a file or to standard output inside Docker,
you can set up fluentd to tail the log files and push to ElasticSearch every time one line is
finished (just like the tail
command). In case you don’t know, Docker redirect its standard output
to a file on disk. You can find a sample fluentd setup here
https://github.com/tmtxt/clojure-pedigree/tree/master/images/fluentd.
The most important thing that you need to notice is its configuration file, the td-agent.conf
file. You will need to modify it to fit your requirements