In this post I'm going to describe how I used Heka to drain the logs from all my Docker containers running on my Tutum nodes.

tl;dr

You can get all source code on Github.
If you need help you can open an issue on Github.
Or send me a message on twitter @ianneub.

The goal

Our goal with this project is to get the logs off of our Tutum nodes and into Elasticsearch where we can aggregate, search, and report on our log files in one place. We also want to integrate our Tutum service and stack names into the log messages, so that we can use human readable ID's and not UUID's.

Heka

Heka is a log processing system written in Go and made by the great team at Mozilla. Seriously, the guys are awesome, look them up in IRC in channel #heka @ irc.mozilla.org. The system is similar to Logstash, but IMHO is faster, and a lot less resource intensive.

Heka sends messages through a build pipeline where each step in the pipeline receives a message as input, processes it, and then passes it to the next step in the pipeline. That pipeline consists of Inputs -> Decoders -> Encoders -> Outputs. There are other optional steps that are beyond the scope of this tutorial. You can find the documentation here.

First, we need to install and setup Heka in a Docker container. This container will run on every node and will be able to pull the logs from the Docker API, process them, and send the results to our Elasticsearch server.

For the sake of simplicity I won't go into the details of building Heka here, but will use my ready-to-go image hosted on the Docker hub.

Docker image: ianneub/heka | source

First, let's create our Heka config in a file named config.toml:

[DockerLogInput]
decoder = "TutumDecoder"

[TutumDecoder]
auth = "!!TUTUM_AUTH!!"

[ESJsonEncoder]
index = "hekad-%{2006.01.02}"
es_index_from_timestamp = true
type_name = "%{Logger}"

[ElasticSearchOutput]
message_matcher = "TRUE"
server = "http://es-server:9200"
flush_interval = 5000
flush_count = 10
encoder = "ESJsonEncoder"

The [DockerLogInput] section tells Heka that we want to use the DockerLog input to collect the log files. And that we want to decode each message with the TutumDecoder.

The [TutumDecoder] section sets up the decoder, and gives it your Tutum API token. This will let the decoder lookup the name of your containers, and get the service and stack names associated with them. Note that the auth setting includes the string !!TUTUM_AUTH!!. This is a placeholder and will be replaced with an environment variable using our container's startup script.

The [ESJsonEncoder] section defines what the data going into Elasticsearch will look like and what index to put it in. More info on ESJSonEncoder.

Finally the [ElasticSearchOutput] section defines how we want to connect to Elasticsearch and tells Heka to use the ESJsonEncoder that we setup previously. Of course you can modify the server option as you need to.

Great! Now Heka is configured to drain logs from Docker, process them using the Tutum API, and then send the logs to Elasticsearch.

Startup script

As of this writing Heka does not support using environment variables in its config file. (Since Heka 0.8, Heka has supported environment variables in the config file. So you don't need to do this step if you replace the !!TUTUM_AUTH!! with %ENV[TUTUM_AUTH]. I will leave this section here as it is a decent example of this pattern that you can use with other Docker projects.) So we will need to pass our API key in using a shell script before we start Heka. Let's write a simple bash script to replace the !!TUTUM_AUTH!! placeholder string that we used in the config file.

Create a file named run.sh with the contents like below:

#!/bin/bash

perl -p -i -e 's/!!TUTUM_AUTH!!/$ENV{TUTUM_AUTH}/' /app/config.toml

exec /app/hekad --config /app/config.toml

This will run the great perl -p -i -e command to rewrite the /app/config.toml file and replace !!TUTUM_AUTH!! with the environment variable TUTUM_AUTH. TUTUM_AUTH will be provided by Tutum at the container runtime.

Docker image

Right on! Now let's write a simple Dockerfile that will become our Heka drain image:

FROM ianneub/heka:0.10

COPY config.toml /app/config.toml

COPY run.sh /run.sh
RUN chmod +x /run.sh

CMD ["/run.sh"]

This Dockerfile will copy the config file and run.sh, set the run.sh file as executable, and tell Docker to start it by default when the container launches.

Build and push the container image

Now we have everything we need to build and push the container to Tutum (or any Docker registry). (Replace my username with yours)

$ docker build -t tutum.co/ianneub/heka-drain .
$ docker push tutum.co/ianneub/heka-drain

Run the container on Tutum

Last step! Let's create the service on Tutum.

Create a new service and choose the container image that you just created.

Change the deployment strategy to "Every node" and click next to the environment variables.

Under API Roles choose "Full access" and continue on to the volume setup.

Add a new volume with these settings:
Container path: /var/run/docker.sock
Host path: /var/run/docker.sock
Click "Add".

The volume is what will give our Heka log drain container access to the Docker logs on the node.

That should be it! Click on "Create and Deploy".

At this point Tutum should deploy your container to all the nodes in your cluster. Once started they will begin forwarding all the messages from all the containers on the node (including the containers running Tutum software). Logs from your stacks and services will be appropriately tagged with the service name and id using these fields:

  • TutumServiceId
  • TutumServiceName
  • TutumStackId
  • TutumStackName

Congratulations!

If you have any questions or issues please open a Github issue, or message me @ianneub.