Overview

Recently, while working with logs in the Kubernetes environment, I was in need to use fluentD. As it was my first time playing with it, I was quite keen to do it.

Now that my task is done, I’m quite content, but during the initial phase, I’ve found a few problems: - Lack of examples in the documentation - No “playground” which would help me to learn this tool efficiently

In the end, I’ve managed to build a quick skeleton, which helped me to implement my solution and keep my sanity. In this quick post, I wanted to share this skeleton.

Setup

The full environment will consist of: - custom Docker image, build on top of official fluentD image, allowing us to install additional plugins - fluentD configuration, allowing us to provide some dummy data, play with it and send to standard output - docker run command

Docker

First the Docker image, that’s quite easy:

FROM fluent/fluentd:v1.10.0-debian-1.0
USER root
RUN fluent-gem install fluent-plugin-multi-format-parser
RUN fluent-gem install fluent-plugin-record-modifier --no-document
# any other plugin..

Please note: - this example is based on Debian version, but I’m pretty sure you can use any other if needed - as it’s local environment, we can use root user quite freely

We can build this image with:

docker build -t local-fluentd-playground:v1 .

command.

FluentD configuration

Create fluentd.conf file:

# --- 1. Create dummy data -----
<source>
  @type dummy
  dummy {
    "log": {
        "level": "Information",
        "message": {
            "app": "application-name",
            "text": "some text"
        }
    }
  }
  tag loaded_data
</source>

# --- 2. Any transformation here -----
<filter>
  @type record_transformer
  enable_ruby
  <record>
    new_key ${record.dig("log","message", "app")}
  </record>
  remove_keys $.log.message.text
</filter>

# --- 3. Flush to standard output -----
<match>
  @type stdout
</match>

Please note: - You can provide your test data using dummy source. - Although transformation doesn’t matter in this post, the example one will use custom plugin and: - Create new key new_field with value coming from log.message.app - Remove the key log.message.text - Display data on standard output

Run and test

We can run this example with:

docker run -v C:/Path-to-config/fluentd.conf:/fluentd/etc/fluentd.conf -e FLUENTD_CONF=fluentd.conf local-fluentd-playground:v1

Please note: - Using volumes, you can very quickly change the configuration and re-run new container. This creates a nice fast feedback loop. - This example was run on Windows. You might need to volume your configuration file with different syntax on Linux.

The output is:

2020-05-09 20:07:37 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluentd.conf"
2020-05-09 20:07:37 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.0'
2020-05-09 20:07:37 +0000 [info]: gem 'fluentd' version '1.10.0'
...
2020-05-09 20:07:37 +0000 [info]: using configuration file: <ROOT>
  <source>
    @type dummy
    dummy {"log":{"level":"Information","message":{"app":"application-name","text":"some text"}}}
    tag "loaded_data"
  </source>
  <filter>
    @type record_transformer
    enable_ruby
    remove_keys $.log.message.text
    <record>
      new_key ${record.dig("log","message", "app")}
    </record>
  </filter>
  <match>
    @type stdout
  </match>
</ROOT>
...
2020-05-09 20:07:38.068323070 +0000 loaded_data: {"log":{"level":"Information","message":{"app":"application-name"}},"new_key":"application-name"}

Summary

Although the setup is quite easy, I’ve found it quite useful to quickly verify my fluentD configuration. I hope this might help someone else too!