Skip to content

Fluentd

Using Fluentd's out_http plugin, and the headers syntax that catches almost everyone.

Fluentd's out_http plugin has been built in since v1.7, so if Fluentd is already deployed this adds no infrastructure. If you are choosing fresh, Fluent Bit is much lighter for the same job.

Read The datafeed contract first.

Minimal configuration

<match app.**>
  @type http
  endpoint https://stroom.example.com/<base>/datafeed
  http_method post

  headers {"Feed":"FLUENTD_EVENTS","Type":"Events","Compression":"NONE"}

  <format>
    @type json
  </format>

  <buffer>
    @type file
    path /var/log/fluentd/buffer/stroom
    flush_interval 5s
    retry_forever true
  </buffer>
</match>

Fails silently

headers is a hash written inline as JSON on one line, not a <headers> section.

This is the most common mistake in out_http configurations. Writing it as a section does not produce the headers, and the request goes out without a feed name.

Authentication

Add the Authorization header to the same hash. Fluentd's parser evaluates embedded Ruby inside double-quoted strings, so the token can come from the environment:

  headers {"Feed":"FLUENTD_EVENTS","Compression":"NONE","Authorization":"Bearer #{ENV['STROOM_TOKEN']}"}

Confirm the interpolation actually resolved. An unexpanded expression reaching Stroom shows up as an authentication failure, not a configuration error, which sends you looking in the wrong place. fluentd --dry-run prints the parsed configuration and answers it in seconds.

Do not reach for the <auth> section. It exists, but its method accepts only basic and aws_sigv4. There is no bearer option, so a data feed key has to go through the headers hash, which is how Stroom expects to receive it anyway.

For header values derived from record fields rather than the environment, the separate headers_from_placeholders parameter is the correct one. Read Choosing a feed before deriving the feed name that way.

Buffering

The file buffer matters: Fluentd's default in-memory buffer loses data on crash or restart. retry_forever true keeps retrying rather than discarding a chunk after the retry limit, which is right for audit data provided you monitor buffer size.

Batch shape

With <format> @type json, each buffered chunk becomes a post body. Whether that is a JSON array or newline-delimited depends on the json_array parameter. Set it explicitly, and make sure the receiving pipeline's parser matches.

Compression

out_http gained a compress parameter in v1.17.1, taking text or gzip. On anything older the option does not exist, so check your version before relying on it.

If you set compress gzip, leave "Compression":"NONE" in the headers hash. Fluentd sets Content-Encoding: gzip, which Stroom's web server decodes before Stroom reads the body. Changing Compression to GZIP as well makes Stroom decompress the body a second time, and every post fails with 500 and Stroom-Status 400.

Several feeds

One <match> block sends to one feed. Split by tag with several <match> blocks, keeping the set small and explicit. See Choosing a feed.

How long: immediate

Posting is synchronous, but buffered: flush_interval controls how often a chunk is sent, so allow at least one interval before concluding nothing arrived.

Check it worked

Validate and print the parsed configuration, then confirm the data landed:

fluentd --dry-run -c <config-file>

Read the printed configuration to check both the headers hash and any environment interpolation. Then find the stream on the feed in Stroom.

The check that matters most

The stream is on the feed in Stroom, and its content is readable text.

The dry run is unusually useful here because it shows the hash as parsed, which is where the two commonest mistakes on this page show up. It still does not prove delivery.

This does not prove it worked

A dry run passing proves the configuration parses, not that the headers are the ones you meant. Read what it printed.

An unexpanded #{ENV[...]} is not a configuration error, so nothing fails until Stroom rejects the request.

If it isn't working

In order. Each step rules out a class of cause, so don't skip ahead.

  1. Prove the endpoint without Fluentd. Use the curl test in The datafeed contract.
  2. Read the parsed configuration. --dry-run prints it. Check the headers hash is a hash, and that the token expanded.
  3. Are records reaching the match? Temporarily add a @type stdout match with the same pattern to inspect them.
  4. Is the buffer growing? The monitor agent exposes queue length and retry counts, which are the signals that matter once this is live.
  5. Does every post fail with 500 and Stroom-Status 400? compress gzip is set and the headers hash says "Compression":"GZIP". Set it back to NONE.
  6. Does the pipeline expect the wrong shape? Set json_array explicitly rather than relying on the default.
  7. Check parameter names against your installed version, since this page is written from the plugin documentation rather than from a run.

All documentation