Fluent Bit

Fluent Bit is a single static C binary, smaller and lighter than Vector, with a large input catalogue and a generic http output. Either is a good default for anything greenfield.

Read The datafeed contract first.

Minimal configuration

[SERVICE]
    flush     5
    log_level info

[INPUT]
    Name tail
    Path /var/log/app/*.log
    Tag  app.log

[OUTPUT]
    Name   http
    Match  app.log
    Host   stroom.example.com
    Port   443
    URI    /<base>/datafeed
    tls    on
    Format json
    Header Feed APP_LOGS
    Header Type Events
    Header Compression NONE
    Header Authorization Bearer ${STROOM_TOKEN}

Each header is its own Header <key> <value> line. The Authorization value contains a space, which is fine: only the first token after Header is treated as the key.

Set STROOM_TOKEN in the environment of the Fluent Bit process. ${VAR} is expanded from the environment at config load, so do not put the key in the file.

Batch shape

The http output's Format controls the post body. json sends a JSON array of records, json_lines sends newline-delimited JSON, and msgpack sends binary.

json_lines is usually the easiest to split downstream. Whichever you choose, the receiving pipeline's parser has to match, so tell whoever builds it.

Compression

[OUTPUT]
    Name     http
    Match    app.log
    Host     stroom.example.com
    Port     443
    URI      /<base>/datafeed
    tls      on
    Format   json_lines
    compress gzip
    Header   Feed APP_LOGS
    Header   Compression NONE
    Header   Authorization Bearer ${STROOM_TOKEN}

Buffering

By default Fluent Bit buffers in memory only, so a restart or a Stroom outage loses data. For production, enable filesystem-backed storage:

[SERVICE]
    storage.path              /var/lib/fluent-bit/storage
    storage.sync              normal
    storage.backlog.mem_limit 64M

[INPUT]
    Name         tail
    Path         /var/log/app/*.log
    Tag          app.log
    storage.type filesystem

storage.type filesystem is set per input; storage.path once in [SERVICE]. Setting the path without the per-input type is a common mistake and leaves you buffering in memory.

Other inputs

Swap the [INPUT] block; the output is unchanged. The catalogue includes tail, syslog, systemd, kubernetes_events, winevtlog, exec, http and docker_events, plus metric inputs.

Two naming points worth knowing:

• Container logs are normally collected with tail over the container log files. The docker input collects Docker metrics, which is not the same thing.
• winevtlog is the current Windows Event Log input; older configurations may reference winlog.

Feed is a static header per output, so one output means one feed. To send different sources to different feeds, define one [OUTPUT] per feed and select with Match or Match_Regex on the tags. See Choosing a feed.

fluent-bit --dry-run -c fluent-bit.conf