Connecting a Shipper
Pointing a log shipper you already run at Stroom, usually with configuration alone.
If you already run a log shipper, you almost certainly do not need to write anything. Most of them can POST with custom headers, which is all Stroom's datafeed endpoint asks for.
Read The datafeed contract first. Every recipe here depends on it, and skipping it is how you end up with silently corrupted data through the Compression header, or data in the wrong permission domain through the feed name.
Pick a route
| If you… | Use | Infrastructure added |
|---|---|---|
| Control your own agents, or are starting fresh | Vector or Fluent Bit | One small binary |
| Already run rsyslog | rsyslog | None |
| Already run syslog-ng | syslog-ng | None |
| Already run Logstash | Logstash | None |
| Already run Fluentd | Fluentd | None |
| Already run Apache NiFi | Apache NiFi | None |
| Already run Cribl Stream | Cribl Stream | None |
| Run agents locked to another vendor's protocol | Protocol shims | One small binary |
| Have a system that can only write files | ZIP drop folder | None, but a proxy is required |
| Want to read a Kafka topic | Vector or Fluent Bit | One small binary, no code |
| Need database change data capture | Debezium Server | One process, plus a header-injecting hop |
Vector and Fluent Bit are the recommended default for anything greenfield: a single small binary, a large source catalogue, and good buffering.
One of these is not configuration-only, and the table marks it in bold. Debezium Server cannot set the headers Stroom requires, so it needs a second process in front of it. Worth knowing before it is costed as simple.
Reading a Kafka topic needs no code at all: Vector and Fluent Bit both have a Kafka source, so a topic reaches Stroom through the same HTTP sink as everything else here.
What every recipe has in common
Four decisions, whichever tool you use:
- The feed name. See Choosing a feed. Decide it before you configure anything.
- Compression. A shipper's own compression option sends
Content-Encoding: gzip, which Stroom accepts as it is, so leave theCompressionheader atNONE. SettingCompression: GZIPas well decompresses the body twice and fails every post with500. SetCompression: GZIPonly when the body was compressed by something that does not setContent-Encoding. - Batch shape. One event per post, newline-delimited, or a JSON array. The receiving pipeline's parser has to match, so tell whoever builds it which you chose.
- Buffering. What the shipper does when Stroom is unreachable. For audit data, apply backpressure rather than dropping.
Note
Each recipe's Stroom-side facts are verified against the Stroom source. The shipper's own configuration is written from that tool's documentation and has not been run here, and plugin names and options move between releases. Each page says so, and each ends with how to validate it locally.