Debezium Server

For database change data capture without requiring Kafka. Neither Vector nor Fluent Bit does CDC, so this fills a genuine gap.

Read The datafeed contract first.

The working shape

[ database ] → [ Debezium Server ] → [ header-injecting hop ] → [ Stroom ]

The hop can be a reverse proxy you already run, or the same Vector binary used elsewhere in these guides. Vector is tidier if you have no proxy in the path, because it also gives you disk buffering and retries.

Vector as the hop

[sources.debezium_in]
type     = "http_server"
address  = "0.0.0.0:8080"
encoding = "json"

[sinks.stroom]
type           = "http"
inputs         = ["debezium_in"]
uri            = "https://stroom.example.com/<base>/datafeed"
method         = "post"
encoding.codec = "json"
framing.method = "newline_delimited"

[sinks.stroom.request.headers]
Feed          = "CDC_APPDB"
Type          = "Events"
Compression   = "NONE"
Authorization = "Bearer ${STROOM_TOKEN}"

[sinks.stroom.buffer]
type      = "disk"
max_size  = 268435488
when_full = "block"

Then point Debezium at http://localhost:8080.

nginx as the hop

location /cdc {
    proxy_pass https://stroom.example.com/<base>/datafeed;
    proxy_set_header Feed          CDC_APPDB;
    proxy_set_header Type          Events;
    proxy_set_header Compression   NONE;
    proxy_set_header Authorization "Bearer $stroom_token";
}

Lighter if nginx is already in the path, but it gives you no buffering: if Stroom is down, Debezium sees the failure and retries against its own configuration rather than a durable queue.

Debezium server configuration

In conf/application.properties:

debezium.sink.type=http
debezium.sink.http.url=http://localhost:8080
debezium.sink.http.timeout.ms=60000
debezium.sink.http.retries=5
debezium.sink.http.retry.interval.ms=1000

debezium.source.connector.class=io.debezium.connector.postgresql.PostgresConnector
debezium.source.database.hostname=db.example.com
debezium.source.database.user=debezium
debezium.source.database.password=${DB_PASSWORD}
debezium.source.database.dbname=appdb
debezium.source.topic.prefix=app_cdc
debezium.source.offset.storage.file.filename=/var/lib/debezium/offsets.dat

Later 3.x releases add batch.enabled and batch.max-size. Check whether your version has them before relying on batching.

Authentication

The HTTP sink's authentication types do not include a plain static bearer token, so a Stroom data feed key cannot be supplied from Debezium.

Authenticate at the hop instead, as both options above do, and restrict the hop's own listener to localhost or a private network. The hop is now holding the credential, so treat it accordingly.

Delivery semantics

Not documented. Debezium Server makes no at-least-once or exactly-once statement for its HTTP sink. Retries plus offset-based resume imply at-least-once in practice, but treat that as an inference rather than a guarantee, and do not cite it as documented behaviour in a design.

Since the datafeed endpoint is itself at-least-once, plan for duplicates downstream regardless.

Deriving the feed

CDC events carry database, schema and table, which are naturally stable and low-cardinality. But the header is set at the hop, not by Debezium, so the simple approach is one hop listener per feed: a Debezium Server instance, or a distinct hop port, per logical dataset, with the feed name hardcoded there.

For per-table feeds through a single hop, Vector can template the header from the event body, at the cost of the collision and cardinality risks in Choosing a feed. Never derive the feed from anything per-row.

nc -l 8080