Stroom Proxy and Data Receipt

Client systems send data to Stroom with a plain HTTP POST to its /datafeed endpoint. The body of the request is the data, and HTTP headers describe it. Stroom stores each accepted request as a stream in the feed the headers name, ready for processing.

Clients can post straight to Stroom, or to Stroom Proxy, which sits at the edge of the network and passes data on. Both present the same /datafeed API, so a client sends data the same way to either. To set up a sender, see How data gets in and Connecting a shipper.

From Stroom 7.14, data can also reach Stroom without a post to Stroom itself. Proxy forwards it to an S3 bucket, and Stroom picks it up by polling an AWS SQS queue for the bucket's S3 create events (the S3 Event Notifications Poll job), or from a notification posted to its /s3event/v1/notify endpoint, which is intended mainly for testing. Stroom then records a stream that points at the S3 object, and reads it through an S3 v1 (Read only) volume. See Volumes and Running a Proxy.

Sending data

A minimal post with curl looks like this:

curl --data-binary @events.log "https://<stroom-host>/stroom/datafeed" \
  -H "Feed:MY_SYSTEM-EVENTS" \
  -H "System:MY_SYSTEM" \
  -H "Environment:OPS" \
  -H "Authorization: Bearer ${STROOM_TOKEN}"

The payload must be text once it is uncompressed. It can be sent in one of three forms:

• Uncompressed: plain text, with no compression header.
• GZIP: gzip-compressed text, with Compression:GZIP or the standard Content-Encoding: gzip, but never both.
• ZIP: a ZIP archive, with Compression:ZIP. This is the only form that can carry more than one file, and the only one that can carry context data alongside the events.

The ZIP format

Inside a ZIP, files are grouped by their base name (the name without its extension). Each group is one item of data, with optional sidecar files:

A file and its sidecars must sit in the same directory inside the ZIP. Stroom rejects a ZIP that has two event files with the same base name.

Header arguments

Header names are case insensitive. Stroom keeps the headers, along with attributes it adds itself such as the receipt time, as the stream's metadata, so you can filter on them later.

By default Stroom requires every client to authenticate. Out of the box it accepts client certificates and OAuth tokens. It can also be set up to accept data feed keys, which work much like API keys and are issued for sending data only.

Response codes

Stroom answers every post with an HTTP status, plus a Stroom-Status header holding a more precise code and a Stroom-Error header holding the reason in words. The same message is in the response body. A client should capture all three.

The common cases are above. Every code, with the exact message Stroom returns, is in Datafeed response codes, which is generated from the source. The headers, compression and authentication rules are in The datafeed contract.

If the response is not 200, the client should keep the data and try again later. It should only delete data once Stroom or Stroom Proxy has accepted it.

Receipt IDs

Each successful post is given a unique receipt ID, which is written into the body of the 200 response and stored in the stream's ReceiptId attribute. It looks like this:

1738332835967_0000_P_node1

The parts are the receipt time in milliseconds since the epoch, a sequence number for posts received in the same millisecond, P or S for Stroom Proxy or Stroom, and the proxy ID or Stroom node name. When data passes through a chain of proxies, each hop, Stroom included, replaces ReceiptId with its own receipt ID and appends it to the ReceiptIdPath attribute, so ReceiptIdPath shows where the data has been and holds the ID the original sender was given. Proxy IDs and node names must be unique across the estate for this to work.

Stroom Proxy

Stroom Proxy is the front door for data. It turns many small posts into fewer, larger streams, which Stroom processes more efficiently, and it keeps accepting data while Stroom is offline.

Proxy can:

• Filter incoming data on its headers, and receive, drop or reject it.
• Split ZIPs that hold data for more than one feed or stream type.
• Aggregate data by feed and stream type into larger batches.
• Forward data to one or more destinations: Stroom, another proxy over HTTP, a directory on disk, or, from Stroom 7.14, an S3 bucket.
• Instantly forward data to a single HTTP or file destination (not S3) as it arrives, with no aggregation.
• Scan directories for ZIP files in Stroom's ZIP format and send them on (see ZIP drop folder).

Proxy is store-and-forward. It writes each post to its local disk before it answers the client, and works through a series of directory queues from there. From Stroom 7.14, by default it also forces each post to durable storage (fsync) before it answers, and does the same as data moves between queues. These settings are under proxyConfig.fsync (see Durability), and turning off receivingMode speeds up receipt at the cost of that guarantee. Stroom itself does the same for its stream store with data.store.fsyncMode (see Configuration properties). Data that fails to forward is kept and retried.

Aggregation

With aggregation on (the default), Proxy closes an aggregate and forwards it when it reaches any of these limits, set under proxyConfig.aggregator:

An aggregated batch arrives in Stroom as a single multi-part stream.

Deciding what to accept

Stroom and Stroom Proxy both check incoming data before they accept it. The property receive.receiptCheckMode chooses how:

Use the same mode on Stroom and on every proxy in front of it.

Feed status

Every feed has a Feed Status on its Settings tab:

• Receive: the data is accepted.
• Reject: the client gets a 406 with Stroom-Status 110.
• Drop: the data is thrown away, but the client gets a 200 as if it had been received. Use this when the sender should not know its data is being discarded.

Proxy has no access to feed settings, so it asks the Stroom (or proxy) downstream of it and caches the answer.

Data receipt rules

Data receipt rules give finer control. Open them from Administration > Data Receipt Rules (you need the Manage Data Receipt Rules permission). Each rule is an expression over the stream's metadata, which means its headers, and an action: Receive, Reject or Drop. Rules are tested in order and the first match wins. If no rule matches, the data is rejected, so add a final rule with no terms and a Receive action if you want to accept everything else. Data that passes the rules must still be for a feed that exists.

Proxies fetch the rules from Stroom periodically, so a change takes a short time to reach them.

Creating feeds automatically

Two optional features save creating a feed by hand for every new source. Both are off by default.

• Feed name generation (receive.feedNameGenerationEnabled) builds the feed name from other headers when Feed is not sent. The default template is ${accountid}-${component}-${format}-${schema}, and those four headers then become mandatory. Configure it the same way on Stroom and on Proxy.
• Content templates (autoContentCreation.enabled) create the feed, and a pipeline or processor filter to process it, when the first data for a new feed arrives. Manage them from Administration > Content Templates.

Related pages

• Feeds: the feed settings that receipt depends on.
• Streams: what a post becomes once it is stored.
• Processing: how received data is picked up by pipelines.
• Volumes and storage: where received streams are written.
• Is data arriving? and Proxy problems: if data is not getting through.