Skip to content

Purge Reference Data

How reference data is retained, when stale lookups need a purge, and what that costs.

Reference data is loaded from reference streams into an off-heap store, and lookups are served from there rather than from the streams. The store is a cache in the sense that it can be rebuilt, but it is not cleared by clearing caches: it has its own retention and its own purge.

Stale lookups are the symptom. A reference feed has been updated, or a bad load has been corrected, and lookups still return the old values.

How retention works

Entries are purged by age. The controlling setting is purgeAge, in ISO-8601 duration form, and it defaults to 30 days. The job that applies it is Ref Data Off-heap Store Purge.

Two consequences worth holding on to:

  • Data older than purgeAge goes whether or not anything still refers to it. If lookups need to reach back further than purgeAge, raise it rather than discovering the gap later.
  • The purge is a scheduled job. If it is disabled, nothing is purged and the store grows until it hits its size limit.

The procedure

  1. Fix the reference data itself. Correct the reference feed and reload it. Purging a store that will be repopulated from the same bad source achieves nothing.
  2. Check the purge job is enabled, and what purgeAge is set to.
  3. Run the purge, or wait for the scheduled run if the timing suits.
  4. Re-run the lookup. The next lookup reloads what it needs from the reference streams.

Purging one stream

The scheduled job purges by age. To remove one reference stream's entries now, call the API: DELETE /refData/v1/purgeByStream/{refStreamId}, which you can also reach from Help → API Specification.

Find the stream id on the feed's Data tab: select the stream, then read Stream Id from the Info tab in the lower pane.

The optional nodeName query parameter restricts the purge to one node. Leave it unset to purge every node, which is almost always what you want, since the store is per node.

When the loader's XSLT has changed

Changing a reference loader's XSLT does not change data already in the store, and purging alone is not enough either: Stroom caches the compiled stylesheet.

  1. Purge the stream, as above.
  2. Clear the XSLT Pool cache under Monitoring → Caches, or the reload will re-run the old compiled stylesheet and appear to change nothing. Clearing Reference Data - Effective Stream Cache as well makes Stroom re-evaluate which streams are available. See Clear caches.
  3. Perform a lookup. The reload is lazy, so nothing reloads until something asks.

If the reference data is loaded by a processor filter rather than on demand, delete the old processing tasks as well, or the filter will not re-run.

How long: not immediate

The purge is a background job over a potentially large store, and it commits in batches. On a big store expect it to take a while, and expect lookups during that period to be slower than usual.

Reloading afterwards is lazy: the store repopulates when something asks for a value, not when the purge finishes.

Check it worked

Run a lookup that was returning a stale value, with an event time in the range you care about, and confirm it returns the corrected value.

The check that matters most

A lookup that was stale now returns the corrected value, for an event at the right time.

Reference lookups are resolved against the effective time of the event being processed, not the time you run the test, so a lookup that is right "now" can still be wrong for the data you actually process. Test at a time that matters.

This does not prove it worked

A smaller store is not proof the stale entries went. It proves some entries went.

A correct lookup in the Stepper on one record is not proof across the feed, if several reference streams overlap in their effective times. The one that wins is the latest whose effective time is not after the event.

Destroys data

Purging deletes entries that may be the only loaded copy of data you still need.

The store rebuilds from reference streams. If the reference stream a purged entry came from has itself been deleted by data retention, the entry does not come back and lookups for that period fail quietly, returning nothing rather than an error.

Check that the reference streams still exist before purging aggressively, and treat reference feeds as data to retain for at least as long as the events that look them up.

If it isn't working

In order. Each step rules out a class of cause, so don't skip ahead.

  1. Is the reference data itself correct now? Read the reference stream. Most "stale lookup" reports are a reference feed that was never actually fixed.
  2. Does the effective time overlap the events? Lookups select the reference stream whose effective time is the latest not after the event. A corrected stream with an effective time of today does not fix events from last week.
  3. Is the purge job enabled? If it is off, nothing is being purged on any schedule.
  4. Is the store full? A store at its size limit stops accepting new entries, so a corrected reference feed may have loaded nothing at all. Check the size against its limit.
  5. Is the <map> name right? A reference load writes into a named map. If the name in the XSLT does not match what the lookup asks for, nothing is stale: nothing was ever there.
  6. Does the lookup report a loader state of NEW? An error naming Current loader state: NEW means the reference pipeline failed to start, not that the lookup is wrong. Look for a syntax error in its XSLT, a missing parser or ReferenceDataFilter, or a user without Use permission on the pipeline or feed.

All documentation