Reset a Processor Filter

A processor filter scans for data matching its criteria and tracks how far it has got. A new filter starts with an empty tracker, so it does pick up streams that were already in the data store when you created it. The tracker only moves forward, though, and editing the filter does not reset it. If you change a filter's criteria after its tracker has moved past older streams, it never goes back to them: a filter can be enabled, correct, and happily processing new arrivals while never revisiting the streams its new criteria match.

The symptom is specific. The filter's criteria were changed. New data processes. Older data that the new criteria match does not. Nothing reports an error.

The procedure

There is no "rewind" control. Resetting the tracker means replacing the filter:

Confirm the criteria are right now. If the filter's feed or stream type is wrong, it is not a tracker problem and recreating it will change nothing.
Note what the filter is: its pipeline, criteria and priority. You are about to recreate it by hand.
Delete the filter.
Create it again with the same settings. The new filter starts with an empty tracker and scans the data store from the beginning.

For a bounded set of old streams, reprocessing a selection is usually better: it is scoped to what you choose rather than to everything the criteria match.

How long: not immediate

A new filter does not start instantly. The task creator polls on an interval, so expect a wait before the first task appears, then minutes to hours while it works through the backlog it has just discovered.

Check it worked

Watch the filter's tracker. After the recreate it should show a position early in the data and advance. Then look for output streams whose parents are the old raw streams.

The check that matters most

Output streams appear with parent streams that the old tracker had already passed.

That is the whole point of the reset: processing data the edited filter would never go back to. New output from new data proves nothing, because that was already working.

This does not prove it worked

The filter being enabled is not proof it is scanning. It was enabled before, and the old data still was not processed.

A tracker showing 100% is not proof everything was processed. It means the filter has reached the end of what it considers its scope, which is exactly the thing in question when old data was skipped.

Fails silently

Both the Processor and the Filter must be enabled, and they are separate controls.

A pipeline has a parent Processor and one or more Filters beneath it. A filter shown as enabled under a disabled processor processes nothing. Neither screen says so, and the filter looks correct in every respect.

Check both before concluding you have a tracker problem. This is the most common cause of "nothing is processing" by a wide margin, and recreating the filter will not fix it.

If it isn't working

Are both the processor and the filter enabled? See above. Check this first every time.
Do the criteria match the old streams? Compare the filter's feed, stream type and status against a stream you can see that was skipped. A filter on Events will never match Raw Events.
Is the old data within the filter's scope at all? Some criteria include a time bound. Data outside it is not skipped so much as excluded.
Did the recreate actually take? Confirm the new filter exists, is enabled, and has a tracker at a different position from the one you deleted.
Are there errors on the old data specifically? Old data is often older shapes of data. A pipeline written for this month's format may fail on last year's, which is a translation problem rather than a filter one.