How we built the security dashboard on this website

The threat-detection dashboard on our home page is not a mock-up. It is a real Stroom instance fed about 32,000 synthetic security events over seven days. Everything in it is fictional, including a scripted intrusion for the dashboard to find. This post covers how we built it, because the set-up is a good template for any new data source, and because we hit four problems worth passing on.

What it creates

Everything lives in one Security Demo folder, so it can be removed in a single step without touching anything else in the instance:

• a feed, SECURITY_DEMO-EVENTS
• one XSLT that maps events to records:2
• an index of 21 fields, partitioned monthly
• an indexing pipeline, inheriting the stock Indexing template
• a processor filter scoped to this feed and stream type only
• the Security Operations dashboard: KPI tiles, events over time, a geo map, severity, activity heat maps, a MITRE ATT&CK tactic → technique → host sunburst, and an alerts table

Two choices are deliberate. The feed's stream type is Records, not Events, because existing processor filters index any Events stream, and the demo data would have leaked into their index. And the XSLT copies every child of <Event> generically instead of naming fields one by one. Element names match index field names, so adding a field means changing the generator and the index, never the XSLT.

The workflow

1. Generate the data: ./generate_events.py writes one XML file per day. The dashboard's time terms are relative (now()-7d), so regenerating refreshes it.
2. Build and validate the content pack: ./build_content.py, then ./validate.py. The validator checks every column, field reference and formatting rule against the index, because a pane pointing at a missing field comes up silently empty instead of showing an error.
3. Import the zip, and tick Enable Filters.
4. Upload the data zip from the feed's Data tab. Each day's file arrives as its own stream.

Indexing starts on its own within a minute.

Four things that caught us out

1. Import filenames are sanitised. The importer looks for a doc's files by its name with every non-alphanumeric character replaced by _. The feed SECURITY_DEMO-EVENTS therefore has to be in files named SECURITY_DEMO_EVENTS.*. Get it wrong and the doc imports with no content, and the error you get (feedDoc is null) names neither the file nor the rule.

2. The "Enable Filters" checkbox overrides the filter. Import sets each processor filter's enabled flag from that checkbox, and it defaults to off. Leave it unticked and nothing is processed, and nothing reports a problem.

3. Search extraction needs a stream a pipeline wrote. Extraction seeks to an event by its segment, and a stream uploaded straight into a feed has no segments. Every row failed with Segment number … not within bounds. We stored every index field and read values straight from the index instead. A table with no extraction pipeline falls back to the index's defaultExtractionPipeline, so that has to be absent too.

4. Re-importing does not clear the Index Field Cache. An index's field definitions are cached, and re-importing the index does not invalidate the cache. After changing index fields, go to Monitoring → Caches → Index Field Cache and clear it on every node, or restart Stroom, which is more certain.

Using this as a template

Swap the generator for a real feed and this becomes a working starting point: one folder, one feed, a generic translation, an index and a dashboard. If you want one built for your own data, that is what our content development work is.