Skip to content

Glossary

What Stroom's terms mean, in one place.

Stroom's vocabulary, defined once. Terms cross-reference each other, and every one has an anchor, so a link to a definition goes straight to it.

Account

Refers to a user account in Stroom's internal Identity Provider. An Account holds the credentials a person authenticates with, and exists only where Stroom is its own Identity Provider, unlike a User which exists in every deployment.

See also:

API

Application Programming Interface. An interface that one system can present so other systems can use it to communicate. Stroom has a number of APIs, e.g. its many REST APIs and its /datafeed interface for data receipt.

See also:

REST

API Key

API Keys are a form of authentication token that are created within Stroom for use by Stroom-Proxy instances or other clients that want to use Stroom's API. It is an encrypted string that contains details of the user and the expiration date of the token. Possession of a valid API Key for a user account means that you can do anything that the user can do in the user interface via the API.

API Keys should therefore be protected carefully and treated like a password. If you are using an external Identity Provider then tokens for use with the Identity Provider are generated by the external Identity Provider.

See also:

Application permission

This is a permission that is not specific to a single document. It applies to all documents or is not related to documents in any way.

Application permissions are generally associated with a screen or functional area of the Stroom application. A lot of the application permissions tend to be more applicable to system administrators but allow fine grained control of the different functional areas in Stroom so these functions can be devolved to other users.

Examples of application permissions are Manage Users, Pipeline Stepping and Data - View.

See also:

Application permissions

Byte order mark

A special Unicode character at the start of a text stream that indicates the byte order (or endianness) of the stream.

See also:

Byte Order Mark.

Character encoding

Character Encoding is the means of encoding character data (i.e. text) into binary form. Therefore to decode character data from a stream of bytes, the character encoding must be known (or guessed).

Common examples of character encodings are ASCII, UTF-8 and UTF-16.

Each feed has a defined character encoding for the data and context. This allows Stroom to decode the data sent into that Feed.

See also:

Character encoding

Condition

A Condition in an query expression term, e.g. =, >, in, etc.

Content

Content in Stroom typically means the user created documents/entities created in Stroom and as seen in the explorer tree. Content can be created/modified by Stroom users and imported/exported for sharing between different Stroom instances.

See also:

Content management

Context data

This is an additional stream of contextual data that is sent along side the main event stream. It provides a means for the sending system to send additional data that relates only to the event stream it is sent alongside.

This can be useful where the sending system has no control over the data in the event stream and the event stream does not contain contextual information such as what machine it is running on or the location of that machine.

The contextual information (such as hostname, FQDN, physical location, etc.) can be sent in a Context Stream so that the two can be combined together during pipeline processing using stroom:lookup().

See also:

Coordinated Universal Time (UTC)

Coordinated Universal Time (UTC), also known as Zulu time, is the international standard by which the world regulates clocks and time. It is essentially a successor to Greenwich Mean Time (GMT). UTC has the time zone offset of +00:00 and does not change for daylight saving. All international time zones are relative to UTC.

Stroom currently works internally in UTC, though it is possible to change the display time zone via User Preferences to display times in another time zone.

See also:

UTC (Coordinated Universal Time)

Cron

Cron is a command line utility found on most linux/unix systems that is used for scheduling background tasks. Cron expressions (or variants of them) are widely used in other schedulers.

Stroom uses a scheduler called Quartz which supports cron expressions for scheduling. The full details of the cron syntax supported by Quartz can be found here.

See also:

CSV

Comma Separated Values is a file format with typically one record per line and fields delimited by a ,. Fields may be optionally enclosed with double quotes, though there is no fixed standard for CSV data, particularly when it comes to escaping of double quotes and/or commas.

Dashboard

A Dashboard is a configurable entity for querying one or more Data Sources and displaying the results as a table, a visualisation or some other form.

See also:

Data source

The source of data for a Query, e.g. a Lucene based Index, a SQL Statistics Data source, etc.

Data sources include:

  • Lucene based search index data sources.
  • Elasticsearch index data sources.
  • Solr index data sources.
  • Plan B state store data sources.
  • Stroom's SQL Statistics data sources.
  • Searchable data sources for searching the internals of Stroom.

A data source will have a DocRef to identify it and will define the set of Fields that it presents. Each Field will have:

  • A name
  • A set of Conditions that it supports. E.g. a Feed field would likely support is but not >.
  • A flag to indicate if it is queryable or not. I.e. a queryable field could be referenced in the query expression tree and in a Dashboard table, but a non-queryable field could only be referenced in the Dashboard table.

See also:

Data splitter

Data Splitter is a pipeline element for converting text data (e.g. CSV, fixed width, delimited, multi-line) into XML for onward processing.

See also:

Dictionary

A entity for storing static content, e.g. lists of terms for use in a query with the in dictionary condition. They can also be used to hold arbitrary text for use in XSLT with the dictionary function.

See also:

Doc Ref

A Doc Ref (or Document Reference) is an identifier used to identify most documents/entities in Stroom, e.g. an XSLT will have a Doc Ref.

It is comprised of the following parts:

  • UUID - A Universally Unique Identifier to uniquely identify the document/entity.
  • Type - The type of the document/entity, e.g. Index, XSLT, Dashboard, etc.
  • Name - The name given to the document/entity.

Doc Refs are used heavily in the REST API for identifying the document/entity to be acted on.

See also:

Document

Document

Typically refers to an item that can be created in the Explorer Tree, e.g. a Feed, a Pipeline, a Dashboard, etc. May also be known as an Entity.

See also:

Entity

Document permission

Document permissions control the access that users and/or groups have to a Document.

See also:

Elasticsearch

Elasticsearch is an Open Source and commercial search index product. Stroom can be connected to one or more Elasticsearch clusters so that event indexing and search is handled by Elasticsearch rather than internally.

See also:

Elasticsearch

ELFF

The Extended Log File Format. A W3C standard format for log files produced by web servers.

See also:

Extended Log File Format

Entity

Typically refers to an item that can be created in the Explorer Tree, e.g. a Feed, a Pipeline, a Dashboard, etc. May also be known as a Document.

See also:

Document

Event

An event is a single auditable event, e.g. a user logging in to a system. A Stream typically contains multiple events.

In a Raw Events an event is typically represented as block of XML or JSON, a single line for CSV data. In an Events Stream an event is identified by its Event ID which its position in that stream (as a one-based number). The Event ID combined with a Stream ID provide a unique identifier for an event within a Stroom instance.

See also:

Events

This is a Stream Type in Stroom. An Events stream consists of processed/cooked data that has been demarcated into individual Events.

Typically in Stroom an Events stream will contain data conforming to the event-logging XML Schema which provides a normalised form for all Raw Events to be transformed into.

See also:

Stream Type

Explorer tree

The left hand navigation tree, described in User interface. The Explorer Tree is used for finding, opening, creating, renaming, copying, moving and deleting Documents.

It can also be used to control the access permissions of entities and folders. The tree can be filtered using the quick filter, see Finding things for more details.

See also:

Document

Expression tree

A tree of expression terms that each evaluate to a boolean (True/False) value. Terms can be grouped together within an expression operator (AND, OR, NOT).

For example:

AND (
  Feed is CSV_FEED
  Type = Raw Events
)

Expression Trees are used in Processor Filters and Query expressions.

See also:

Feed

A Feed is a means of organising and categorising data in Stroom. A Feed contains multiple Streams of data that have been ingested into Stroom or output by a Pipeline. Typically a Feed will contain Streams of data that are all from one system and have a common data format.

See also:

Field

A named data Field within some form of record or entity, and where each Field can have an associated value. In Stroom, Fields can be the Fields in an Index (or other queryable Data Source or the fields of Metadata associated with a Stream, e.g. Stream ID, Feed, creation time, etc.

See also:

Filter

A Filter may refer to a Processor Filter or a Filter Element in a Pipeline.

See also:

Fully Qualified Domain Name (FQDN)

The Fully Qualified Domain Name (FQDN) is the complete, unambiguous address of a device or service on the internet, specifying all domain levels including the hostname, domain name, and top-level domain. For example server57.some.domain.com.

See also:

Fully Qualified Domain Name

Git

Git is a free and open source distributed version control system. It is used for controlling, organizing, and tracking different versions in history of computer files, typically text files but also any other type of file. It allows all changes made to a file to be viewed and tracked over time and for branching/merging of the repository for separate strands of work.

The source code for the Stroom software is stored in a Git repository. Stroom also uses Git for managing user content that is held in one or more Git repositories.

See also:

Git

Group (users)

A named group of users to which application and document permissions can be assigned. Users can belong to multiple groups. A Group can belong to multiple groups. Groups allow permissions to be assigned to the group such that members of that group inherit those permissions.

See also:

Users and groups

Identity Provider (IDP)

An Identity Provider is a system or service that can authenticate a user and assert their identity. Identity providers can support single sign on (SSO), which allows the user to sign in once to the Identity Provider so they are then authenticated to all systems using that IDP.

Examples of identity providers are Google, Cognito, Keycloak and Microsoft Azure/Entra AD. Stroom has its own built in IDP or can be configured to use a 3rd party IDP.

Index

A Data Source that is backed by a Lucene based search index.

See also:

Lucene index

IP address

The Internet Protocol (IP) address, e.g. 192.168.0.1. Typically an IP address is assumed to be an IPv4 address.

ISO 8601

This is an international standard for representing dates, times and durations. By default Stroom displays date/times in ISO 8601.

Valid examples of ISO 8601 dates/times are:

2010-01-01T23:59Z
2010-01-01T23:59:59Z
2010-01-01T23:59:59.123Z
2010-01-01T23:59:59+02:00
2010-01-01T23:59:59.123+02

See also:

ISO_8601

JAR

Java Archive is a file format for distributing Java class files, associated metadata and resource files. It is a compressed archive based on the ZIP format, so can be inspected with any tool capable of reading a ZIP file. Stroom and Stroom-Proxy are distributed as JAR files.

JSON

JavaScript Object Notation is a file/data format for storing/transmitting structured data. It has similarities to XML, is less verbose, but is more simplistic. Stroom accepts data in JSON format and can output to JSON.

See also:

Wikipedia JSONParser

Markdown

Markdown is a simple markup language for creating rich formatted text using a text editor. Due to the simplicity of the Markdown it is still very readable in its raw form that contains the markup. Markdown is used in Stroom on the Documentation tab of each Document type and in the Documentation Document type.

Stroom uses the Showdown markdown converter to render users' markdown content into formatted text.

Note

Markdown is a somewhat loose standard so different markdown processors support different amounts of markdown syntax. For a definitive guide to the syntax supported in Stroom, see the above link.

See also:

Metadata

Metadata refers to the data that describes the Stream data. It is sometimes referred to as just Meta.

See also:

Namespace

In Stroom Namespace typically refers to an XML Namespace. Namespaces are used in XML to distinguish different elements, e.g. where an XSLT is transforming XML in the records:2 Namespace into XML in the event-logging:3 Namespace.

An XSLT will define short aliases for Namespaces to make them easier to reference within the XSLT document. For example, in this snippet of an XML document, the aliases are: stroom, evt, xsl, xsi.

<xsl:stylesheet
  xmlns="event-logging:3"
  xpath-default-namespace="records:2"
  xmlns:stroom="stroom"
  xmlns:evt="event-logging:3"
  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
  version="2.0">

See also:

Parser

A Parser is a Pipeline element for parsing Raw Events into a structured form. For example the Data Splitter Parser that parses text data into Records and Fields.

See also:

Pipeline

A Pipeline is an entity that is constructed to take a single input of stream data and process/transform it with one or more outputs. A Pipeline can have many elements within it to read, process or transform the data flowing through it.

See also:

Pipelines

Pipeline element

An element within a Pipeline that performs some action on the data flowing through it.

See also:

Processor

A Processor belongs to a Pipeline. It controls the processing of data through its parent Pipeline using one or more Processor Filters.

The Processor can be enabled/disabled to enable/disable the processing of data through the Pipeline. A processor will have one or more Processor Filters associated with it.

See also:

Pipeline Processor Filter

Processor filter

A Processor Filter is used to used to find Streams to process through the Pipeline associated with the Processor Filter. A Processor Filter consists of an expression tree to select which Streams to process and a tracker to track the what Streams have been processed.

For example a typical Processor Filter would have an Expression Tree that selected all Streams of type Raw Events in a particular Feed. A filter could also select a single Stream by its ID, e.g. when Re-Processing a Stream.

A Pipeline can have multiple Processor Filters. Filters can be enabled/disabled independently of their parent Processor to control processing.

See also:

Property

A configuration Property for configuring Stroom. Properties can be set in the user interface or via the config.yml configuration file.

See also:

Properties

Query

The search Query in a Dashboard that selects the data to display. The Query is constructed using an Expression Tree of terms.

See also:

StroomQL and data sources Dashboard Expression Tree

Raw Events

This is a Stream Type used for Streams received by Stroom. Streams received by Stroom will be in a variety of text formats (CSV, delimited, fixed width, XML, JSON, etc.). Until they have been processed by a pipeline they are essentially just unstructured character data with no concept of what is a record/event. A Parser in a pipeline is required to provide the demarcation between records/events.

See also:

Stream Stream Type Parser

Re-processing

The act of repeating the processing of a set of input data (Stream) that have already been processed at least once. Re-Processing can be done for an individual Stream or multiple Streams using a Processor Filter.

See also:

Reprocess data Stream Processor Filter

Records

This is a Stream Type for Streams containing data conforming to the records:2 XML Schema. It also refers more generally to any XML conforming to the records:2 XML Schema which is used in a number of places in Stroom, including as the output format for the DSParser and input for the IndexingFilter.

See also:

REST

REST (Representational State Transfer) is essentially an architectural style that dictates how data should be handled and "transferred" across a network. REST APIs typically use JSON to send data between the client and the server, and the HTTP methods GET, PUT, PATCH, POST and DELETE.

See also:

Search extraction

The process of extracting un-indexed Field values from the source Event to be used in search results.

See also:

Stored and extracted fields Field Event

Searchable

A Searchable is the term given the special searchable data sources that appear at the root of the explorer tree picker when selecting a data source. These data sources are special internal data sources that are not user managed content, unlike an Index. They provide the means to search various aspects of Stroom's internals, such as the Meta Store or Processor Tasks.

See also:

Index

SQL

Structured Query Language (SQL). The language used in the MySQL database to query and manipulate the data.

SQL is used by Stroom's database migration scripts to modify the structure of the database schema and the data contained within it. Sometimes it is necessary for Stroom administrators to manually run SQL statements/queries as part of pre-migration checks.

See also:

Stepper

The Stepper is a tool in Stroom for developing and debugging a Pipeline. It allows the user to simulate passing a Stream through a pipeline with the ability to step from one record/event to the next or to jump to records/events based on filter criteria.

The parsers and translations can be edited while in the Stepper with the element output updating to show the effect of the change. The stepper will not write data to the file system or stream stores.

See also:

Pipeline debugging Pipeline Stream

Stream

A Stream is the unit of data that Stroom works with and will typically contain many Events.

See also:

Stream Type

All Streams must have a Stream Type. The list of Stream Types is configured using the Property stroom.data.meta.metaTypes.

Additional Stream Types can be added however the list of Stream Types must include the following built-in types:

Some Stream Types, such as Meta Data and Context only exist as child streams within another Stream.

See also:

StroomQl

Stroom Query Language is Stroom's own query language. It has similarities with Structured Query Language (SQL) as used in databases. StroomQL is sometimes referred to as sQL to distinguish it from SQL.

See also:

Stroom Query Language

Table

A Table is the tabular part of a Dashboard or Query that contains the data.

Token

Typically refers to an authentication token that may be used for user authentication. A Stroom API Key is a form of authentication token.

Tokens are generally set in the HTTP header Authorization with a value of the form Bearer TOKEN_GOES_HERE. Tokens may contain information, e.g. a JSON Web Tokens (JWT) or simply be long strings of random characters (to essentially make a very secure password), like API Keys.

Tokens are associated with a Stroom User so have the same or less permissions than that user. Tokens also typically have an expiry time after which they will no longer work.

See also:

API Key

Tracker

A Tracker is associated with a Processor Filter and keeps track of the Stream that the Processor Filter has already processed.

See also:

Transport Layer Security (TLS)

Transport Layer Security (TLS) is the evolution of Secure Sockets Layer (SSL) and refers to the encryption of traffic between client and server.

TLS is typically used in Stroom for communications between Stroom-Proxy and Stroom, between Stroom nodes and when communicating with external systems (e.g. an Elasticsearch cluster or an HTTPAppender destination).

Unix Epoch

The Unix epoch is 00:00:00 UTC on 1st January 1970. Some timestamps in Stroom are represented as the number of milliseconds since the Unix epoch, e.g. 1738331628276, and may be referred to as epoch ms or epoch milliseconds.

User

Refers to a Stroom User that is linked to either an Account in Stroom's internal Identity Provider or a user account in an external Identity Provider. A Stroom User is primarily concerned with authorisation (i.e. application/document permissions and group memberships) rather than authentication, though disabling one also refuses that person at authentication and ends any access they currently hold.

See also:

UUID

A Universally Unique Identifier for uniquely identifying something. UUIDs are used as the identifier in Doc Refs. An example of a UUID is 4ffeb895-53c9-40d6-bf33-3ef025401ad3.

See also:

Visualisation

A document comprising some Javascript code for visualising data, e.g. pie charts, heat maps, line graphs etc. Visualisations are not baked into Stroom, they are content, so can be created/modified/shared by Stroom users.

See also:

Visualisations Dashboard

Volume

In Stroom a Volume is a logical storage area that Stroom can write data to. Volumes are associated with a path on a file system that can either be local to the Stroom node or on a shared file system. A Data Volume can instead be an S3 bucket, see Volumes and storage.

Stroom has two types of Volume; Index Volumes and Data Volumes.

  • Index Volume - Where the Lucene Index Shards are written to. An Index Volume must belong to a Volume Group.
  • Data Volume - Where streams are written to. When writing Stream data Stroom will pick a data volume using a volume selector as configured by the Property stroom.data.filesystemVolume.volumeSelector.

See also:

Volume Group Volumes and storage

Volume group

A Volume Group is a collection of one or more Index Volumes. Index volumes must belong to a volume group and Indexes are configured to write to a particular Volume Group.

When Stroom is writing data to a Volume Group it will choose which of the Volumes in the group to write to using a volume selector as configured by the Property stroom.volumes.volumeSelector.

See also:

Volume Volumes and storage

XML

Extensible Markup Language is a markup language for storing/transmitting structured data. It is the working format for most Pipeline processing in Stroom and is the standard normalised format for event data.

See also:

Pipeline XMLParser pipeline element

XML Schema

XML Schema is a language used to define the permitted structure of an XML document. An XML Schema can be used to validate an XML document to ensure it conforms to that schema such that onward processing of the XML document can be done with confidence that the document is correct.

The event-logging XML Schema is an example of an XML Schema.

See also:

XML Schema Schema filter

XPath

XPath is an expression language for selecting a node or nodes in an XML document. It is used heavily in XSLT to define the match criteria for templates and to select values.

See also:

XSLT XPath

XSLT

Extensible Stylesheet Language Transformations is a language for transforming XML documents into other XML documents. XSLTs are the primary means of transforming data in Stroom.

All data is converted into a basic form of XML and then XSLTs are used to decorate and transform it into a common form. XSLTs are also used to transform XML Events data into non-XML forms or XML with a different schema for indexing, statistics or for sending to other systems.

See also:

Parsers and translation

YAML

YAML Ain't Markup Language. A human readable data format often used for configuration files. YAML is used in stroom for various things, e.g. Stroom & Stroom Proxy's main configuration file, Content Store definition files. YAML files will typically have the file extension .yaml or .yml.

See also:

yaml.org

ZIP

A compressed file format for storing a one or more files with an associated directory structure. Stroom and Stroom Proxy use the ZIP format for exporting content and data as well as its Proxy ZIP format for holding multiple streams of data with associated meta data.

See also:

The ZIP format

All documentation