Skip to content

Index Problems

Shard states, fields that are silently ignored, and the configuration mistake that indexes nothing at all.

An index can be empty, incomplete, or wrong on one field, and all three look similar from a dashboard. This page is about the index itself. If the question is simply "is search working", start at Is search working?.

Shard states

Documents are written into shards. Each has a status, and only one of them is a problem:

StatusMeaning
NEWCreated, not yet opened.
OPENING, OPENIn use, being written to.
CLOSING, CLOSEDNot being written to. Normal for a finished shard.
DELETEDMarked for deletion; the Index Shard Delete job removes the files.
CORRUPTUnreadable. Its documents cannot be searched.

A CORRUPT shard does not recover. The documents in it have to be rebuilt from the source streams, which is only possible while those streams still exist. See Reset an index.

How long: not immediate

Indexing follows processing, so it inherits its latency.

Once indexed, documents are searchable straight away, because a search reads each shard through its open writer. Only the Doc Count on the Shards tab lags, until the Index Writer Flush job commits the shard, every 10 minutes by default.

Check it worked

Open the index, go to its Shards tab and select refresh, since the list is empty until you do. Add up the Doc Count column, which counts the documents committed to each shard; it can trail the real count until the next flush. Then confirm a search reads them back, with one row per indexed stream:

from "MY_INDEX"
eval events = count()
group by StreamId
select StreamId, events

The check that matters most

Rows from the query, no Error streams, and, once the shards are committed, a non-zero Doc Count.

The Doc Count proves documents were written, the query proves they are readable, and the absence of errors proves the indexing filter did not fail on the way. Any one alone can mislead.

This does not prove it worked

The field list proves nothing about contents. It is the registry of fields the index can hold, populated from configuration, and it looks complete and healthy when zero documents have been written. This is the most common way people conclude indexing worked when it did not.

Shards existing proves nothing either. They are created before documents go into them. Read the Doc Count.

from "MY_INDEX" select count() is not a count. Without a group by, it returns one row per document, each holding 1.

Fails silently

An index with no volume group indexes nothing.

A Lucene index needs a volume group naming the volumes its shards are written to. Without one the index document looks entirely correct, the field list is populated, the pipeline runs, and the indexing filter fails at processing time with Index volume group name must be specified.

The only sign is an Error stream on the feed. The index reports zero documents and offers no explanation. If an index has never held anything, check this before anything else.

Fails silently

A static index silently ignores fields it has not been told about.

Send a field that is not defined in the index and it is dropped. No error, no warning, no entry in a log. The index is smaller than you expect and searches on that field return nothing, as though the data never had it.

Either define every field you intend to search, or use a dynamic index, which creates fields from the incoming data at the cost of control over their types.

If it isn't working

In order. Each step rules out a class of cause, so don't skip ahead.

  1. Is the Doc Count zero, or does the query fail outright? A failure means the data source cannot be resolved or the query is invalid: a different problem, and one to fix first.
  2. Are there Error streams? Read one. The volume group failure appears only here.
  3. Did processing produce output at all? No Events streams means this is a processing problem. See Is data processing?.
  4. Is the indexing filter in the pipeline and pointed at this index? A filter whose index property is empty runs without complaint and indexes nothing.
  5. Does the XSLT output match what the filter consumes? Static indexing needs records:2; dynamic needs index-documents:1. The wrong namespace produces valid, empty output.
  6. Is one field wrong rather than all of them? That is usually a type change applied after data was written. Existing shards hold the old type and cannot be migrated; the index has to be reset.
  7. Does the search find documents but show nothing useful? That is extraction. The index needs a Default Extraction Pipeline, and extraction needs StreamId and EventId in the index, normally added by an IdEnrichmentFilter running before the indexing filter.
  8. Any CORRUPT shards? They will not recover. Reset and rebuild.

All documentation