Kibana 5 internals - Dashboards, Visualizations and Index Patterns

Kibana is one of the Elastic products, part of the Elastic stack (formerly known as ELK).

Kibana is a an open source web app written with Angular and running over a thin Node.js server (which acts as a webserver).

Kibana is able to connect to a single Elasticsearch node.
It's possible to put a load balancer to several nodes, but the best practices suggest a Coordinating or a Tribe (deprecated) or a Cross-Cluster node (in case you need it).

Keep in mind: Kibana has no actual reason to exist without an Elasticsearch cluster to connect to.
The opposite is not true: Elasticsearch can live on its own and be quite useful without Kibana and can be even connected to Grafana if you really need visualizations.

Kibana has very little local state/persistence: the configuration files and some cache files. It completely relies on Elasticsearch for most of the features (e.g. authentication if you own X-Pack) and for storing all the data required to run.

Kibana provides a cool interface to build dashboards.

Kibana 5.0 dashboard screenshot from Elastic website

The dashboards are made up of several visualizations.

A visualization can be something static (a Markdown field to show some comment or help message) or dynamic (a pie chart, a table or a histogram).

The dynamic visualizations gets generated from the data retrieved from Elasticsearch.
The query to Elasticsearch is normally generated behind the scenes thanks to the UI. It's possible even to write your own query and use the output in the visualization.
The queries targeting Elasticsearch are most of the time aggregations.

A visualization typically refers to data fields exposed by an a index pattern.

What is an index pattern? Let's talk about index templates first.

If you are familiar with the Elastic stack, you've probably heard about index templates.
Index templates are one of the key parts of the Elasticsearch configuration.
An index template tells how the data you send into an index(or indices having an index name pattern) of an Elasticsearch cluster:

• should be analyzed
• if a text must be tokenized for free text search or taken as it is
• if a float should be maybe mapped into a scaled float to save some space
• if a field shouldn't be indexed at all
• ...
• should be stored and distributed
• number of shards
• number of replicas
• how many times the index gets refreshed
• if you want to keep the original document or not
• ...

An index template is not able on its own to enumerate all the fields present in an index.
The only way to get them would be to perform a GET on an index, but you cannot define how Kibana should consider it.

Index patterns to the rescue!

An index pattern tells Kibana the fields you can query and what is their type.
Other advantages are:

• it's able to target multiple indices (but you could do that with aliases on Elasticsearch)
• you can define scripted fields

It's important to trigger a refresh of the index pattern if fields in the targeted index (or indices) have changed (added, removed or change of type occurred).

What happens if you want to export a single Dashboard?

You have several options:

• the Elasticsearch API allows to backup all Kibana state, typically stored in the .kibana index
• All dashboards, visualizations, index patterns and the dynamic configuration will be saved
• Play with the new experimental Kibana import/export API (available since 5.5.0, it will be ready for prime time on 6.0)
• Write a script in your preferred language

The latter option implies your tool:

• gets the dashboard from the .kibana index, dashboard type by title
• gets the panelsJSON field
• unmarshalls the json data
• gets all the visualization ids
• exports all the visualization ids getting them from the .kibana index, visualization type
• on each visualization
• gets the kibanaSavedObjectMeta field
• unmarshalls the json value
• gets the index field within the query
• get the index pattern from the .kibana index, index-pattern type
• scripted fields are stored within an index-pattern

Unfortunately, there's no way to ensure this will not break in the near future.

Several changes will come with Elastic stack 6.0 and 7.0, such as:

• Elasticsearch index mapping types will disappear
• Kibana will have multi tenancy?
• Internal storage changes on Kibana (can occurr even on minor versions), as the API is not public

Elastic has written a post on Kibana internals on 2016, worth taking a look.

There's no silver bullet solution to move your dashboard out from your cluster in a future-proof manner.

Hope you find this post useful!

Bye!