Wednesday 27 September 2017

Kibana 5 internals - Dashboards, Visualizations and Index Patterns

Kibana is one of the Elastic products, part of the Elastic stack (formerly known as ELK).

Kibana is a an open source web app written with Angular and running over a thin Node.js server (which acts as a webserver).

Kibana is able to connect to a single Elasticsearch node.
It's possible to put a load balancer to several nodes, but the best practices suggest a Coordinating or a Tribe (deprecated) or a Cross-Cluster node (in case you need it).

Keep in mind: Kibana has no actual reason to exist without an Elasticsearch cluster to connect to.
The opposite is not true: Elasticsearch can live on its own and be quite useful without Kibana and can be even connected to Grafana if you really need visualizations.

Kibana has very little local state/persistence: the configuration files and some cache files. It completely relies on Elasticsearch for most of the features (e.g. authentication if you own X-Pack) and for storing all the data required to run.

Kibana provides a cool interface to build dashboards.

Kibana 5.0 dashboard screenshot from Elastic website
The dashboards are made up of several visualizations.

A visualization can be something static (a Markdown field to show some comment or help message) or dynamic (a pie chart, a table or a histogram).

The dynamic visualizations gets generated from the data retrieved from Elasticsearch.
The query to Elasticsearch is normally generated behind the scenes thanks to the UI. It's possible even to write your own query and use the output in the visualization.
The queries targeting Elasticsearch are most of the time aggregations.

A visualization typically refers to data fields exposed by an a index pattern.

What is an index pattern? Let's talk about index templates first.

If you are familiar with the Elastic stack, you've probably heard about index templates.
Index templates are one of the key parts of the Elasticsearch configuration.
An index template tells how the data you send into an index(or indices having an index name pattern) of an Elasticsearch cluster:

  • should be analyzed
    • if a text must be tokenized for free text search or taken as it is
    • if a float should be maybe mapped into a scaled float to save some space
    • if a field shouldn't be indexed at all
    • ...
  • should be stored and distributed
    • number of shards
    • number of replicas
    • how many times the index gets refreshed
    • if you want to keep the original document or not
    • ...

An index template is not able on its own to enumerate all the fields present in an index.
The only way to get them would be to perform a GET on an index, but you cannot define how Kibana should consider it.

Index patterns to the rescue!

An index pattern tells Kibana the fields you can query and what is their type.
Other advantages are:

  • it's able to target multiple indices (but you could do that with aliases on Elasticsearch)
  • you can define scripted fields

It's important to trigger a refresh of the index pattern if fields in the targeted index (or indices) have changed (added, removed or change of type occurred).

What happens if you want to export a single Dashboard?

You have several options:

  • the Elasticsearch API allows to backup all Kibana state, typically stored in the .kibana index
    • All dashboards, visualizations, index patterns and the dynamic configuration will be saved
  • Play with the new experimental Kibana import/export API (available since 5.5.0, it will be ready for prime time on 6.0)
  • Write a script in your preferred language

The latter option implies your tool:
  • gets the dashboard from the .kibana index, dashboard type by title
  • gets the panelsJSON field
  • unmarshalls the json data
  • gets all the visualization ids
  • exports all the visualization ids getting them from the .kibana index, visualization type
  • on each visualization
    • gets the kibanaSavedObjectMeta field
    • unmarshalls the json value
    • gets the index field within the query
  • get the index pattern from the .kibana index, index-pattern type
    • scripted fields are stored within an index-pattern

Unfortunately, there's no way to ensure this will not break in the near future.
Several changes will come with Elastic stack 6.0 and 7.0, such as:
  • Elasticsearch index mapping types will disappear
  • Kibana will have multi tenancy?
  • Internal storage changes on Kibana (can occurr even on minor versions), as the API is not public

Elastic has written a post on Kibana internals on 2016, worth taking a look.

There's no silver bullet solution to move your dashboard out from your cluster in a future-proof manner.

Hope you find this post useful!



Sadhana Rathore said...

Excellent post, thanks for this. I gathered lots of information from this and I am happy about it. Do share more updates.
AngularJS Training in Chennai
AngularJS Training institute in Chennai
Angular 7 Training in Chennai
ccna Training in Chennai
ReactJS Training in Chennai
PHP Training in Chennai
Tally course in Chennai
ui ux design course in Chennai
AngularJS Training in Anna Nagar

Aishu said...

Superb. I really enjoyed very much with this article here. Really it is an amazing article I had ever read. I hope it will help a lot for all. Thank you so much for this amazing posts and please keep update like this excellent article.thank you for sharing such a great blog with us.
IELTS Coaching in chennai

German Classes in Chennai

GRE Coaching Classes in Chennai

TOEFL Coaching in Chennai

spoken english classes in chennai | Communication training

Real-Time Teaching said...

Informative blog and useful information for all users. Keep sharing more blogs with us.
UI Development Training in Hyderabad
RPA Training in Hyderabad
Python Training in Hyderabad
Mean Stack developer Training in Hyderabad

cyber pc said...

awesome article, it's miles especially beneficial! I quietly commenced in this, and i am becoming more acquainted alongside amid it higher! Delights, refrain take leisure motion greater and tallying superb! Office 2013 Free Download With Crack Full Version

cyber pc said...

You delivered such an exceptional piece to retrieve, giving every difficulty enlightenment for us to attain records. thanks for sharing such information once us because of which my several concepts had been cleared. Office TimeLine Product Key

cyber pc said...

thank you for an tempting blog. What else can also I attain that form of data written in the sort of answer entre? i've an conflict that i am just now full of animatronics upon, and i've been upon the lookout for such info. Happy Eid Mubarak Wishes