Introduction To Elastic Search, Kibana and X-Pack

– Elasticsearch is a search engine. It is powered by Lucene, an open-source full-text search library.
It provides a distributed full-text search engine with an HTTP web interface and schema-free JSON documents.

– Kibana is the data visualization plugin for elasticsearch.

– X-Pack is an Elastic Stack extension that bundles security, alerting, monitoring, reporting, and graph capabilities into one easy-to-install package.

Installation :

Required:

Elasticsearch requires at least Java 8

$ java -version

java version “1.8.0_92”

Java(TM) SE Runtime Environment (build 1.8.0_92-b14)

Java HotSpot(TM) 64-Bit Server VM (build 25.92-b14, mixed mode)

Steps:

1) Download elastic search and kibana from https://www.elastic.co/start

* Following example is for Mac

$ ls

elasticsearch-5.5.0.tar.gz kibana-5.5.0-darwin-x86_64.tar.gz

2) Extract ElasticSearch and verify files and folders

$ tar zxf elasticsearch-5.5.0.tar.gz

$ ls

elasticsearch-5.5.0 elasticsearch-5.5.0.tar.gz kibana-5.5.0-darwin-x86_64.tar.gz

$ cd elasticsearch-5.5.0

$ ls

LICENSE.txt NOTICE.txt README.textile bin config lib modules plugins

3) Run elasticsearch by typing command bin/elasticsearch from the parent folder.

$ bin/elasticsearch

[2017–08–19T02:41:02,805][INFO ][o.e.n.Node ] [] initializing …

….

[2017–08–19T02:41:08,001][INFO ][o.e.n.Node ] [CkWGb2R] starting …

[2017–08–19T02:41:08,233][INFO ][o.e.t.TransportService ] [CkWGb2R] publish_address {127.0.0.1:9300}, bound_addresses {[fe80::1]:9300}, {[::1]:9300}, {127.0.0.1:9300}

[2017–08–19T02:41:11,509][INFO ][o.e.n.Node ] [CkWGb2R] started

* Elastic is running on the node CkWGb2R which is created by default. Node name can be changed in config/elasticsearch.yml (node.name) before staring the elasticsearch.

* Its running on 127.0.0.1(localhost) port 9200

4) Open browser and load http://localhost:9200. You can see the following information.

{

“name” : “CkWGb2R”,

“cluster_name” : “elasticsearch”,

“cluster_uuid” : “OxfxTDEJSju3VUlX8GFpqw”,

“version” : {

“number” : “5.5.0”,

“build_hash” : “260387d”,

“build_date” : “2017–06–30T23:16:05.735Z”,

“build_snapshot” : false,

“lucene_version” : “6.6.0”

},

“tagline” : “You Know, for Search”

}

Cluster :

– A cluster is a collection of one or more nodes (servers) that together holds entire data and provides federated indexing and search capabilities across all nodes.

– A cluster is identified by a unique name which by default is “elasticsearch”.

Node:

– A node is a single server that is part of the cluster, stores your data, and participates in the cluster’s indexing and search capabilities.

– A node is identified by a name which by default is a random UUID that is assigned to the node at startup. One can define any node name if do not want the default.

5) Extract Kibana

$ tar zxf kibana-5.5.0-darwin-x86_64.tar.gz

$ ls

elasticsearch-5.5.0 elasticsearch-5.5.0.tar.gz kibana-5.5.0-darwin-x86_64 kibana-5.5.0-darwin-x86_64.tar.gz

6) Run Kibana in another terminal by typing command bin/kibana from the parent folder.

$ bin/kibana

log [21:21:53.232] [info][status][plugin:kibana@5.5.0] Status changed from uninitialized to green — Ready

….

log [21:21:53.907] [info][listening] Server running at http://localhost:5601

log [21:21:53.909] [info][status][ui settings] Status changed from uninitialized to yellow — Elasticsearch plugin is yellow

….

log [21:21:59.907] [info][status][ui settings] Status changed from yellow to green — Ready

7) Access http://localhost:5601 in browser

Kibana Dev Tools

– Go to http://localhost:5601, kibana UI -> Dev Tools -> Console.

– Left panel to write query and execution.

– Right panel will show the result.

Type “GET /” and run.

This will fetch the current node and cluster information that we have already seen.

Terminology

Index:

– An index is a collection of documents having more or less similar characteristics. For ex, index for product data, index for customer data etc.

– And index is identified by a lowercase name which is used to refer to the index when performing indexing, search, update, and delete operations against the documents in it.

Type:

– A type is a logical category/partition of your index whose semantics is completely up to you.

– A type is defined for documents that have a set of common fields.

– For example, let’s assume you run a e-commerce platform and store all your data in a single index. In this index, you may define a type for user data, another type for product data, and yet another type for feedback data.

Document:

– A document is a basic unit of information that can be indexed.

– For example, you can have a document for a single customer, another document for a single product, and yet another for a single order.

– This document is expressed in JSON

Create Document

Kibana UI -> Dev Tools -> Console

Type the following and RUN.

POST /my-index/my-type/1

{

“body” : “test”

}

* This will create a document of id 1 of type ‘my-type’ under index ‘my-index’. In order to index a document, we must tell Elasticsearch which type in the index it should go to.

* Elasticsearch does not require you to explicitly create an index first before you can index documents into it. In the previous example, Elasticsearch automatically created the ‘my-index’ index

* Elasticsearch has fast search responses because, instead of searching the text directly, it searches an index like retrieving pages in a book related to a keyword by scanning the index at the back of a book.

Get Document

GET /my-index/my-type/1

Search (API)

GET my-index/_search

{

“query”: {

“match”: {

“body”: “test”

}

}

}

Output

{

“took”: 24,

“timed_out”: false,

“_shards”: {

“total”: 5,

“successful”: 5,

“failed”: 0

},

“hits”: {

“total”: 1,

“max_score”: 0.2876821,

“hits”: [

{

“_index”: “my-index”,

“_type”: “my-type”,

“_id”: “1”,

“_score”: 0.2876821,

“_source”: {

“body”: “test”

}

}

]

}

}

Delete Document

DELETE /my-index/my-type/1

Output:

{

“found”: true,

“_index”: “my-index”,

“_type”: “my-type”,

“_id”: “1”,

“_version”: 2,

“result”: “deleted”,

“_shards”: {

“total”: 2,

“successful”: 1,

“failed”: 0

}

}

GET /my-index/my-type/1

{

“_index”: “my-index”,

“_type”: “my-type”,

“_id”: “1”,

“found”: false

}

Install X-Pack

* Install at Elastic Side

– stop elastic search by Ctrl+C

– Install x-pack plugin

$ bin/elasticsearch-plugin install x-pack

-> Downloading x-pack from elastic

….

-> Installed x-pack

-Start elastic again

$ bin/elasticsearch

….

[2017–08–19T03:22:28,975][INFO ][o.e.p.PluginsService ] [CkWGb2R] loaded plugin [x-pack]

….

[2017–08–19T03:22:33,927][INFO ][o.e.n.Node ] [CkWGb2R] starting …

….

[2017–08–19T03:22:37,479][INFO ][o.e.n.Node ] [CkWGb2R] started

….

* Install x-pack UI at kibana side

– stop kibana by Ctrl+C

– Install x-pack plugin

$ bin/kibana-plugin install x-pack

Attempting to transfer from x-pack

Attempting to transfer from https://artifacts.elastic.co/downloads/kibana-plugins/x-pack/x-pack-5.5.0.zip

Transferring 119276235 bytes………………..

Transfer complete

Retrieving metadata from plugin archive

Extracting plugin archive

Extraction complete

Optimizing and caching browser bundles…

Plugin installation complete

– Restart kibana

$ bin/kibana

log [22:01:43.618] [info][listening] Server running at http://localhost:5601

log [22:01:43.620] [info][status][ui settings] Status changed from uninitialized to green — Ready

* Go to http://localhost:5601

You can find the login screen. This comes because of x-pack which gives role based permissions and authentication.