34
loading...
Deploy Kafka + Filebeat + ELK - Docker Edition - Part 2
This article is the last part of a two part series where we will deploy ELK stack using docker/docker-compose.
In this article, we will be configuring Logstash, Elasticsearch and Kibana. If you haven't gone through the previous article, check out this link
In this article, we will be configuring Logstash, Elasticsearch and Kibana. If you haven't gone through the previous article, check out this link
Logstash is a server-side data processing pipeline that consumes data from different sources and send it to elasticsearch. We touched on its importance when comparing with filebeat in the previous article.
Now to install logstash, we will be adding three components
Now to install logstash, we will be adding three components
Pipeline configuration will include the information about your input (kafka in our case), any filteration that needs to be done, and output (aka elasticsearch).
Create a folder named pipeline and add this configuration file to it.
Create a folder named pipeline and add this configuration file to it.
input {
kafka{
bootstrap_servers => "KAFKA_SERVERS_IP:KAFKA_SERVERS_PORT"
topics => ["applogs"]
}
}
filter {
json {
source => "message"
}
}
## Add your filters / logstash plugins configuration here
output {
elasticsearch {
hosts => ["ELASTICSEARCH_IP:9200"]
user => 'elastic'
password => 'somesecretpassword'
}As you can see, In the input section, we are listening to kafka on topic applogs.
In my case, I have added a filter which will jsonify the data that comes under the key named "message". There are several plugin filters to choose from.
The output is redirected towards elastic search, with username password configured for authentication.
Create a folder named settings and add this configuration file to it
In my case, I have added a filter which will jsonify the data that comes under the key named "message". There are several plugin filters to choose from.
The output is redirected towards elastic search, with username password configured for authentication.
Create a folder named settings and add this configuration file to it
http.host: "0.0.0.0"
path.config: /usr/share/logstash/pipeline
path.logs: /var/log/logstash
config.reload.automatic: true
log.level: debug
xpack.monitoring.enabled: falseThe docker-compose file looks like this:
version: '2'
services:
logstash:
image: docker.elastic.co/logstash/logstash:5.6.3
ports:
- "10000:10000"
volumes:
- ./settings/:/usr/share/logstash/config/
- ./pipeline/:/usr/share/logstash/pipeline/
container_name: logstashHere the configuration files mentioned above have been mounted.
To run the above file
docker-compose up -dNow, You will get an error as we have not setup elastic search yet. Now lets go ahead and install Elasticsearch and Kibana
Kibana is an open source user interface that helps you monitor and visualise data, which in our case, is being provided by elasticsearch.
Elasticsearch, as the name suggests, is a search and analytics engine for different types of data.
Elasticsearch, as the name suggests, is a search and analytics engine for different types of data.
version: "2"
services:
elasticsearch:
image: "docker.elastic.co/elasticsearch/elasticsearch:7.13.2"
container_name: elasticsearch
environment:
- discovery.type=single-node
- cluster.routing.allocation.disk.threshold_enabled=true
- cluster.routing.allocation.disk.watermark.low=65%
- cluster.routing.allocation.disk.watermark.high=70%
- xpack.security.enabled=true
- xpack.security.audit.enabled=true
- ELASTIC_PASSWORD=somethingsecret
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- bootstrap.memory_lock=true
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- ./data:/usr/share/elasticsearch/data
ports:
- "9200:9200"
networks:
- eknetwork
kibana:
depends_on:
- elasticsearch
image: "docker.elastic.co/kibana/kibana:7.13.2"
container_name: kibana
ports:
- "5601:5601"
environment:
- ELASTICSEARCH_URL=http://localhost:9200
- ELASTICSEARCH_USERNAME=elastic
- ELASTICSEARCH_PASSWORD=somethingsecretpassword
networks:
- eknetwork
networks:
eknetwork:
driver: bridgeMake sure to add same elasticsearch password as the one you have provided in your logstash pipeline configuration file.
To run the above file
docker-compose up -dNow that all our processes are running, You would need to configure Kibana's source from the UI so that you are able to see data on the UI. If you have done this setup on the server and want to configure UI immediately, I would suggest you to create a firewall rule for your public IP on port 5601
ufw allow from YOUR_PUBLIC_IP to any port 5601Now go to your favorite browser, type server's ip and port and you will see something like this:
And Voila! Your setup is complete. Just don't forget to add log-rotation to your docker instances.
I have added all the configuration and docker file here
I have added all the configuration and docker file here
34