Follow this tutorial to get the stack up: https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elastic-stack-on-ubuntu-18-04#step-4-—-installing-and-configuring-filebeat
Configure Filebeat to read data from keep-core and keep-ecsda:
nano /etc/filebeat/filebeat.yml
Add the files you want to your inputs for filebeat.yml
filebeat.inputs:
# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.
- type: log
# Change to true to enable this input configuration.
enabled: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
- /root/logs/keep.log
- /root/logs-ecdsa/keep.log
Add a logstash filter to process connected peers. I named this file 20-keep-filter.conf and placed it in /etc/logstash/conf.d/:
filter {
if [log][file][path] == "/root/logs/keep.log" {
grok {
patterns_dir => "./patterns"
match => { "message" => "%{TIMESTAMP_ISO8601} %{LOGLEVEL} %{NOTSPACE:module} %{JAVACLASS}:%{INT:line_number}: number of connected peers: \\[%{INT:peer_count}\\]%{GREEDYDATA:message}"}
remove_tag => ["_grokparsefailure"]
add_field => { "subType" => "total_peers" }
remove_tag => ["_grokparsefailure"]
}
if "_grokparsefailure" in [tags] {
grok {
patterns_dir => "./patterns"
# check if the log line is has 'caused by'
match => { "message" => "%{TIMESTAMP_ISO8601} %{LOGLEVEL} %{NOTSPACE:module} %{JAVACLASS}:%{INT:line_number}: %{GREEDYDATA:message}" }
add_field => { "subType" => "all" }
remove_tag => ["_grokparsefailure"]
}
}
mutate {
convert => {
"peer_count" => "integer"
}
}
}
}
In the Dashboard, to display the Metric for peer_count (one of the fields that the above filter will create), create a visualization > metric:
Metric Aggregation: Top Hit
Field: peer_count (this is our data field)
Aggregate with: concatenate
Size: 1
Sort on: @timestamp
Order: Descending