https://s3-us-west-2.amazonaws.com/secure.notion-static.com/27794350-e4c4-438b-896e-44fb41efc235/Untitled.png

Follow this tutorial to get the stack up: https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elastic-stack-on-ubuntu-18-04#step-4-—-installing-and-configuring-filebeat

Configure Filebeat to read data from keep-core and keep-ecsda:

nano /etc/filebeat/filebeat.yml

Add the files you want to your inputs for filebeat.yml

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

- type: log

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /root/logs/keep.log
    - /root/logs-ecdsa/keep.log

Add a logstash filter to process connected peers. I named this file 20-keep-filter.conf and placed it in /etc/logstash/conf.d/:

filter {
  if [log][file][path] == "/root/logs/keep.log" {
    grok {
      patterns_dir => "./patterns"
      match => { "message" => "%{TIMESTAMP_ISO8601} %{LOGLEVEL} %{NOTSPACE:module} %{JAVACLASS}:%{INT:line_number}: number of connected peers: \\[%{INT:peer_count}\\]%{GREEDYDATA:message}"}
      remove_tag => ["_grokparsefailure"]
      add_field => { "subType" => "total_peers" }
      remove_tag => ["_grokparsefailure"]
    }
    if "_grokparsefailure" in [tags] {
      grok {
        patterns_dir => "./patterns"
        # check if the log line is has 'caused by'
        match => { "message" => "%{TIMESTAMP_ISO8601} %{LOGLEVEL} %{NOTSPACE:module} %{JAVACLASS}:%{INT:line_number}: %{GREEDYDATA:message}" }
        add_field => { "subType" => "all" }
        remove_tag => ["_grokparsefailure"]
      }
    }
    mutate {
      convert => {
        "peer_count" => "integer"
      }
    }
  }
}

In the Dashboard, to display the Metric for peer_count (one of the fields that the above filter will create), create a visualization > metric:

Metric Aggregation: Top Hit
Field: peer_count (this is our data field)
Aggregate with: concatenate
Size: 1
Sort on: @timestamp
Order: Descending