This article describes in detail how the yuuvis® RAD metrics-manager works, on what technologies it is based, and what it is used for.
...
We decided to use the Elasticsearch database and its stack for this purpose as it has great aggregation functions and can handle queries very fast, even on millions of records. In detail, the tools that yuuvis® RAD metrics-manager comprises are:
logstash
Logstash is a server-side data processing pipeline that ingests data from many sources like tcp or one of the elastic beats, transforms it, and then sends it to Elasticsearch. All metrics-manager tools use logstash to send data to Elasticsearch.filebeat
Filebeat is a small and simple tool that reads log files and sends the data line by line to Elasticsearch using logstash.metricbeat
Metricbeat is another tool of the beats family that can read system metrics like CPU load or disc I/O and sends the data to Elasticsearch using logstash.elastalert
Elastalert is a third-party tool that can be used to alert users over various channels on anomalies, spikes, or other patterns of interest from data in Elasticsearch. This can be done by creating definition files that specify the conditions that need to be met for an alert to trigger.kibana
Kibana is a frontend application that lets you visualize the data in Elasticsearch indices by running aggregations or similar queries and plotting the results in diagrams, graphs, timelines, etc. You can restrict the visualization to specific time ranges or view the entire data at once.
...