In Energotel's IP network we need to store and visualisate few thousands of sFlow samples per second. Common open source tools (mainly Java based) require huge amount of system resources and becomes quicky unresposible.
By using open source tools GoFlow2, Kafka, Clickhouse and Grafana we have built own internal tool for parsing, storing and visualisating sFlow data. The network size forced us to do many optimisations we want to describe and share the know-how:
- change default Clickhouse granularity
- avoid storing calculated data (using Clickhouse's ALIAS data type)
- implement double-sampling (data skipping to render wide time-window graphs)
- use big cache for uncompressed data
and many more.
The tool is in use in Energotel for a few months. Screenshots from Grafana raised questions from many people, what the tool we use.