Loki
Fleeting- Référence externe : https://github.com/grafana/loki
grafana/loki: Like Prometheus, but for logs.
label best practices
- Référence externe : https://grafana.com/docs/loki/latest/get-started/labels/bp-labels/
don’t add a label for something until you know you need it! Use filter expressions (|= “text”, |~ “regex”, …) and brute force those logs. It works – and it’s fast.
— https://grafana.com/docs/loki/latest/get-started/labels/bp-labels/ ()
when would you need labels?? A little farther down is a section on chunk_target_size. If you set this to 1MB (which is reasonable), this will try to cut chunks at 1MB compressed size, which is about 5MB-ish of uncompressed logs (might be as much as 10MB depending on compression). If your logs have sufficient volume to write 5MB in less time than max_chunk_age, or many chunks in that timeframe, you might want to consider splitting it into separate streams with a dynamic label.
— https://grafana.com/docs/loki/latest/get-started/labels/bp-labels/ ()
metric which can help you understand why chunks are flushed sum by (reason) (rate(loki_ingester_chunks_flushed_total{cluster=“dev”}[1m])).
— https://grafana.com/docs/loki/latest/get-started/labels/bp-labels/ ()
don’t add a level dynamic label, just |= “level=debug” instead
— https://grafana.com/docs/loki/latest/get-started/labels/bp-labels/ ()
Ensure the labels have low cardinality, ideally limited to tens of values.
— https://grafana.com/docs/loki/latest/get-started/labels/bp-labels/ ()
Use labels with long-lived values, such as the initial segment of an HTTP path: /load, /save, /update.Do not extract ephemeral values like a trace ID or an order ID into a label; the values should be static, not dynamic
— https://grafana.com/docs/loki/latest/get-started/labels/bp-labels/ ()
Only add labels that users will frequently use in their queries.Don’t increase the size of the index and fragment your log streams if nobody is actually using these labels. This will degrade performance.
— https://grafana.com/docs/loki/latest/get-started/labels/bp-labels/ ()
general rule, you should try to keep any single tenant in Loki to less than 100,000 active streams, and less than a million streams in a 24-hour period. These values are for HUGE tenants, sending more than 10 TB a day. If your tenant is 10x smaller, you should have at least 10x fewer labels.
— https://grafana.com/docs/loki/latest/get-started/labels/bp-labels/ ()
Use the Loki series API to get an idea of what your log streams look like and see if there might be ways to reduce streams and cardinality.
— https://grafana.com/docs/loki/latest/get-started/labels/bp-labels/ ()
logcli series command added the –analyze-labels flag specifically for debugging high cardinality labels
— https://grafana.com/docs/loki/latest/get-started/labels/bp-labels/ ()