Log to Metrics
Generate metrics from logs
Last updated
Generate metrics from logs
Last updated
The Log To Metrics Filter plugin allows you to generate log-derived metrics. It currently supports modes to count records, provide a gauge for field values or create a histogram. You can also match or exclude specific records based on regular expression patterns for values or nested values. This filter plugin does not actually act as a record filter and does not change or drop records. All records will pass this filter untouched and generated metrics will be emitted into a seperate metric pipeline.
Please note that this plugin is an experimental feature and is not recommended for production use. Configuration parameters and plugin functionality are subject to change without notice.
The plugin supports the following configuration parameters:
tag
Defines the tag for the generated metrics record
Yes
metric_mode
Defines the mode for the metric. Valid values are [counter, gauge or histogram]
Yes
metric_name
Sets the name of the metric.
Yes
metric_description
Sets a help text for the metric.
Yes
bucket
Defines a bucket for histogram
Yes, for mode histogram
e.g. 0.75
add_label
Add a custom label NAME and set the value to the value of KEY
label_field
Includes a record field as label dimension in the metric.
Name of record key. Supports Record Accessor notation for nested fields.
value_field
Specify the record field that holds a numerical value
Yes, for modes [gauge and histogram]
Name of record key. Supports Record Accessor notation for nested fields.
kubernetes_mode
If enabled, it will automatically put pod_id, pod_name, namespace_name, docker_id and container_name into the metric as labels. This option is intended to be used in combination with the kubernetes filter plugin, which fills those fields.
Regex
Include records in which the content of KEY matches the regular expression.
KEY REGEX
Exclude
Exclude records in which the content of KEY matches the regular expression.
KEY REGEX
Flush_Interval_Sec
The interval for metrics emission, in seconds. If Flush_Interval_Sec and Flush_Interval_Nsec are either both unset or both set to 0, the filter emits metrics immediately after each filter match. Otherwise, if either parameter is set to a non-zero value, the filter emits metrics at the specified interval. Longer intervals help lower resource consumption in high-load situations. Default value: 0.
Flush_Interval_Nsec
The interval for metrics emission, in nanoseconds. This parameter works in conjunction with Flush_Interval_Sec. Default value: 0.
The following example takes records from two dummy inputs and counts all messages passing through the log_to_metrics filter. It then generates metric records which are provided to the prometheus_exporter:
You can then use e.g. curl command to retrieve the generated metric:
The gauge mode needs a value_field specified, where the current metric values are generated from. In this example we also apply a regex filter and enable the kubernetes_mode option:
You can then use e.g. curl command to retrieve the generated metric:
As you can see in the output, only one line is printed, as the records from the first input plugin are ignored, as they do not match the regex.
The filter also allows to use multiple rules which are applied in order, you can have many Regex and Exclude entries as required (see grep filter plugin).
If you execute the above curl command multiple times, you see, that in this example the metric value stays at 60, as the messages generated by the dummy plugin are not changing. In a real-world scenario the values would change and return the last processed value.
Metric label_values
As you can see, the label sets defined by add_label and label_field are added to the metric. The lines in the metric represent every combination of labels. Only actually used combinations are displayed here. To see this, you can add a dummy dummy input to your configuration.
The metric output would then look like:
You can also see, that all the kubernetes labels have been attached to the metric, accordingly.
Similar to the gauge mode, histogram needs a value_field specified, where the current metric values are generated from. In this example we also apply a regex filter and enable the kubernetes_mode option:
You can then use e.g. curl command to retrieve the generated metric:
As you can see in the output, there are per default the buckets 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0 and +Inf, in which values are sorted into. A sum and a counter are also part of this metric. You can specify own buckets in the config, like in the following example:
Please note, that the +Inf bucket will always be included implicitly. The buckets in a histogram are cumulative, so a value added to one bucket will add to all larger buckets, too.
This filter also attaches Kubernetes labels to each metric, identical to the behavior of label_field. This results in two sets for the histogram.
