Conditional processing
Last updated
Was this helpful?
Last updated
Was this helpful?
Conditional processing lets you selectively apply to logs based on the value of fields that those logs contain. This feature lets you create processing pipelines that only process records that meet certain criteria, and ignore the rest.
Conditional processing is available in Fluent Bit version 4.0 and greater.
You can turn a standard processor into a conditional processor by adding acondition
block to the processor's YAML configuration settings.
These condition
blocks use the following syntax:
Each processor can only have a single condition
block, but that condition can
include multiple rules. These rules are stored as items in the condition.rules
array.
The condition.op
parameter specifies the condition's evaluation logic. It has
two possible values:
Each item in the condition.rules
array must include values for the following parameters:
field
op
value
Rules are evaluated against each log that passes through your data pipeline. For example, given a rule with these parameters:
This rule evaluates to true
for a log that contains the string 'status':200
, but evaluates to false
for a log that contains the string 'status':403
.
You can use $field
syntax to access a top-level field, and $field['child']['subchild']
to access nested fields.
The conditions.rules.op
parameter has the following possible values:
eq
: equal to
neq
: not equal to
gt
: greater than
lt
: less than
gte
: greater than or equal to
lte
: less than or equal to
regex
: matches a regular expression
not_regex
: does not match a regular expression
in
: is included in the specified array
not_in
: is not included in the specified array
This example applies a condition that only processes logs that contain the
string {"request": {"method": "POST"
:
and
This example applies a condition that only processes logs when all of the specified rules are met:
or
This example applies a condition that only processes logs when one or more of the specified rules are met:
This example uses an array for the value of condition.rules.value
:
This example uses multiple processors with conditional processing enabled for each:
This configuration adds an alert
field to error logs from critical services,
and adds a paging_required
field to errors that contain specific critical patterns.
and
: A log entry meets this condition when all of the rules in the condition.rules
are .
or
: A log entry meets this condition when one or more rules in the condition.rules
array are .
The field within your logs to evaluate. The value of this parameter must use to access the fields inside logs.
The to evaluate whether the rule is true. This parameter (condition.rules.op
) is distinct from the condition.op
parameter and has different possible values.
The value of the specified log field to use in your comparison. Optionally, you can provide .
The conditions.rules.field
parameter uses to reference fields inside logs.