Validating your Data and Structure
Last updated
Was this helpful?
Last updated
Was this helpful?
Fluent Bit supports multiple sources and formats. In addition, it provides filters that you can use to perform custom modifications. As your pipeline grows, it's important to validate your data and structure.
Fluent Bit users are encouraged to integrate data validation in their continuous integration (CI) systems.
In a normal production environment, inputs, filters, and outputs are defined in configuration files. Fluent Bit provides the filter, which you can use to validate keys and values from your records and take action when an exception is found.
A simplified view of the data processing pipeline is as follows:
Consider the following pipeline, which uses a JSON file as its data source and has two filters:
to exclude certain records.
to alter records' content by adding and removing specific keys.
Add data validation between each step to ensure your data structure is correct.
This example uses the filter.
Expect filters set rules aiming to validate criteria like:
Does the record contain key A
?
Does the record not contain key A
?
Does the key A
value equal NULL
?
Is the key A
value not NULL
?
Does the key A
value equal B
?
Consider a JSON file data.log
with the following content:
The following Fluent Bit configuration file configures a pipeline to consume the log, while applying an Expect filter to validate that the keys color
and label
exist:
To extend the pipeline, add a Grep filter to match records that map label
containing a key called name
with value the abc
, and add an Expect filter to re-validate that condition:
When deploying in production, consider removing any Expect filters from your configuration file. These filters are unnecessary unless you need 100% coverage of checks at runtime.
Every Expect filter configuration exposes rules to validate the content of your records using .
If the JSON parser fails or is missing in the input (parser json
), the Expect filter triggers the exit
action.