1.9
Installation
Administration
Data Pipeline
Stream Processing
Validating your Data and Structure
Fluent Bit is a powerful log processing tool that can deal with different sources and formats, in addition it provides several filters that can be used to perform custom modifications. This flexibility is really good but while your pipeline grows, it's strongly recommended to validate your data and structure.
We encourage Fluent Bit users to integrate data validation in their CI systems
A simplified view of our data processing pipeline is as follows:
In a normal production environment, many Inputs, Filters, and Outputs are defined in the configuration, so integrating a continuous validation of your configuration against expected results is a must. For this requirement, Fluent Bit provides a specific Filter called Expect which can be used to validate expected Keys and Values from your records and takes some action when an exception is found.
How it Works
As an example, consider the following pipeline where your source of data is a normal file with JSON content on it and then two filters: grep to exclude certain records and record_modifier to alter the record content adding and removing specific keys.
Ideally you want to add checkpoints of validation of your data between each step so you can know if your data structure is correct, we do this by using expect filter.
Expect filter sets rules that aims to validate certain criteria like:
- does the record contain a key A ?
- does the record not contains key A?
- does the record key A value equals NULL ?
- does the record key A value a different value than NULL ?
- does the record key A value equals B ?
Every expect filter configuration can expose specific rules to validate the content of your records, it supports the following configuration properties:
Property
Description
key_exists
Check if a key with a given name exists in the record.
key_not_exists
Check if a key does not exist in the record.
key_val_is_null
check that the value of the key is NULL.
key_val_is_not_null
check that the value of the key is NOT NULL.
key_val_eq
check that the value of the key equals the given value in the configuration.
action
action to take when a rule does not match. The available options are
warn or exit. On warn, a warning message is sent to the logging layer when a mismatch of the rules above is found; using exit makes Fluent Bit abort with status code 255.Start Testing
Consider the following JSON file called
data.log with the following content:1
{"color": "blue", "label": {"name": null}}
2
{"color": "red", "label": {"name": "abc"}, "meta": "data"}
3
{"color": "green", "label": {"name": "abc"}, "meta": null}
Copied!
The following Fluent Bit configuration file will configure a pipeline to consume the log above apply an expect filter to validate that keys
color and label exists:1
[SERVICE]
2
flush 1
3
log_level info
4
parsers_file parsers.conf
5
​
6
[INPUT]
7
name tail
8
path ./data.log
9
parser json
10
exit_on_eof on
11
​
12
# First 'expect' filter to validate that our data was structured properly
13
[FILTER]
14
name expect
15
match *
16
key_exists color
17
key_exists $label['name']
18
action exit
19
​
20
[OUTPUT]
21
name stdout
22
match *
Copied!
note that if for some reason the JSON parser failed or is missing in the
tail input (line 9), the expect filter will trigger the exit action. As a test, go ahead and comment out or remove line 9.As a second step, we will extend our pipeline and we will add a grep filter to match records that map
label contains a key called name with value abc, then an expect filter to re-validate that condition:1
[SERVICE]
2
flush 1
3
log_level info
4
parsers_file parsers.conf
5
​
6
[INPUT]
7
name tail
8
path ./data.log
9
parser json
10
exit_on_eof on
11
​
12
# First 'expect' filter to validate that our data was structured properly
13
[FILTER]
14
name expect
15
match *
16
key_exists color
17
key_exists label
18
action exit
19
​
20
# Match records that only contains map 'label' with key 'name' = 'abc'
21
[FILTER]
22
name grep
23
match *
24
regex $label['name'] ^abc$
25
​
26
# Check that every record contains 'label' with a non-null value
27
[FILTER]
28
name expect
29
match *
30
key_val_eq $label['name'] abc
31
action exit
32
​
33
# Append a new key to the record using an environment variable
34
[FILTER]
35
name record_modifier
36
match *
37
record hostname ${HOSTNAME}
38
​
39
# Check that every record contains 'hostname' key
40
[FILTER]
41
name expect
42
match *
43
key_exists hostname
44
action exit
45
​
46
[OUTPUT]
47
name stdout
48
match *
Copied!
Deploying in Production
When deploying your configuration in production, you might want to remove the expect filters from your configuration since it's an unnecessary extra work unless you want to have a 100% coverage of checks at runtime.
Export as PDF
Copy link