The Record Modifier Filter plugin allows to append fields or to exclude specific fields.
Configuration Parameters
The plugin supports the following configuration parameters: Remove_key and Allowlist_key are exclusive.
Key Description Append fields. This parameter needs key and value pair.
If the key is matched, that field is removed.
If the key is not matched, that field is removed.
An alias of Allowlist_key
for backwards compatibility.
If set, the plugin appends uuid to each record. The value assigned becomes the key in the map.
Getting Started
In order to start filtering records, you can run the filter from the command line or through the configuration file.
This is a sample in_mem record to filter.
Copy {"Mem.total"=>1016024, "Mem.used"=>716672, "Mem.free"=>299352, "Swap.total"=>2064380, "Swap.used"=>32656, "Swap.free"=>2031724}
Append fields
The following configuration file is to append product name and hostname (via environment variable) to record.
fluent-bit.conf fluent-bit.yaml
Copy [INPUT]
Name mem
Tag mem . local
[OUTPUT]
Name stdout
Match *
[FILTER]
Name record_modifier
Match *
Record hostname $ { HOSTNAME }
Record product Awesome_Tool
Copy pipeline :
inputs :
- name : mem
tag : mem.local
filters :
- name : record_modifier
match : '*'
record :
- hostname ${HOSTNAME}
- product Awesome_Tool
outputs :
- name : stdout
match : '*'
You can also run the filter from command line.
Copy $ fluent-bit -i mem -o stdout -F record_modifier -p 'Record=hostname ${HOSTNAME}' -p 'Record=product Awesome_Tool' -m '*'
The output will be
Copy [0] mem.local: [1492436882.000000000, {"Mem.total"=>1016024, "Mem.used"=>716672, "Mem.free"=>299352, "Swap.total"=>2064380, "Swap.used"=>32656, "Swap.free"=>2031724, "hostname"=>"localhost.localdomain", "product"=>"Awesome_Tool"}]
Remove fields with Remove_key
The following configuration file is to remove 'Swap.*' fields.
fluent-bit.conf fluent-bit.yaml
Copy [INPUT]
Name mem
Tag mem . local
[OUTPUT]
Name stdout
Match *
[FILTER]
Name record_modifier
Match *
Remove_key Swap . total
Remove_key Swap . used
Remove_key Swap . free
Copy pipeline :
inputs :
- name : mem
tag : mem.local
filters :
- name : record_modifier
match : '*'
remove_key :
- Swap.total
- Swap.used
- Swap.free
outputs :
- name : stdout
match : '*'
You can also run the filter from command line.
Copy $ fluent-bit -i mem -o stdout -F record_modifier -p 'Remove_key=Swap.total' -p 'Remove_key=Swap.free' -p 'Remove_key=Swap.used' -m '*'
The output will be
Copy [ 0 ] mem . local : [ 1492436998.000000000 , { "Mem.total" = > 1016024 , "Mem.used" = > 716672 , "Mem.free" = > 295332 } ]
Remove fields with Allowlist_key
The following configuration file is to remain 'Mem.*' fields.
fluent-bit.conf fluent-bit.yaml
Copy [INPUT]
Name mem
Tag mem . local
[OUTPUT]
Name stdout
Match *
[FILTER]
Name record_modifier
Match *
Allowlist_key Mem . total
Allowlist_key Mem . used
Allowlist_key Mem . free
Copy pipeline :
inputs :
- name : mem
tag : mem.local
filters :
- name : record_modifier
match : '*'
Allowlist_key :
- Mem.total
- Mem.used
- Mem.free
outputs :
- name : stdout
match : '*'
You can also run the filter from command line.
Copy $ fluent-bit -i mem -o stdout -F record_modifier -p 'Allowlist_key=Mem.total' -p 'Allowlist_key=Mem.free' -p 'Allowlist_key=Mem.used' -m '*'
The output will be
Copy [ 0 ] mem . local : [ 1492436998.000000000 , { "Mem.total" = > 1016024 , "Mem.used" = > 716672 , "Mem.free" = > 295332 } ]
Last updated 4 months ago