Record Modifier The Record Modifier Filter plugin allows to append fields or to exclude specific fields.
Configuration Parameters
The plugin supports the following configuration parameters: Remove_key and Allowlist_key are exclusive.
Getting Started
In order to start filtering records, you can run the filter from the command line or through the configuration file.
This is a sample in_mem record to filter.
Copy {"Mem.total"=>1016024, "Mem.used"=>716672, "Mem.free"=>299352, "Swap.total"=>2064380, "Swap.used"=>32656, "Swap.free"=>2031724} Append fields
The following configuration file is to append product name and hostname (via environment variable) to record.
fluent-bit.conf
Copy [INPUT]
Name mem
Tag mem . local
[OUTPUT]
Name stdout
Match *
[FILTER]
Name record_modifier
Match *
Record hostname $ { HOSTNAME }
Record product Awesome_Tool fluent-bit.yaml
Copy pipeline :
inputs :
- name : mem
tag : mem.local
filters :
- name : record_modifier
match : '*'
record :
- hostname ${HOSTNAME}
- product Awesome_Tool
outputs :
- name : stdout
match : '*' You can also run the filter from command line.
Copy $ fluent-bit -i mem -o stdout -F record_modifier -p 'Record=hostname ${HOSTNAME}' -p 'Record=product Awesome_Tool' -m '*' The output will be
Copy [0] mem.local: [1492436882.000000000, {"Mem.total"=>1016024, "Mem.used"=>716672, "Mem.free"=>299352, "Swap.total"=>2064380, "Swap.used"=>32656, "Swap.free"=>2031724, "hostname"=>"localhost.localdomain", "product"=>"Awesome_Tool"}] Remove fields with Remove_key
The following configuration file is to remove 'Swap.*' fields.
fluent-bit.conf
Copy [INPUT]
Name mem
Tag mem . local
[OUTPUT]
Name stdout
Match *
[FILTER]
Name record_modifier
Match *
Remove_key Swap . total
Remove_key Swap . used
Remove_key Swap . free fluent-bit.yaml
Copy pipeline :
inputs :
- name : mem
tag : mem.local
filters :
- name : record_modifier
match : '*'
remove_key :
- Swap.total
- Swap.used
- Swap.free
outputs :
- name : stdout
match : '*' You can also run the filter from command line.
Copy $ fluent-bit -i mem -o stdout -F record_modifier -p 'Remove_key=Swap.total' -p 'Remove_key=Swap.free' -p 'Remove_key=Swap.used' -m '*' The output will be
Copy [ 0 ] mem . local : [ 1492436998.000000000 , { "Mem.total" = > 1016024 , "Mem.used" = > 716672 , "Mem.free" = > 295332 } ] Remove fields with Allowlist_key
The following configuration file is to remain 'Mem.*' fields.
fluent-bit.conf
Copy [INPUT]
Name mem
Tag mem . local
[OUTPUT]
Name stdout
Match *
[FILTER]
Name record_modifier
Match *
Allowlist_key Mem . total
Allowlist_key Mem . used
Allowlist_key Mem . free fluent-bit.yaml
Copy pipeline :
inputs :
- name : mem
tag : mem.local
filters :
- name : record_modifier
match : '*'
Allowlist_key :
- Mem.total
- Mem.used
- Mem.free
outputs :
- name : stdout
match : '*' You can also run the filter from command line.
Copy $ fluent-bit -i mem -o stdout -F record_modifier -p 'Allowlist_key=Mem.total' -p 'Allowlist_key=Mem.free' -p 'Allowlist_key=Mem.used' -m '*' The output will be
Copy [ 0 ] mem . local : [ 1492436998.000000000 , { "Mem.total" = > 1016024 , "Mem.used" = > 716672 , "Mem.free" = > 295332 } ]