The Record Modifier Filter plugin allows to append fields or to exclude specific fields.
The plugin supports the following configuration parameters: Remove_key and Whitelist_key are exclusive.
Key
Description
In order to start filtering records, you can run the filter from the command line or through the configuration file.
This is a sample in_mem record to filter.
The following configuration file is to append product name and hostname (via environment variable) to record.
You can also run the filter from command line.
The output will be
The following configuration file is to remove 'Swap.*' fields.
You can also run the filter from command line.
The output will be
The following configuration file is to remain 'Mem.*' fields.
You can also run the filter from command line.
The output will be
Record
Append fields. This parameter needs key and value pair.
Remove_key
If the key is matched, that field is removed.
Whitelist_key
If the key is not matched, that field is removed.
{"Mem.total"=>1016024, "Mem.used"=>716672, "Mem.free"=>299352, "Swap.total"=>2064380, "Swap.used"=>32656, "Swap.free"=>2031724}[INPUT]
Name mem
Tag mem.local
[OUTPUT]
Name stdout
Match *
[FILTER]
Name record_modifier
Match *
Record hostname ${HOSTNAME}
Record product Awesome_Tool$ fluent-bit -i mem -o stdout -F record_modifier -p 'Record=hostname ${HOSTNAME}' -p 'Record=product Awesome_Tool' -m '*'[0] mem.local: [1492436882.000000000, {"Mem.total"=>1016024, "Mem.used"=>716672, "Mem.free"=>299352, "Swap.total"=>2064380, "Swap.used"=>32656, "Swap.free"=>2031724, "hostname"=>"localhost.localdomain", "product"=>"Awesome_Tool"}][INPUT]
Name mem
Tag mem.local
[OUTPUT]
Name stdout
Match *
[FILTER]
Name record_modifier
Match *
Remove_key Swap.total
Remove_key Swap.used
Remove_key Swap.free$ fluent-bit -i mem -o stdout -F record_modifier -p 'Remove_key=Swap.total' -p 'Remove_key=Swap.free' -p 'Remove_key=Swap.used' -m '*'[0] mem.local: [1492436998.000000000, {"Mem.total"=>1016024, "Mem.used"=>716672, "Mem.free"=>295332}][INPUT]
Name mem
Tag mem.local
[OUTPUT]
Name stdout
Match *
[FILTER]
Name record_modifier
Match *
Whitelist_key Mem.total
Whitelist_key Mem.used
Whitelist_key Mem.free$ fluent-bit -i mem -o stdout -F record_modifier -p 'Whitelist_key=Mem.total' -p 'Whitelist_key=Mem.free' -p 'Whitelist_key=Mem.used' -m '*'[0] mem.local: [1492436998.000000000, {"Mem.total"=>1016024, "Mem.used"=>716672, "Mem.free"=>295332}]