Record Modifier

The Record Modifier Filter plugin allows to append fields or to exclude specific fields.

Configuration Parameters

The plugin supports the following configuration parameters: Remove_key and Allowlist_key are exclusive.

Key	Description
Record	Append fields. This parameter needs key and value pair.
Remove_key	If the key is matched, that field is removed.
Allowlist_key	If the key is not matched, that field is removed.
Whitelist_key	An alias of Allowlist_key for backwards compatibility.
Uuid_key	If set, the plugin appends uuid to each record. The value assigned becomes the key in the map.

Getting Started

In order to start filtering records, you can run the filter from the command line or through the configuration file.

This is a sample in_mem record to filter.

{"Mem.total"=>1016024, "Mem.used"=>716672, "Mem.free"=>299352, "Swap.total"=>2064380, "Swap.used"=>32656, "Swap.free"=>2031724}

Append fields

The following configuration file is to append product name and hostname (via environment variable) to record.

fluent-bit.conf

[INPUT]
    Name mem
    Tag  mem.local

[OUTPUT]
    Name  stdout
    Match *

[FILTER]
    Name record_modifier
    Match *
    Record hostname ${HOSTNAME}
    Record product Awesome_Tool

fluent-bit.yaml

pipeline:
    inputs:
        - name: mem
          tag: mem.local
    filters:
        - name: record_modifier
          match: '*'
          record: 
             - hostname ${HOSTNAME}
             - product Awesome_Tool
    outputs:
        - name: stdout
          match: '*'

You can also run the filter from command line.

$ fluent-bit -i mem -o stdout -F record_modifier -p 'Record=hostname ${HOSTNAME}' -p 'Record=product Awesome_Tool' -m '*'

The output will be

[0] mem.local: [1492436882.000000000, {"Mem.total"=>1016024, "Mem.used"=>716672, "Mem.free"=>299352, "Swap.total"=>2064380, "Swap.used"=>32656, "Swap.free"=>2031724, "hostname"=>"localhost.localdomain", "product"=>"Awesome_Tool"}]

Remove fields with Remove_key

The following configuration file is to remove 'Swap.*' fields.

fluent-bit.conf

[INPUT]
    Name mem
    Tag  mem.local

[OUTPUT]
    Name  stdout
    Match *

[FILTER]
    Name record_modifier
    Match *
    Remove_key Swap.total
    Remove_key Swap.used
    Remove_key Swap.free

fluent-bit.yaml

pipeline:
    inputs:
        - name: mem
          tag: mem.local
    filters:
        - name: record_modifier
          match: '*'
          remove_key: 
             - Swap.total
             - Swap.used
             - Swap.free
    outputs:
        - name: stdout
          match: '*'

You can also run the filter from command line.

$ fluent-bit -i mem -o stdout -F  record_modifier -p 'Remove_key=Swap.total' -p 'Remove_key=Swap.free' -p 'Remove_key=Swap.used' -m '*'

The output will be

[0] mem.local: [1492436998.000000000, {"Mem.total"=>1016024, "Mem.used"=>716672, "Mem.free"=>295332}]

Remove fields with Allowlist_key

The following configuration file is to remain 'Mem.*' fields.

fluent-bit.conf

[INPUT]
    Name mem
    Tag  mem.local

[OUTPUT]
    Name  stdout
    Match *

[FILTER]
    Name record_modifier
    Match *
    Allowlist_key Mem.total
    Allowlist_key Mem.used
    Allowlist_key Mem.free

fluent-bit.yaml

pipeline:
    inputs:
        - name: mem
          tag: mem.local
    filters:
        - name: record_modifier
          match: '*'
          Allowlist_key: 
             - Mem.total
             - Mem.used
             - Mem.free
    outputs:
        - name: stdout
          match: '*'

You can also run the filter from command line.

$ fluent-bit -i mem -o stdout -F  record_modifier -p 'Allowlist_key=Mem.total' -p 'Allowlist_key=Mem.free' -p 'Allowlist_key=Mem.used' -m '*'

The output will be

[0] mem.local: [1492436998.000000000, {"Mem.total"=>1016024, "Mem.used"=>716672, "Mem.free"=>295332}]