Record modifier

The Record Modifier filter lets you append fields to a record, or exclude specific fields.

Configuration parameters

The plugin supports the following configuration parameters:

Key	Description
allowlist_key	If the key isn't matched, that field is removed. You can use this or remove_key.
record	Append fields. This parameter needs a key/value pair.
remove_key	If the key is matched, that field is removed. You can use this or allowlist_key.
uuid_key	If set, the plugin appends UUID to each record. The value assigned becomes the key in the map.
whitelist_key	An alias of allowlist_key for backwards compatibility.

Get started

To start filtering records, run the filter from the command line or through a configuration file.

This is a sample in_mem record to filter.

{"Mem.total"=>1016024, "Mem.used"=>716672, "Mem.free"=>299352, "Swap.total"=>2064380, "Swap.used"=>32656, "Swap.free"=>2031724}

Append fields

The following configuration file appends a product name and hostname to a record using an environment variable:

fluent-bit.yaml

pipeline:
  inputs:
    - name: mem
      tag: mem.local

  filters:
    - name: record_modifier
      match: '*'
      record:
       - hostname ${HOSTNAME}
       - product Awesome_Tool

  outputs:
    - name: stdout
      match: '*'

fluent-bit.conf

You can run the filter from command line:

fluent-bit -i mem -o stdout -F record_modifier -p 'Record=hostname ${HOSTNAME}' -p 'Record=product Awesome_Tool' -m '*'

The output looks something like:

[0] mem.local: [1492436882.000000000, {"Mem.total"=>1016024, "Mem.used"=>716672, "Mem.free"=>299352, "Swap.total"=>2064380, "Swap.used"=>32656, "Swap.free"=>2031724, "hostname"=>"localhost.localdomain", "product"=>"Awesome_Tool"}]

Remove fields with Remove_key

The following configuration file removes Swap.* fields:

fluent-bit.yaml

pipeline:
  inputs:
    - name: mem
      tag: mem.local

  filters:
    - name: record_modifier
      match: '*'
      remove_key:
       - Swap.total
       - Swap.used
       - Swap.free

  outputs:
    - name: stdout
      match: '*'

fluent-bit.conf

You can also run the filter from command line.

fluent-bit -i mem -o stdout -F  record_modifier -p 'Remove_key=Swap.total' -p 'Remove_key=Swap.free' -p 'Remove_key=Swap.used' -m '*'

The output looks something like:

[0] mem.local: [1492436998.000000000, {"Mem.total"=>1016024, "Mem.used"=>716672, "Mem.free"=>295332}]

Retain fields with Allowlist_key

The following configuration file retains Mem.* fields.

fluent-bit.yaml

pipeline:
  inputs:
    - name: mem
      tag: mem.local

  filters:
    - name: record_modifier
      match: '*'
      allowlist_key:
       - Mem.total
       - Mem.used
       - Mem.free

  outputs:
    - name: stdout
      match: '*'

fluent-bit.conf

You can also run the filter from command line:

fluent-bit -i mem -o stdout -F record_modifier -p 'Allowlist_key=Mem.total' -p 'Allowlist_key=Mem.free' -p 'Allowlist_key=Mem.used' -m '*'

The output looks something like:

[0] mem.local: [1492436998.000000000, {"Mem.total"=>1016024, "Mem.used"=>716672, "Mem.free"=>295332}]