SlackGitHubCommunity MeetingsSandboxWebinars

Exec

The Exec input plugin lets you execute external programs and collects event logs.

This plugin invokes commands using a shell. Its inputs are subject to shell metacharacter substitution. Careless use of untrusted input in command arguments could lead to malicious command execution.

Container support

This plugin needs a functional /bin/sh and won't function in all the distro-less production images.

The debug images use the same binaries so even though they have a shell, there is no support for this plugin as it's compiled out.

Configuration parameters

The plugin supports the following configuration parameters:

Key
Description

Command

The command to execute, passed to popen without any additional escaping or processing. Can include pipelines, redirection, command-substitution, or other information.

Parser

Specify the name of a parser to interpret the entry as a structured message.

Interval_Sec

Polling interval (seconds).

Interval_NSec

Polling interval (nanosecond).

Buf_Size

Size of the buffer. See unit sizes for allowed values.

Oneshot

Only run once at startup. This allows collection of data precedent to Fluent Bit startup (Boolean, default: false).

Exit_After_Oneshot

Exit as soon as the one-shot command exits. This allows the exec plugin to be used as a wrapper for another command, sending the target command's output to any Fluent Bit sink, then exits. (Boolean, default: false).

Propagate_Exit_Code

When exiting due to Exit_After_Oneshot, cause Fluent Bit to exit with the exit code of the command exited by this plugin. Follows shell conventions for exit code propagation. (Boolean, default: false).

Threaded

Indicates whether to run this input in its own thread. Default: false.

Get started

You can run the plugin from the command line or through the configuration file:

Command line

The following example will read events from the output of ls.

fluent-bit -i exec -p 'command=ls /var/log' -o stdout

which should return something like the following:

Fluent Bit v1.x.x
* Copyright (C) 2019-2020 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2018/03/21 17:46:49] [ info] [engine] started
[0] exec.0: [1521622010.013470159, {"exec"=>"ConsoleKit"}]
[1] exec.0: [1521622010.013490313, {"exec"=>"Xorg.0.log"}]
[2] exec.0: [1521622010.013492079, {"exec"=>"Xorg.0.log.old"}]
[3] exec.0: [1521622010.013493443, {"exec"=>"anaconda.ifcfg.log"}]
[4] exec.0: [1521622010.013494707, {"exec"=>"anaconda.log"}]
[5] exec.0: [1521622010.013496016, {"exec"=>"anaconda.program.log"}]
[6] exec.0: [1521622010.013497225, {"exec"=>"anaconda.storage.log"}]

Configuration file

In your main configuration file append the following:

pipeline:
    inputs:
        - name: exec
          tag: exec_ls
          command: ls /var/log
          interval_sec: 1
          interval_nsec: 0
          buf_size: 8mb
          oneshot: false

    outputs:
        - name: stdout
          match: '*'

Use as a command wrapper

To use Fluent Bit with the exec plugin to wrap another command, use the Exit_After_Oneshot and Propagate_Exit_Code options:

pipeline:
    inputs:
        - name: exec
          tag: exec_oneshot_demo
          command: 'for s in $(seq 1 10); do echo "count: $s"; sleep 1; done; exit 1'
          oneshot: true
          exit_after_oneshot: true
          propagate_exit_code: true

    outputs:
        - name: stdout
          match: '*'

Fluent Bit will output:

[0] exec_oneshot_demo: [[1681702172.950574027, {}], {"exec"=>"count: 1"}]
[1] exec_oneshot_demo: [[1681702173.951663666, {}], {"exec"=>"count: 2"}]
[2] exec_oneshot_demo: [[1681702174.953873724, {}], {"exec"=>"count: 3"}]
[3] exec_oneshot_demo: [[1681702175.955760865, {}], {"exec"=>"count: 4"}]
[4] exec_oneshot_demo: [[1681702176.956840282, {}], {"exec"=>"count: 5"}]
[5] exec_oneshot_demo: [[1681702177.958292246, {}], {"exec"=>"count: 6"}]
[6] exec_oneshot_demo: [[1681702178.959508200, {}], {"exec"=>"count: 7"}]
[7] exec_oneshot_demo: [[1681702179.961715745, {}], {"exec"=>"count: 8"}]
[8] exec_oneshot_demo: [[1681702180.963924140, {}], {"exec"=>"count: 9"}]
[9] exec_oneshot_demo: [[1681702181.965852990, {}], {"exec"=>"count: 10"}]

then exits with exit code 1.

Translation of command exit codes to Fluent Bit exit code follows the usual shell rules for exit code handling. Like with a shell, there is no way to differentiate between the command exiting on a signal and the shell exiting on a signal. Similarly, there is no way to differentiate between normal exits with codes greater than 125 and abnormal or signal exits reported by Fluent Bit or the shell. Wrapped commands should use exit codes between 0 and 125 inclusive to allow reliable identification of normal exit. If the command is a pipeline, the exit code will be the exit code of the last command in the pipeline unless overridden by shell options.

Parsing command output

By default the exec plugin emits one message per command output line, with a single field exec containing the full message. Use the Parser directive to specify the name of a parser configuration to use to process the command input.

Security concerns

Take great care with shell quoting and escaping when wrapping commands.

A script like the following can ruin your day if someone passes it the argument $(rm -rf /my/important/files; echo "deleted your stuff!")'

#!/bin/bash
# This is a DANGEROUS example of what NOT to do, NEVER DO THIS
exec fluent-bit \
  -o stdout \
  -i exec \
  -p exit_after_oneshot=true \
  -p propagate_exit_code=true \
  -p command='myscript $*'

The previous script would be safer if written with:

  -p command='echo '"$(printf '%q' "$@")" \

It's generally best to avoid dynamically generating the command or handling untrusted arguments.

Last updated

Was this helpful?