The stdin plugin supports retrieving a message stream from the standard input interface (stdin) of the Fluent Bit process. In order to use it, specify the plugin name as the input, e.g:
If the stdin stream is closed (end-of-file), the stdin plugin will instruct Fluent Bit to exit with success (0) after flushing any pending output.
If no parser is configured for the stdin plugin, it expects valid JSON input data in one of the following formats:
A JSON object with one or more key-value pairs: { "key": "value", "key2": "value2" }
A 2-element JSON array in format, which may be:
[TIMESTAMP, { "key": "value" }] where TIMESTAMP is a floating point value representing a timestamp in seconds; or
from Fluent Bit v2.1.0, [[TIMESTAMP, METADATA], { "key": "value" }] where TIMESTAMP has the same meaning as above and and METADATA is a JSON object.
Multi-line input JSON is supported.
Any input data that is not in one of the above formats will cause the plugin to log errors like:
To handle inputs in other formats, a parser must be explicitly specified in the configuration for the stdin plugin. See for sample configuration.
The Fluent Bit event timestamp will be set from the input record if the 2-element event input is used or a custom parser configuration supplies a timestamp. Otherwise the event timestamp will be set to the timestamp at which the record is read by the stdin plugin.
A better example to demonstrate how it works will be through a Bash script that generates messages and writes them to . Write the following content in a file named test.sh:
Now lets start the script and :
An input event timestamp may also be supplied. Replace test.sh with:
Re-run the sample command. Note that the timestamps output by Fluent Bit are now one day old because Fluent Bit used the input message timestamp.
Additional metadata is also supported on Fluent Bit v2.1.0 and above by replacing the timestamp with a 2-element object, e.g.:
On older Fluent Bit versions records in this format will be discarded. Fluent Bit will log:
if the log level permits.
To capture inputs in other formats, specify a parser configuration for the stdin plugin.
For example, if you want to read raw messages line-by-line and forward them you could use a parser.conf that captures the whole message line:
then use that in the parser clause of the stdin plugin in the fluent-bit.conf:
Fluent Bit will now read each line and emit a single message for each input line:
In real-world deployments it is best to use a more realistic parser that splits messages into real fields and adds appropriate tags.
The plugin supports the following configuration parameters:
$ fluent-bit -i stdin -o stdoutBuffer_Size
Set the buffer size to read data. This value is used to increase buffer size. The value must be according to the Unit Size specification.
16k
Parser
The name of the parser to invoke instead of the default JSON input parser
Threaded
Indicates whether to run this input in its own thread.
false
[debug] [input:stdin:stdin.0] invalid JSON message, skipping
[error] [input:stdin:stdin.0] invalid record found, it's not a JSON map or array#!/bin/sh
for ((i=0; i<=5; i++)); do
echo -n "{\"key\": \"some value\"}"
sleep 1
done$ bash test.sh | fluent-bit -q -i stdin -o stdout
[0] stdin.0: [[1684196745.942883835, {}], {"key"=>"some value"}]
[0] stdin.0: [[1684196746.938949056, {}], {"key"=>"some value"}]
[0] stdin.0: [[1684196747.940162493, {}], {"key"=>"some value"}]
[0] stdin.0: [[1684196748.941392297, {}], {"key"=>"some value"}]
[0] stdin.0: [[1684196749.942644238, {}], {"key"=>"some value"}]
[0] stdin.0: [[1684196750.943721442, {}], {"key"=>"some value"}]#!/bin/sh
for ((i=0; i<=5; i++)); do
echo -n "
[
$(date '+%s.%N' -d '1 day ago'),
{
\"realtimestamp\": $(date '+%s.%N')
}
]
"
sleep 1
done$ bash test.sh | fluent-bit -q -i stdin -o stdout
[0] stdin.0: [[1684110480.028171300, {}], {"realtimestamp"=>1684196880.030070}]
[0] stdin.0: [[1684110481.033753395, {}], {"realtimestamp"=>1684196881.034741}]
[0] stdin.0: [[1684110482.036730051, {}], {"realtimestamp"=>1684196882.037704}]
[0] stdin.0: [[1684110483.039903879, {}], {"realtimestamp"=>1684196883.041081}]
[0] stdin.0: [[1684110484.044719457, {}], {"realtimestamp"=>1684196884.046404}]
[0] stdin.0: [[1684110485.048710107, {}], {"realtimestamp"=>1684196885.049651}]#!/bin/sh
for ((i=0; i<=5; i++)); do
echo -n "
[
[
$(date '+%s.%N' -d '1 day ago'),
{\"metakey\": \"metavalue\"}
],
{
\"realtimestamp\": $(date '+%s.%N')
}
]
"
sleep 1
done$ bash ./test.sh | fluent-bit -q -i stdin -o stdout
[0] stdin.0: [[1684110513.060139417, {"metakey"=>"metavalue"}], {"realtimestamp"=>1684196913.061017}]
[0] stdin.0: [[1684110514.063085317, {"metakey"=>"metavalue"}], {"realtimestamp"=>1684196914.064145}]
[0] stdin.0: [[1684110515.066210508, {"metakey"=>"metavalue"}], {"realtimestamp"=>1684196915.067155}]
[0] stdin.0: [[1684110516.069149971, {"metakey"=>"metavalue"}], {"realtimestamp"=>1684196916.070132}]
[0] stdin.0: [[1684110517.072484016, {"metakey"=>"metavalue"}], {"realtimestamp"=>1684196917.073636}]
[0] stdin.0: [[1684110518.075428724, {"metakey"=>"metavalue"}], {"realtimestamp"=>1684196918.076292}][ warn] unknown time format 6[PARSER]
name stringify_message
format regex
Key_Name message
regex ^(?<message>.*)[INPUT]
Name stdin
Tag stdin
Parser stringify_message
[OUTPUT]
Name stdout
Match *pipeline:
inputs:
- name: stdin
tag: stdin
parser: stringify_message
outputs:
- name: stdout
match: '*'$ seq 1 5 | /opt/fluent-bit/bin/fluent-bit -c fluent-bit.conf -R parser.conf -q
[0] stdin: [1681358780.517029169, {"message"=>"1"}]
[1] stdin: [1681358780.517068334, {"message"=>"2"}]
[2] stdin: [1681358780.517072116, {"message"=>"3"}]
[3] stdin: [1681358780.517074758, {"message"=>"4"}]
[4] stdin: [1681358780.517077392, {"message"=>"5"}]
$