s3_key_formatoption lets you inject the tag into the s3 key using the following syntax:
$TAG=> the full tag
$TAG[n]=> the nth part of the tag (index starting at zero). This syntax is copied from the rewrite tag filter. By default, “parts” of the tag are separated with dots, but you can change this with
store_diris used to temporarily store data before it is uploaded. If Fluent Bit is stopped suddenly it will try to send all data and complete all uploads before it shuts down. If it can not send some data, on restart it will look in the
store_dirfor existing data and will try to send it.
store_dir, ensuring that uploads can be completed when Fluent Bit restarts (assuming it has access to persistent disk and the
store_dirfiles will still be present on restart).
store_dirfrom previous executions- some considerations apply. This might occur if you run Fluent Bit on AWS Fargate.
workerswhich enables outputs to have dedicated threads. This
s3plugin has partial support for workers. The plugin can only support a single worker; enabling multiple workers will lead to errors/indeterminate behavior.
libarrowas the prerequisite.
FLB_ARROWmust be turned on at compile time:
arrowmakes Fluent Bit to convert payload into Apache Arrow format.
pyarrowto analyze the uploaded data: