false
.upload_chunk_size
is reached.s3_key_format_tag_delimiters
option. Add extension directly after the last piece of the format string to insert a key suffix. If you want to specify a key suffix and you are in use_put_object
mode, you must specify $UUID as well. More explanations can be found in use_put_object
option. See the in depth examples and tutorial in the documentation.s3_key_format
. If $UUID is not included, a random string will be appended at the end of the format string and the key extension cannot be customized in this case.use_put_object
is on. If Apache Arrow support was enabled at compile time, you can set 'arrow' to this option.tls.verify
as false
. For more details about the properties available and general configuration, please refer to the TLS/SSL section.s3:PutObject
permission.s3_key_format
option lets you inject the tag into the s3 key using the following syntax:$TAG
=> the full tag$TAG[n]
=> the nth part of the tag (index starting at zero). This syntax is copied from the rewrite tag filter. By default, “parts” of the tag are separated with dots, but you can change this with s3_key_format_tag_delimiters
.my_app_name-logs.prod
.$TAG[0]
= my_app_name$TAG[1]
= logs$TAG[2]
= prod/prod/my_app_name/2020/01/01/00/00/00/bgdHN1NM.gz
.store_dir
is used to temporarily store data before it is uploaded. If Fluent Bit is stopped suddenly it will try to send all data and complete all uploads before it shuts down. If it can not send some data, on restart it will look in the store_dir
for existing data and will try to send it.store_dir
, ensuring that uploads can be completed when Fluent Bit restarts (assuming it has access to persistent disk and the store_dir
files will still be present on restart).store_dir
from previous executions- some considerations apply. This might occur if you run Fluent Bit on AWS Fargate.workers
which enables outputs to have dedicated threads. This s3
plugin has partial support for workers. The plugin can only support a single worker; enabling multiple workers will lead to errors/indeterminate behavior.your-bucket
by referring the client docs.libarrow
as the prerequisite.FLB_ARROW
must be turned on at compile time:Compression
to arrow
makes Fluent Bit to convert payload into Apache Arrow format.pyarrow
to analyze the uploaded data: