Amazon Kinesis Data Streams

Send logs to Amazon Kinesis Streams

The Amazon Kinesis Data Streams output plugin lets you ingest your records into the Kinesis service.

This is the documentation for the core Fluent Bit Kinesis plugin written in C. It has all the core features of the aws/amazon-kinesis-streams-for-fluent-bit Golang Fluent Bit plugin released in 2019. The original Golang plugin is named kinesis, and this new high performance and highly efficient Kinesis plugin is called kinesis_streams to prevent conflicts or confusion.

This kinesis_streams plugin always uses a random partition key when uploading records to Kinesis through the PutRecords API.

For information about how AWS credentials are fetched, see AWS credentials.

Configuration parameters

Key	Description	Default
region	The AWS region.	none
stream	The name of the Kinesis Streams Delivery stream that you want log records sent to.	none
time_key	Add the timestamp to the record under this key.	false
time_key_format	The strftime compliant format string for the timestamp. Supports millisecond precision with %3N and supports nanosecond precision with %9N and %L. For example, adding %3N to support millisecond %Y-%m-%dT%H:%M:%S.%3N. This option is used with time_key.	%Y-%m-%dT%H:%M:%S
log_key	By default, the whole log record will be sent to Kinesis. If you specify a key name with this option, then only the value of that key will be sent to Kinesis. For example, if you are using the Fluentd Docker log driver, you can specify log_key log and only the log message will be sent to Kinesis.	none
role_arn	ARN of an IAM role to assume (for cross account access).	none
endpoint	Specify a custom endpoint for the Kinesis API.	none
port	TCP port of the Kinesis Streams service.	443
sts_endpoint	Custom endpoint for the STS API.	none
auto_retry_requests	Immediately retry failed requests to AWS services once. This option doesn't affect the normal Fluent Bit retry mechanism with backoff. Instead, it enables an immediate retry with no delay for networking errors, which might help improve throughput when there are transient/random networking issues.	true
external_id	Specify an external ID for the STS API, can be used with the role_arn parameter if your role requires an external ID.	none
profile	AWS profile name to use.	default
workers	The number of workers to perform flush operations for this output.	1

Get started

To send records into Amazon Kinesis Data Streams, you can run the plugin from the command line or through the configuration file.

Command line

The kinesis_streams plugin can read the parameters from the command line through the -p (property) argument. For example:

fluent-bit -i cpu -o kinesis_streams -p stream=my-stream -p region=us-west-2 -m '*' -f 1

Configuration file

In your main configuration file append the following:

fluent-bit.yaml

pipeline:

  outputs:
    - name: kinesis_steams
      match: '*'
      region: us-east-1
      stream: my-stream

fluent-bit.conf

[OUTPUT]
  Name  kinesis_streams
  Match *
  region us-east-1
  stream my-stream

Permissions

The following AWS IAM permissions are required to use this plugin:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": [
      "kinesis:PutRecords"
    ],
    "Resource": "*"
  }]
}

Container images

Amazon distributes a container image with Fluent Bit and these plugins.

GitHub

The aws-for-fluent-bit container image is available on GitHub.

Amazon ECR Public Gallery

The aws-for-fluent-bit container image is available on the Amazon ECR Public Gallery. Use the following command to download images with different tags:

docker pull public.ecr.aws/aws-observability/aws-for-fluent-bit:TAG

Replace TAG with the tag of the image you want to download. You can also use the value latest to download the latest image.

If you see errors for image pull limits, try to log into the gallery with your AWS credentials:

aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws

You can use the Fluent Bit SSM Public Parameters to find the Amazon ECR image URI in your region:

aws ssm get-parameters-by-path --path /aws/service/aws-for-fluent-bit/

For more details, see the Amazon ECR Public official doc.

Docker Hub

The aws-for-fluent-bit container image is available on Docker Hub.

More information

For more information, see the aws-for-fluent-bit README on GitHub.