Fluent Bit: Official Manual
SlackGitHubCommunity MeetingsSandbox and LabsWebinars
3.0
3.0
  • Fluent Bit v3.0 Documentation
  • About
    • What is Fluent Bit?
    • A Brief History of Fluent Bit
    • Fluentd & Fluent Bit
    • License
  • Concepts
    • Key Concepts
    • Buffering
    • Data Pipeline
      • Input
      • Parser
      • Filter
      • Buffer
      • Router
      • Output
  • Installation
    • Getting Started with Fluent Bit
    • Upgrade Notes
    • Supported Platforms
    • Requirements
    • Sources
      • Download Source Code
      • Build and Install
      • Build with Static Configuration
    • Linux Packages
      • Amazon Linux
      • Redhat / CentOS
      • Debian
      • Ubuntu
      • Raspbian / Raspberry Pi
    • Docker
    • Containers on AWS
    • Amazon EC2
    • Kubernetes
    • macOS
    • Windows
    • Yocto / Embedded Linux
    • Buildroot / Embedded Linux
  • Administration
    • Configuring Fluent Bit
      • Classic mode
        • Format and Schema
        • Configuration File
        • Variables
        • Commands
        • Upstream Servers
        • Record Accessor
      • YAML Configuration
        • Configuration File
      • Unit Sizes
      • Multiline Parsing
    • Transport Security
    • Buffering & Storage
    • Backpressure
    • Scheduling and Retries
    • Networking
    • Memory Management
    • Monitoring
    • HTTP Proxy
    • Hot Reload
    • Troubleshooting
  • Local Testing
    • Validating your Data and Structure
    • Running a Logging Pipeline Locally
  • Data Pipeline
    • Pipeline Monitoring
    • Inputs
      • Collectd
      • CPU Log Based Metrics
      • Disk I/O Log Based Metrics
      • Docker Log Based Metrics
      • Docker Events
      • Dummy
      • Elasticsearch
      • Exec
      • Exec Wasi
      • Fluent Bit Metrics
      • Forward
      • Head
      • HTTP
      • Health
      • Kafka
      • Kernel Logs
      • Kubernetes Events
      • Memory Metrics
      • MQTT
      • Network I/O Log Based Metrics
      • NGINX Exporter Metrics
      • Node Exporter Metrics
      • Podman Metrics
      • Process Log Based Metrics
      • Process Exporter Metrics
      • Prometheus Scrape Metrics
      • Prometheus Remote Write
      • Random
      • Serial Interface
      • Splunk
      • Standard Input
      • StatsD
      • Syslog
      • Systemd
      • Tail
      • TCP
      • Thermal
      • UDP
      • OpenTelemetry
      • Windows Event Log
      • Windows Event Log (winevtlog)
      • Windows Exporter Metrics
    • Parsers
      • Configuring Parser
      • JSON
      • Regular Expression
      • LTSV
      • Logfmt
      • Decoders
    • Processors
      • Content Modifier
      • Metrics Selector
      • SQL
    • Filters
      • AWS Metadata
      • CheckList
      • ECS Metadata
      • Expect
      • GeoIP2 Filter
      • Grep
      • Kubernetes
      • Log to Metrics
      • Lua
      • Parser
      • Record Modifier
      • Modify
      • Multiline
      • Nest
      • Nightfall
      • Rewrite Tag
      • Standard Output
      • Sysinfo
      • Throttle
      • Type Converter
      • Tensorflow
      • Wasm
    • Outputs
      • Amazon CloudWatch
      • Amazon Kinesis Data Firehose
      • Amazon Kinesis Data Streams
      • Amazon S3
      • Azure Blob
      • Azure Data Explorer
      • Azure Log Analytics
      • Azure Logs Ingestion API
      • Counter
      • Datadog
      • Elasticsearch
      • File
      • FlowCounter
      • Forward
      • GELF
      • Google Chronicle
      • Google Cloud BigQuery
      • HTTP
      • InfluxDB
      • Kafka
      • Kafka REST Proxy
      • LogDNA
      • Loki
      • Microsoft Fabric
      • NATS
      • New Relic
      • NULL
      • Observe
      • Oracle Log Analytics
      • OpenSearch
      • OpenTelemetry
      • PostgreSQL
      • Prometheus Exporter
      • Prometheus Remote Write
      • SkyWalking
      • Slack
      • Splunk
      • Stackdriver
      • Standard Output
      • Syslog
      • TCP & TLS
      • Treasure Data
      • Vivo Exporter
      • WebSocket
  • Stream Processing
    • Introduction to Stream Processing
    • Overview
    • Changelog
    • Getting Started
      • Fluent Bit + SQL
      • Check Keys and NULL values
      • Hands On! 101
  • Fluent Bit for Developers
    • C Library API
    • Ingest Records Manually
    • Golang Output Plugins
    • WASM Filter Plugins
    • WASM Input Plugins
    • Developer guide for beginners on contributing to Fluent Bit
Powered by GitBook
On this page
  • Configuration Parameters
  • Configuration File
  • Resource Labels
  • Using the resource_labels parameter
  • Troubleshooting Notes
  • Upstream connection error
  • Fail to process local_resource_id
  • Occasional Crashing with >1 Workers
  • Other implementations

Was this helpful?

Export as PDF
  1. Data Pipeline
  2. Outputs

Stackdriver

Last updated 11 months ago

Was this helpful?

Stackdriver output plugin allows to ingest your records into service.

Before to get started with the plugin configuration, make sure to obtain the proper credentials to get access to the service. We strongly recommend to use a common JSON credentials file, reference link:

Your goal is to obtain a credentials JSON file that will be used later by Fluent Bit Stackdriver output plugin.

Configuration Parameters

Key
Description
default

google_service_credentials

Absolute path to a Google Cloud credentials JSON file

Value of environment variable $GOOGLE_APPLICATION_CREDENTIALS

service_account_email

Account email associated to the service. Only available if no credentials file has been provided.

Value of environment variable $SERVICE_ACCOUNT_EMAIL

service_account_secret

Private key content associated with the service account. Only available if no credentials file has been provided.

Value of environment variable $SERVICE_ACCOUNT_SECRET

metadata_server

Prefix for a metadata server. Can also set environment variable $METADATA_SERVER.

location

The GCP or AWS region in which to store data about the resource. If the resource type is one of the generic_node or generic_task, then this field is required.

namespace

A namespace identifier, such as a cluster name or environment. If the resource type is one of the generic_node or generic_task, then this field is required.

node_id

A unique identifier for the node within the namespace, such as hostname or IP address. If the resource type is generic_node, then this field is required.

job

An identifier for a grouping of related task, such as the name of a microservice or distributed batch. If the resource type is generic_task, then this field is required.

task_id

A unique identifier for the task within the namespace and job, such as a replica index identifying the task within the job. If the resource type is generic_task, then this field is required.

export_to_project_id

The GCP project that should receive these logs.

Defaults to the project ID of the google_service_credentials file, or the project_id from Google's metadata.google.internal server.

resource

Set resource type of data. Supported resource types: k8s_container, k8s_node, k8s_pod, k8s_cluster, global, generic_node, generic_task, and gce_instance.

global, gce_instance

k8s_cluster_name

The name of the cluster that the container (node or pod based on the resource type) is running in. If the resource type is one of the k8s_container, k8s_node or k8s_pod, then this field is required.

k8s_cluster_location

The physical location of the cluster that contains (node or pod based on the resource type) the container. If the resource type is one of the k8s_container, k8s_node or k8s_pod, then this field is required.

labels_key

The value of this field is used by the Stackdriver output plugin to find the related labels from jsonPayload and then extract the value of it to set the LogEntry Labels.

labels

Optional list of comma separated of strings specifying key=value pairs. The resulting labels will be combined with the elements in obtained from labels_key to set the LogEntry Labels. Elements from labels will override duplicate values from labels_key.

log_name_key

The value of this field is used by the Stackdriver output plugin to extract logName from jsonPayload and set the logName field.

tag_prefix

Set the tag_prefix used to validate the tag of logs with k8s resource type. Without this option, the tag of the log must be in format of k8s_container(pod/node).* in order to use the k8s_container resource type. Now the tag prefix is configurable by this option (note the ending dot).

k8s_container., k8s_pod., k8s_node.

severity_key

Specify the name of the key from the original record that contains the severity information.

project_id_key

The value of this field is used by the Stackdriver output plugin to find the gcp project id from jsonPayload and then extract the value of it to set the PROJECT_ID within LogEntry logName, which controls the gcp project that should receive these logs.

autoformat_stackdriver_trace

Rewrite the trace field to include the projectID and format it for use with Cloud Trace. When this flag is enabled, the user can get the correct result by printing only the traceID (usually 32 characters).

false

Workers

Enables dedicated thread(s) for this output.

1

custom_k8s_regex

Set a custom regex to extract field like pod_name, namespace_name, container_name and docker_id from the local_resource_id in logs. This is helpful if the value of pod or node name contains dots.

(?<pod_name>[a-z0-9](?:[-a-z0-9]*[a-z0-9])?(?:\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-(?<docker_id>[a-z0-9]{64})\.log$

resource_labels

compress

Set payload compression mechanism. The only available option is gzip. Default = "", which means no compression.

Configuration File

If you are using a Google Cloud Credentials File, the following configuration is enough to get started:

[INPUT]
    Name  cpu
    Tag   cpu

[OUTPUT]
    Name        stackdriver
    Match       *

Example configuration file for k8s resource type:

local_resource_id is used by stackdriver output plugin to set the labels field for different k8s resource types. Stackdriver plugin will try to find the local_resource_id field in the log entry. If there is no field logging.googleapis.com/local_resource_id in the log, the plugin will then construct it by using the tag value of the log.

The local_resource_id should be in format:

  • k8s_container.<namespace_name>.<pod_name>.<container_name>

  • k8s_node.<node_name>

  • k8s_pod.<namespace_name>.<pod_name>

This implies that if there is no local_resource_id in the log entry then the tag of logs should match this format. Note that we have an option tag_prefix so it is not mandatory to use k8s_container(node/pod) as the prefix for tag.

[INPUT]
    Name               tail
    Tag_Regex          var.log.containers.(?<pod_name>[a-z0-9](?:[-a-z0-9]*[a-z0-9])?(?:\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-(?<docker_id>[a-z0-9]{64})\.log$
    Tag                custom_tag.<namespace_name>.<pod_name>.<container_name>
    Path               /var/log/containers/*.log
    Parser             docker
    DB                 /var/log/fluent-bit-k8s-container.db

[OUTPUT]
    Name        stackdriver
    Match       custom_tag.*
    Resource    k8s_container
    k8s_cluster_name test_cluster_name
    k8s_cluster_location  test_cluster_location
    tag_prefix  custom_tag.

Resource Labels

Currently, there are four ways which fluent-bit uses to assign fields into the resource/labels section.

  1. Resource Labels API

  2. Monitored Resource API

  3. Local Resource Id

  4. Credentials / Config Parameters

Note that the project_id resource label will always be set from the service credentials or fetched from the metadata server and cannot be overridden.

Using the resource_labels parameter

The resource_labels configuration parameter offers an alternative API for assigning the resource labels. To use, input a list of comma separated strings specifying resource labels plaintext assignments (new=value), mappings from an original field in the log entry to a destination field (destination=$original) and/or environment variable assignments (new=${var}).

For instance, consider the following log entry:

{
  "keyA": "valA",
  "toplevel": {
    "keyB": "valB"
  }
}

Combined with the following Stackdriver configuration:

[OUTPUT]
    Name stackdriver
    Match *
    Resource_Labels  keyC=$keyA,keyD=$toplevel['keyB'],keyE=valC

This will produce the following log:

{
  "resource": {
    "type": "global",
    "labels": {
      "project_id": "fluent-bit",
      "keyC": "valA",
      "keyD": "valB"
      "keyE": "valC"
    }
  },
  "entries": [
    {
      "jsonPayload": {
        "keyA": "valA",
        "toplevel": {
           "keyB": "valB"
        }
      },
    }
  ]
}

This makes the resource_labels API the recommended choice for supporting new or existing resource types that have all resource labels known before runtime or available on the payload during runtime.

[OUTPUT]
    Name            stackdriver
    Match           *
    Resource        k8s_container
    Resource_Labels cluster_name=my-cluster,location=us-central1-c,container_name=$kubernetes['container_name'],namespace_name=$kubernetes['namespace_name'],pod_name=$kubernetes['pod_name']

Troubleshooting Notes

Upstream connection error

An upstream connection error means Fluent Bit was not able to reach Google services, the error looks like this:

[2019/01/07 23:24:09] [error] [oauth2] could not get an upstream connection

This belongs to a network issue by the environment where Fluent Bit is running, make sure that from the Host, Container or Pod you can reach the following Google end-points:

Fail to process local_resource_id

The error looks like this:

[2020/08/04 14:43:03] [error] [output:stackdriver:stackdriver.0] fail to process local_resource_id from log entry for k8s_container

Do following check:

  • If the log entry does not contain the local_resource_id field, does the tag of the log match for format?

  • If tag_prefix is configured, does the prefix of tag specified in the input plugin match the tag_prefix?

Occasional Crashing with >1 Workers

When the number of Workers is greater than 1, Fluent Bit may intermittently crash.

Other implementations

logging.googleapis.com/labels. See for more info.

logging.googleapis.com/logName. See for more info.

logging.googleapis.com/severity. See for more info.

logging.googleapis.com/projectId. See for more info.

An optional list of comma separated strings specifying resource labels plaintext assignments (new=value) and/or mappings from an original field in the log entry to a destination field (destination=$original). Nested fields and environment variables are also supported using the . If configured, all resource labels will be assigned using this API only, with the exception of project_id. See for more details.

If resource_labels is correctly configured, then fluent-bit will attempt to populate all resource/labels using the entries specified. Otherwise, fluent-bit will attempt to use the monitored resource API. Similarly, if the monitored resource API cannot be used, then fluent-bit will attempt to populate resource/labels using configuration parameters and/or credentials specific to the resource type. As mentioned in the section, fluent-bit will attempt to use or construct a local resource ID for a K8s resource type which does not use the resource labels or monitored resource API.

For instance, for a K8s resource type, resource_labels can be used in tandem with the to pack all six resource labels. Below is an example of what this could look like for a k8s_container resource:

resource_labels also supports validation for required labels based on the input resource type. This allows fluent-bit to check if all specified labels are present for a given configuration before runtime. If validation is not currently supported for a resource type that you would like to use this API with, we encourage you to open a pull request for it. Adding validation for a new resource type is simple - all that is needed is to specify the resources associated with the type alongside the required labels .

Github reference:

Github reference:

Stackdriver officially supports a .

We plan to support some . Use cases of special fields is .

Google Cloud Stackdriver Logging
Creating a Google Service Account for Stackdriver
Kubernetes filter
here
#761
https://www.googleapis.com
https://logging.googleapis.com
#7552
logging agent based on Fluentd
special fields in structured payloads
here
Configuration File
http://metadata.google.internal
Stackdriver Special Fields
Stackdriver Special Fields
Stackdriver Special Fields
Stackdriver Special Fields
record accessor syntax
Resource Labels