Google Cloud Storage
Send logs to Google Cloud Storage
The Google Cloud Storage (gcs) output plugin lets you upload the records received through the input plugin to a Google Cloud Storage bucket. Records are buffered locally and uploaded as objects when the configured upload timeout elapses.
This plugin is available in Fluent Bit version 5.1 and greater. It's included in official builds. If you compile Fluent Bit from source, the FLB_OUT_GCS build option is enabled by default.
Google Cloud configuration
Before using the plugin, you must:
Identify or create a Google Cloud service account for Fluent Bit, unless you use Workload Identity Federation with direct resource access. If you plan to authenticate with a credentials file, also create a JSON key for that service account. A JSON key isn't required when you use metadata server authentication, because that mode uses the service account attached to the Compute Engine instance or Google Kubernetes Engine node and reads no credentials file. A JSON key also isn't required when you use Workload Identity Federation, because that mode exchanges an external OpenID Connect (
OIDC) token for a Google access token.Create the bucket that receives your data. Fluent Bit doesn't create buckets.
Grant the identity that Fluent Bit authenticates as permission to write objects to that bucket. For credentials file, metadata server, and impersonated federation modes, this is the service account. For Workload Identity Federation with
google_service_accountunset, this is the federated principal itself, so grant that principal bucket access directly. See Workload Identity Federation. The plugin requests thehttps://www.googleapis.com/auth/devstorage.read_writeOAuth 2.0scope.Make the credentials available to Fluent Bit. See Authentication.
The plugin uploads objects to storage.googleapis.com over TLS using the Cloud Storage JSON API.
Authentication
The plugin supports two authentication paths. Setting enable_identity_federation to true selects Workload Identity Federation. Otherwise, the plugin uses a service account key or the metadata server.
Service account key or metadata server
The plugin looks for service account credentials in the following order, and uses the first source it finds:
The
google_service_credentialsoption, set to the absolute path of a service account JSON key file.The
GOOGLE_APPLICATION_CREDENTIALSenvironment variable, set to that path.The
GOOGLE_SERVICE_CREDENTIALSenvironment variable, set to that path. This variable is retained for backward compatibility. If both environment variables are set, the plugin usesGOOGLE_APPLICATION_CREDENTIALSand logs a warning.The Compute Engine or Google Kubernetes Engine metadata server. When no credentials file is configured, the plugin requests an access token from the address set in
metadata_server. Use this mode when Fluent Bit runs on a Compute Engine instance or a Google Kubernetes Engine node whose attached service account can write to the bucket.
The metadata server isn't a fallback for a failed credentials file. It's used only when no credentials file is configured through the option or either environment variable. If a credentials path is configured but the file can't be opened, read, or parsed as a service account key, the plugin logs an error such as cannot open credentials file or invalid JSON credentials file and fails to initialize. If you intend to use the metadata server, verify that the plugin logs using GCE/GKE metadata server authentication at startup.
Workload Identity Federation
Workload Identity Federation lets Fluent Bit authenticate to Cloud Storage without a service account key. Use it when Fluent Bit runs outside Google Cloud, or on Google Kubernetes Engine where the workload receives a projected OIDC token. Set enable_identity_federation to true to select this mode.
When this mode is enabled, the plugin ignores the credentials file, both credentials environment variables, and the metadata server. Setting both enable_identity_federation and google_service_credentials fails at startup with 'google_service_credentials' and 'enable_identity_federation' are mutually exclusive.
This mode requires the following options. If any of them is missing, the plugin logs '<option>' is required when 'enable_identity_federation' is true and fails to initialize:
project_number: the number of the Google Cloud project that owns the workload identity pool.pool_id: the workload identity pool ID.provider_id: the workload identity pool provider ID.identity_token_file: the path to the file holding theOIDCsubject token.
The plugin builds the STS audience from the first three of these values as //iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/POOL_ID/providers/PROVIDER_ID, and logs it at startup. This string must match the audience configured on the pool provider.
At token time, the plugin reads identity_token_file and exchanges its contents at https://sts.googleapis.com/v1/token for a federated access token. The plugin re-reads the file on every exchange and never caches its contents, so platforms that rotate the token in place, such as a kubelet projected token, are handled without a restart.
The federated token can be used two ways:
Impersonation: set
google_service_accountto the email address of a service account that can write to the bucket. The plugin callshttps://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/SERVICE_ACCOUNT:generateAccessTokenand uses the returned token. Grant the pool principal theroles/iam.workloadIdentityUserrole on that service account.Direct resource access: leave
google_service_accountunset. The plugin uses the federated token directly against Cloud Storage, so the pool principal itself must have permission to write objects to the bucket. The bucket must also have uniform bucket-level access enabled. On a bucket that uses fine-grained access, you can still add a federated principal to theACLsand IAM policies, but the grant doesn't take effect and uploads fail. This requirement doesn't apply to impersonation, because those requests are made as the service account.
The plugin caches the resulting token until 5 minutes before the expiry reported by the token service, then exchanges a new one. If the token service doesn't report a usable expiry, the plugin falls back to its default refresh interval.
Use subject_token_type only if your provider issues something other than a JWT. The default is urn:ietf:params:oauth:token-type:jwt.
The following configuration authenticates through a Google Kubernetes Engine projected token and impersonates a service account:
Configuration parameters
bucket
Name of the Cloud Storage bucket that receives the objects.
none
canned_acl
Predefined ACL applied to uploaded objects. Accepted values: authenticated-read, bucket-owner-full-control, bucket-owner-read, private, project-private, and public-read. The camel case forms used by the Cloud Storage API, such as publicRead, are also accepted.
none
compression
Compression applied to uploaded objects. Values aren't case-sensitive. When format is json_lines, accepted values are none and gzip. When format is parquet, this option selects the page-level codec inside the Parquet file and accepted values are none, snappy, gzip, and zstd. See Parquet format.
none
content_type
Value of the Content-Type metadata set on uploaded objects.
application/json, or application/vnd.apache.parquet when format is parquet
enable_identity_federation
Authenticate with Workload Identity Federation instead of a service account key. Mutually exclusive with google_service_credentials. See Workload Identity Federation.
false
format
Output format for uploaded objects. Accepted values: json_lines and parquet. See Parquet format. Setting json logs a warning and is treated as json_lines.
json_lines
gcs_key_format
Format string for object names in the bucket. See Object key format.
fluent-bit-logs/$TAG/%Y/%m/%d/%H/%M/%S
gcs_key_format_tag_delimiters
Characters used to split the tag into the parts referenced by $TAG[n] in gcs_key_format.
.
google_service_account
The email address of the service account to impersonate after the token exchange. When unset, the federated token is used directly against Cloud Storage. Only used when enable_identity_federation is true. See Workload Identity Federation.
none
google_service_credentials
Absolute path to a Google Cloud service account credentials JSON file. See Authentication.
none
identity_token_file
Path to the file that holds the OIDC subject token used as the federation credential source. Required when enable_identity_federation is true. See Workload Identity Federation.
none
metadata_server
Address of the Compute Engine or Google Kubernetes Engine metadata server used to request an access token when no credentials file is configured. See Authentication.
http://metadata.google.internal
pool_id
Workload identity pool ID. Required when enable_identity_federation is true. See Workload Identity Federation.
none
preserve_data_ordering
When an upload request fails, the last received chunk might swap with a later chunk, resulting in data shuffling. This option prevents shuffling by using queue logic for uploads.
false
project_number
Number of the Google Cloud project that owns the workload identity pool. Required when enable_identity_federation is true. See Workload Identity Federation.
none
provider_id
Workload identity pool provider ID. Required when enable_identity_federation is true. See Workload Identity Federation.
none
send_content_md5
Send the Content-MD5 header with uploads so that Cloud Storage verifies object integrity.
false
static_file_path
Disables the behavior where a random suffix appends to the object name when $UUID isn't provided in gcs_key_format.
false
store_chunk_limit
Maximum number of buffered chunks kept in store_dir. Set to 0 for unlimited.
0
store_dir
Directory used to locally buffer data before uploading it.
/tmp/fluent-bit/gcs
store_dir_limit_size
Limits the amount of data buffered in store_dir to limit disk usage. When the limit is reached, data is discarded. Set to 0 for unlimited.
0
subject_token_type
OIDC subject token type presented to the Google Security Token Service. Only used when enable_identity_federation is true. See Workload Identity Federation.
urn:ietf:params:oauth:token-type:jwt
total_file_size
Maximum size of the file currently being buffered for a tag. When that file reaches this size, Fluent Bit uploads it as an object, even when upload_timeout hasn't elapsed. Files already sealed and waiting to upload don't count toward this size, so a tag can hold more than this amount on disk. The minimum accepted value is 1M. Set to 0 to upload only when upload_timeout elapses. See Buffering. Supported in v5.1.3 or later.
100M
unify_tag
Whether to buffer records from every tag into a single file instead of one file per tag. See Unified tag buffering.
false
unify_tag_name
Logical tag that replaces the record tag when unify_tag is enabled. It's stored as the buffer chunk metadata and used for $TAG in gcs_key_format. It doesn't set the name of the local buffer file. See Unified tag buffering.
fluent-bit-buffer-file-unify-tag.log
upload_timeout
When this amount of time elapses, Fluent Bit uploads the buffered data and starts a new object. Set to 60m to upload a new object every hour.
10m
Object key format
The gcs_key_format option supports the same formatters as the Amazon S3 output plugin:
$TAG: The full tag.$TAG[n]: The nth part of the tag, with the index starting at zero. Tag parts are separated using the characters set ingcs_key_format_tag_delimiters.$UUID: A random string.$INDEX: An integer that increments with each upload. The value is stored instore_dirso it survives a restart.strftime formatters such as
%Y,%m, and%d. The time used is the timestamp of the first record in the object.
When gcs_key_format doesn't contain $UUID and static_file_path is false, Fluent Bit appends a random suffix to each object name so that concurrent uploads don't overwrite each other. Set static_file_path to true to keep the object name exactly as the format string produces it.
Parquet format
Setting format to parquet converts log records to Apache Parquet columnar format before uploading them. Parquet objects are directly queryable by BigQuery, Spark, and Presto without additional transformation.
The compression option controls the page-level codec applied inside the Parquet file:
compression value
Parquet page codec
Notes
snappy
Snappy
Fast, moderate compression ratio. Commonly used with Parquet, but Fluent Bit defaults compression to none.
zstd
Zstandard
Better ratio, slightly slower.
gzip
Gzip
Best ratio, slowest.
none
Uncompressed
No page-level compression.
Unless you set content_type explicitly, Parquet objects are uploaded with the Content-Type metadata application/vnd.apache.parquet.
Example: Parquet with Snappy compression
Buffering
This plugin buffers records as files in store_dir and uploads them when upload_timeout elapses, so it requires a writeable filesystem. In Fluent Bit version 5.1.3 and greater, the plugin also uploads the file currently being buffered for a tag when it reaches total_file_size. Because the plugin has its own buffering system, the storage.total_limit_size parameter isn't meaningful. Use store_dir_limit_size and store_chunk_limit to limit disk usage instead.
The total_file_size size trigger is available in Fluent Bit version 5.1.3 and greater. When adding a batch of records would push a buffered file past total_file_size, Fluent Bit seals that file, schedules it for immediate upload, and starts a new file for the incoming records. A sealed file never receives more data, even if its upload fails and is retried later. Because the check runs against whole batches, an uploaded object can be slightly smaller or larger than total_file_size. Set total_file_size to 0 to disable the size trigger and upload only on upload_timeout. A non-zero value smaller than 1M fails at startup with 'total_file_size' must be at least 1M (or 0 to disable the size trigger).
Unified tag buffering
By default, each tag buffers into its own file, which is inefficient when a pipeline produces many small chunks under many tags, such as one tag per container. Set unify_tag to true to buffer records from every tag together under the single logical tag set by unify_tag_name. Fluent Bit generates local buffer file names internally, so unify_tag_name doesn't appear on disk.
When unify_tag is enabled, the value of unify_tag_name replaces the record tag for the rest of the upload path. As a result, $TAG and $TAG[n] in gcs_key_format resolve to unify_tag_name instead of the original tag. If your object names depend on the tag, remove those formatters from gcs_key_format before enabling this option.
Networking and TLS configuration
This plugin inherits the core Fluent Bit networking and TLS settings. The upload connection to storage.googleapis.com always uses TLS on port 443, so you don't need to turn TLS on. Use these settings to tune how that connection behaves:
Networking Setup:
net.*properties such as connection timeouts, keepalive behavior, and the source address. Fluent Bit version 5.1.3 and greater applies these to the GCS upload connection. They don't apply to the separate connections used to obtain credentials from the Google Security Token Service, IAM Credentials, or metadata server endpoints.Security and TLS:
tls.*properties such astls.ca_file,tls.verify, andtls.vhost, which control certificate validation for the upload connection.
Set only the properties your environment requires. Most of them take a value or a path rather than an on or off flag, and the defaults are suitable for connecting to Cloud Storage.
Get started
The following configuration buffers CPU metrics and uploads a gzip-compressed object to the my-logs bucket every five minutes:
Last updated
Was this helpful?