Stackdriver special fields

When the google-logging-agent receives a structured log record, it treats some fields specially, allowing users to set specific fields in the LogEntry object that get written to the Logging API.

LogEntry fields

Fluent Bit support some special fields for setting fields on the LogEntry object:

JSON log field

LogEntry field

Type

Description

logging.googleapis.com/logName

logName

string

The log name to write this log to.

logging.googleapis.com/labels

labels

object<string, string>

The labels for this log.

logging.googleapis.com/severity

severity

LogSeverity Enum

The severity of this log.

logging.googleapis.com/monitored_resource

resource

MonitoredResource (without type)

Resource labels for this log.

logging.googleapis.com/operation

operation

LogEntryOperation

Additional information about a potentially long-running operation.

logging.googleapis.com/insertId

insertId

string

A unique identifier for the log entry. It's used to order logEntries.

logging.googleapis.com/sourceLocation

sourceLocation

LogEntrySourceLocation

Additional information about the source code location that produced the log entry.

logging.googleapis.com/http_request

httpRequest

HttpRequest

A common proto for logging HTTP requests.

logging.googleapis.com/trace

trace

string

Resource name of the trace associated with the log entry.

logging.googleapis.com/traceSampled

traceSampled

boolean

The sampling decision associated with this log entry.

logging.googleapis.com/spanId

spanId

string

The ID of the trace span associated with this log entry.

timestamp

object (protobuf Timestamp object format)

An object including the seconds and nanoseconds fields that represent the time.

timestampSecond and timestampNanos

timestamp

int

The seconds and nanoseconds that represent the time.

Other special fields

JSON log field

Description

logging.googleapis.com/projectId

Changes the project ID that this log will be written to. Ensure that you are authenticated to write logs to this project.

logging.googleapis.com/local_resource_id

Overrides the configured local_resource_id.

Use special fields

To use a special field, you must add a field with the right name and value to your log. Given an example structured log (internally in MessagePack but shown in JSON for demonstration):

{
  "log": "Hello world!"
}

To use the logging.googleapis.com/logName special field, add it to your structured log as follows:

{
  "log": "Hello world!",
  "logging.googleapis.com/logName": "my_log"
}

For the special fields that map to LogEntry prototypes, add them as objects with field names that match the proto. For example, to use the logging.googleapis.com/operation:

{
  "log": "Hello world!",
  "logging.googleapis.com/operation": {
    "id": "test_id",
    "producer": "test_producer",
    "first": true,
    "last": true
  }
}

Adding special fields to logs is best done through the modify filter for basic fields, or a Lua script using the lua filter for more complex fields.

Basic type special fields

Special fields with basic types (except for the logging.googleapis.com/insertId field) will follow this pattern (demonstrated with the logging.googleapis.com/logName field):

If the special field matches the type, it will be moved to the corresponding LogEntry field. For example:

{
  ...
  "logging.googleapis.com/logName": "my_log"
  ...
}

the logEntry will be:

{
  "jsonPayload": {
    ...
  }
  "logName": "my_log"
  ...
}

If the field is non-empty but an invalid, it will be left in the jsonPayload. For example:
```
{
  ...
  "logging.googleapis.com/logName": 12345
  ...
}
```

the logEntry will be:

{
  "jsonPayload": {
    "logging.googleapis.com/logName": 12345
    ...
  }
}

Exceptions [#exceptions-basic]

Insert ID

If the logging.googleapis.com/insertId field has an invalid type, the log will be rejected by the plugin and not sent to Cloud Logging.

Trace sampled

If theautoformat_stackdriver_trace is set to true, the value provided in the trace field will be formatted into the format that Cloud Logging expects along with the detected Project ID (from the Google Metadata server, configured in the plugin, or provided using a special field).

For example, if autoformat_stackdriver_trace is enabled, this:

{
  ...
  "logging.googleapis.com/projectId": "my-project-id",
  "logging.googleapis.com/trace": "12345"
  ...
}

Will become this:

{
  "jsonPayload": {
    ...
  },
  "projectId": "my-project-id",
  "trace": "/projects/my-project-id/traces/12345"
  ...
}

`timestampSecond` and `timestampNano`

The timestampSecond and timestampNano fields don't map directly to the timestamp field in LogEntry so the parsing behaviour deviates from other special fields. Read more in the Timestamp section.

Proto special fields

For special fields that expect the format of a prototype from the LogEntry (except for the logging.googleapis.com/monitored_resource field) will follow this pattern (demonstrated with the logging.googleapis.com/operation field):

If any sub-fields of the proto are empty or in incorrect type, the plugin will set these sub-fields empty. For example:

{
  "logging.googleapis.com/operation": {
    "id": 123, #incorrect type
    # no producer here
    "first": true,
    "last": true
  }
  ...
}

the logEntry will be:

{
  "jsonPayload": {
    ...
  }
  "operation": {
    "first": true,
    "last": true
  }
  ...
}

If the field itself isn't a map, the plugin will leave this field untouched. For example:

{
  ...
  "logging.googleapis.com/operation": "some string",
  ...
}

the logEntry will be:

{
  "jsonPayload": {
    "logging.googleapis.com/operation": "some string",
    ...
  }
  ...
}

If there are extra sub-fields, the plugin will add the recognized fields to the corresponding field in the LogEntry, and preserve the extra sub-fields in jsonPayload. For example:

{
  ...
  "logging.googleapis.com/operation": {
    "id": "test_id",
    "producer": "test_producer",
    "first": true,
    "last": true,
    "extra1": "some string",
    "extra2": 123,
    "extra3": true
  }
  ...
}

the `logEntry will be:

{
  "jsonPayload": {
    "logging.googleapis.com/operation": {
      "extra1": "some string",
      "extra2": 123,
      "extra3": true
    }
    ...
  }
  "operation": {
    "id": "test_id",
    "producer": "test_producer",
    "first": true,
    "last": true
  }
  ...
}

Exceptions [#exceptions-proto]

`MonitoredResource` ID

The logging.googleapis.com/monitored_resource field is parsed in a special way, meaning it has some important exceptions:

The type field from the MonitoredResource proto isn't parsed out of the special field. It's read from the resource plugin configuration option. If it's supplied in the logging.googleapis.com/monitored_resource special field, it won't be recognized.

The labels field is expected to be an object<string, string>. If any fields have a value that's not a string, the value is ignored and not preserved. The plugin logs an error and drops the field.

If no valid labels field is found, or if all entries in the labels object provided are invalid, the logging.googleapis.com/monitored_resource field is dropped in favour of automatically setting resource labels using other available information based on the configured resource type.

Timestamp

Fluent Bit supports the following formats of time-related fields:

timestamp

Log body contains a timestamp field that includes the seconds and nanoseconds fields.

{
  "timestamp": {
    "seconds": CURRENT_SECONDS,
    "nanos": CURRENT_NANOS
  }
}

timestampSeconds/timestampNanos
Log body contains both the timestampSeconds and timestampNanos fields.
```
{
  "timestampSeconds": CURRENT_SECONDS,
  "timestampNanos": CURRENT_NANOS
}
```

If one of the following JSON timestamp representations is present in a structured record, the plugin collapses them into a single representation in the timestamp field in the LogEntry object.

Without time-related fields, the plugin will set the current time as timestamp.

Format 1

Set the input log as follows:

{
  "timestamp": {
    "seconds": 1596149787,
    "nanos": 12345
  }
  ...
}

the logEntry will be:

{
  "jsonPayload": {
    ...
  }
  "timestamp": "2020-07-30T22:56:27.000012345Z"
  ...
}

Format 2

Set the input log as follows:

{
  "timestampSeconds":1596149787,
  "timestampNanos": 12345
  ...
}

the logEntry will be:

{
  "jsonPayload": {
    ...
  }
  "timestamp": "2020-07-30T22:56:27.000012345Z"
  ...
}

If the timestamp object or the timestampSeconds and timestampNanos fields end up being invalid, they will remain in the jsonPayload untouched.

PreviousStackdriver NextStandard output

Last updated 2 days ago

Was this helpful?

LogEntry fields

Other special fields

Use special fields

Basic type special fields

Exceptions [#exceptions-basic]

Insert ID

Trace sampled

timestampSecond and timestampNano

Proto special fields

Exceptions [#exceptions-proto]

MonitoredResource ID

Timestamp

Format 1

Format 2

`timestampSecond` and `timestampNano`

`MonitoredResource` ID