arrow-left

All pages
gitbookPowered by GitBook
1 of 1

Loading...

GELF

GELF is Graylogarrow-up-right Extended Log Format. The GELF output plugin allows to send logs in GELF format directly to a Graylog input using TLS, TCP or UDP protocols.

The following instructions assumes that you have a fully operational Graylog server running in your environment.

hashtag
Configuration Parameters

According to GELF Payload Specificationarrow-up-right, there are some mandatory and optional fields which are used by Graylog in GELF format. These fields are determined with Gelf\*_Key_ key in this plugin.

hashtag
TLS / SSL

GELF output plugin supports TLS/SSL, for more details about the properties available and general configuration, please refer to the section.

hashtag
Notes

  • If you're using Fluent Bit to collect Docker logs, note that Docker places your log in JSON under key log. So you can set log as your Gelf_Short_Message_Key to send everything in Docker logs to Graylog. In this case, you need your log value to be a string; so don't parse it using JSON parser.

  • The order of looking up the timestamp in this plugin is as follows:

hashtag
Configuration File Example

If you're using Fluent Bit for shipping Kubernetes logs, you can use something like this as your configuration file:

By default, GELF tcp uses port 12201 and Docker places your logs in /var/log/containers directory. The logs are placed in value of the log key. For example, this is a log saved by Docker:

If you use and use a Parser like the docker parser shown above, it decodes your message and extracts data (and any other present) field. This is how this log in looks like after decoding:

Now, this is what happens to this log:

  1. Fluent Bit GELF plugin adds "version": "1.1" to it.

  2. The , unnests fields inside log key. In our example, it puts data alongside stream and time.

Finally, this is what our Graylog server input sees:

udp

Gelf_Short_Message_Key

A short descriptive message (MUST be set in GELF)

short_message

Gelf_Timestamp_Key

Your log timestamp (SHOULD be set in GELF)

timestamp

Gelf_Host_Key

Key which its value is used as the name of the host, source or application that sent this message. (MUST be set in GELF)

host

Gelf_Full_Message_Key

Key to use as the long message that can i.e. contain a backtrace. (Optional in GELF)

full_message

Gelf_Level_Key

Key to be used as the log level. Its value must be in (between 0 and 7). (Optional in GELF)

level

Packet_Size

If transport protocol is udp, you can set the size of packets to be sent.

1420

Compress

If transport protocol is udp, you can set this if you want your UDP packets to be compressed.

true

Value of Gelf_Timestamp_Key provided in configuration

  • Value of timestamp key

  • If you're using Docker JSON parserarrow-up-right, this parser can parse time and use it as timestamp of message. If all above fail, Fluent Bit tries to get timestamp extracted by your parser.

  • Timestamp does not set by Fluent Bit. In this case, your Graylog server will set it to the current timestamp (now).

  • Your log timestamp has to be in UNIX Epoch Timestamparrow-up-right format. If the Gelf_Timestamp_Key value of your log is not in this format, your Graylog server will ignore it.

  • If you're using Fluent Bit in Kubernetes and you're using Kubernetes Filter Pluginarrow-up-right, this plugin adds host value to your log by default, and you don't need to add it by your own.

  • The version of GELF message is also mandatory and Fluent Bit sets it to 1.1 which is the current latest version of GELF.

  • If you use udp as transport protocol and set Compress to true, Fluent Bit compresses your packets in GZIP format, which is the default compression that Graylog offers. This can be used to trade more CPU load for saving network bandwidth.

  • We used this data key as Gelf_Short_Message_Key; so GELF plugin changes it to short_message.

  • Kubernetes Filterarrow-up-right adds host name.

  • Timestamp is generated.

  • Any custom field (not present in GELF Payload Specificationarrow-up-right) is prefixed by an underline.

  • Key

    Description

    default

    Match

    Pattern to match which tags of logs to be outputted by this plugin

    Host

    IP address or hostname of the target Graylog server

    127.0.0.1

    Port

    The port that your Graylog GELF input is listening on

    12201

    Mode

    TLS/SSLarrow-up-right
    Tail Inputarrow-up-right
    stdoutarrow-up-right
    Nest Filterarrow-up-right

    The protocol to use (tls, tcp or udp)

    [INPUT]
        Name                    tail
        Tag                     kube.*
        Path                    /var/log/containers/*.log
        Parser                  docker
        DB                      /var/log/flb_kube.db
        Mem_Buf_Limit           5MB
        Refresh_Interval        10
    
    [FILTER]
        Name                    kubernetes
        Match                   kube.*
        Merge_Log_Key           log
        Merge_Log               On
        Keep_Log                Off
        Annotations             Off
        Labels                  Off
    
    [FILTER]
        Name                    nest
        Match                   *
        Operation               lift
        Nested_under            log
    
    [OUTPUT]
        Name                    gelf
        Match                   kube.*
        Host                    <your-graylog-server>
        Port                    12201
        Mode                    tcp
        Gelf_Short_Message_Key  data
    
    [PARSER]
        Name                    docker
        Format                  json
        Time_Key                time
        Time_Format             %Y-%m-%dT%H:%M:%S.%L
        Time_Keep               Off
    {"log":"{\"data\": \"This is an example.\"}","stream":"stderr","time":"2019-07-21T12:45:11.273315023Z"}
    [0] kube.log: [1565770310.000198491, {"log"=>{"data"=>"This is an example."}, "stream"=>"stderr", "time"=>"2019-07-21T12:45:11.273315023Z"}]
    {"version":"1.1", "short_message":"This is an example.", "host": "<Your Node Name>", "_stream":"stderr", "timestamp":1565770310.000199}
    standard syslog levelsarrow-up-right