Running a Logging Pipeline Locally

You can test logging pipelines locally to observe how they handles log messages. This guide explains how to use Docker Compose to run Fluent Bit and Elasticsearch locally, but you can use the same principles to test other plugins.

Create a configuration file

Start by creating a Fluent Bit configuration file to test.

fluent-bit.conf

[INPUT]
  Name dummy
  Dummy {"top": {".dotted": "value"}}

[OUTPUT]
  Name es
  Host elasticsearch
  Replace_Dots On

Use Docker Compose

Use Docker Compose to run Fluent Bit (with the configuration file mounted) and Elasticsearch.

docker-compose.yaml

version: "3.7"

services:
  fluent-bit:
    image: fluent/fluent-bit
    volumes:
      - ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf
    depends_on:
      - elasticsearch
  elasticsearch:
    image: elasticsearch:7.6.2
    ports:
      - "9200:9200"
    environment:
      - discovery.type=single-node

View indexed logs

To view indexed logs, run the following command:

curl "localhost:9200/_search?pretty" \
  -H 'Content-Type: application/json' \
  -d'{ "query": { "match_all": {} }}'

Reset index

To reset your index, run the following command:

curl -X DELETE "localhost:9200/fluent-bit?pretty"