Run a logging pipeline locally

You can test logging pipelines locally to observe how they handles log messages. This guide explains how to use Docker Compose to run Fluent Bit and Elasticsearch locally, but you can use the same principles to test other plugins.

Create a configuration file

Start by creating one of the corresponding Fluent Bit configuration files to start testing.

fluent-bit.yaml

pipeline:
  inputs:
    - name: dummy
      dummy: '{"top": {".dotted": "value"}}'
      
  outputs:       
    - name: es
      host: elasticsearch
      replace_dots: on

fluent-bit.conf

Use Docker Compose

Use Docker Compose to run Fluent Bit (with the configuration file mounted) and Elasticsearch.

docker-compose.yaml

version: "3.7"

services:
  fluent-bit:
    image: fluent/fluent-bit
    volumes:
      - ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf
    depends_on:
      - elasticsearch
  elasticsearch:
    image: elasticsearch:7.6.2
    ports:
      - "9200:9200"
    environment:
      - discovery.type=single-node

View indexed logs

To view indexed logs, run the following command:

curl "localhost:9200/_search?pretty" \
  -H 'Content-Type: application/json' \
  -d'{ "query": { "match_all": {} }}'

Reset index

To reset your index, run the following command:

curl -X DELETE "localhost:9200/fluent-bit?pretty"