Running a Logging Pipeline Locally

You may wish to test a logging pipeline locally to observe how it deals with log messages. The following is a walk-through for running Fluent Bit and Elasticsearch locally with Docker Compose which can serve as an example for testing other plugins locally.
Create a Configuration File
Refer to the Configuration File section to create a configuration to test.
fluent-bit.conf:
1
[INPUT]
2
  Name dummy
3
  Dummy {"top": {".dotted": "value"}}
4
​
5
[OUTPUT]
6
  Name es
7
  Host elasticsearch
8
  Replace_Dots On
Copied!
Docker Compose
Use Docker Compose to run Fluent Bit (with the configuration file mounted) and Elasticsearch.
docker-compose.yaml:
1
version: "3.7"
2
​
3
services:
4
  fluent-bit:
5
    image: fluent/fluent-bit
6
    volumes:
7
      - ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf
8
    depends_on:
9
      - elasticsearch
10
  elasticsearch:
11
    image: elasticsearch:7.6.2
12
    ports:
13
      - "9200:9200"
14
    environment:
15
      - discovery.type=single-node
Copied!
View indexed logs
To view indexed logs run:
1
curl "localhost:9200/_search?pretty" \
2
  -H 'Content-Type: application/json' \
3
  -d'{ "query": { "match_all": {} }}'
Copied!
To "start fresh", delete the index by running:
1
curl -X DELETE "localhost:9200/fluent-bit?pretty"
Copied!

Local Testing - Previous

Validating your Data and Structure

Next - Data Pipeline

Pipeline Monitoring

Last modified 1yr ago

Export as PDF

Copy link

Contents

Create a Configuration File

Docker Compose

View indexed logs