Prometheus text file

The Prometheus text file input plugin allows Fluent Bit to read metrics from Prometheus text format files (.prom files) on the local filesystem. Use this plugin to collect custom metrics that are written to files by external applications or scripts, similar to the Prometheus Node Exporter text file collector.

Configuration parameters

Key	Description	Default
alias	Sets an alias. Use for multiple instances of the same input plugin. If no alias is specified, a default name is assigned using the plugin name followed by a dot and a sequence number.	none
log_level	Specifies the log level for output plugin. If not set here, plugin uses global log level in service section.	info
log_supress_interval	Suppresses log messages from output plugin that appear similar within a specified time interval. 0 no suppression.	0
mem_buf_limit	Set a memory buffer limit for the input plugin. If the limit is reached, the plugin will pause until the buffer is drained. The value is in bytes. If set to 0, the buffer limit is disabled.	0
path	File or directory path pattern. Supports glob patterns with * wildcard (for example, /var/lib/prometheus/*.prom).	none
routable	If set to true, the data generated by the plugin will be routable, meaning that it can be forwarded to other plugins or outputs. If set to false, the data will be discarded.	true
scrape_interval	Interval in seconds between file scans.	10s
storage.pause_on_chunks_overlimit	Enable pausing on an input when they reach their chunks limit.	none
storage.type	Sets the storage type for this input, one of: filesystem, memory or memrb.	memory
tag	Set a tag for the events generated by this input plugin.	none
threaded	Enable threading on an input.	false
thread.ring_buffer.capacity	Set custom ring buffer capacity when the input runs in threaded mode.	1024
thread.ring_buffer.window	Set custom ring buffer window percentage for threaded inputs.	5

Get started

Basic configuration

The following configuration will monitor /var/lib/prometheus/textfile directory for .prom files every 15 seconds:

pipeline:
  inputs:
    - name: prometheus_textfile
      tag: custom_metrics
      path: '/var/lib/prometheus/textfile/*.prom'
      scrape_interval: 15
  outputs:
    - name: prometheus_exporter
      match: custom_metrics
      host: 192.168.100.61
      port: 2021

Prometheus text format

The plugin expects files to be in the standard Prometheus text exposition format. Here's an example of a valid .prom file:

# HELP custom_counter_total A custom counter metric
# TYPE custom_counter_total counter
custom_counter_total{instance="server1",job="myapp"} 42

# HELP custom_gauge A custom gauge metric
# TYPE custom_gauge gauge
custom_gauge{environment="production"} 1.23

# HELP custom_histogram_bucket A custom histogram
# TYPE custom_histogram_bucket histogram
custom_histogram_bucket{le="0.1"} 10
custom_histogram_bucket{le="0.5"} 25
custom_histogram_bucket{le="1.0"} 40
custom_histogram_bucket{le="+Inf"} 50
custom_histogram_sum 125.5
custom_histogram_count 50

Use cases

Custom application metrics

Applications can write custom metrics to .prom files, and this plugin will collect them:

# Script writes metrics to file
echo "# HELP app_requests_total Total HTTP requests" > /var/lib/prometheus/textfile/app.prom
echo "# TYPE app_requests_total counter" >> /var/lib/prometheus/textfile/app.prom
echo "app_requests_total{status=\"200\"} 1500" >> /var/lib/prometheus/textfile/app.prom
echo "app_requests_total{status=\"404\"} 23" >> /var/lib/prometheus/textfile/app.prom

Batch job metrics

Cron jobs or batch processes can write completion metrics:

#!/bin/bash
# Backup script writes completion metrics
BACKUP_START=$(date +%s)
# ... perform backup ...
BACKUP_END=$(date +%s)
DURATION=$((BACKUP_END - BACKUP_START))

cat > /var/lib/prometheus/textfile/backup.prom << EOF
# HELP backup_duration_seconds Time taken to complete backup
# TYPE backup_duration_seconds gauge
backup_duration_seconds ${DURATION}

# HELP backup_last_success_timestamp_seconds Last successful backup timestamp
# TYPE backup_last_success_timestamp_seconds gauge
backup_last_success_timestamp_seconds ${BACKUP_END}
EOF

System integration

External monitoring tools can write metrics that Fluent Bit will collect and forward.

Integration with other plugins

OpenTelemetry destination

pipeline:
  inputs:
    - name: prometheus_textfile
      tag: textfile_metrics
      path: /var/lib/prometheus/textfile
    - name: node_exporter_metrics
      tag: system_metrics
      scrape_interval: 15
  outputs:
    - name: opentelemetry
      match: '*'
      host: 192.168.56.4
      port: 2021