mem_buf_limit
: if a plugin has enqueued more than the mem_buf_limit
, it won't be able to ingest more until that data can be delivered or flushed properly. In this scenario the input plugin in question is paused.mem_buf_limit
is good for certain scenarios and environments, it helps to control the memory usage of the service, but at the costs that if a file gets rotated while paused, you might lose that data since it won't be able to register new records. This can happen with any input source plugin. The goal of mem_buf_limit
is memory control and survival of the service.up
which means "the chunk content is up in memory".up
in memory.up
in memory in total (considering all Chunks), this value is controlled by service property storage.max_chunks_up
. The active Chunks that are up
are ready for delivery and the ones that still are receiving records. Any other remaining Chunk is in a down
state, which means that's only in the filesystem and won't be up
in memory unless is ready to be delivered.mem_buf_limit
and storage.type
as filesystem
, when reaching the mem_buf_limit
threshold, instead of the plugin being paused, all new data will go to Chunks that are down
in the filesystem. This allows to control the memory usage by the service but also providing a a guarantee that the service won't lose any data.storage.total_limit_size
which limits the number of Chunks that exist in the file system for a certain logical output destination. If one of destinations reaches the storage.total_limit_size
, the oldest Chunk from its queue for that logical output destination will be discarded.filesystem
storage type, this property sets the maximum number of Chunks that can be up
in memory. This helps to control memory usage.http_server
option has been enabled in the main [SERVICE]
section, this option registers a new endpoint where internal metrics of the storage layer can be consumed. For more details refer to the Monitoring section.