mem_buf_limit: if a plugin has enqueued more than the
mem_buf_limit, it won't be able to ingest more until that data can be delivered or flushed properly. In this scenario the input plugin in question is paused.
mem_buf_limitis good for certain scenarios and environments, it helps to control the memory usage of the service, but at the costs that if a file gets rotated while paused, you might lose that data since it won't be able to register new records. This can happen with any input source plugin. The goal of
mem_buf_limitis memory control and survival of the service.
upwhich means "the chunk content is up in memory".
upin memory in total (considering all Chunks), this value is controlled by service property
storage.max_chunks_up. The active Chunks that are
upare ready for delivery and the ones that still are receiving records. Any other remaining Chunk is in a
downstate, which means that's only in the filesystem and won't be
upin memory unless is ready to be delivered.
filesystem, when reaching the
mem_buf_limitthreshold, instead of the plugin being paused, all new data will go to Chunks that are
downin the filesystem. This allows to control the memory usage by the service but also providing a a guarantee that the service won't lose any data.
storage.total_limit_sizewhich limits the number of Chunks that exist in the file system for a certain logical output destination. If one of destinations reaches the
storage.total_limit_size, the oldest Chunk from its queue for that logical output destination will be discarded.
filesystemstorage type, this property sets the maximum number of Chunks that can be
upin memory. This helps to control memory usage.
http_serveroption has been enabled in the main
[SERVICE]section, this option registers a new endpoint where internal metrics of the storage layer can be consumed. For more details refer to the Monitoring section.