What is the maximum data size of logs a collector can hold on DLP environment when buffering is enabled ?

Badge +1

In the case of DLP environment with collector and data node, if the connection is lost between two, what is the maximum data size of logs a collector can hold?

3 replies

Userlevel 2
Badge +3

The maximum data size the collector can hold is depended on the storage that you have allocated to the collector node. Typically the storage provisioned is less in DLP environment as the collector doesn’t store logs. 

However during network outages it does store them but as a buffer. After the network connection is regained it starts sending the logs to the Data Node. 

Userlevel 1

I assume the disk buffer size is dependent on available disk in the Collector, or is it configurable?



Userlevel 1

The buffer size and folder in which the buffer operates can be adjusted.

It is stored by kafka service inside directory ( /opt/makalu/storage/lpc/kdata) in its own convention/format.
Base directory for buffering (/opt/makalu/storage) can be changed by modifying "storage_base_path" configuration parameter of "lpc_kafka_producer" service. ( tuned in lp_services_config file.)
Maximum buffer size (and other buffer parameters) can be defined in lp_services config as well under name "kafka_server_properties".
Customers without root acces would need to ask their partner or open a support ticket in our service desk to configure those settings.