Please follow the documentation for LPAgent at https://docs.logpoint.com/docs/logpoint-agent/en/latest/index.html carefully. Sometimes simple steps like processing policy or even the collector/fetcher are missed while configuring devices.
Important Considerations for LPAgent:
https://docs.logpoint.com/docs/logpoint-agent/en/latest/Important%20Considerations.html
Logs should be coming into the system in few minutes once the setup is done, but sometimes we don't see any logs in the search. Below we have described few common issues faced on customer environment and their debugging procedures.
Service Status:
Check if lpagent process is running on windows side & lpa_manager and lpanx_collector services are running on LogPoint.
> netstat -nap | egrep ':7998|:7999'____________________________________________________________tcp 0 0 0.0.0.0:7999 0.0.0.0:* LISTEN 30445/lpanx_collecttcp 0 0 10.10.1.74:7999 10.10.1.67:50459 ESTABLISHED 30445/lpanx_collect tcp 0 0 10.10.1.74:7998 10.10.1.6:64664 ESTABLISHED 30445/lpanx_collect tcp 0 0 10.10.1.74:7999 10.10.1.65:49956 ESTABLISHED 30445/lpanx_collect# 7998 or 7999 should be on listening state and # various established session should be seen with windows ip
LPAgent Issue:
- There is a bug on lpagent installation for 6.11.x+ LP where necessary dependencies aren't installed properly. We can verify it by running below mentioned cmd. The fix should be released on future version but current workaround is to reinstall the plugin, make sure normalization packages and lpagent collector are removed from devices before deleting the application.
> tail -f /opt/immune/var/log/service/lpanx_collector/current____________________________________________________________FileNotFoundError: [Errno 2] No such file or directory: '/opt/nxlog/bin/lpanx_collector'
- Nxlog has deprecated batching, so if you have enabled batching mode, please disable it and restart the
lpa_managerandlpanx_collectorservice. - Orphan or duplicate process of
lpanx_collectorrunning, only one instance should be running on the server> ps -ef | grep /opt/nxlog/etc/lpanx_collector.conf | grep -v grep____________________________________________________________loginsp+ 30445 30404 0 2021 ? 04:22:00 /opt/nxlog/bin/lpanx_collector -f -c /opt/nxlog/etc/lpanx_collector.conf
Config & Cert Discrepancy:
If the windows side has only encrypted port allowed and logpoint agent is re-installed or cert is regenerated, windows won't accept connection with new certs. In such cases we can manually copy the conf and certs to the windows side.
Doing manual changes on large number of servers can be hassle. Therefore for syncing all the severs automatically following steps are recommended:
- Remove logpoint agent and associated device from configuration
- Re-install lpagent application on logpoint and allow 7994 port in windows side
- Add a new device and set the connection on TCP mode
- Once the connection is done, change the mode to TLS and push the config
File Permissions and Types:
Sometimes event logs are received but file collection module isn’t receiving any logs, either the file path wasn’t set correctly or the lpagent didn’t had enough permissions to read those files on the system.
Evtx files can't be fetched using file collection module.
Queue in Normalization:
Windows logs can cause queue in normalization layer if the normalization packages are in wrong sequence. For example; always make sure windows normalization policy has LPA_Windows above DNSCompiledNormalizer (has some regex checks which takes longer time). After that restart the normalizer service and queue should be decreased, 2nd and 3rd column would have persistent large values if queue is present in below output.
netstat -nap | grep :5505____________________________________________________________tcp 0 0 127.0.0.1:5505 0.0.0.0:* LISTEN - tcp 0 0 127.0.0.1:58306 127.0.0.1:5505 ESTABLISHED - tcp 0 0 127.0.0.1:58092 127.0.0.1:5505 ESTABLISHED - tcp 0 0 127.0.0.1:5505 127.0.0.1:58292 ESTABLISHED - tcp 0 0 127.0.0.1:5505 127.0.0.1:58284 ESTABLISHED - tcp 0 0 127.0.0.1:58284 127.0.0.1:5505 ESTABLISHED –
Connection Issue:
Firewall Rules:
Port 7994-5 is used by LP to share conf and certs with windows. Inbound firewall for 7994 and 7995 needs to be open at windows.
[7994] – Unencrypted Communication Channel[7995] – Encrypted Communication Channel
Port 7998-9 is used to collect logs from windows by a service called lpanx_collector, outbound firewall is needed for 7998-7999 from windows to LP
[7998] – lpanx_collector [Unencrypted data channel][7999] – lpanx_collector [Encrypted data channel]
Note: This rules applies to the Collector LP Machine not Backend, if backend and collector LP are different. Please make sure the firewall or switch between logpoint and windows also have necessary rules and routes setup.
How to test if the endpoints are reachable and connected from Logpoint:
For unencrypted connection:
> telnet <windows_ip> 7994____________________________________________________________Trying windows_ip ...Connected to windows_ip# the windows is reachable from logpoint side> telnet <windows_ip> 7994____________________________________________________________Trying windows_ip ...telnet: Unable to connect to remote host: Connection refused# the windows is not reachable from logpoint side
For encrypted connection:
> openssl s_client -connect <windows_ip>:7995____________________________________________________________CONNECTED(0000003) # if reachable and certs are correct____________________________________________________________connect: Connection refusedconnect:errno=111 # if unreachable or certs are invalid
Current status:
> netstat -nap | egrep ":7999|:7998" | grep ESTABLISHED# Output contains all the established connections > netstat -nap | egrep ":7999|:7998" | grep -v ESTABLISHED# Output contains all the connections that logpoint is trying to make but fails# 5th column is the windows ip
On windows side you can just check the logs files to see if the connection is being broken or refused at C:/Program Files/lpagent/data/lpagent.log
CPU/Memory Spike:
- Unsupported file formats can cause overhead on nxlog processor.
- Selecting mem buffer instead of disk buffer can also cause memory spike if the connection isn't stable.
- Older LPAgent version had a memory leak, so please upgrade the plugin to latest version.
Note: LogPoint Agent is powered by NxLog and is just a user friendly way of sending logs via windows. If you are comfortable with nxlog, it can directly send logs to syslog port of LP which would eradicate some of the issue with LPAgent. For example: collection of custom evtx files.
Some of the commands might require root access, please consult support if above mentioned steps are confusing or didn't help you figure out the cause for the issue.