When we encounter cases when report fails to generate, There might be several reasons why this happened. Here are the list of things that can be attempted to handle report generation issues:
- Try running the report again
Sometimes, report might fail when there is high load in the system. And just trying to run the report again after sometime might result successful generation of report.
- Try increasing the heap_size of the report
There is an option in the report template (in the right side) to increase the heap_size of the report job. Large reports (reports with more number of queries and large time ranges) require larger memory.
So, increasing the heap_size incrementally from 512 MB to few GBs (up to 4GB) can be helpful.
- Try searching for each of the query in the report in search, if they are working fine or not
This is kind of a brute force way to figure out the problematic query or query which might be causing the report failure.
If any of the repo is down or if any of the query has issues, then report might fail. So, even though this step might take some time, this can be really helpful.
- Try splitting the report if there are too many queries
Sometimes, due to large size of the report (both in number of queries as well as in terms of timerange), the amount of data to be handled can be rocket high, hundreds of millions of events.
Splitting the report both in terms of queries as well as splitting the timerange of the report can also be helpful.
- For Scheduled Reports
If the report is scheduled, it is wise to see if there are any other reports scheduled at the same time or not. If that is the case then, trying to schedule less reports simultaneously can help. This will reduce the load in the system.
- Contact to support
Even after trying above steps, if the report is still failing then it's time for some expert eyes. Please contact the support in that scenario. But try to attach the following information in the beginning itself so that support staff can act quickly in the issue.- Attach the screenshot of the error message as seen in the report jobs status.
- Also, attach the following file from the backend: /opt/immune/var/log/report/report.log
This file contains more details about the failed report.
Some Potential Scenarios
1. Report Gets Timed Out
Sometimes large reports might take more time to complete than the default time out period for the reports (30 minutes). If the report times out, the timeout period needs to be increased to accommodate for large reports. This can be done by support staff.
2. Report stays at running state and neither completes nor fails
If the indexsearcher service is restarted midway when a report is generating, then the report can stay in running state. In that case, we need to check if any related repo's indexsearcher service has restarted when report was running.
And if that is true, cancel the report job and start generating report again.