Indexsearcher write.lock Issues?

Has anyone of you seen errors during running a search over a larger timerange (+24 hours) from the indexsearchers reporting a missing “write.lock” file?

Here is an example log of such a case:

2022-12-11_00:00:45.94320 00:00:45.943 [pool-1-thread-3] WARN c.i.shared.lib.logging.util.LogUtil - Exception occurred during index closing;2022/08/11/1660176000
2022-12-11_00:00:45.94330 00:00:45.943 [pool-1-thread-3] WARN c.i.shared.lib.logging.util.LogUtil - Exception occurred during index closing;2022/08/09/1660003200
2022-12-11_00:00:45.97880 java.nio.file.NoSuchFileException: /opt/makalu/storage/indexes/M365/2022/08/11/1660176000/write.lock
2022-12-11_00:00:45.97888 at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
2022-12-11_00:00:45.97888 at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
2022-12-11_00:00:45.97888 at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
2022-12-11_00:00:45.97889 at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
2022-12-11_00:00:45.97889 at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
2022-12-11_00:00:45.97890 at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
2022-12-11_00:00:45.97891 at java.nio.file.Files.readAttributes(Files.java:1737)
2022-12-11_00:00:45.97892 at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:177)
2022-12-11_00:00:45.97892 at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:43)
2022-12-11_00:00:45.97892 at org.apache.lucene.index.SegmentInfos.write(SegmentInfos.java:516)
2022-12-11_00:00:45.97892 at org.apache.lucene.index.SegmentInfos.prepareCommit(SegmentInfos.java:809)
2022-12-11_00:00:45.97893 at org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:4439)
2022-12-11_00:00:45.97893 at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2874)
2022-12-11_00:00:45.97893 at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2977)
2022-12-11_00:00:45.97893 at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2944)
2022-12-11_00:00:45.97893 at com.logpoint.indexsearcher.indexer.ImmuneIndex.closeAll(ImmuneIndex.java:179)
2022-12-11_00:00:45.97894 at com.logpoint.indexsearcher.indexer.IndexManager.deleteExtraIndexer(IndexManager.java:1147)
2022-12-11_00:00:45.97894 at com.logpoint.indexsearcher.indexer.Indexer.checkPartition(Indexer.java:134)
2022-12-11_00:00:45.97894 at com.logpoint.indexsearcher.indexer.Indexer.createPartition(Indexer.java:347)
2022-12-11_00:00:45.97894 at com.logpoint.indexsearcher.indexer.Indexer.index(Indexer.java:266)
2022-12-11_00:00:45.97894 at com.logpoint.indexsearcher.indexer.MultiThreadedRunnableIndex.index(MultiThreadedRunnableIndex.java:148)
2022-12-11_00:00:45.97894 at com.logpoint.indexsearcher.indexer.MultiThreadedRunnableIndex$IndexingThread.run(MultiThreadedRunnableIndex.java:323)
2022-12-11_00:00:45.97895 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2022-12-11_00:00:45.97895 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2022-12-11_00:00:45.97895 at java.lang.Thread.run(Thread.java:748)
2022-12-11_00:00:45.97908 java.lang.RuntimeException: /opt/makalu/storage/indexes/M365/2022/08/11/1660176000/write.lock
2022-12-11_00:00:45.97910 at com.logpoint.indexsearcher.indexer.Indexer.createPartition(Indexer.java:378)
2022-12-11_00:00:45.97911 at com.logpoint.indexsearcher.indexer.Indexer.index(Indexer.java:266)
2022-12-11_00:00:45.97911 at com.logpoint.indexsearcher.indexer.MultiThreadedRunnableIndex.index(MultiThreadedRunnableIndex.java:148)
2022-12-11_00:00:45.97911 at com.logpoint.indexsearcher.indexer.MultiThreadedRunnableIndex$IndexingThread.run(MultiThreadedRunnableIndex.java:323)
2022-12-11_00:00:45.97911 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2022-12-11_00:00:45.97911 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2022-12-11_00:00:45.97912 at java.lang.Thread.run(Thread.java:748)
2022-12-11_00:00:45.97927 00:00:45.979 [pool-1-thread-3] ERROR c.i.shared.lib.logging.util.LogUtil - IndexingThread; indexingThread; /opt/makalu/storage/indexes/M365/2022/08/11/1660176000/write.lock
2022-12-11_00:00:48.00775 Exception in thread "Thread-1" java.lang.RuntimeException: java.nio.file.NoSuchFileException: /opt/makalu/storage/indexes/M365/2022/08/11/1660176000/write.lock
2022-12-11_00:00:48.00777 at com.logpoint.indexsearcher.indexer.SaveOnShutdown.run(SaveOnShutdown.java:21)
2022-12-11_00:00:48.00789 Caused by: java.nio.file.NoSuchFileException: /opt/makalu/storage/indexes/M365/2022/08/11/1660176000/write.lock
2022-12-11_00:00:48.00790 at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
2022-12-11_00:00:48.00790 at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
2022-12-11_00:00:48.00790 at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
2022-12-11_00:00:48.00790 at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
2022-12-11_00:00:48.00791 at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
2022-12-11_00:00:48.00792 at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
2022-12-11_00:00:48.00792 at java.nio.file.Files.readAttributes(Files.java:1737)
2022-12-11_00:00:48.00792 at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:177)
2022-12-11_00:00:48.00792 at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:43)
2022-12-11_00:00:48.00793 at org.apache.lucene.index.SegmentInfos.write(SegmentInfos.java:516)
2022-12-11_00:00:48.00793 at org.apache.lucene.index.SegmentInfos.prepareCommit(SegmentInfos.java:809)
2022-12-11_00:00:48.00793 at org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:4439)
2022-12-11_00:00:48.00793 at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2874)
2022-12-11_00:00:48.00793 at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2977)
2022-12-11_00:00:48.00794 at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2944)
2022-12-11_00:00:48.00794 at com.logpoint.indexsearcher.indexer.ImmuneIndex.closeAll(ImmuneIndex.java:179)
2022-12-11_00:00:48.00794 at com.logpoint.indexsearcher.indexer.IndexManager.close(IndexManager.java:145)
2022-12-11_00:00:48.00795 at com.logpoint.indexsearcher.indexer.Indexer.shutdownIndexer(Indexer.java:206)
2022-12-11_00:00:48.00796 at com.logpoint.indexsearcher.indexer.SaveOnShutdown.run(SaveOnShutdown.java:19)
2022-12-11_00:00:48.79674 Starting indexsearcher_M365
2022-12-11_00:01:10.64122 creating new msg retriever

We see this often (3-4 times per week) but irregularly. Sometimes (as in the error log above) the indexsearcher is crashing and restarting, which “solves” the problem temporarily for a few hours or days.

We excpect that this also causes alert rules to not run properly.