[Bug] Core dumps are triggered by rocksdb compacting when shutdown the bookkeeper #4705

Activity

1. AnonHxy

   added

   type/bug

   on Jan 20, 2026
2. yangxianjungree commented on Aug 27, 2026

   

   yangxianjungree

   on Aug 27, 2026

   Last edited by yangxianjungree

   More actions

   We also hit shutdown crashes with entryLocationCompactionInterval enabled. Sharing what we observed; it looks related to this issue but is a different stack.

   The stack in the issue body is a caught RocksDBException: Database shutdown from entryLocationCompact → compactRange. In our environment the JVM SIGSEGV'd. The crashing thread was always db-storage-cleanup-* (the single-thread cleanupExecutor), and the Java frames were the checkpoint cleanup task queued on that executor, not compact:

   SingleDirectoryDbLedgerStorage.lambda$checkpoint$9
     → EntryLocationIndex.removeOffsetFromDeletedLedgers
         → RocksDBBatch.flush → RocksDB.write0          // locations DB
     // or
     → LedgerMetadataIndex.removeDeletedLedgers
         → KeyValueStorageRocksDB.delete → rocksdb_delete_helper  // ledgers DB
   

   What we saw:

   1. entryLocationCompact() submits a sync compactRange() onto cleanupExecutor and occupies that thread for a long time.
   2. checkpoint() / flush() finally submits a different task onto the same executor: removeOffsetFromDeletedLedgers() then removeDeletedLedgers(). That task sits in the ThreadPoolExecutor queue behind compact.
   3. SingleDirectoryDbLedgerStorage.shutdown() does cleanupExecutor.shutdown() + awaitTermination(1, SECONDS) (return value unused), then ledgerIndex.close() and entryLocationIndex.close().
   4. After compact's Java task returns, the queued checkpoint cleanup runs flush()/delete(). Those write paths are not covered by the closedLock that PR Fix the coredump that occurs when calling KeyValueStorageRocksDB.count after rocksdb has been closed #4581 added for count(). We then get a native write/delete vs close race. Compact widened the window; it was not the crashing frame.

   #4706 looks helpful for the compactRange vs close case in this issue. For our queued-checkpoint-cleanup SIGSEGV, we are not sure it would fix the scenario: close() waits on the compacting flag rather than draining cleanupExecutor, so after compact finishes the checkpoint cleanup task may still run, and ledgerIndex.close() still happens first. Happy to share more stacks if useful.