Segment index writers do not fsync before close, write corruption stays silent #19576

Summary

We observed IllegalStateException: Unknown type N / IndexOutOfBoundsException thrown by RoaringBitmap's RangeBitmap when queries touch certain segments' range indexes. Forensic byte-diffing of affected segments (comparing corrupted local replica vs. a clean replica vs. deep store) found a consistent signature:

• A contiguous 128KiB-aligned storage block in the .range_index file.
• The first 4KB/16KB sub-page of that block is correct; the remaining sub-pages are exact byte-for-byte copies of the first sub-page.
• Deep store always has the clean bytes -- corruption is local-disk-only.
• Corruption is non-deterministic given identical input.
• Both committers (segment builders) and followers (segment downloaders) are independently affected.

Root cause theory

The full write path (BitSlicedRangeIndexCreator, MmapFileWriter, RoaringBitmap's Appender/container serializers) was read end-to-end with no application-level bug found. The duplication pattern (exact copy of the preceding 128KiB block) is most consistent with a storage/virtualization-layer fault below the page cache, but this couldn't be conclusively confirmed.

The confirmed Pinot-side gap

Independent of root cause: nothing in the segment write path ever confirms bytes reached physical storage before a checksum is computed over them, and nothing re-verifies bytes at load time.

1. Every segment index writer closed its output via a bare FileChannel.close()/unmap, relying on the OS's own lazy writeback schedule. A write that never reached disk (disk-full, storage fault) succeeded silently.
2. hasSameCRC() only compares two already-recorded checksum values; it never re-hashes live on-disk bytes, and doesn't run at all on a plain committer's own commit transition.
3. Because RoaringBitmap's RangeBitmap.SingleEvaluation.evaluateHorizontalSliceRange() only fully evaluates a bit-plane container when a query predicate happens to route into the "evaluate" branch, corruption can sit silent and undetected indefinitely, dependent on query predicate values rather than whether the segment is queried at all.

Related prior issue: #11368 ("Pinot Does not FSync and Relies on FD Close") raised the same issue against BaseJsonIndexCreator.java specifically, and was closed with a fix there.