Skip to main content

Limitations & Future Roadmap (v1.0.0)

TangoDB v1.0.0 is a high-performance storage engine baseline focused on high-throughput off-heap ingest, WAL crash recovery, Bloom Filter / Sparse Index fast point reads, and LSM-tree persistence primitives.

Features Implemented in v1.0.0 Release​

  • ✅ Write-Ahead Logging (WAL) & Automated Crash Recovery: Replays append-only .wal logs on startup with Little-Endian CRC32C checksum verification, sequence restoration, delete tombstones, and obsolete log cleanup.
  • ✅ SSTable Bloom Filters (O(1) Fast Rejection): Embedded 1% false-positive Bfilter bit array into .sgdb files to reject missing key lookups in nanoseconds without reading disk blocks.
  • ✅ Sparse Index Binary Search (O(log(N/16))): Embedded SstSparseIndex sampling key byte offsets every 16 records, boosting disk point read throughput by +34% (up to 512.5 ops/sec).
  • ✅ Zero-Allocation Read/Write Pipelines: ThreadLocal<ByteArrayWrapper> key reuse for 40.77M ops/sec MemTable put throughput, and positional FileChannel.read eliminating OS mmap leaks and Windows file lock retention.

Current V1 Limitations​

  1. Off-Heap Memory Sizing Bounds: Native memory allocated via FFM Arena blocks bypasses standard JVM -Xmx heap limits and requires OS native memory availability.

  2. Basic Multi-Way Merge Compaction: The current CompactionManager executes basic multi-way merge compaction across flushed SSTables rather than a full multi-tiered Leveled Compaction tree (L0..Lmax).

Future Engineering Roadmap​

 ┌──────────────────────┐    ┌──────────────────────┐    ┌──────────────────────┐
 │ Leveled Compaction   │ ──►│ Vector API           │ ──►│ Block Cache &        │
 │ Tiering (L0..Lmax)   │    │ Hash Acceleration    │    │ Range Scans          │
 └──────────────────────┘    └──────────────────────┘    └──────────────────────┘

1. Leveled Compaction Architecture​

Evolve the CompactionManager into a multi-level LSM compaction pipeline (L0, L1... Ln) to bound write amplification and optimize space amplification.

2. Vector API Acceleration for Bloom Filters​

Leverage Java's Vector API (jdk.incubator.vector) to vectorize Murmur3 and hash calculations across SIMD vector registers, accelerating Bloom filter hash evaluations.

3. Block Cache & Range Scans​

Introduce an off-heap LRU Block Cache for SSTable data blocks and extend the TangoDB API to support concurrent range scans (iterator(startKey, endKey)).