Skip to main content

TangoDB Overview & Status (v1.0.0)

TangoDB is an embedded high-performance key-value storage engine built from scratch in modern Java.

The project explores how modern JVM capabilities — specifically the Foreign Function & Memory (FFM) API, off-heap memory management, WAL crash recovery, Bloom Filters & Sparse Indexing, and lock-free concurrent data structures — can be leveraged to build a storage engine designed for extreme throughput, minimal GC latency, concurrent workloads, and predictable memory usage.

Core Architecture Highlights​

  1. Off-Heap Native Allocation: Storing raw key-value record payloads off-heap using Java's Foreign Function & Memory (FFM) API (java.lang.foreign.MemorySegment and Arena).
  2. Write-Ahead Logging (WAL) Recovery: Automated crash recovery replaying .wal log entries with CRC32C binary checksum verification and global sequence restoration.
  3. SSTable Bloom Filters & Sparse Indexes: Embedded Bloom Filter (1% false-positive rate) for instant O(1) negative lookup rejection and Sparse Index for O(log N) binary search range navigation inside .sgdb files.
  4. LSM-Tree Storage Pipeline: Log-Structured Merge-Tree storage model featuring concurrent active MemTables, immutable frozen MemTables, asynchronous background flushing (FlushManager), and background compaction (CompactionManager).
  5. Zero-Allocation Read & Write Paths: ThreadLocal key wrappers and positional channel reads eliminating heap allocation churn and OS memory leaks.

Performance Highlights (v1.0.0)​

In JMH (Java Microbenchmark Harness) microbenchmarks, TangoDB v1.0.0 achieves:

  • In-Memory MemTable Ingest: 40,769,881 ops/sec (~40.77 Million key-value put() calls per second, ~24.5 ns/op).
  • Indexed SSTable Disk Reads: 512.50 ops/sec (+34% throughput boost via Sparse Index & Bloom Filter).
  • End-to-End Database Writes: 1,545.96 ops/sec (~0.64 ms/op in WalType.SYNC mode).
Benchmark                               (records)   Mode  Cnt         Score      Error  Units
memtable.MemtablePutBenchmark.put             N/A  thrpt    3  40769881.228 ±24019114  ops/s
endToEnd.TangoPutBenchmark.put                N/A  thrpt    3      1545.959 ±  500.907  ops/s
sstable.SstManagerReadBenchmark.read          N/A  thrpt    3       512.502 ± 3871.974  ops/s
sstable.SstManagerWriteBenchmark.write     100000  thrpt    3        59.042 ±    1.272  ops/s

Explore the complete design breakdown, optimization journey, and raw benchmark scores in Performance & Benchmarks.