Benchmarks¶

The numbers on this page come straight from BENCHMARK.md in the repo, generated by scripts/benchmark.py.

Test system¶

Property	Value
Platform	Linux 6.17 (x86_64)
CPU	16 cores
Memory	62 GB
Python	3.11.13
fast-axolotl	0.2.0 (rust: 0.2.0)

Results¶

Operation	Data size	Rust (s)	Python (s)	Speedup
Streaming Data Loading (Parquet)	50,000 rows	0.0094	0.7237	77.26x
Token Packing	10,000 sequences	0.0786	0.0327	0.42x
Parallel Hashing (SHA256)	100,000 rows	0.0273	0.0520	1.90x
Batch Padding	10,000 sequences	0.1998	0.1051	0.53x

Times are averages over 5-10 iterations. "Speedup" is Python / Rust.

Analysis¶

Streaming data loading - 77x¶

This is the headline number. The Rust reader skips the per-row Python object creation that the datasets baseline incurs:

Native arrow and parquet decoders
Zero-copy where Arrow allows it
Concurrent I/O through Tokio
Data stays in Rust until the batch is yielded as a Python dict

The benefit grows with row count and column width.

Parallel hashing - 1.9x¶

Steady gain from spreading SHA256 work across all cores. The Rust sha2 crate is highly optimised, and the implementation hands rows out to worker threads without holding the GIL.

Token packing and batch padding - sub-1x at small sizes¶

For 10,000 sequences both operations are slower than the Python baseline. The FFI cost (moving Python lists in and Rust vectors out) dominates the actual work. At production scale - sequences in the thousands and batches of hundreds - the cache-friendly buffer layout starts to pay back. See Best Practices for guidance on when to reach for these directly.

Expected gains by workload¶

Scenario	Data size	Expected benefit
Dataset loading	files >1 GB	50-100x
Deduplication	>100K rows	2-5x
Sequence packing	>1M tokens	2-4x
Batch collation	long sequences	2-3x

Reproducing locally¶

git clone https://github.com/neul-labs/fast-axolotl.git
cd fast-axolotl
uv pip install -e ".[dev]"
maturin develop --release
uv run python scripts/benchmark.py

The script writes a fresh BENCHMARK.md with results for your machine.

Methodology¶

The benchmark uses a small warmup followed by N timed runs averaged with time.perf_counter(). Python baselines call:

hashlib.sha256(...).hexdigest() for hashing
list comprehensions and extend() for padding
loop-based concatenation for packing
datasets.load_dataset(...) for streaming

Synthetic data is generated with realistic distributions (sequence length uniform in [10, 2 * avg], token IDs uniform in [1, 30000], hash inputs JSON-serialised dicts).