MLAS
MLAS (Microsoft Linear Algebra Subprograms) is ONNX Runtime’s in-tree linear-algebra layer. On RISC-V SpaceMiT X60 it exposes an IME (smt.vmadot) backend for int4 / int8 QNBit GEMM — the kernel behind ONNX Runtime’s MatMulNBits nodes (Q4×16 and SQ8 / Q8×16 m1pack paths).
Benchmark source: opensolvers/benchmarks/onnx — bench_qnbit_mlas.cpp links directly against libonnxruntime_mlas.a and calls the same entry points as ORT’s matmul_nbits.cc.
End-to-end story (Qwen / SmolLM2 / TinyLlama decode numbers): ONNX Runtime app. Raw IME: RV2. llama.cpp: app; IME1 scale-build +4.3%. Toolchain: RV2 xsmtvdot.
What it probes
../ime benchmarks raw smt.vmadot int8 GEMM. MLAS sits one layer up: 4-bit symmetric weights, BlkLen=32, external per-block scales — the packing and compute path ORT uses for quantised LLM weights.
The harness calls:
MlasQNBitGemmPackQuantBData— pack quantised B (nbits=4, no zero-point),MlasQNBitGemmBatch<float>— run the GEMM,
with thread_pool=nullptr so results are single-thread kernel rates, not ORT graph throughput.
Single-thread kernel rates (X60, one core)
Stock CompInt8 before m1pack (historical baseline):
| Shape (M×K×N) | Kernel rate | packed B |
|---|---|---|
| 1 × 4096 × 11008 | 0.47 GOP/s | 22.5 MB |
| 1 × 11008 × 4096 | 0.39 GOP/s | 22.5 MB |
| 1 × 4096 × 4096 | 0.48 GOP/s | 8.4 MB |
Shipped m1pack (Q4×16 panels + M1 asm; BlkLen=32) lands near ~9.5–10.4 GOP/s on 1×4096×11008. Panel loop (N-outer / M-inner, StrideN=16) adds ~19% on 4×4096×11008 (2.10 → 2.51 GOP/s); M=1 neutral. TCM-resident packed B does not help ORT e2e on RV2 — see RV2 IME/TCM.
These isolate kernel efficiency. Production ONNX inference fans the same kernel across cores (~6× scaling at M=1 decode).
RISC-V pack gotcha
The X60 IME backend registers only the plain SQ4BitGemmPackQuantBData (memcpy), not the x64-style ...AndBlkSum variant. Scales stay external and are passed at compute time via QuantBScale. A second “finalize” pack call with QuantBData=nullptr (x64 recipe) memcpys from NULL and segfaults — the harness does a single data-only pack.
Kernel tuning note
Once the correct ORT compute path was confirmed (accuracy_level=4 → SQNBIT_CompInt8), a hand-written RVV gemv “fast path” inside the IME kernel was 28% slower than the stock smt.vmadot tile path at M=1 and was reverted. Config beat micro-optimisation — see ONNX app.
Reproduce
make board CXX=$GCC14/bin/g++
LD_LIBRARY_PATH=$GCC14/lib64:$LD_LIBRARY_PATH ./qnbit-mlas-bench 1 4096 11008 50
Toolchain: ONNX Runtime 1.29.0, foss/2025b, X60 smt.vmadot (XsmtVdot v1.0), -march=rv64gcv_zvl256b_zfh_zvfh.