Nvidia Launches SOL‑ExecBench, a Unified Real‑World DL Kernel Benchmark for Speculative

Nvidia’s SOL‑ExecBench is built as an open‑source framework that runs inside a Docker container equipped with the NVIDIA Container Toolkit, allowing developers to evaluate custom GPU kernels under identical hardware and software conditions. The repository ships a set of scripts that first pull down two benchmark datasets—the SOL‑ExecBench core suite and the FlashInfer trace set—into a local data directory, then compile a container image and drop the user into an interactive shell where the sol‑execbench CLI can be invoked (GitHub NVIDIA/SOL‑ExecBench). The CLI accepts either a directory containing a definition.json and a workload.jsonl pair or explicit file paths, and it consumes a user‑provided solution.json that describes the kernel implementation in one of the supported DSLs (PyTorch, Triton, CUTLASS, cuDNN, CuTe, cuTile, or native CUDA C++). This design abstracts away the myriad build‑time dependencies that typically plague kernel‑level benchmarking, ensuring that each run is reproducible across any system that meets the minimal requirement of an NVIDIA driver version 580 or newer.

At the heart of the benchmark is a two‑step validation process. First, each submitted kernel is compared against a reference implementation to verify numerical correctness; any deviation beyond a tight tolerance triggers a failure, preventing “reward hacking” where a kernel might artificially lower runtime by sacrificing output fidelity. Second, the framework measures execution time under a controlled environment—GPU clocks are locked, power limits are fixed, and the same data tensors are streamed to the kernel on each iteration—to produce a deterministic latency figure (GitHub NVIDIA/SOL‑ExecBench). The benchmark’s emphasis on correctness mirrors the approach taken by the Hugging Face blog post that introduced the dataset, which stresses that speculative decoding workloads must be evaluated not just for speed but also for the integrity of the generated tokens.

Performance is reported using the SOL‑Score, a metric that normalises raw runtime against the theoretical roofline of Nvidia’s B200 GPU. The roofline model, derived analytically with Nvidia’s SOLAR tool, represents the maximum attainable throughput given the device’s memory bandwidth and compute capacity. By expressing a kernel’s speed as a fraction of this ceiling, the SOL‑Score provides a hardware‑agnostic yardstick that can be compared across different kernel languages and algorithmic strategies. According to the GitHub README, submissions are ranked on a public leaderboard that aggregates SOL‑Scores across the entire benchmark suite, allowing developers to see how their custom implementations stack up against community baselines and the official reference kernels.

The benchmark suite itself is deliberately diverse, covering a range of real‑world deep‑learning primitives that are critical to speculative decoding pipelines. Examples include attention projection kernels, matrix‑multiplication kernels, and token‑generation loops that have been extracted from production‑grade models such as FlashInfer. Each problem definition bundles a JSON‑encoded description of the kernel’s input shapes, data types, and expected computational pattern, along with a workload file that enumerates a sequence of execution traces. This structure enables the benchmark to simulate realistic memory access patterns and compute intensities rather than relying on synthetic micro‑benchmarks that often over‑estimate performance. The Hugging Face blog notes that the dataset will soon be accompanied by an ArXiv paper, underscoring the academic rigor behind the problem selection.

Beyond the technical mechanics, SOL‑ExecBench aims to address a broader workflow friction point for AI engineers. Historically, developers have had to stitch together disparate tools—profilers, custom scripts, and ad‑hoc validation suites—to assess the efficacy of a new kernel written in, say, Triton versus an existing CUTLASS implementation. By consolidating these steps into a single, reproducible pipeline, Nvidia hopes to lower the barrier to entry for kernel optimisation and to curb the “reward hacking” problem that can arise when performance is measured in isolation. The open‑source nature of the project, combined with the Hugging Face CLI integration (pip install huggingface‑hub[cli]), also encourages community contributions and makes it straightforward to publish new solutions directly to the leaderboard.

In practice, the benchmark’s utility will be judged by how well it scales to emerging hardware generations and novel DSLs. The current release targets the B200 GPU, but the roofline‑based SOL‑Score methodology is extensible: a new analytical model can be generated for future architectures, and the same benchmark definitions can be reused. This forward‑looking design aligns with Nvidia’s broader strategy of fostering an ecosystem where kernel developers can iterate rapidly, verify correctness, and benchmark performance against a common standard—ultimately accelerating the deployment of speculative decoding techniques across the AI stack.