Throughput Benchmark

Overview

The Throughput Benchmark assesses a validator’s raw arithmetic capability by measuring performance on large matrix multiplications.

This benchmark focuses on fundamental numerical throughput, independent of model-specific inference tasks.

Procedure

• Validators multiply pairs of random matrices:

  $$C = A \times B$$

  with:

  • $A \in \mathbb{R}^{n \times k}$
  • $B \in \mathbb{R}^{k \times m}$

• Matrix sizes increase until the validator can no longer complete the operation within the latency threshold.

• Each multiplication involves:

  $$\text{FLOPs} = 2 \times n \times m \times k$$

• The validator’s throughput is computed as FLOPs per second at the maximum successful input size.

Scoring

• The benchmark returns the largest matrix size $n$ completed correctly and timely.
• This size serves as the validator’s throughput score and affects reputation.

Significance

• Provides a direct measure of raw compute throughput.
• Complements inference benchmarks by focusing on generic numerical tasks.
• Enables transparent evaluation of validator hardware performance.