Skip to content

How the benchmarks were taken

The results are on the benchmark page: five libraries over four models, with the numbers under each chart. This page is how they were taken.

Every published number is the median of a measurement's rounds, and every band is the first to the third quartile of the same rounds. Nine rounds is the floor. Not the fastest round — it is a best-of-n and n is not equal, since the harness calibrates by duration. Not the mean — one round in forty of a 20 ms measurement took 1.5 s here, which drags a mean to 2.9x its median.

That choice cost us: nine cells flipped against lpspec, all on the gurobi sink, where our build alternates between a fast and a slow state round after round and no other library's does (#1288).

How to reproduce it

uv run --locked bench/reproduce.py

bench/reproduce.py.lock freezes every version, git commits included — two of the five libraries install from git and one of those is a branch, so without it "the versions that produced this number" is unrepeatable. --locked refuses to start if the resolution has drifted.

Everything the tables are drawn from is in bench/results — one file per sink and case, since each is measured in a process of its own, and each carries the machine, the versions, the commit and every round of every measurement. A case the box could not finish leaves no file behind, so the directory holds what ran and nothing else. pixi run table prints it as one long CSV and commits nothing — the JSON is the archive because it keeps the rounds. Re-taking rather than reproducing is pixi run refresh, which writes the tables into their fences and the chart's data literal into its own.

First model against every model after it

One model built, then built again in the same process.

Marginal cost per model

Build only, repeated in one process. first is the first recorded round and steady the best of the rounds after it, so the pair is what a rolling horizon pays for its second window against its first. The harness warms up before it records, so neither column carries the one-time import cost: the median gap between them is +8.7 ms on lpspec and +1.5 ms on linopy and +5.7 ms on pyomo and +17.2 ms on gurobipy-loop and +29.4 ms on gurobipy-matrix.

Read down a column, not across the row. The build is not the same work in every library — one that defers materialising its coefficients to its writer spends almost nothing here and pays it at the seam — so these columns carry no ratios. The tables above measure to a common artifact and are where a comparison belongs.

case vars lpspec: first lpspec: steady linopy: first linopy: steady pyomo: first pyomo: steady gurobipy-loop: first gurobipy-loop: steady gurobipy-matrix: first gurobipy-matrix: steady
dispatch 10k 35.2 ms 28.1 ms 27.4 ms 26.1 ms 36.8 ms 35.0 ms 27.0 ms 26.9 ms 28.4 ms 28.6 ms
fleet 12k 138.0 ms 137.6 ms 164.9 ms 164.2 ms 42.1 ms 38.9 ms 100.6 ms 73.6 ms 25.9 ms 24.6 ms
dispatch 100k 45.1 ms 34.7 ms 55.5 ms 26.9 ms 290.7 ms 282.5 ms 230.1 ms 213.0 ms 112.3 ms 86.3 ms
fleet 120k 155.4 ms 162.6 ms 167.9 ms 166.7 ms 783.1 ms 771.6 ms 861.8 ms 868.0 ms 186.6 ms 158.5 ms
dispatch 1M 141.9 ms 136.9 ms 37.4 ms 35.3 ms 4914.9 ms 4874.3 ms 2678.5 ms 2709.3 ms 919.3 ms 852.7 ms
fleet 1.2M 350.9 ms 322.4 ms 189.2 ms 188.3 ms 6118.4 ms 6135.4 ms 8697.9 ms 8308.3 ms 1697.9 ms 1667.3 ms
dispatch 10M 652.8 ms 569.6 ms 164.5 ms 158.6 ms — — 28775.9 ms 28746.3 ms 9084.8 ms 9012.6 ms
fleet 12M 1590.7 ms 1519.8 ms 470.2 ms 468.3 ms — — — — 17277.6 ms 17067.0 ms

The same size, reached by widening

Entity counts x N, snapshots fixed — the same sizes as the ladder, in a different shape.

The width ladder

Entity counts x N with the snapshot count held fixed, through the highs sink. Each rung matches one of the size ladder rungs above variable for variable — w10 is s, w1000 is l — so the pair reads as one model at one size in two shapes.

case entities x variables wall: lpspec wall: linopy wall: pyomo wall ÷ linopy wall ÷ pyomo peak: lpspec peak: linopy peak: pyomo peak ÷ linopy peak ÷ pyomo
storage 1 10k 0.04 s 0.08 s 0.31 s 0.55x 0.14x 0.22 GB 0.24 GB 0.19 GB 0.90x 1.16x
storage 10 100k 0.06 s 0.10 s 2.91 s 0.62x 0.02x 0.25 GB 0.25 GB 0.32 GB 0.96x 0.76x
transport 1 9.8k 0.04 s 0.06 s 0.26 s 0.71x 0.16x 0.22 GB 0.25 GB 0.19 GB 0.87x 1.13x
transport 10 98k 0.06 s 0.31 s 2.45 s 0.19x 0.02x 0.25 GB 0.84 GB 0.35 GB 0.30x 0.72x

Not measured yet

Listed so that a claim with no table under it is visible as one.

  • Solve time. Every number stops at the hand-off; the simplex is the solver's work whoever filled the model.
  • The LP-file round trip. The tables price writing a file, never reading one back.
  • Sizes past l. xl and 2xl exist in the harness and no run publishes them.
  • The width ladder past w10. w100 and w1000 are left out of the published run rather than measured and dropped. transport/w100 on linopy peaks at 14.26 GB, and a measurement holds the model twice, so the cell wants more machine than the box has; the budget cannot stop it either, projecting the next rung linearly off a w10 cell that took under a gigabyte. What is lost is the runner rather than the rung (#1416). The last numbers taken there — lpspec 0.11 s and 0.59 GB against linopy 53.53 s and 14.26 GB at transport/w100 — are in #1285, on the machine that could hold them.
  • Anything about expressiveness. Four models say nothing about a fifth.

Method

One process per measurement, ru_maxrss for peak rather than a tracker, import excluded from the timing and teardown included. A run refuses to start on a machine that is already working. The rest — every flag, every default switched off and what it costs — is in bench/README.md.

Peak carries an allocator cost that only the polars arms pay. polars ships its own jemalloc settings, so a peak measured through it holds pages freed and not yet returned; an arm on the system allocator never enters jemalloc at all. Ours moves 12–27% with the decay clock on and off where linopy's does not move at three digits (#896). It runs against us and is left in.

memray never times anything. Its tracker slows an allocation-heavy engine several-fold and overcounts reserved arenas. Peak RSS is the metric; memray is for attribution.