Relationship to linopy¶
Everything about linopy in one place, because it is otherwise the kind of thing that gets mentioned everywhere and explained nowhere. Three separate relationships, and conflating them is what made the rest of the docs noisy:
| What | Where it matters | |
|---|---|---|
| Not a dependency | solving a model never imports it | packaging |
| The oracle | how we know the answers are right | testing |
| The other consumer | it reads the same YAML, natively | where to go for a linopy.Model |
1. It is not a runtime dependency¶
lps.solve, lps.build, lps.write and lps.check go YAML → polars → HiGHS
or file, and import nothing from linopy, xarray or pandas. CI proves it: the
bare-install job runs the whole suite with none of them present.
pip install "lpspec[linopy]" adds linopy, xarray and pandas, which buys two
things and nothing else — the differential oracle below, and the to_pandas /
to_dataarray bridges out of a result. A bare install is a complete one.
Nothing a bare install can reach names linopy, including in a traceback. The
public exception tree is rooted at LpspecError, with no alias
(#389) — a name from this
extra has no business reaching a caller who never installed it.
2. It is the oracle¶
Correctness here is not "the tests pass"; it is the same YAML, built both
ways, produces the same model. The differential suite builds a model through
the relational engine and through linopy.Model.from_spec, and compares.
That is only meaningful because both paths consume the same resolved AST and neither may hold its own opinion about what a name means — the narrow waist in the architecture notes. If they resolved names independently, the suite would be comparing two dialects rather than checking one language. It is also why the two pins move together: the oracle lowering with a different math-spec than this package is two languages again, by the back door.
It also has a known blind spot, which is why the model gallery exists: a shared misreading passes the differential suite green. Only an outside published optimum catches that, and docs/examples/index.md is where those live.
Where a concept is already linopy's, we copy its name — solve statuses,
status / termination_condition as two axes with is_ok as the rollup, the
shape of a result. Our audience arrives from linopy and PyPSA, and a second
vocabulary for one fact is a tax on all of them. But copy it, do not import
it: the engine may not import linopy, so the tables live here and a test
imports linopy to assert the copy still matches. A copy nobody checks is a copy
that rots.
3. It is the other consumer of the language¶
The same file, built as a linopy.Model — by linopy. Model.from_spec
reads a math-spec program natively
(PyPSA/linopy#922), so the way to
get one is to ask linopy for it:
import linopy
m = linopy.Model.from_spec('spec.yaml', sources={...})
m.solve()
m.spec.expressions['co2'].solution # a named quantity, read back
There used to be a second lane in this package doing the same job —
lpspec.linopy.build, a peer of lps.build picked by an import. It is gone.
Keeping it meant maintaining a smaller implementation of something linopy owns:
its own reads back through model.spec, survives to_netcdf and Model.copy,
and typesets the model it built, none of which the lane did.
What this package takes on instead is that the two agree. The differential
suite builds every model here and there and compares them, which is section 2
— and it is a stronger claim than it was, because the two no longer share a
reader: linopy.spec.attach enforces the language's binding rules in its own
code, where the lane went through this package's sources.py and could only
ever have agreed with itself.
What was lost with the lane, and is worth knowing:
check(spec, sink='linopy')— asking, before any data, whether linopy will take a file. Capability now covers sinks only.- The two constructs below used to raise a
LaneErrornaming the wall and the route around it. One of them is linopy's to say now, and it says it as a library exception.
Where they part: accepting is not building¶
Neither is a language limit: both files pass check, and each is built by
the one the other cannot.
The first is linopy's: an objective carrying a constant. linopy.Objective's
expression setter rejects any expression whose const is nonzero — "Constant
values in objective function not supported." — and there is no slot to put one
in, which is why PyPSA carries n.objective_constant out of band. So a model
like examples/ports/osemosys_utopia.yaml, whose objective owes a fixed cost
on capacity that already stood in 1990, builds here and raises there
(#894).
The second is a quadratic constraint, which add_constraints refuses
outright — a QuadraticExpression is not something it takes, and no
reformulation of it is exact. This engine builds one and gurobi or an .lp
file carries it.
The third is this engine's, and it is the mirror: an operator acting along a
dimension a constant part does not carry, beside a term that does — say
sum(x * k + d, over=t) where d is a scalar. A constant part compiles to its
own frame here, so a fragment with no rows for t has no slots for the
operator to act on, and under a mask which slots those are is known only to the
rows. linopy has no such split — the operand is one masked expression — so it
builds the file as written.
It is one wall, reached by all four operators that act along a dimension
(sum(over=), sum(by=), shift, sum_back), which is why they share a
refusal rather than each wording its own. The message names the rewrite that
reaches the same number — declare the parameter over the dimension and supply
it there (#1137).
Finding that wall is what turned up a real disagreement behind it: sum_back
read a constant at a slot the variable was absent from, where every other
operator drops it, so the two answered 2.5 and 3.0 on a file neither
refused. A reduction consumes its operand before any row exists, so absence has
to be pushed into the operand first — sum and sum(by=) did that and the
window did not. Fixed by giving the window the same pass, with a differential
test over every operator that moves along a dimension
(#1142).
Where they part today by accident¶
Two differences are not deliberate on either side: they are places the oracle
reads the language differently from the reference, and every one of them is a
strict xfail in the suite naming it, so the day it is fixed upstream the
suite goes red and the marker comes out.
| What | Which is right |
|---|---|
| a sparse coefficient table — rows only where the value is nonzero | this engine. Rule 8 reads a missing row as the value that contributes nothing, which for a coefficient is 0; the oracle refuses it wherever a parameter is used. Nine of the ten ported models are refused on this alone. |
a dtype: datetime dimension whose labels arrive as datetime.date |
this engine. Labels are canonicalised against the declared dtype (#1076), so one instant is one label whichever library spelled it; the oracle compares the objects it was handed. |
The same language, and the same data¶
Both accept exactly the same language, and structurally: both run the same
to_program gate, so a construct one refuses the other refuses in the same
sentence — it is math-spec's sentence, raised before either attaches a source.
The data is where they differ by design, and the difference is containers
rather than rules. This package reads tables: a parquet path, any table
exporting the Arrow PyCapsule protocol, a pd.Series carrying its dims in an
index, a dict or a sequence over one dimension, or one number. linopy reads
pandas and xarray, and takes a dimension's labels as a sequence where this
takes a one-column table. Neither reads the other's spelling of a dimension
index, and neither takes an xr.DataArray as a source here — this package
reads tables and hands arrays back.
The rules underneath are the language's and are the same on both sides: labels
come from sources or from what the file declares and never from a parameter,
a dimension the file declares and the caller supplies is refused, and a
dimension with neither has no index and is refused rather than derived.
What we deliberately do not take¶
Array operations (merge, reindex, stack), the Python modeling API, and the
solver layer. The first is data prep
(the limits), the second is
hard rule 5 — the model is the file you review
and diff — and the third is
#106, where we adopt linopy's
design for declared solver capabilities without adopting its code.
The modeling API is the one a reader arriving from linopy misses first, and
what replaces it is two notebook pages: Change a model
for the loops — update for new numbers, a longer table for more rows, a
patched dict for new math — and Fix, relax, remove for
the verbs, which are the same loops aimed at fix, relax and
remove_constraints. What neither replaces is the debugging: an IIS. Both
pages say so — a built row is read with
row, in linopy's own form.
Where linopy is genuinely ahead, and why none of it is a ceiling question, is the honest snapshot in the roadmap.
What is owed to linopy rather than merely true of it — and the same for Calliope, whose math language this surface is derived from — is prior art and credit.