Inlining policy

vecxt uses Scala 3's inline where it is load-bearing and plain defs everywhere else. This page explains the rule, why it is drawn where it is, and what went wrong when it was not.

Current state: the JVM array layer (src-jvm/{double,float,int}arrays.scala) is converted. The JS and Native mirrors and the matrix layer still carry the old universal-inline convention and are being worked through — so if you open src-js/floatarrays.scala and find 114 inline def, that is a backlog, not a policy.

If you are contributing a new operation, the checklist near the bottom is the short version.

The rule

Use inline when the compiler erases something the JIT cannot recover: closure identity, a compile-time constant, or a concrete type at the point where elements are touched. Do not use inline merely to avoid a call.

Corollary: inline only pays when the compiler finishes the job. An inline f: A => B parameter erases a closure. A closure handed onward to a non-inline method is not erased by anything — inline copies it to every call site. See "Closures that don't get erased" below.

Everything else here is justification.

Two different inliners

It is tempting to reason that "inlining is nearly always a win, so inline everywhere is nearly always a win." The premise is true and the conclusion does not follow, because the folklore is about a different mechanism.

C2's inlining is nearly always a win because of the selectivity, not because of the inlining. It inlines what is hot and small, declines what is not, and decides separately at each call site using data that does not exist at compile time.

Scala's inline keeps the expansion and discards the selectivity. That is not "more inlining"; it is a strictly less informed policy applied unconditionally.

They are also not additive — Scala's inline spends the budget C2's inliner needs.

The measurements

mill bytecodeAudit.test reports emitted bytecode size per method into out/bytecode-audit/method-sizes.csv, and fails the build on the size cliffs described below. The probe (experiments/src/Kernel.scala) chains N element-wise array operations in one method, measured before de-inlining and after:

Marginal cost per operation before de-inlining: ~85 bytes. Each leaf is a SIMD loop plus a scalar tail, around 110 bytes.

The thresholds these run into:

chain04 at 391 bytes is the finding that matters. Five ordinary operations, past FreqInlineSize — so a user's method containing one such expression could no longer be inlined into its caller. Our inlining disabled theirs.

Two things this also corrects:

• HugeMethodLimit needs ~94 chained operations. Reachable in a real numeric kernel — a pricing routine, a forward pass written as chained array ops — but not in ordinary code. Earlier drafts of this page implied a user could fall off that cliff casually. They can't.
• 65535 is not reachable by chaining at all (~770 operations). vecxt did hit it as a compile error, and the mechanism was layering with fan-out: an inline ndarray operation expanding several inline matrix operations, each expanding several inline array operations. Three levels with a fan-out of five is a 125× multiplier on a ~110-byte leaf. Layering multiplies; chaining adds.

MaxInlineLevel (15 frames) is the related limit: Panama's Vector API works by inlining a deep chain of @ForceInline intrinsics, and when that chain runs out of depth DoubleVector stops being register-resident and becomes a heap allocation per operation. The practical effect of bloat is usually inward rather than outward — a method already carrying expanded loop bodies has a large IR graph and no room left to absorb the inlining Panama depends on.

The limit that is not measured in bytecodes

Every threshold above is a bytecode count, which is why static analysis can enforce them. There is a fifth one that is not, and for SIMD kernels it is the tightest of them.

InlineSmallCode (2500) applies to a callee that has already been compiled: if its nmethod's machine code exceeds the limit, C2 will not inline it into a new caller, however hot that caller is. Nothing about bytecode size predicts this, because the ratio between the two is not a constant.

Measured on a LogCompilation run of the jitAudit kernels — Microsoft OpenJDK 25.0.4, x86-64, 8-lane double species — reading stub_offset - insts_offset off each c2 <nmethod> element:

Roughly 200 bytes of fixed overhead, plus 7–10× the bytecode for vectorised code with a masked tail.

At that ratio InlineSmallCode is reached at around 260–300 bytecodes, which is below FreqInlineSize. So for a SIMD kernel the 325-byte budget is not the binding constraint, and a kernel that satisfies it can still be one C2 declines to inline. The @HotPath annotation asserts the bytecode budget because that is what bytecode analysis can see; it should be read as necessary rather than sufficient.

The forwarder row is the one worth staring at. vecxt.all.clamp! is an eleven-bytecode export, and its compiled form is 1696 bytes because C2 inlined the kernel into it. A budget expressed in bytecodes — which is what @Thin asserts — cannot see that at all.

Two things are deliberately not claimed here. The ratio is one workload on one CPU at one lane width, so treat the direction as established and the crossover point as approximate. And whether HotSpot's MaxTrivialSize/MaxInlineSize fast paths let a small callee bypass the InlineSmallCode veto is unverified — if they do not, the forwarder row describes a real hazard rather than a curiosity.

Enforcing this needs the compiled size, which only the JVM's own compilation log carries — and the check that would have read it (D2 of #105) was deliberately not built. The reasoning is in jitAudit/package.mill; the short version is that intrinsic ids and inline-failure strings are JDK-internal, drift between releases, and drift in a way indistinguishable from the regression the check is looking for.

So this limit is documented and unenforced. What covers it indirectly is the allocation measurement: a kernel that stops being inlined also stops having its Vector temporaries scalarised, which shows up as bytes per operation. That catches the consequence rather than the cause, and only for kernels annotated @AllocFree.

Where inline is not negotiable

Higher-order functions

inline def reduce(inline f: (Double, Double) => Double): Double

Without inline, f.apply is a real call site. Called from several places with different lambda classes it becomes megamorphic: C2 gives up past two receiver types and emits a virtual call it will not inline. Nothing downstream survives — no scalar replacement, no optimisation across the boundary.

With a generic element type it is worse. Scala 3 has no @specialized, so f: A => B erases to Function1[Object, Object] and every primitive boxes on every element. No specialised JFunction variant helps, because A is not known at the definition site.

inline erases the closure entirely — body spliced in, no lambda, nothing boxed, no call site to become megamorphic. This is the strongest case for inline in the language and the ndarray layer relies on it throughout.

Operator dispatchers

private inline def logicalIdx(inline op: VectorOperators.Comparison, num: Double): Array[Boolean] =
  ...
  inline op match
    case VectorOperators.LT => while i < n do idx(i) = vec(i) < num; i += 1
    case VectorOperators.GT => ...

Two things depend on inline. The inline op match reduces at compile time to exactly one tail loop — and a non-reducible inline match is a compile error, not a silent runtime fallback, so the reduction is guaranteed rather than hoped for. And the Vector API only intrinsifies when the operator reaches C2 as a constant.

De-inlining these turns op into a runtime parameter, at which point the vector operations quietly stop being intrinsified. The test suite stays green. Only -XX:+PrintIntrinsics or a benchmark reveals it.

Type specialisation, but only where the element type is abstract

This is the case most easily got wrong, so note precisely what dimCheck does:

protected[vecxt] object dimCheck:
  // element type abstract -> inline, so the concrete type is visible at expansion
  inline def apply[A, B](a: Array[A], b: Array[B]) =
    if a.length != b.length then throw VectorDimensionMismatch(a.length, b.length)

  // element type known -> plain def, one copy in the library
  def apply(a: Array[Double], b: Array[Double]) =
    if a.length != b.length then throw VectorDimensionMismatch(a.length, b.length)
  def apply(a: Array[Float], b: Array[Float]) = ...
  def apply(a: Array[Int], b: Array[Int]) = ...
  def apply(a: Array[Long], b: Array[Long]) = ...

Array[A] with unbounded A erases to Object, so a.length cannot compile to an arraylength bytecode — it routes through ScalaRunTime.array_length, a runtime match over nine array types, megamorphic when shared across the Double, Float and Int paths. The generic arms therefore need inline. The concrete overloads do not: the element type is already known, a.length is one instruction, and at ~5 bytecodes C2 inlines them for free.

"Is it generic?" is the wrong question. "Is it generic at the point where it touches elements?" is the right one. Measured:

So inside extension (m: Matrix[Double]), A is not abstract even though the class is generic — element access is a cheap cast plus a typed load with no inline at all. Only extension [A](m: Matrix[A]) needs it, and only in methods that reach an element. A method taking using ClassTag[A] can allocate Array[A] without inline; reading arr(i) with A abstract still boxes. Allocation and access are different questions.

Closures that don't get erased

This is the failure mode that crosses no threshold and appears in no metric.

inline def +(vec2: Array[Double]): Array[Double] =
  dimCheck(vec, vec2)
  vec.clone.tap(_ += vec2)          // do not do this

tap is scala.util.chaining, an ordinary method taking f: A => Unit as a runtime parameter. The closure must therefore be materialised — inline does not erase it, it relocates it. Each call site in user code receives its own $anonfun method (~34 bytes), its own invokedynamic linkage spinning a LambdaMetafactory hidden class on first execution, and a capturing closure allocation per invocation.

A 41-method probe file produced 19 synthesised lambda methods and 19 hidden classes from source containing no lambdas. Their combined bytecode exceeded the method that generated them.

Write it out instead:

def +(vec2: Array[Double]): Array[Double] =
  dimCheck(vec, vec2)
  val out = vec.clone
  out += vec2
  out

Before assuming inline is helping, grep the body for .tap(, .pipe(, .foreach(, .map(. If a lambda is handed to a non-inline callee, inline makes it worse, not better.

Where inline is not helping

For the bulk of the arithmetic surface — element-wise operations on Array[Double], Array[Float], Array[Int] — none of the above applies:

def *(d: Array[Double]): Array[Double] =
  dimCheck(vec, d)
  val n     = vec.length
  val bound = spd.loopBound(n)          // spd: species, spdl: lane count
  val out   = new Array[Double](n)
  var i = 0
  while i < bound do
    DoubleVector.fromArray(spd, vec, i).mul(DoubleVector.fromArray(spd, d, i)).intoArray(out, i)
    i += spdl
  end while
  while i < n do
    out(i) = vec(i) * d(i); i += 1
  end while
  out

No closure. No abstract element type. No operator to keep constant. Extension methods compile to static methods on the module class, so the call is monomorphic by construction and cannot become megamorphic. The DoubleVector values are born and die inside one loop iteration, so they never cross the method boundary and escape analysis sees the same graph either way.

Plain def. C2 inlines it when hot, per call site, with better information than we have.

On escape analysis specifically

A common argument for inline around Panama is that vector values are box-like pre-Valhalla, so inlining is needed to keep them in registers. Escape analysis runs on C2's IR after C2's own inlining, so the question is only ever whether C2 inlined the call. Scala-level inlining can only help where C2 would otherwise have declined — and it makes methods bigger, which makes C2 more likely to decline. The mechanism undercuts itself at exactly the margin where it would matter.

Where it genuinely applies is closures, which is why the higher-order functions above keep inline.

The bigger lever: intermediates

Worth stating plainly, because it dwarfs everything above.

chain04 — (a * b + a - b + a).sum — makes five passes over memory and allocates four intermediate arrays. The hand-written loop makes one pass and allocates nothing. At 10k elements that is ~320KB of garbage per call.

No inline decision changes this. Fusion does. If you are optimising a chain of vecxt operations, the intermediates are the larger number and the bytecode is a rounding error beside them.

Bounds checking, and a bug worth remembering

vecxt used to carry a compile-time flag:

object BoundsCheck:
  type BoundsCheck = Boolean
  object DoBoundsCheck:
    inline given yes: BoundsCheck = true
    inline given no: BoundsCheck = false

Importing DoBoundsCheck.no erased dimension checks. Because inline if doCheck can only reduce if the flag is a compile-time constant at the point of expansion, every method in the chain had to be inline to propagate it. That single requirement is what pushed inline across most of the library — 993 inline def against 57 plain def at its peak.

It has been removed, for two reasons.

It conflated a correctness check with an optimisation

Most members of the check family validated arguments and threw. One did not:

def apply[A, B](a: Matrix[A], b: Matrix[B])(using inline doCheck: BoundsCheck): Boolean =
  inline if doCheck then
    a.isDenseColMajor && b.isDenseColMajor && a.rowStride == b.rowStride || ...
  else true

This is not validation. It is capability detection — "are these two matrices laid out such that a flat loop over the backing arrays is valid?" — and matrix element-wise operations branch on it to choose between a contiguous fast path and a stride-aware slow path.

Returning true unconditionally under DoBoundsCheck.no meant transposed, sliced, and strided matrices took the contiguous path. Wrong values, or reads past the logical extent, with no exception. Across 13 call sites and all four platforms.

• Validation — "throw if the caller is wrong." Optional in principle; skipping it trades a diagnostic for speed.
• Capability detection — "which algorithm is valid here?" Never optional. Skipping it does not lose a diagnostic; it runs the wrong code.

The two are easy to confuse when both are spelled somethingCheck and live in the same file.

The checks probably paid for themselves anyway

An upfront a.length != b.length is exactly the fact C2 needs to prove b(i) is in range for i < a.length, and therefore to eliminate the per-access range check inside the loop. You cannot remove the JVM's own bounds checks — that is memory safety — you can only give C2 enough information to prove them redundant. The dimension check is that information; it is why java.util.Arrays does upfront checks.

So the flag plausibly cost more than it saved on two-array operations, while forcing inline across the whole library to exist at all. (Untested prediction: a differential benchmark should show no penalty on two-array ops and exactly one compare on array-plus-scalar ops.)

Checks are now unconditional and kept in the same method as the loop they guard, so the length fact and the accesses are in one compilation unit.

Loop bounds and lazy val

Write the loop limit as a local val:

val n     = vec.length
val bound = spd.loopBound(n)
var i = 0
while i < bound do ...

not while i < spd.loopBound(vec.length) do.

loopBound is @ForceInline and reduces to length & -laneCount, so in C2 steady state the difference is nil. The reason to hoist is loop recognition: for C2 to treat the loop as counted — and so unroll it, eliminate range checks, and align it — the limit must be provably invariant, which depends on loopBound having been inlined. Hoisting removes that dependency and also removes the per-iteration call in the interpreter and C1 tiers.

For the same reason, never read a lazy val in a loop condition. Scala 3 compiles lazy val to a bitmap read with volatile semantics, and a volatile read is a memory barrier C2 cannot hoist — so the loop is not counted and loses unrolling and range-check elimination. Matrix.numel and NDArray.numel were lazy vals for this reason and are now eager vals. A lazy val caching cheap arithmetic is a pessimisation twice over.

Also hoist repeated field reads out of loops — val a = m.raw before the loop, not m.raw(i) inside it. C2 will hoist the getter and its checkcast, but there is no reason to depend on it.

Checklist for new operations

1. Takes a function parameter? → inline, with inline on the parameter too.
2. Needs a VectorOperators constant, or an inline match on one? → inline.
3. Generic at the point where it touches an element? → inline. Generic in signature only, or concrete element type → plain def.
4. Body creates a lambda for a non-inline callee (.tap, .pipe, .foreach, .map)? → rewrite without the lambda. inline will duplicate it, not erase it.
5. None of the above? → plain def. Hoist the loop bound, hoist field reads, keep the dimension check in the same method as the loop.

And separately:

1. Writing something named ...Check that returns a value the caller branches on? → It is not a check. Name it as a predicate, keep it unconditional, and put it with the other predicates.

Verification

# every static check: size bands against the budgets below, the specialization scan,
# and the bytes-per-operation ratchet. Writes out/bytecode-audit/{report.md,method-sizes.csv}
mill bytecodeAudit.test

# hidden lambdas: should return only lambdaDeserialize
javap -c -p 'probe.Kernel$.class' | grep invokedynamic

# what the Scala compiler expanded, per call site
scalac -Xprint:inlining Kernel.scala

# C2's decisions, and whether the vector ops are still intrinsified
java -XX:+UnlockDiagnosticVMOptions -XX:+PrintInlining -XX:+PrintIntrinsics \
     --add-modules=jdk.incubator.vector ...

PrintIntrinsics is the important one. Losing intrinsification is the failure mode that does not break a single test.

Thresholds quoted here are HotSpot defaults and do not transfer to OpenJ9 or GraalVM native-image. MaxInlineSize, FreqInlineSize and InlineSmallCode are visible via java -XX:+PrintFlagsFinal -version | grep -i inline; HugeMethodLimit is a develop flag and will not appear there, though DontCompileHugeMethods will.

That listing also records which of them are portable. MaxInlineSize and MaxInlineLevel are plain C2 product flags, but FreqInlineSize and InlineSmallCode are C2 pd product — platform-dependent, so both can legitimately differ on another architecture. Two of the four budgets this page relies on are properties of the machine, not of HotSpot.

What is enforced, and what is still convention

The rule above is a convention. Three parts of it are now checked on every PR by bytecodeAudit (#105), which reads the budgets off the running JVM rather than trusting the numbers in this page — including the HugeMethodLimit caveat in the paragraph above, which the report labels as an assumed constant rather than a discovered one:

The rest — closure identity, compile-time constants — is still convention. C5 checks it and is Phase 3.

Two things the checks cannot see, and both matter for reading this page.

An inline def body is expanded into its callers rather than emitted, so it has no bytecode of its own. A generic inline def is audited only through whatever non-inline callers exist. That is why @HotPath and @Thin are defined as properties of emitted methods, and why putting one on an inline def is a build failure rather than a no-op.

And bytecodeAudit reads bytecode, so none of C1/C2/C3 can see the InlineSmallCode limit described above. The two budgets those checks enforce are proxies for a machine-code constraint that is, for vectorised kernels, tighter than either of them. A passing @HotPath therefore means "inside the bytecode budget", not "C2 will inline this".

The dynamic tier could close that gap and deliberately does not. jitAudit implements D1 (allocation per @AllocFree kernel), D3 (species reporting), D4 (SIMD-vs-scalar differential) and D6 (harness canary), plus an escape-analysis-off cross-check. It does not implement the two checks that would have parsed -XX:+LogCompilation output — D2, intrinsic confirmation, and D5, deopt churn. The line is which API a check depends on: ThreadMXBean, VectorSpecies, arithmetic and a product-grade -XX: flag are public and contractual; the compiler's diagnostic XML is neither, and its vocabulary drifts in a way that cannot be told apart from the regression being looked for. jitAudit/package.mill has the full reasoning and the measurements that led to it.

The honest summary of the page, then: the static checks enforce bytecode budgets that are necessary but not sufficient, the dynamic checks catch the consequences of losing vectorisation rather than the cause, and the gap between them is documented rather than closed.