numbox.utils
numbox.utils.highlevel
Dynamically defining StructRef
Defining numba StructRef requires writing a lot of boilerplate code.
A utility for concise definition of StructRef types that supports caching
is provided in numbox.utils.highlevel.make_structref(). To use it,
define a separate module such as type_classes.py such as:
from numba.experimental.structref import register
from numba.core.types import StructRef
@register
class DataStructTypeClass(StructRef):
pass
Then in a different module main.py define:
from numba.core.types import float32, unicode_type
from numpy import isclose
from numbox.utils.highlevel import make_structref
from type_classes import DataStructTypeClass
def derive_output(struct_):
if struct_.control == "double":
return struct_.value * 2
else:
return struct_.value
data_struct = make_structref(
"DataStruct",
{"value": float32, "control": unicode_type},
DataStructTypeClass,
struct_methods={
"derive_output": derive_output
}
)
if __name__ == "__main__":
data_1 = data_struct(3.14, "double")
data_2 = data_struct(2.17, "something else")
assert isclose(data_1.derive_output(), 6.28)
assert isclose(data_2.derive_output(), 2.17)
- numbox.utils.highlevel.cres(sig, **kwargs)[source]
Returns Python proxy to FunctionType rather than CPUDispatcher returned by njit
The proxy is a
DeriveWAP, typed asDeriveFunctionType, so that an exception raised inside the compiled body propagates out of a first-class call instead of being discarded. Seenumbox.utils.derive_wapfor why that requires a numbox-owned type. On numba 0.60, which has nojit_addrslot to populate, a plainCompileResultWAPis returned and the previous behaviour is kept.
- numbox.utils.highlevel.cres_if_available(lib, sig, **kwargs)[source]
Like
cres(sig, **kwargs), but stubs out the wrapper if the C symbol matchingfunc.__name__is absent fromlib.Use for binding sets that target multiple library versions where some symbols may only exist in newer releases. Callers get a stub that raises
NotImplementedErrorinstead of a confusing LLVM link error at call time.
- numbox.utils.highlevel.hash_type(ty: Type) str[source]
Process-stable content hash of a numba type; see
_type_identity().
- numbox.utils.highlevel.make_structref(struct_name: str, struct_fields: Iterable[str] | dict[str, Type], struct_type_class: type | Type, *, struct_methods: dict[str, Callable] | None = None, jit_options: dict | None = None, ns: dict | None = None)[source]
Makes structure type with struct_name and struct_fields from the StructRef type class.
A unique struct_type_class for each structref needs to be provided. If caching of code that will be using the created struct type is desired, these type class(es) need/s to be defined in a python module that is not executed. (Same requirement is also to observed even when the full definition of StructRef is entirely hard-coded rather than created dynamically.)
In particular, that’s why struct_type_class cannot be incorporated into the dynamic compile / exec routine here.
Dictionary of methods to be bound to the created structref can be provided as well. Struct methods will get inlined into the caller if numba deems it to be optimal (even if jit_options says otherwise), therefore changing the methods code without poking the jitted caller can result in a stale cache - when the latter is cached. This is not an exclusive limitation of a dynamic structref creation via this function and is equally true when the structref definition is coded explicitly.
Anchor file
The generated
code_txtis written to a content-addressed file under numba’s cache directory and that file – nothighlevel.py– is used as thecompile()anchor. See the “Cache-anchor mechanism” section indocs/numbox.utils.rstfor the rationale.
numbox.utils.derive_wap
A first-class FunctionType value carries three addresses in its data model, and which
one numba calls decides whether an exception can leave the call. The jit_addr slot holds
the numba calling convention entry point, which unwinds; when it is empty numba calls the C
wrapper instead, which it documents as not supporting exceptions.
numba populates that slot for a Dispatcher and leaves it empty for everything else, so
the compile result behind numbox.utils.highlevel.cres() would arrive with the slot
unset. DeriveWAP captures the entry point from the compile
result and DeriveFunctionType fills the slot from it, on
both unboxing and constant lowering.
The three slots are directly observable through
numbox.utils.lowlevel.get_func_tuple(), which is the clearest way to see what the type
adds. jit_addr is populated and matches the entry point the wrapper captured, while
numba’s own _get_jit_address still yields 0 for the same value, because it resolves an
address only for a Dispatcher. The example needs numba 0.61 or later:
from numba import float64
from numba.experimental.function_type import _get_jit_address
from numbox.utils.derive_wap import DeriveWAP
from numbox.utils.highlevel import cres
from numbox.utils.lowlevel import get_func_tuple
sig = float64(float64, float64)
@cres(sig, cache=True)
def add(x, y):
return x + y
assert isinstance(add, DeriveWAP)
c_addr, py_addr, jit_addr = get_func_tuple(add)
assert jit_addr != 0
assert jit_addr == add.jit_address
assert _get_jit_address(add, sig) == 0
Nothing above runs on an earlier numba. _get_jit_address does not exist there to import,
the tuple has two entries rather than three, and cres returns a plain
CompileResultWAP rather than a DeriveWAP, because
without the slot there is nothing for the mechanism to populate.
numbox.utils.highlevel.cres() is not the only source of one. From numba 0.61 onward a
@proxy binding’s .as_func is a DeriveWAP too,
minted from the proxied body’s compile result, with the same exception semantics and the
same effect on a jitted caller that reaches it as a constant. See numbox.core.proxy.
test/utils/test_lowlevel.py::test_get_func_tuple pins every assertion above except the
first; the wrapper’s type is pinned separately by
test/utils/test_derive_wap.py::test_cres_mints_a_derive_wap.
First-class function values whose exceptions propagate out of the call.
The mechanism
numba lowers a first-class FunctionType call two ways and picks between them on
the function model’s jit_addr slot. A populated slot selects the numba calling
convention plus return_status_propagate, so an exception unwinds into the caller.
An empty slot selects the C wrapper, which numba documents as not supporting
exceptions: it discards the exception, zero-fills the return value and reports the
failure only as an unraisable on stderr.
numba fills the slot for a Dispatcher and leaves it empty for everything else,
_get_jit_address returning 0. A bare compile result therefore arrives at a call
site with no entry point that could carry an exception out.
DeriveWAP captures the compile result’s numba-callconv entry point, and
DeriveFunctionType populates jit_addr from it on both unboxing and
constant lowering. A value of this type carries a usable entry point wherever it
goes.
Using it
numbox.utils.highlevel.cres() mints these values, so a cres-compiled function
is already one:
@cres(float64(float64))
def f(x):
if x < 0:
raise ValueError("negative")
return x
Filling the slot is necessary but not sufficient: the call site decides which
convention it emits, and one that reads the wrapper address unconditionally gets the
C wrapper however the slot is filled. A consumer opts in by reading jit_addr and
emitting call_conv.call_function plus return_status_propagate when it is
non-null; numba’s own version of that is
numba.core.lowering.Lower.__call_first_class_function_pointer.
numbox.core.work.work._call_derive is the worked example in this repository, and
numbox.core.work walks through it. It selects the propagating convention at
compile time for a DeriveFunctionType, and tests the slot at runtime for a
plain FunctionType, which numba does populate for an njit dispatcher passed as a
FunctionType-typed argument.
Limits
DeriveFunctionType.can_convert_to() documents the one direction that does not
survive: a value handed from Python into a parameter declared as a plain
FunctionType degrades to the C convention on the way in.
Where numba has no such slot jit_addr_supported() is false, the whole mechanism
is inert and cres returns a plain CompileResultWAP.
Standing on numba
The decorators below are numba’s public extension API, save for lower_constant,
which numba.extending does not re-export. What they register against is not:
FunctionModel, CompileResultWAP, Conversion, box_function_type and
lower_get_wrapper_address all sit outside numba.extending, and the constant
lowering drives context.declare_function and context.active_code_library
directly. What does hold is that no numba internals are patched: nothing here
replaces numba behaviour, it only registers against it.
- class numbox.utils.derive_wap.DeriveFunctionType(*args, **kwargs)[source]
Bases:
FunctionTypeFirst-class function type whose values always carry a populated
jit_addr.Kept distinct from
FunctionTypeso that_call_derivecan select the propagating calling convention at compile time, with no runtime branch, for every derive numbox itself compiled.- can_convert_to(typingctx, other)[source]
Permit passing a
DeriveWAPwhere a plainFunctionTypeof the same signature is declared.Such a call site keeps working unchanged, which is the point, but it keeps its old behaviour too: crossing the Python boundary into a parameter declared as a plain
FunctionTypedegrades the value to the C convention, so a derive supplied that way still discards its exception. Reaching the same declared type by a cast within jitted scope does not degrade it, becauselower_cast_derive_to_function_type()is an identity on a sharedFunctionModeland the populatedjit_addrsurvives. One source-level call site therefore has two different exception semantics depending on where the value came from.
- class numbox.utils.derive_wap.DeriveWAP(cres)[source]
Bases:
CompileResultWAPCompileResultWAPthat also captures the numba-callconv entry point.CompileResultWAPrecords only the cfunc wrapper address, which is the entry point that cannot carry an exception out.
- numbox.utils.derive_wap.jit_addr_supported() bool[source]
Whether the running numba exposes the
jit_addrslot.The slot was added to
FunctionModelin numba 0.61. Where it is absent the mechanism has nothing to populate, so numbox leaves first-class calls to numba rather than shipping a half-installed variant.
- numbox.utils.derive_wap.rewrap_derive(derive)[source]
Upgrade a foreign
CompileResultWAPso its exceptions propagate.A derive compiled by numbox’s own
cres()is already aDeriveWAP. One built directly against numba is not, and would keep the swallowing convention. The compile result it carries is all that is needed to upgrade it.Anything else, including
None, is returned unchanged so callers can apply this unconditionally.On a numba without the
jit_addrslot there is nothing to upgrade into: the struct has no field to hold the entry point, so producing aDeriveWAPthere would only yield a value that cannot be unboxed.The upgraded wrapper is memoized onto the object it upgrades, and that is required rather than an optimization.
py_addrholds the derive’s address without taking a reference, so a wrapper minted fresh per call would be freed as soon as the caller returned, leaving every Work built from it pointing at released memory. Hanging it off the original ties its lifetime to the object the caller already holds, which is the lifetime the address assumed all along.
numbox.utils.preprocessing
Cache-anchor mechanism
make_structref writes the generated code_txt to a
content-addressed file under numba’s cache directory and uses that
file – not highlevel.py – as the compile() anchor. The
content-addressing is what keeps numba’s per-overload cache correct
when two versions of the generated code differ only in co_consts.
Numba’s per-overload cache key
(numba.core.caching.Cache._index_key) is:
(sig, codegen.magic_tuple(), hash(co_code), hash(closure_cells))
It hashes co_code only, not co_consts. Python’s
LOAD_CONST
opcode encodes an index into co_consts rather than the value
itself, so two methods differing only in a numeric literal
(return self.x + 1 vs return self.x + 1000) produce identical
co_code. With a shared anchor file both would resolve to the same
cache_subpath (numba’s
_CacheLocator.get_suitable_cache_subpath derives the cache subdir
from a hash of co_filename) and the second to compile would
silently load the first’s binary. Per-content anchors segregate the
two by cache_subpath so the collision never arises.
Structural body changes – different operators, additional
statements, renamed variables – produce different co_code and
therefore different _index_key values regardless of the anchoring
scheme; those invalidate cleanly even under a shared anchor. The
narrow failure mode protected by content-addressing is constant-only
edits (numeric or string literals, default arg values) where
co_code is identical across versions.
Python 3.14’s
LOAD_SMALL_INT
opcode inlines small integers directly into co_code, narrowing
the failure mode on that version to constants outside the inline
range. Earlier supported versions (3.10–3.13) collide on any
constant edit.
See also numba.core.caching.Cache._index_key and
numba.core.caching._SourceFileBackedLocatorMixin.get_source_stamp
in numba’s source for the cache key construction and source-stamp
validity check.
Source-anchor machinery for dynamically-exec’d code.
Content-addressed anchors keep numba’s per-overload cache correct
when two exec’d code blocks differ only in co_consts. See the
“Cache-anchor mechanism” section in docs/numbox.utils.rst for
the rationale and references.
numbox.utils.digest
Content-addressed digest for cache keys and anchor identifiers.
digest produces a short, stable hash that invalidates when the subject’s
repr, the user functions, the resolved jit_options, or the numba/numbox
versions change. Plain Python functions are fingerprinted with the shared
closure/global-aware walker (numbox.utils.fingerprint._fingerprint_function)
– so two callbacks with identical source but different captured closure-cell or
referenced-global values key distinctly, which a bare code-object hash would
miss. Callables with no canonical fingerprint (a partial, a builtin, a callable
object, or a function closing over an un-canonicalizable value) fall back to
cloudpickle of the object/code, which also captures bound state. The SQLite UDAF
registration anchors are one consumer; any content-addressed cache that mixes a
type/identifier with user callbacks can reuse it.
numbox.utils.fingerprint
Content fingerprinting of Python values and functions for cache keys.
A value canonicalizer (_canon_value) and a deep function fingerprint (_fingerprint_function) that capture everything numba freezes into a compiled artifact: code-object bytecode/consts/names, default arguments, closure-cell values, and the values of referenced module-level globals (recursing into helper functions and dispatchers, with cycle protection). Stronger than hashing the bare code object – two functions with identical source but different captured closure/global values fingerprint differently. Shared by numbox.core.variable.compile_kernel (kernel cache digest) and numbox.utils.digest (SQLite UDAF cache key).
_Unfingerprintable is raised for any value with no canonical form; callers decide how to degrade (compile_kernel marks the kernel uncached, digest falls back to cloudpickle of the code object).
numbox.utils.lowlevel
- numbox.utils.lowlevel.array_data_p(arr)[source]
Return the data pointer of a numpy array as signed intp.
arr.ctypes.dataisuint64under numba; the cast aligns with the signed-pointer convention used by numbox binding signatures. Callable from Python and@njitcontexts.
- numbox.utils.lowlevel.extract_struct_member(context: BaseContext, builder: IRBuilder, struct_fe_ty: StructRef, struct_obj, member_name: str, incref: bool = False)[source]
For the given struct object of the given numba (front-end) type extract member with the given name (must be literal, available at compile time)
- numbox.utils.lowlevel.get_func_p_from_func_struct(builder: IRBuilder, func_struct)[source]
Extract void* function pointer from the low-level FunctionType structure
- numbox.utils.lowlevel.get_str_from_p_as_int(p)[source]
Given pointer to null-terminated array of characters as an integer p, return unicode string object copying the original string’s data.
Bytes are read through an unsigned 8-bit view, so each byte maps to its own codepoint 0..255 (per-byte Latin-1). A multi-byte UTF-8 payload is returned byte-for-byte, not decoded to its Unicode text.
- numbox.utils.lowlevel.get_unicode_data_p(s)[source]
Given Python unicode string, return pointer to its data payload, array of null-terminated characters. See https://github.com/numba/numba/blob/release0.61/numba/cpython/unicode.py#L83
- numbox.utils.lowlevel.load_at(p, ty)[source]
Load a value of type
tyfrom raw pointerp(carried asintp).Caller is responsible for
ppointing at a live region of memory whose LLVM layout matchesty.
- numbox.utils.lowlevel.load_unaligned(p, ty)[source]
Like
load_at()but emits analign=1load, legal on a misaligned address (e.g. a packed structured-dtype field) whereload_atis UB.
- numbox.utils.lowlevel.populate_structref(context, builder, signature, structref_type_, structref_, args, ordered_args_names, decref_old=False)[source]
Store args with the corresponding ordered names ordered_args_names in structref with type structref_type_ and payload at data_pointer.
Based on numba.experimental.structref::define_attributes::struct_setattr_impl
Do not call decref_old when populating a newly-created structref, as there’s nothing to decref there.
- numbox.utils.lowlevel.store_at(p, v)[source]
Store
vat raw pointerp(LLVM type derived fromv’s numba type).Caller is responsible for
ppointing at a writable region of memory whose LLVM layout matchesv’s type, and for castingvto the intended width (e.g.store_at(p, int32(value))to write 4 bytes).
- numbox.utils.lowlevel.store_unaligned(p, v)[source]
Like
store_at()but emits analign=1store, legal on a misaligned address (e.g. a packed structured-dtype field) wherestore_atis UB.
numbox.utils.cstrings
Allocating C strings for the bindings layer
The bindings family (numbox.core.bindings.libc, numbox.core.bindings.sqlite.*, etc.)
takes intp pointers for every text argument. Producing a valid
NUL-terminated UTF-8 C string from a Python str is non-trivial:
get_unicode_data_p() returns a pointer to
the Python string’s internal data payload, which CPython stores as
UCS-1/2/4 depending on contents – only safe for ASCII inputs.
c_string() is a Python-side context
manager that allocates a real C buffer with the UTF-8 encoding of the
input and yields the pointer with safe lifetime tied to the with
block:
from numbox.utils.cstrings import c_string
from numbox.core.bindings.sqlite.exec import sqlite3_exec
with c_string("CREATE TABLE t(x INTEGER)") as sql_p:
sqlite3_exec(db_p, sql_p, 0, 0, 0)
# buffer freed here automatically
For concurrent multi-string calls, use a single with with multiple
context managers, or contextlib.ExitStack when the count is
dynamic:
with c_string("main") as schema_p, c_string("t") as table_p, c_string("b") as col_p:
sqlite3_blob_open(db_p, schema_p, table_p, col_p, rowid, flags,
addressof(blob_p))
Python-only. c_string cannot be used inside @njit – numba
does not support arbitrary context managers (raises
UnsupportedBytecodeError), and ctypes objects can’t be
manipulated under JIT anyway. For @njit callers that need a C
string, pre-allocate a numpy uint8 buffer containing the UTF-8
bytes plus a trailing NUL at Python level, then pass
array_data_p() into the JIT kernel.
Python-side helpers for passing C strings into the bindings layer.
The bindings layer (numbox.core.bindings.sqlite.*, numbox.core.bindings.libc, etc.)
takes intp pointers for every text argument. Producing a valid
NUL-terminated UTF-8 C string from a Python str is non-trivial:
get_unicode_data_p in lowlevel.py returns a pointer to the
Python string’s internal data payload, which CPython stores as
UCS-1/2/4 depending on contents – only safe for ASCII inputs.
This module’s c_string() allocates a real C buffer with the
UTF-8 encoding of the input and yields the pointer with safe
lifetime management via the with statement.
Python-only. c_string is a context manager; numba does not
support arbitrary context managers inside @njit (raises
UnsupportedBytecodeError), and ctypes objects can’t be
manipulated under JIT anyway. For @njit callers that need a C
string, pre-allocate a numpy uint8 buffer containing the UTF-8
bytes + a trailing NUL outside the JIT scope, then pass
array_data_p(buf) from numbox.utils.lowlevel into the JIT
kernel.
- numbox.utils.cstrings.c_string(s)[source]
Yield an
intppointer to a freshly-allocated NUL-terminated UTF-8 C string fors.Usage:
from numbox.utils.cstrings import c_string from numbox.core.bindings.sqlite.exec import sqlite3_exec with c_string("CREATE TABLE t(x INTEGER)") as sql_p: sqlite3_exec(db_p, sql_p, 0, 0, 0) # buffer freed here
The underlying
ctypesbuffer lives for the duration of thewithblock. Once the block exits, the Python reference is dropped and ctypes frees the memory; the pointer is then dangling and must not be used.Embedded NULs in
sare rejected withValueError. Passing a string containing"\x00"would silently truncate the C string at the first NUL — a footgun that CPython’s ownsqlite3.connectrejects similarly on filename inputs.For concurrent multi-string needs, nest
withstatements or usecontextlib.ExitStack:from contextlib import ExitStack with ExitStack() as stack: a_p = stack.enter_context(c_string("a")) b_p = stack.enter_context(c_string("b")) # both pointers valid here
Python-only – not callable inside
@njit. See module docstring for the JIT alternative.
numbox.utils.pysqlite_bridge
Bridging Python sqlite3 connections to the bindings layer
CPython’s stdlib sqlite3.Connection holds the underlying
sqlite3 *db C handle as a private field inside its PyObject
struct. extract_connection_ptr()
reads that field via ctypes so callers can pass the handle into
numbox’s JIT-callable SQLite bindings:
import numbox.utils.pysqlite_bridge as pysqlite_bridge
import sqlite3
from numbox.core.bindings.sqlite.conn import sqlite3_changes
conn = sqlite3.connect("app.db")
conn.execute("CREATE TABLE t(x INTEGER)")
conn.execute("INSERT INTO t VALUES (1), (2), (3)")
db_p = pysqlite_bridge.extract_connection_ptr(conn)
assert sqlite3_changes(db_p) == 3
Mirrors the pattern in numbduck.pybridge.
Build configurations are handled at runtime. The sqlite3 *db field
offset is computed by _pyobject_head_fields(), which adapts to the running
interpreter: Py_DEBUG builds (detected via sys.gettotalrefcount)
prepend the _ob_next / _ob_prev trace pointers, and free-threaded
builds (Py_GIL_DISABLED) use the no-GIL object header — so release, debug,
and free-threaded builds all read the field at the correct offset.
The macOS shared-cache caveat and its DYLD_INSERT_LIBRARIES workaround are
described in the module docstring below; extract_connection_ptr() also
validates (via libraries_coordinated()) that numbox’s bindings and
Python’s sqlite3 resolve to the same libsqlite3 before returning a pointer,
raising rather than handing back a pointer to a mismatched library.
Extract raw SQLite C API pointers from Python sqlite3 objects.
Bridges CPython’s stdlib sqlite3 module to the numba-callable bindings layer by exposing the underlying
sqlite3 *db handle that sits inside a Python sqlite3.Connection. Mirrors the pattern in
numbduck.pybridge.
macOS caveat
On macOS, the system sqlite is in the dyld shared cache (structure defined in Apple’s
DyldSharedCache.h),
mapped into every process at launch. LLVM’s JIT linker resolves sqlite3_* symbols via
dlsym(RTLD_DEFAULT), which returns the first match in load order — the system copy. Python’s
_sqlite3.so may use a different sqlite (statically linked on python.org framework builds, or
dynamically linked to Homebrew’s copy).
If the system sqlite and Python’s sqlite differ enough in version or internal layout, passing the pointer from
extract_connection_ptr() to numbox’s @njit bindings can produce wrong results or segfaults.
Workaround: force the process to load your Python’s sqlite first:
DYLD_INSERT_LIBRARIES=/path/to/your/libsqlite3.dylib python my_script.py
Common paths:
Homebrew (Apple Silicon):
/opt/homebrew/opt/sqlite/lib/libsqlite3.dylibHomebrew (Intel):
/usr/local/opt/sqlite/lib/libsqlite3.dylibConda-forge:
$CONDA_PREFIX/lib/libsqlite3.dylibpython.org framework builds statically link sqlite — no external dylib available; install Homebrew sqlite and use its path.
On Linux there is typically only one sqlite on the system, so no workaround is needed.
- numbox.utils.pysqlite_bridge.extract_connection_ptr(conn)[source]
Return the raw
sqlite3*underlying a Pythonsqlite3.Connection.Uses
_PysqliteConnectionto read thedbfield at its platform-correct offset inside the PyObject struct, then validates the pointer by callingsqlite3_errmsg()(a healthy connection returns"not an error").Before reading the pointer, checks (via
libraries_coordinated()) that numbox’s bindings and Python’ssqlite3module resolve to the same libsqlite3. If they differ — the macOS shared-cache situation in the module docstring — it raises rather than pass the pointer to a mismatched library, which could segfault.Parameters
conn : sqlite3.Connection
Returns
- int
sqlite3*as a Python int (intp-compatible). The pointer is borrowed from conn: it is owned by the PythonConnectionand stays valid only while conn is alive and open. Anintcannot keep conn referenced, so the keep-alive is the caller’s responsibility.
Warning
The caller must retain conn for the entire lifetime of any
@njituse of the returned pointer, and must not use the pointer afterconn.close()or after conn is garbage-collected — doing so is a use-after-free (dereferencing a danglingsqlite3*). This mirrors theSQLITE_STATICownership contract onbind_text/bind_blob: the borrowed memory must outlive every use.Raises
- TypeError
If conn is not a
sqlite3.Connection.- RuntimeError
If numbox’s bindings and Python’s
sqlite3use different libsqlite3 instances, or if the extracted pointer fails the validation call.
- numbox.utils.pysqlite_bridge.libraries_coordinated()[source]
True if numbox’s
@njitbindings and Python’ssqlite3resolve to the same libsqlite3.Compares
sqlite3_libversion()(what numbox’s bindings link against) withsqlite3.sqlite_version(what Python’ssqlite3module links against). When they differ — the macOS shared-cache situation described in the module docstring — passing a connection pointer across the two is unsafe, andextract_connection_ptr()raises rather than proceed.Returns
bool