numbox.utils

numbox.utils.highlevel

Dynamically defining StructRef

Defining numba StructRef requires writing a lot of boilerplate code. A utility for concise definition of StructRef types that supports caching is provided in numbox.utils.highlevel.make_structref(). To use it, define a separate module such as type_classes.py such as:

from numba.experimental.structref import register
from numba.core.types import StructRef


@register
class DataStructTypeClass(StructRef):
    pass

Then in a different module main.py define:

from numba.core.types import float32, unicode_type
from numpy import isclose
from numbox.utils.highlevel import make_structref
from type_classes import DataStructTypeClass


def derive_output(struct_):
    if struct_.control == "double":
        return struct_.value * 2
    else:
        return struct_.value


data_struct = make_structref(
    "DataStruct",
    {"value": float32, "control": unicode_type},
    DataStructTypeClass,
    struct_methods={
        "derive_output": derive_output
    }
)


if __name__ == "__main__":
    data_1 = data_struct(3.14, "double")
    data_2 = data_struct(2.17, "something else")
    assert isclose(data_1.derive_output(), 6.28)
    assert isclose(data_2.derive_output(), 2.17)

numbox.utils.highlevel.cres(sig, **kwargs)[source]: Returns Python proxy to FunctionType rather than CPUDispatcher returned by njit

numbox.utils.highlevel.cres_if_available(lib, sig, **kwargs)[source]

Like cres(sig, **kwargs), but stubs out the wrapper if the C symbol matching func.__name__ is absent from lib.

Use for binding sets that target multiple library versions where some symbols may only exist in newer releases. Callers get a stub that raises NotImplementedError instead of a confusing LLVM link error at call time.

numbox.utils.highlevel.determine_field_index(struct_ty, field_name)[source]

numbox.utils.highlevel.hash_type(ty: Type)[source]

numbox.utils.highlevel.make_structref(struct_name: str, struct_fields: Iterable[str] | dict[str, Type], struct_type_class: type | Type, *, struct_methods: dict[str, Callable] | None = None, jit_options: dict | None = None, ns: dict | None = None)[source]

Makes structure type with struct_name and struct_fields from the StructRef type class.

A unique struct_type_class for each structref needs to be provided. If caching of code that will be using the created struct type is desired, these type class(es) need/s to be defined in a python module that is not executed. (Same requirement is also to observed even when the full definition of StructRef is entirely hard-coded rather than created dynamically.)

In particular, that’s why struct_type_class cannot be incorporated into the dynamic compile / exec routine here.

Dictionary of methods to be bound to the created structref can be provided as well. Struct methods will get inlined into the caller if numba deems it to be optimal (even if jit_options says otherwise), therefore changing the methods code without poking the jitted caller can result in a stale cache - when the latter is cached. This is not an exclusive limitation of a dynamic structref creation via this function and is equally true when the structref definition is coded explicitly.

Anchor file

The generated code_txt is written to a content-addressed file under numba’s cache directory and that file – not highlevel.py – is used as the compile() anchor. See the “Cache-anchor mechanism” section in docs/numbox.utils.rst for the rationale.

numbox.utils.highlevel.make_structref_code_txt(struct_name: str, struct_fields: Iterable[str] | dict[str, Type], struct_type_class: type | Type, struct_methods: dict[str, Callable] | None = None)[source]

numbox.utils.highlevel.prune_type(ty)[source]

numbox.utils.preprocessing

Cache-anchor mechanism

make_structref writes the generated code_txt to a content-addressed file under numba’s cache directory and uses that file – not highlevel.py – as the compile() anchor. The content-addressing is what keeps numba’s per-overload cache correct when two versions of the generated code differ only in co_consts.

Numba’s per-overload cache key (numba.core.caching.Cache._index_key) is:

(sig, codegen.magic_tuple(), hash(co_code), hash(closure_cells))

It hashes co_code only, not co_consts. Python’s LOAD_CONST opcode encodes an index into co_consts rather than the value itself, so two methods differing only in a numeric literal (return self.x + 1 vs return self.x + 1000) produce identical co_code. With a shared anchor file both would resolve to the same cache_subpath (numba’s _CacheLocator.get_suitable_cache_subpath derives the cache subdir from a hash of co_filename) and the second to compile would silently load the first’s binary. Per-content anchors segregate the two by cache_subpath so the collision never arises.

Structural body changes – different operators, additional statements, renamed variables – produce different co_code and therefore different _index_key values regardless of the anchoring scheme; those invalidate cleanly even under a shared anchor. The narrow failure mode protected by content-addressing is constant-only edits (numeric or string literals, default arg values) where co_code is identical across versions.

Python 3.14’s LOAD_SMALL_INT opcode inlines small integers directly into co_code, narrowing the failure mode on that version to constants outside the inline range. Earlier supported versions (3.10–3.13) collide on any constant edit.

See also numba.core.caching.Cache._index_key and numba.core.caching._SourceFileBackedLocatorMixin.get_source_stamp in numba’s source for the cache key construction and source-stamp validity check.

Source-anchor machinery for dynamically-exec’d code.

Content-addressed anchors keep numba’s per-overload cache correct when two exec’d code blocks differ only in co_consts. See the “Cache-anchor mechanism” section in docs/numbox.utils.rst for the rationale and references.

numbox.utils.digest

Content-addressed digest for cache keys and anchor identifiers.

digest produces a short, stable hash that invalidates when the subject’s repr, the user functions, the resolved jit_options, or the numba/numbox versions change. Plain Python functions are fingerprinted with the shared closure/global-aware walker (numbox.utils.fingerprint._fingerprint_function) – so two callbacks with identical source but different captured closure-cell or referenced-global values key distinctly, which a bare code-object hash would miss. Callables with no canonical fingerprint (a partial, a builtin, a callable object, or a function closing over an un-canonicalizable value) fall back to cloudpickle of the object/code, which also captures bound state. The SQLite UDAF registration anchors are one consumer; any content-addressed cache that mixes a type/identifier with user callbacks can reuse it.

numbox.utils.digest.digest(subject, fns)[source]: Content hash that invalidates when subject (by repr), the user functions (closure/global/const-sensitive via the shared fingerprint walker, with a cloudpickle fallback), the resolved jit_options, or the numba/numbox versions change.

numbox.utils.fingerprint

Content fingerprinting of Python values and functions for cache keys.

A value canonicalizer (_canon_value) and a deep function fingerprint (_fingerprint_function) that capture everything numba freezes into a compiled artifact: code-object bytecode/consts/names, default arguments, closure-cell values, and the values of referenced module-level globals (recursing into helper functions and dispatchers, with cycle protection). Stronger than hashing the bare code object – two functions with identical source but different captured closure/global values fingerprint differently. Shared by numbox.core.variable.compile_kernel (kernel cache digest) and numbox.utils.digest (SQLite UDAF cache key).

_Unfingerprintable is raised for any value with no canonical form; callers decide how to degrade (compile_kernel marks the kernel uncached, digest falls back to cloudpickle of the code object).

numbox.utils.lowlevel

numbox.utils.lowlevel.array_data_p(arr)[source]

Return the data pointer of a numpy array as signed intp.

arr.ctypes.data is uint64 under numba; the cast aligns with the signed-pointer convention used by numbox binding signatures. Callable from Python and @njit contexts.

numbox.utils.lowlevel.cast(source, dest_ty)[source]: Cast source to the type dest_ty

numbox.utils.lowlevel.deref_payload(p, ty)[source]: Dereference payload of structref p as type ty

numbox.utils.lowlevel.extract_struct_member(context: BaseContext, builder: IRBuilder, struct_fe_ty: StructRef, struct_obj, member_name: str, incref: bool = False)[source]: For the given struct object of the given numba (front-end) type extract member with the given name (must be literal, available at compile time)

numbox.utils.lowlevel.get_func_p_as_int_from_func_struct(func_)[source]

numbox.utils.lowlevel.get_func_p_from_func_struct(builder: IRBuilder, func_struct)[source]: Extract void* function pointer from the low-level FunctionType structure

numbox.utils.lowlevel.get_func_tuple(func)[source]

numbox.utils.lowlevel.get_ll_func_sig(context: BaseContext, func_ty: FunctionType)[source]

numbox.utils.lowlevel.get_str_from_p_as_int(p)[source]

Given pointer to null-terminated array of characters as an integer p, return unicode string object copying the original string’s data.

Bytes are read through an unsigned 8-bit view, so each byte maps to its own codepoint 0..255 (per-byte Latin-1). A multi-byte UTF-8 payload is returned byte-for-byte, not decoded to its Unicode text.

numbox.utils.lowlevel.get_unicode_data_p(s)[source]: Given Python unicode string, return pointer to its data payload, array of null-terminated characters. See https://github.com/numba/numba/blob/release0.61/numba/cpython/unicode.py#L83

numbox.utils.lowlevel.load_at(p, ty)[source]

Load a value of type ty from raw pointer p (carried as intp).

Caller is responsible for p pointing at a live region of memory whose LLVM layout matches ty.

numbox.utils.lowlevel.load_unaligned(p, ty)[source]: Like load_at() but emits an align=1 load, legal on a misaligned address (e.g. a packed structured-dtype field) where load_at is UB.

numbox.utils.lowlevel.populate_structref(context, builder, signature, structref_type_, structref_, args, ordered_args_names, decref_old=False)[source]

Store args with the corresponding ordered names ordered_args_names in structref with type structref_type_ and payload at data_pointer.

Based on numba.experimental.structref::define_attributes::struct_setattr_impl

Do not call decref_old when populating a newly-created structref, as there’s nothing to decref there.

numbox.utils.lowlevel.store_at(p, v)[source]

Store v at raw pointer p (LLVM type derived from v’s numba type).

Caller is responsible for p pointing at a writable region of memory whose LLVM layout matches v’s type, and for casting v to the intended width (e.g. store_at(p, int32(value)) to write 4 bytes).

numbox.utils.lowlevel.store_unaligned(p, v)[source]: Like store_at() but emits an align=1 store, legal on a misaligned address (e.g. a packed structured-dtype field) where store_at is UB.

numbox.utils.lowlevel.tuple_of_struct_ptrs_as_int(tup)[source]

numbox.utils.lowlevel.uniformize_tuple_of_structs(tup, uniform_ty=numba.VoidTypeClass())[source]

numbox.utils.cstrings

Allocating C strings for the bindings layer

The bindings family (numbox.core.bindings.libc, numbox.core.bindings.sqlite.*, etc.) takes intp pointers for every text argument. Producing a valid NUL-terminated UTF-8 C string from a Python str is non-trivial: get_unicode_data_p() returns a pointer to the Python string’s internal data payload, which CPython stores as UCS-1/2/4 depending on contents – only safe for ASCII inputs.

c_string() is a Python-side context manager that allocates a real C buffer with the UTF-8 encoding of the input and yields the pointer with safe lifetime tied to the with block:

from numbox.utils.cstrings import c_string
from numbox.core.bindings.sqlite.exec import sqlite3_exec

with c_string("CREATE TABLE t(x INTEGER)") as sql_p:
    sqlite3_exec(db_p, sql_p, 0, 0, 0)
# buffer freed here automatically

For concurrent multi-string calls, use a single with with multiple context managers, or contextlib.ExitStack when the count is dynamic:

with c_string("main") as schema_p, c_string("t") as table_p, c_string("b") as col_p:
    sqlite3_blob_open(db_p, schema_p, table_p, col_p, rowid, flags,
                      addressof(blob_p))

Python-only. c_string cannot be used inside @njit – numba does not support arbitrary context managers (raises UnsupportedBytecodeError), and ctypes objects can’t be manipulated under JIT anyway. For @njit callers that need a C string, pre-allocate a numpy uint8 buffer containing the UTF-8 bytes plus a trailing NUL at Python level, then pass array_data_p() into the JIT kernel.

Python-side helpers for passing C strings into the bindings layer.

The bindings layer (numbox.core.bindings.sqlite.*, numbox.core.bindings.libc, etc.) takes intp pointers for every text argument. Producing a valid NUL-terminated UTF-8 C string from a Python str is non-trivial: get_unicode_data_p in lowlevel.py returns a pointer to the Python string’s internal data payload, which CPython stores as UCS-1/2/4 depending on contents – only safe for ASCII inputs.

This module’s c_string() allocates a real C buffer with the UTF-8 encoding of the input and yields the pointer with safe lifetime management via the with statement.

Python-only. c_string is a context manager; numba does not support arbitrary context managers inside @njit (raises UnsupportedBytecodeError), and ctypes objects can’t be manipulated under JIT anyway. For @njit callers that need a C string, pre-allocate a numpy uint8 buffer containing the UTF-8 bytes + a trailing NUL outside the JIT scope, then pass array_data_p(buf) from numbox.utils.lowlevel into the JIT kernel.

numbox.utils.cstrings.c_string(s)[source]

Yield an intp pointer to a freshly-allocated NUL-terminated UTF-8 C string for s.

Usage:

from numbox.utils.cstrings import c_string
from numbox.core.bindings.sqlite.exec import sqlite3_exec

with c_string("CREATE TABLE t(x INTEGER)") as sql_p:
    sqlite3_exec(db_p, sql_p, 0, 0, 0)
# buffer freed here

The underlying ctypes buffer lives for the duration of the with block. Once the block exits, the Python reference is dropped and ctypes frees the memory; the pointer is then dangling and must not be used.

Embedded NULs in s are rejected with ValueError. Passing a string containing "\x00" would silently truncate the C string at the first NUL — a footgun that CPython’s own sqlite3.connect rejects similarly on filename inputs.

For concurrent multi-string needs, nest with statements or use contextlib.ExitStack:

from contextlib import ExitStack
with ExitStack() as stack:
    a_p = stack.enter_context(c_string("a"))
    b_p = stack.enter_context(c_string("b"))
    # both pointers valid here

Python-only – not callable inside @njit. See module docstring for the JIT alternative.

numbox.utils.pysqlite_bridge

Bridging Python `sqlite3` connections to the bindings layer

CPython’s stdlib sqlite3.Connection holds the underlying sqlite3 *db C handle as a private field inside its PyObject struct. extract_connection_ptr() reads that field via ctypes so callers can pass the handle into numbox’s JIT-callable SQLite bindings:

import numbox.utils.pysqlite_bridge as pysqlite_bridge
import sqlite3
from numbox.core.bindings.sqlite.conn import sqlite3_changes

conn = sqlite3.connect("app.db")
conn.execute("CREATE TABLE t(x INTEGER)")
conn.execute("INSERT INTO t VALUES (1), (2), (3)")
db_p = pysqlite_bridge.extract_connection_ptr(conn)
assert sqlite3_changes(db_p) == 3

Mirrors the pattern in numbduck.pybridge.

Build configurations are handled at runtime. The sqlite3 *db field offset is computed by _pyobject_head_fields(), which adapts to the running interpreter: Py_DEBUG builds (detected via sys.gettotalrefcount) prepend the _ob_next / _ob_prev trace pointers, and free-threaded builds (Py_GIL_DISABLED) use the no-GIL object header — so release, debug, and free-threaded builds all read the field at the correct offset.

The macOS shared-cache caveat and its DYLD_INSERT_LIBRARIES workaround are described in the module docstring below; extract_connection_ptr() also validates (via libraries_coordinated()) that numbox’s bindings and Python’s sqlite3 resolve to the same libsqlite3 before returning a pointer, raising rather than handing back a pointer to a mismatched library.

Extract raw SQLite C API pointers from Python sqlite3 objects.

Bridges CPython’s stdlib sqlite3 module to the numba-callable bindings layer by exposing the underlying sqlite3 *db handle that sits inside a Python sqlite3.Connection. Mirrors the pattern in numbduck.pybridge.

macOS caveat

On macOS, the system sqlite is in the dyld shared cache (structure defined in Apple’s DyldSharedCache.h), mapped into every process at launch. LLVM’s JIT linker resolves sqlite3_* symbols via dlsym(RTLD_DEFAULT), which returns the first match in load order — the system copy. Python’s _sqlite3.so may use a different sqlite (statically linked on python.org framework builds, or dynamically linked to Homebrew’s copy).

If the system sqlite and Python’s sqlite differ enough in version or internal layout, passing the pointer from extract_connection_ptr() to numbox’s @njit bindings can produce wrong results or segfaults.

Workaround: force the process to load your Python’s sqlite first:

DYLD_INSERT_LIBRARIES=/path/to/your/libsqlite3.dylib python my_script.py

Common paths:

Homebrew (Apple Silicon): /opt/homebrew/opt/sqlite/lib/libsqlite3.dylib
Homebrew (Intel): /usr/local/opt/sqlite/lib/libsqlite3.dylib
Conda-forge: $CONDA_PREFIX/lib/libsqlite3.dylib
python.org framework builds statically link sqlite — no external dylib available; install Homebrew sqlite and use its path.

On Linux there is typically only one sqlite on the system, so no workaround is needed.

numbox.utils.pysqlite_bridge.extract_connection_ptr(conn)[source]

Return the raw sqlite3* underlying a Python sqlite3.Connection.

Uses _PysqliteConnection to read the db field at its platform-correct offset inside the PyObject struct, then validates the pointer by calling sqlite3_errmsg() (a healthy connection returns "not an error").

Before reading the pointer, checks (via libraries_coordinated()) that numbox’s bindings and Python’s sqlite3 module resolve to the same libsqlite3. If they differ — the macOS shared-cache situation in the module docstring — it raises rather than pass the pointer to a mismatched library, which could segfault.

Parameters

conn : sqlite3.Connection

Returns

int: sqlite3* as a Python int (intp-compatible). The pointer is borrowed from conn: it is owned by the Python Connection and stays valid only while conn is alive and open. An int cannot keep conn referenced, so the keep-alive is the caller’s responsibility.

Warning

The caller must retain conn for the entire lifetime of any @njit use of the returned pointer, and must not use the pointer after conn.close() or after conn is garbage-collected — doing so is a use-after-free (dereferencing a dangling sqlite3*). This mirrors the SQLITE_STATIC ownership contract on bind_text / bind_blob: the borrowed memory must outlive every use.

Raises

TypeError: If conn is not a sqlite3.Connection.
RuntimeError: If numbox’s bindings and Python’s sqlite3 use different libsqlite3 instances, or if the extracted pointer fails the validation call.

numbox.utils.pysqlite_bridge.libraries_coordinated()[source]

True if numbox’s @njit bindings and Python’s sqlite3 resolve to the same libsqlite3.

Compares sqlite3_libversion() (what numbox’s bindings link against) with sqlite3.sqlite_version (what Python’s sqlite3 module links against). When they differ — the macOS shared-cache situation described in the module docstring — passing a connection pointer across the two is unsafe, and extract_connection_ptr() raises rather than proceed.

Returns

bool

numbox.utils.timer

class numbox.utils.timer.Timer(precision=3)[source]

Bases: object

times = {}

numbox.utils

numbox.utils.highlevel

Dynamically defining StructRef

Anchor file

numbox.utils.preprocessing

Cache-anchor mechanism

numbox.utils.digest

numbox.utils.fingerprint

numbox.utils.lowlevel

numbox.utils.cstrings

Allocating C strings for the bindings layer

numbox.utils.pysqlite_bridge

Bridging Python sqlite3 connections to the bindings layer

macOS caveat

Parameters

Returns

Warning

Raises

Returns

numbox.utils.timer

Bridging Python `sqlite3` connections to the bindings layer