numbarrow.utils

numbarrow.utils.utils

Overview

Low-level pointer utilities for zero-copy access to Arrow memory buffers. Provides Numba-compatible functions that reinterpret a raw memory address (from pyarrow.Buffer.address) as a typed NumPy array, enabling @njit code to read Arrow buffer data directly without copying.

arrays_viewers builds a viewer the first time a dtype is asked for and keeps it, so arrays_viewers[np.int32] is compiled once per process and nothing is compiled that nothing asks for; the adapters ask for uint8 for validity bitmaps and packed booleans, and int32 and int64 for string offsets. Each viewer takes (address, length) and returns a bare view over the memory at that address, with no owner and no read-only flag, valid only while that memory is. The supported way in is arrow_array_adapter(), which returns read-only arrays tied to the Arrow array they view.

Low-level pointer utilities for zero-copy access to Arrow memory buffers.

Provides Numba-compatible functions that reinterpret a raw memory address (obtained from pyarrow.Buffer.address) as a typed NumPy array, enabling @njit code to read Arrow buffer data directly without copying.

numbarrow.utils.utils.numpy_array_from_ptr_factory(dtype_)

Create a JIT-compiled function that views memory at a given address as a NumPy array.

Returns an @njit function with signature (ptr_as_int, sz) -> ndarray that uses numba.carray() to reinterpret sz elements starting at address ptr_as_int as a contiguous C-order NumPy array of dtype_. No data is copied: the returned array is a bare view over the memory at that address, with no owner and no read-only flag, valid only while that memory is. Reading it after the source is gone is undefined, and a write through it changes the source. The supported way in is arrow_array_adapter(), which returns read-only arrays tied to the Arrow array they view; this is the primitive it is built on.

Parameters:

dtype – NumPy dtype for the resulting array (e.g. np.int32)

Returns:

JIT-compiled function (int, int) -> np.ndarray

numbarrow.utils.arrow_array_utils

Overview

Higher-level utilities for extracting data from PyArrow array buffers as NumPy arrays. Handles uniform arrays (fixed-width elements), string arrays (variable-length with offset buffers), struct arrays, and list-of-struct arrays.

Utilities for extracting data from PyArrow array buffers as NumPy arrays.

Handles uniform arrays (fixed-width elements), string arrays (variable-length with offset buffers), struct arrays, and list-of-struct arrays. Validity bitmaps are extracted as uint8 arrays for use with is_null().

exception numbarrow.utils.arrow_array_utils.MissingKeyError

Bases: KeyError

A KeyError whose message reads as written.

KeyError.__str__ reprs its single argument, so a sentence raised through a plain KeyError arrives wrapped in a second pair of quotes, and one that was already rendered arrives mangled. except KeyError still catches this; only the rendering changes.

numbarrow.utils.arrow_array_utils.create_bitmap(bitmap_buf: Buffer | None, offset: int = 0, length: int = 0)

Create numpy array of uint8 type containing bit-map of valid array entries, adjusted for array offset.

The returned array always owns its memory. The offset path below already produced a fresh array through np.packbits and is returned as-is; the offset-0 path has to copy, because there it would otherwise be a view over the Arrow validity buffer reached by raw address. That aliasing is not only a hazard for a caller who writes: the bits change under a caller who only reads, once the source array is collected and its buffer is reused, so a bitmap that read 0b11011011 comes back all-zero and every row looks null. Writing through it nulls out the source, and a short slice hands back bits the slice does not own.

numbarrow.utils.arrow_array_utils.create_str_array(pa_str_array: StringArray | LargeStringArray) → tuple[ndarray | None, ndarray]

Copy data from a densely packed PyArrow string array into a padded NumPy Unicode array.

StringArray uses int32 offsets; LargeStringArray uses int64 offsets.

The padding is NUL, so a value whose last character is NUL cannot be told from a shorter one and raises ValueError rather than coming back truncated. Leading and interior NULs are representable and are preserved.

numbarrow.utils.arrow_array_utils.renamed(exc: Exception, prefix: str) → Exception

The same exception with prefix in front of its message.

The class is kept when it can be rebuilt from one string, which every pyarrow error and a plain TypeError, ValueError, NotImplementedError or OverflowError can; anything else, such as a UnicodeDecodeError with its five constructor arguments, comes back as a ValueError so that the prefix is never lost to a second error raised while building the message. A plain KeyError comes back as a MissingKeyError, since its str() is the repr of its argument and a KeyError rebuilt from that would repr it again. Raise the result from exc to keep the original traceback.

numbarrow.utils.arrow_array_utils.structured_array_adapter(struct_array: StructArray) → tuple[ndarray | None, dict[str, ndarray | None], dict[str, ndarray]]

NumPy adapter of PyArrow StructArray.

Returns a 3-tuple: - struct-level validity bitmap (None when the array carries no validity buffer) - dict mapping field names to per-field validity bitmaps - dict mapping field names to per-field value arrays

numbarrow.utils.arrow_array_utils.structured_list_array_adapter(list_array: ListArray) → tuple[ndarray | None, dict[str, ndarray | None], dict[str, ndarray]]

NumPy adapter of PyArrow array of same-length lists of structures.

Parameters:

list_array – PyArrow array with elements being of pa.ListType. Each list is in turn of the same length, and each element of the list is of pa.StructType.

Returns a 3-tuple of: the struct-level validity bitmap (or None when the elements carry no validity buffer), a dictionary mapping field names to per-field validity bitmaps (each None when that field carries no validity buffer), and a dictionary mapping field names to the contiguous field data arrays.

Whether a field’s data is copied depends on the field’s type. A fixed-width child is a zero-copy view over the contiguous pa.StructArray values, except a date32 child, whose int32 days are cast up to datetime64[D] and so allocate; date64 and timestamp children are views, since their int64 values need no cast. A boolean child is bit-unpacked and a string child is repacked into fixed-width Unicode, so those are copies too. Every returned data array is read-only either way.

The returned arrays are the flattened elements of every list in list_array, with no offsets telling a caller where one row’s elements end and the next row’s begin. Mapping an element back to its row therefore assumes every list has the same length, which is what the :param: contract above requires. A null row breaks that assumption outright, since it contributes no elements, so list_array.null_count must be zero and a non-zero one raises NotImplementedError.

numbarrow.utils.arrow_array_utils.type_repr(arrow_type) → str

str(arrow_type), cut at a fixed width with a note of what was cut.

A message that interpolates a pyarrow type grows with the schema: a thousand-field struct is 13,000 characters at the dispatcher, and a message that size lands in every log line that catches the traceback. The bound leaves the one-field case, which is the one people read, untouched.

numbarrow.utils.arrow_array_utils.uniform_arrow_array_adapter(pa_array: Array) → tuple[ndarray | None, ndarray]

NumPy adapter for PyArrow arrays with uniformly sized elements.

Returns the validity bitmap, which owns its memory, and a zero-copy numpy view over the array’s data buffer. The view is read-only and cannot be made writable: Arrow buffers are immutable by contract, and this is what pyarrow’s own Array.to_numpy(zero_copy_only=True) returns. Declare numba signatures that receive it with readonly=True, which accepts writable arrays too.