numbarrow.core.adapters

Overview

Type-dispatched adapters that convert PyArrow arrays into NumPy arrays for use in Numba @njit compiled functions.

Uses functools.singledispatch to route each PyArrow array type to a handler that extracts the underlying data buffer as a NumPy view (where possible) and the validity bitmap as a uint8 array.

Supported types:

• BooleanArray (requires copy due to bit-packed layout)

• Int32Array, Int64Array, DoubleArray, UInt8Array (zero-copy view)

• Date32Array (copy: int32 days → datetime64[D])

• Date64Array (zero-copy view as datetime64[ms])

• TimestampArray (zero-copy view as datetime64[unit])

• StringArray, LargeStringArray (tuple of bitmap and a copy of data into a fixed-width NumPy Unicode array, whose width counts characters)

• StructArray (returns a 3-tuple: the struct-level validity bitmap, then bitmaps and data as two dicts keyed by field name)

• ListArray of structs (delegates to the StructArray adapter, honouring the list array’s own offset; any other element type raises NotImplementedError). The elements are flattened and no offsets are returned, so a null outer row can be neither reported nor placed in the element-to-row mapping; a list column whose null_count is non-zero raises NotImplementedError.

A row that is null as a whole carries no validity bits in its fields, so the struct-level bitmap is the only record of it. Pass both layers to numbarrow.core.is_null.is_null_struct.

A string value whose last character is NUL raises ValueError: numpy’s fixed-width |U dtype pads with NUL, so a trailing NUL cannot be told from padding and would come back silently truncated. Leading and interior NULs are preserved.

Returned data arrays are read-only. The views are over Arrow buffers the caller does not own and cannot be made writable, which is also why pyarrow’s own to_numpy(zero_copy_only=True) refuses to hand out a writable one; the copies, booleans, date32 and strings, start read-only as well, so the contract does not depend on the type, though a caller who flips the flag on a copy writes into memory that is their own. Declare numba signatures that receive them with readonly=True, which accepts writable arrays too. Returned bitmaps own their memory and are writable.

A bitmap is None when the array carries no validity buffer and a uint8 array otherwise, which is not the same as having no nulls: slice, take, filter and fill_null keep an all-valid buffer, and Arrow IPC, which is what Spark’s transport uses, drops one when a batch has no nulls. Declare a bitmap parameter Optional in an eager numba signature, or it compiles on one batch and raises No matching definition on the next.

A TimestampArray adapts by its unit alone, so a zoned and a naive timestamp holding the same int64 adapt to the same datetime64, as pyarrow’s to_numpy does; the zone is not reported.

Module

Type-dispatched adapters that convert PyArrow arrays into NumPy arrays for use in Numba @njit compiled functions.

Uses functools.singledispatch() to route each PyArrow array type (BooleanArray, Int32Array, Date32Array, etc.) to a handler that extracts the underlying data buffer as a NumPy array and the validity bitmap as a uint8 array. Where possible, data is viewed without copying; types that require layout changes (e.g. Date32 → datetime64[D]) produce a copy.

numbarrow.core.adapters.arrow_array_adapter(pa_array: Array)

numbarrow.core.adapters.arrow_array_adapter(pa_array: BooleanArray)

numbarrow.core.adapters.arrow_array_adapter(pa_array: Date32Array)

numbarrow.core.adapters.arrow_array_adapter(pa_array: Date64Array)

numbarrow.core.adapters.arrow_array_adapter(pa_array: DoubleArray | Int32Array | Int64Array | UInt8Array)

numbarrow.core.adapters.arrow_array_adapter(pa_array: DoubleArray | Int32Array | Int64Array | UInt8Array)

numbarrow.core.adapters.arrow_array_adapter(pa_array: DoubleArray | Int32Array | Int64Array | UInt8Array)

numbarrow.core.adapters.arrow_array_adapter(pa_array: DoubleArray | Int32Array | Int64Array | UInt8Array)

numbarrow.core.adapters.arrow_array_adapter(pa_array: ListArray)

numbarrow.core.adapters.arrow_array_adapter(pa_array: StructArray)

numbarrow.core.adapters.arrow_array_adapter(pa_array: StringArray | LargeStringArray)

numbarrow.core.adapters.arrow_array_adapter(pa_array: StringArray | LargeStringArray)

numbarrow.core.adapters.arrow_array_adapter(pa_array: TimestampArray)

Dispatcher for PyArrow array adapters of various types.

numbarrow.core.adapters.cast_64bit_date_arrow_to_numpy_array(pa_array: Array, np_dtype: dtype)

Can be used to cast PyArrow arrays of date types that are represented by 64-bit integers to numpy arrays of various date types (np.datetime64[…], which are always represented by 64-bit integers whose meaning is determined by the precision, such as, ‘s’, ‘ms’, ‘us’).

Since underlying data layout of both arrays in int64, a copy is avoided,

The associated bitmap (if any) is also returned.