numbarrow.core.is_null

Overview

Arrow uses a packed validity bitmap to track which elements in an array are non-null. Each bit corresponds to one element: bit=1 means valid, bit=0 means null. Bits are packed LSB-first into uint8 bytes.

This module provides three Numba @njit compiled helpers that read such bitmaps:

  • is_null() returns whether one element index is null according to a single bitmap.

  • is_null_struct() answers the same question across a struct’s two validity layers, its own and the field’s, since a value is null when either layer says so. Its index is int64 and each layer takes a read-only uint8 bitmap or None; an index of another integer type converts to int64 at the call.

  • unpack_booleans() expands bit-packed boolean data into a boolean array; Arrow packs boolean values the same way it packs validity bits.

Module

Null detection for Apache Arrow validity bitmaps.

Arrow uses a packed bitmap to track which elements in an array are valid (non-null). Each bit corresponds to one element: bit=1 means valid, bit=0 means null. Bits are packed LSB-first into uint8 bytes — element i lives at byte i // 8, bit position i % 8 within that byte.

numbarrow.core.is_null.is_null(index_: int, bitmap: ndarray) → bool[source]

Check whether element index_ is null according to bitmap.

Arrow validity bitmaps store one bit per element, packed LSB-first into uint8 bytes. A set bit (1) means valid; a cleared bit (0) means null.

index_ must satisfy 0 <= index_ < 8 * len(bitmap). Compiled without bounds checking, which is numba’s default, an index past the bitmap reads memory that is not the bitmap’s; NUMBARROW_JIT_OPTIONS='{"boundscheck": true}' turns that into IndexError. A negative index reads from the bitmap’s end like any numpy index, which is the wrong bit and in bounds, so bounds checking does not catch it.

Parameters:

  • index – zero-based element index

  • bitmap – uint8 array containing the packed validity bitmap

Returns:

True if the element is null (bit is 0), False if valid (bit is 1)

numbarrow.core.is_null.is_null_struct(index_, struct_bitmap, field_bitmap)[source]

Check whether a struct field value is null at either the struct or field layer.

Arrow StructArrays carry a validity bitmap for the struct itself (is this entire row null?) independent of each child field’s bitmap (is this particular field null within a non-null row?). A value is null if either layer marks it as null.

Compiled at import with one signature: an int64 index and, for each layer, a read-only uint8 bitmap or None. Every caller resolves to it, an index of another integer type converting to int64 and a writable bitmap being accepted where a read-only one is declared. One signature is one entry in numba’s on-disk cache, and numba names the next data file by counting the entries in the index it just read, so a second entry is something two processes warming a cold cache can disagree about.

Parameters:

  • index – zero-based element index, converted to int64

  • struct_bitmap – uint8 packed bitmap for struct-level validity, or None

  • field_bitmap – uint8 packed bitmap for field-level validity, or None

Returns:

True if null at either layer

numbarrow.core.is_null.unpack_booleans(offset: int, length: int, packed_data: ndarray) → ndarray[source]

Unpack bit-packed boolean data into a boolean array.

offset + length must not exceed 8 * len(packed_data); past it the read is out of bounds, and only NUMBARROW_JIT_OPTIONS='{"boundscheck": true}' makes that an IndexError.

Parameters:

  • offset – bit offset into packed_data to start reading

  • length – number of boolean values to extract

  • packed_data – uint8 array containing LSB-first packed bits

Returns:

boolean array of length elements