Developing PyArrow#
Coding Style#
We follow a similar PEP8-like coding style to the pandas project. To fix style issues, use the
pre-commit command:
$ pre-commit run --show-diff-on-failure --color=always --all-files python
Unit Testing#
We are using pytest to develop our unit test suite. After building the project you can run its unit tests like so:
$ pushd arrow/python
$ python -m pytest pyarrow
$ popd
Package requirements to run the unit tests are found in
requirements-test.txt and can be installed if needed with pip install -r
requirements-test.txt.
If you get import errors for pyarrow._lib or another PyArrow module when
trying to run the tests, run python -m pytest arrow/python/pyarrow and check
if the editable version of pyarrow was installed correctly.
The project has a number of custom command line options for its test suite. Some tests are disabled by default, for example. To see all the options, run
$ python -m pytest pyarrow --help
and look for the “custom options” section.
Note
There are a few low-level tests written directly in C++. These tests are
implemented in pyarrow/src/arrow/python/python_test.cc,
but they are also wrapped in a pytest-based
test module
run automatically as part of the PyArrow test suite.
Test Groups#
We have many tests that are grouped together using pytest marks. Some of these
are disabled by default. To enable a test group, pass --$GROUP_NAME,
e.g. --parquet. To disable a test group, prepend disable, so
--disable-parquet for example. To run only the unit tests for a
particular group, prepend only- instead, for example --only-parquet.
The test groups currently include:
dataset: Apache Arrow Dataset testsflight: Flight RPC testsgandiva: tests for Gandiva expression compiler (uses LLVM, deprecated since version 24.0.0)hdfs: tests that use libhdfs to access the Hadoop filesystemhypothesis: tests that use thehypothesismodule for generating random test cases. Note that--hypothesisdoesn’t work due to a quirk with pytest, so you have to pass--enable-hypothesislarge_memory: Test requiring a large amount of system RAMorc: Apache ORC testsparquet: Apache Parquet testss3: Tests for Amazon S3tensorflow: Tests that involve TensorFlow
Type Checking#
PyArrow provides type stubs (*.pyi files) for static type checking. These
stubs are located in the pyarrow-stubs/ directory, which is laid out as a
PEP 561 stub-only package for pyarrow, so type checkers and IDEs pick
it up from the arrow/python directory.
Note
The stubs aren’t included in the distributed wheel packages until they are complete (see GH-49831).
Run all the commands in this section from the arrow/python directory,
which holds the type checker configurations in pyproject.toml:
$ cd arrow/python
Maintaining Type Stubs#
Type stubs for PyArrow are maintained in the pyarrow-stubs/
directory. These stubs mirror the structure of the main pyarrow/ package.
When adding or modifying public APIs:
Update the corresponding .pyi stub file in
pyarrow-stubs/to reflect the new or changed function/class signatures.Run type checkers to ensure the stubs are correct and complete.
The stub files will be automatically copied into the built wheel during the build process and will be included when users install PyArrow, enabling type checking in downstream projects and for users’ IDEs (once issue 32609 is sufficiently resolved).
Note
py.typed marker file in the pyarrow/ directory indicates to type
checkers that PyArrow supports type checking according to PEP 561.
Running Type Checkers#
pre-commit#
We support multiple type checkers. The easiest way to run all of them is with
pre-commit, which installs the type checkers and stub packages pinned in
requirements-stubs.txt (with uv). This also lints and formats the stubs
with ruff.
$ pre-commit run --hook-stage manual python-stubs --all-files
Type checker hooks run only on request (--hook-stage manual), while ruff
hooks run on every commit if the pre-commit git hook is set up locally
(with pre-commit install).
Individual Type Checks#
To run a type checker directly instead, install the pinned type checkers and stub packages in your PyArrow development environment:
$ pip install -r requirements-stubs.txt
Checking the PyArrow tests also needs their dependencies from
requirements-test.txt.
To run mypy on the PyArrow codebase:
$ mypy
The mypy configuration is in the [tool.mypy] section of pyproject.toml.
With an editable install, mypy finds pyarrow-stubs automatically. With a
non-editable install (pip install . or a wheel built with
python -m build), the stubs aren’t installed yet, so pass pyarrow-stubs
along with the files you check:
$ mypy pyarrow-stubs pyarrow/tests/test_array.py
To run pyright on the PyArrow codebase:
$ pyright
The pyright configuration is in the [tool.pyright] section of pyproject.toml.
To run ty on the PyArrow codebase:
$ ty check
The ty configuration is in the [tool.ty.*] sections of pyproject.toml.
mypy’s stubtest compares the stubs with the signatures of the installed PyArrow. Run it with:
$ python scripts/run_stubtest.py
Stubs that aren’t written yet are skipped. The known mismatches are listed in
pyarrow-stubs-tests/stubtest_allowlist.txt; remove the entries of the
mismatches you fix, as stubtest fails on unused entries.
Note on type checking in the CI
The Lint CI job runs the python-stubs pre-commit hooks above. A separate
CI job runs the type checkers, the type stub tests (see Testing Type Stubs)
and stubtest against the PyArrow it builds, with ci/scripts/python_test_type_annotations.sh.
Type Checking in an IDE#
Editors also run the type checkers from the folder they open. If you open the
whole arrow repository in VS Code, point the type checker at arrow/python
instead.
Testing Type Stubs#
The tests in pyarrow-stubs-tests/ pin down the annotated types of public
APIs. Each check pairs typing.assert_type(), which every type checker
verifies statically, with a runtime isinstance check:
from typing import TYPE_CHECKING
import pyarrow as pa
from _check import assert_type, check
def test_list_type() -> None:
lst = pa.list_(pa.int32())
check(assert_type(lst, pa.ListType), pa.ListType)
check(assert_type(lst.value_type, pa.DataType), pa.DataType)
if TYPE_CHECKING:
pa.list_(1) # type: ignore[call-overload]
The type checkers check these files as part of the regular runs above, and pytest runs them against the installed PyArrow to make sure the stubs agree with what PyArrow actually returns:
$ pytest pyarrow-stubs-tests
Code under if TYPE_CHECKING: is never executed. It holds the usage that
should fail type checking, which is asserted with a
# type: ignore[<mypy code>] comment: unused ignores are reported as errors
in these tests, so the test fails once the usage is no longer rejected. ty
doesn’t honor mypy error codes, so it doesn’t check invalid usage.
Doctest#
We are using doctest to check that docstring examples are up-to-date and correct. You can also do that locally by running:
$ pushd arrow/python
$ python -m pytest --doctest-modules
$ python -m pytest --doctest-modules path/to/module.py # checking single file
$ popd
for .py files or
$ pushd arrow/python
$ python -m pytest --doctest-cython
$ python -m pytest --doctest-cython path/to/module.pyx # checking single file
$ popd
for .pyx and .pxi files. In this case you will also need to
install the pytest-cython plugin.
Note
Cython .pxi files are included in .pyx files at compile time,
so --doctest-cython cannot be run directly on .pxi files.
In PyArrow, all .pxi files are included into lib.pyx, so run
doctests on that file:
$ python -m pytest --doctest-cython path/to/lib.pyx
Any doctest errors originating from .pxi files will appear under
lib.pyx, not the original .pxi filename.
Testing Documentation Examples#
Documentation examples in .rst files under docs/source/python/ use
doctest syntax and can be tested locally using:
$ pushd arrow/python
$ pytest --doctest-glob="*.rst" docs/source/python/file.rst # checking single file
$ pytest --doctest-glob="*.rst" docs/source/python # checking entire directory
$ popd
The examples use standard doctest syntax with >>> for Python prompts and
... for continuation lines. The conftest.py fixture automatically
handles temporary directory setup for examples that create files.
Debugging#
Debug build#
Since PyArrow depends on the Arrow C++ libraries, debugging can
frequently involve crossing between Python and C++ shared libraries.
For the best experience, make sure you’ve built both Arrow C++
(-DCMAKE_BUILD_TYPE=Debug) and PyArrow
(pip install --no-build-isolation -C cmake.build-type=Debug .)
in debug mode.
Using gdb on Linux#
To debug the C++ libraries with gdb while running the Python unit tests, first start pytest with gdb:
$ gdb --args python -m pytest pyarrow/tests/test_to_run.py -k $TEST_TO_MATCH
To set a breakpoint, use the same gdb syntax that you would when debugging a C++ program, for example:
(gdb) b src/arrow/python/arrow_to_pandas.cc:1874
No source file named src/arrow/python/arrow_to_pandas.cc.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (src/arrow/python/arrow_to_pandas.cc:1874) pending.
See also
Similarly, use lldb when debugging on macOS.
Benchmarking#
For running the benchmarks, see Benchmarks.