Reading and Writing the Apache Parquet Format#
The Apache Parquet project provides a standardized open-source columnar storage format for use in data analysis systems. It was created originally for use in Apache Hadoop with systems like Apache Drill, Apache Hive, Apache Impala, and Apache Spark adopting it as a shared standard for high performance data IO.
Apache Arrow is an ideal in-memory transport layer for data that is being read or written with Parquet files. We have been concurrently developing the C++ implementation of Apache Parquet, which includes a native, multithreaded C++ adapter to and from in-memory Arrow data. PyArrow includes Python bindings to this code, which thus enables reading and writing Parquet files with pandas as well.
Obtaining pyarrow with Parquet Support#
If you installed pyarrow with pip or conda, it should be built with Parquet
support bundled:
>>> import pyarrow.parquet as pq
If you are building pyarrow from source, you must use -DARROW_PARQUET=ON
when compiling the C++ libraries and enable the Parquet extensions when
building pyarrow. If you want to use Parquet Encryption, then you must
use -DPARQUET_REQUIRE_ENCRYPTION=ON too when compiling the C++ libraries.
See the Python Development page for more details.
Reading and Writing Single Files#
- Reading and Writing Single Files
- Reading Parquet and Memory Mapping
- Parquet file writing options
- Omitting the DataFrame index
- Finer-grained Reading and Writing
- Inspecting the Parquet File Metadata
- Filtering / Predicate Pushdown
- Writing Bloom Filters
- Multithreaded Reads
- Compression, Encoding, and File Compatibility
- Using with Spark
- Reading from cloud storage
- Content-Defined Chunking