Parquet Modular Encryption (Columnar Encryption)#

Columnar encryption is supported for Parquet files in C++ starting from Apache Arrow 4.0.0 and in PyArrow starting from Apache Arrow 6.0.0.

Parquet uses the envelope encryption practice, where file parts are encrypted with “data encryption keys” (DEKs), and the DEKs are encrypted with “master encryption keys” (MEKs). The DEKs are randomly generated by Parquet for each encrypted file/column. The MEKs are generated, stored and managed in a Key Management Service (KMS) of user’s choice.

Reading and writing encrypted Parquet files involves passing file encryption and decryption properties to ParquetWriter and to ParquetFile, respectively.

Writing an encrypted Parquet file:

>>> import pyarrow.parquet as pq
>>> encryption_properties = crypto_factory.file_encryption_properties(
...                                  kms_connection_config, encryption_config)
>>> with pq.ParquetWriter(filename, schema,
...                      encryption_properties=encryption_properties) as writer:
...    writer.write_table(table)

Reading an encrypted Parquet file:

>>> decryption_properties = crypto_factory.file_decryption_properties(
...                                                  kms_connection_config)
>>> parquet_file = pq.ParquetFile(filename,
...                               decryption_properties=decryption_properties)

In order to create the encryption and decryption properties, a pyarrow.parquet.encryption.CryptoFactory should be created and initialized with KMS Client details, as described below.

KMS Client#

The master encryption keys should be kept and managed in a production-grade Key Management System (KMS), deployed in the user’s organization. Using Parquet encryption requires implementation of a client class for the KMS server. Any KmsClient implementation should implement the informal interface defined by pyarrow.parquet.encryption.KmsClient as following:

>>> import pyarrow.parquet.encryption as pe
>>> class MyKmsClient(pe.KmsClient):
...
...    """An example KmsClient implementation skeleton"""
...    def __init__(self, kms_connection_configuration):
...       pe.KmsClient.__init__(self)
...       # Any KMS-specific initialization based on
...       # kms_connection_configuration comes here
...
...    def wrap_key(self, key_bytes, master_key_identifier):
...       wrapped_key = ... # call KMS to wrap key_bytes with key specified by
...                         # master_key_identifier
...       return wrapped_key
...
...    def unwrap_key(self, wrapped_key, master_key_identifier):
...       key_bytes = ... # call KMS to unwrap wrapped_key with key specified by
...                       # master_key_identifier
...       return key_bytes

The concrete implementation will be loaded at runtime by a factory function provided by the user. This factory function will be used to initialize the pyarrow.parquet.encryption.CryptoFactory for creating file encryption and decryption properties.

For example, in order to use the MyKmsClient defined above:

>>> def kms_client_factory(kms_connection_configuration):
...    return MyKmsClient(kms_connection_configuration)

>>> crypto_factory = pe.CryptoFactory(kms_client_factory)

An example of such a class for an open source KMS can be found in the Apache Arrow GitHub repository. The production KMS client should be designed in cooperation with an organization’s security administrators, and built by developers with experience in access control management. Once such a class is created, it can be passed to applications via a factory method and leveraged by general PyArrow users as shown in the encrypted parquet write/read sample above.

KMS connection configuration#

Configuration of connection to KMS (pyarrow.parquet.encryption.KmsConnectionConfig used when creating file encryption and decryption properties) includes the following options:

  • kms_instance_url, URL of the KMS instance.

  • kms_instance_id, ID of the KMS instance that will be used for encryption (if multiple KMS instances are available).

  • key_access_token, authorization token that will be passed to KMS.

  • custom_kms_conf, a string dictionary with KMS-type-specific configuration.

Encryption configuration#

pyarrow.parquet.encryption.EncryptionConfiguration (used when creating file encryption properties) includes the following options:

  • footer_key, the ID of the master key for footer encryption/signing.

  • column_keys, which columns to encrypt with which key. Dictionary with master key IDs as the keys, and column name lists as the values, e.g. {key1: [col1, col2], key2: [col3]}. See notes on nested fields below.

  • uniform_encryption, whether to encrypt the footer and all columns with the same footer_key, instead of specifying column_keys individually. Cannot be used together with column_keys.

  • encryption_algorithm, the Parquet encryption algorithm. Can be AES_GCM_V1 (default) or AES_GCM_CTR_V1.

  • plaintext_footer, whether to write the file footer in plain text (otherwise it is encrypted).

  • double_wrapping, whether to use double wrapping - where data encryption keys (DEKs) are encrypted with key encryption keys (KEKs), which in turn are encrypted with master encryption keys (MEKs). If set to false, single wrapping is used - where DEKs are encrypted directly with MEKs.

  • cache_lifetime, the lifetime of cached entities (key encryption keys, local wrapping keys, KMS client objects) represented as a datetime.timedelta.

  • internal_key_material, whether to store key material inside Parquet file footers; this mode doesn’t produce additional files. If set to false, key material is stored in separate files in the same folder, which enables key rotation for immutable Parquet files.

  • data_key_length_bits, the length of data encryption keys (DEKs), randomly generated by Parquet key management tools. Can be 128, 192 or 256 bits.

Note

When double_wrapping is true, Parquet implements a “double envelope encryption” mode that minimizes the interaction of the program with a KMS server. In this mode, the DEKs are encrypted with “key encryption keys” (KEKs, randomly generated by Parquet). The KEKs are encrypted with “master encryption keys” (MEKs) in the KMS; the result and the KEK itself are cached in the process memory.

An example encryption configuration:

>>> encryption_config = pe.EncryptionConfiguration(
...    footer_key="footer_key_name",
...    column_keys={
...       "column_key_name": ["Column1", "Column2"],
...    },
... )

Note

Columns with nested fields (struct or map data types) can be encrypted as a whole, or only individual fields. Configure an encryption key for the root column name to encrypt all nested fields with this key, or configure a key for individual leaf nested fields.

Conventionally, the key and value fields of a map column m have the names m.key_value.key and m.key_value.value, respectively. An inner field f of a struct column s has the name s.f.

With above example, all inner fields are encrypted with the same key by configuring that key for column m and s, respectively.

An example encryption configuration for columns with nested fields, where all columns are encrypted with the same key identified by column_key_id:

>>> import pyarrow as pa
>>> schema = pa.schema([
...   ("MapColumn", pa.map_(pa.string(), pa.int32())),
...   ("StructColumn", pa.struct([("f1", pa.int32()), ("f2", pa.string())])),
... ])

>>> encryption_config = pe.EncryptionConfiguration(
...    footer_key="footer_key_name",
...    column_keys={
...       "column_key_id": [ "MapColumn", "StructColumn" ],
...    },
... )

An example encryption configuration for columns with nested fields, where some inner fields are encrypted with the same key identified by column_key_id:

>>> schema = pa.schema([
...   ("MapColumn", pa.map_(pa.string(), pa.int32())),
...   ("StructColumn", pa.struct([("f1", pa.int32()), ("f2", pa.string())])),
... ])

>>> encryption_config = pe.EncryptionConfiguration(
...    footer_key="footer_key_name",
...    column_keys={
...       "column_key_id": [ "MapColumn.key_value.value", "StructColumn.f1" ],
...    },
... )

Decryption configuration#

pyarrow.parquet.encryption.DecryptionConfiguration (used when creating file decryption properties) is optional and it includes the following options:

  • cache_lifetime, the lifetime of cached entities (key encryption keys, local wrapping keys, KMS client objects) represented as a datetime.timedelta.

External key material and key rotation#

When internal_key_material=False is set on EncryptionConfiguration, key material is stored in a separate file next to the Parquet file instead of in its footer.

Storing key material externally is what enables key rotation: crypto_factory.rotate_master_keys() re-wraps the data encryption keys of a file that uses external key material under new master keys, without rewriting the file itself:

>>> crypto_factory.rotate_master_keys(
...     kms_connection_config, parquet_file_path="table.parquet",
... )

Direct Key Encryption (without KMS)#

pyarrow.parquet.encryption.create_encryption_properties() and pyarrow.parquet.encryption.create_decryption_properties() build encryption/decryption properties directly from a plaintext key, bypassing CryptoFactory and the KMS-based flow.

Note

Only uniform encryption (a single key for the footer and all columns) is supported by these functions. For per-column keys, use the CryptoFactory/EncryptionConfiguration flow described above.

>>> props = pe.create_encryption_properties(
...     footer_key=b'0123456789abcdef',
...     aad_prefix=b'table_id',
...     store_aad_prefix=False,
... )
>>> pq.write_table(table, 'encrypted.parquet', encryption_properties=props)

>>> decryption_props = pe.create_decryption_properties(
...     footer_key=b'0123456789abcdef',
...     aad_prefix=b'table_id',
... )
>>> pq.read_table('encrypted.parquet', decryption_properties=decryption_props)