Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
265799a
Add WAL content for 2.0 release
Andriciuc Aug 5, 2025
c150be4
improve functions.md with small text fix and reword introductory para…
Andriciuc Aug 7, 2025
9e0a879
Merge branch 'TDE_REL_17_STABLE' into wal-2.0-update
Andriciuc Aug 7, 2025
3dd51c1
Merge branch 'TDE_REL_17_STABLE' into wal-2.0-update
Andriciuc Aug 12, 2025
662f178
Merge branch 'TDE_REL_17_STABLE' into wal-2.0-update
Andriciuc Aug 13, 2025
1c12c70
add WAL tool support to limitations, improve flow, add button to setup
Andriciuc Aug 14, 2025
5d4fe68
Update contrib/pg_tde/documentation/docs/index/tde-limitations.md
Andriciuc Aug 18, 2025
8769d25
update limitations with feedback
Andriciuc Aug 18, 2025
9ce0f6a
Merge branch 'wal-2.0-update' of https://github.com/percona/postgres …
Andriciuc Aug 18, 2025
79b5c28
Merge branch 'TDE_REL_17_STABLE' into wal-2.0-update
Andriciuc Aug 18, 2025
3dc94d3
add limitation regarding WAL shipping standy not supported with WAL e…
Andriciuc Aug 19, 2025
09b38f4
remove confusing text from how tde works, link in note to KMS config …
Andriciuc Aug 19, 2025
aa604d0
add mention of open source and enterprise ed being supported for pg_tde
Andriciuc Aug 19, 2025
b3b7e89
Update the Features topic buttons for better clarity (#508)
Andriciuc Aug 20, 2025
c1c9b63
Merge branch 'TDE_REL_17_STABLE' into wal-2.0-update
Andriciuc Aug 20, 2025
86d8d0d
modify limitations and global key config with feedback
Andriciuc Aug 20, 2025
e6bf5a9
add none method to basebackup and link to topic
Andriciuc Aug 21, 2025
a551004
rename WAL tool support to compatibility
Andriciuc Aug 21, 2025
37fa8b2
add Example Patroni configuration for Patroni tool
Andriciuc Aug 21, 2025
58e53e8
add a new paragraph for unsupported tools under WAL tool compat and r…
Andriciuc Aug 21, 2025
b84c6c4
improve supported vs unsupported tools section
Andriciuc Aug 22, 2025
ab32dbb
Merge branch 'release-17.5.3' into wal-2.0-update
Andriciuc Aug 22, 2025
0f00d2a
Merge branch 'release-17.5.3' into wal-2.0-update
Andriciuc Aug 22, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ The following sections break down the key architectural components of this desig
* Indexes
* Sequences
* Temporary tables
* Write Ahead Log (WAL), still in beta. **Do not enable this feature in production environments**.
* Write Ahead Log (WAL)

**Extension** means that `pg_tde` should be implemented only as an extension, possibly compatible with any PostgreSQL distribution, including the open source community version. This requires changes in the PostgreSQL core to make it more extensible. Therefore, `pg_tde` currently works only with the [Percona Server for PostgreSQL](https://docs.percona.com/postgresql/17/index.html) - a binary replacement of community PostgreSQL and included in Percona Distribution for PostgreSQL.

Expand Down Expand Up @@ -82,9 +82,6 @@ Later decisions are made using a slightly modified Storage Manager (SMGR) API: w

### WAL encryption

!!! note
The WAL encryption feature is currently in beta and is not effective unless explicitly enabled. It is not yet production ready. **Do not enable this feature in production environments**.

WAL encryption is controlled globally via a global GUC variable, `pg_tde.wal_encrypt`, that requires a server restart.

WAL keys also contain the [LSN](https://www.postgresql.org/docs/17/wal-internals.html) of the first WAL write after key creation. This allows `pg_tde` to know which WAL ranges are encrypted or not and with which key.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,6 @@

[`pg_waldump` :octicons-link-external-16:](https://www.postgresql.org/docs/current/pgwaldump.html) is a tool to display a human-readable rendering of the Write-Ahead Log (WAL) of a PostgreSQL database cluster.

!!! warning
The WAL encryption feature is currently in beta and is not effective unless explicitly enabled. It is not yet production ready. **Do not enable this feature in production environments**.

To read encrypted WAL records, `pg_waldump` supports the following additional arguments:

* `keyring_path` is the directory where the keyring configuration files for WAL are stored. The following files are included:
Expand Down
5 changes: 1 addition & 4 deletions contrib/pg_tde/documentation/docs/faq.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,9 +94,6 @@ The principal key is used to encrypt the internal keys. The principal key is sto

### WAL encryption

!!! note
WAL encryption is currently in beta and is not effective unless explicitly enabled. It is not yet production ready. **Do not enable this feature in production environments**.

WAL encryption is done globally for the entire database cluster. All modifications to any database within a PostgreSQL cluster are written to the same WAL to maintain data consistency and integrity and ensure that PostgreSQL cluster can be restored to a consistent state. Therefore, WAL is encrypted globally.

When you turn on WAL encryption, `pg_tde` encrypts entire WAL files starting from the first WAL write after the server was started with the encryption turned on.
Expand Down Expand Up @@ -140,7 +137,7 @@ Since the `SET ACCESS METHOD` command drops hint bits and this may affect the pe
You must restart the database in the following cases to apply the changes:

* after you enabled the `pg_tde` extension
* when enabling WAL encryption, which is currently in beta. **Do not enable this feature in production environments**.
* when enabling WAL encryption

After that, no database restart is required. When you create or alter the table using the `tde_heap` access method, the files are marked as those that require encryption. The encryption happens at the storage manager level, before a transaction is written to disk. Read more about [how tde_heap works](index/table-access-method.md#how-tde_heap-works-with-pg_tde).

Expand Down
5 changes: 1 addition & 4 deletions contrib/pg_tde/documentation/docs/functions.md
Original file line number Diff line number Diff line change
Expand Up @@ -242,7 +242,7 @@ SELECT pg_tde_create_key_using_database_key_provider(

### pg_tde_create_key_using_global_key_provider

Creates a principal key at a global key provider with the given name. Use this key later with the `pg_tde_set_` series of functions.
Creates a principal key at a global key provider with the given name. Use this key later with the `pg_tde_set_*` series of functions.

```sql
SELECT pg_tde_create_key_using_global_key_provider(
Expand Down Expand Up @@ -286,9 +286,6 @@ SELECT pg_tde_set_server_key_using_global_key_provider(
);
```

!!! warning
The WAL encryption feature is currently in beta and is not effective unless explicitly enabled. It is not yet production ready. **Do not enable this feature in production environments**.

The `ensure_new_key` parameter instructs the function how to handle a principal key during key rotation:

* If set to `true`, a new key must be unique.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,15 @@ To use an external KMS with `pg_tde`, follow these two steps:
2. Set the [Global Principal Key](set-principal-key.md)

!!! note
While keyfiles may be acceptable for **local** or **testing environments**, KMS integration is the recommended approach for production deployments.
While key files may be acceptable for **local** or **testing environments**, KMS integration is the recommended approach for production deployments.

Select your prefered configuration from the links below:
`pg_tde` has been tested with the following key providers:

[KMIP Configuration :material-arrow-right:](kmip-server.md){.md-button}
[Vault Configuration :material-arrow-right:](vault.md){.md-button}
[Keyring File Configuration (not recommended) :material-arrow-right:](keyring.md){.md-button}
| KMS Provider | Description | Documentation |
|--------------------|-------------------------------------------------------|---------------|
| **KMIP** | Standard Key Management Interoperability Protocol. | [Configure KMIP →](kmip-server.md) |
| **Vault** | HashiCorp Vault integration (KV v2 API, KMIP engine). | [Configure Vault →](vault.md) |
| **Fortanix** | Fortanix DSM key management. | [Configure Fortanix →](kmip-fortanix.md) |
| **Thales** | Thales CipherTrust Manager and DSM. | [Configure Thales →](kmip-thales.md) |
| **OpenBao** | Community fork of Vault, supporting KV v2. | [Configure OpenBao →](kmip-openbao.md) |
| **Keyring file** *(not recommended)* | Local key file for dev/test only. | [Configure keyring file →](keyring.md) |
Original file line number Diff line number Diff line change
@@ -1,10 +1,16 @@
# Global Principal Key configuration

You can configure a default principal key using a global key provider. This key will be used by all databases that do not have their own encryption keys configured. The function **both** sets the principal key and rotates internal keys as needed.
You can configure a default principal key using a global key provider. This key is used by all databases that do not have their own encryption keys configured.

There are two main functions for this:

- [pg_tde_create_key_using_global_key_provider()](../functions.md#pg_tde_create_key_using_global_key_provider) creates a principal key at a global key provider
- [pg_tde_set_default_key_using_global_key_provider()](../functions.md#pg_tde_set_default_key_using_global_key_provider) sets the default principal key and rotates the internal encryption key if one is already configured

## Create a default principal key

!!! note

The sample output below is for demonstration purposes only. Be sure to replace the key name and provider with your actual values.

To create a global principal key, run:
Expand Down Expand Up @@ -57,15 +63,19 @@ SELECT pg_tde_set_default_key_using_global_key_provider(
* `global_vault_provider` is the name of the global key provider you previously configured.

!!! note

If no error is reported, the action completed successfully.

## How key generation works

The key material (actual cryptographic key) is auto-generated by `pg_tde` and stored securely by the configured provider.

!!! note

This process sets the **default principal key for the entire server**. Any database without a key explicitly configured will fall back to this key.

## Next steps

To confirm that encryption is working as expected, follow the validation steps:

[Validate Encryption with pg_tde :material-arrow-right:](../test.md){.md-button}
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Vault configuration

You can configure `pg_tde` to use HashiCorp Vault as a global key provider for managing encryption keys securely.
You can configure `pg_tde` to use HashiCorp Vault as a global key provider for managing encryption keys securely. Both the open source and enterprise editions are supported.

!!! note
This guide assumes that your Vault server is already set up and accessible. Vault configuration is outside the scope of this document, see [Vault's official documentation](https://developer.hashicorp.com/vault/docs) for more information.
Expand Down
10 changes: 2 additions & 8 deletions contrib/pg_tde/documentation/docs/index/how-does-tde-work.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,15 +6,9 @@ To encrypt the data, two types of keys are used:
* The **principal key** to encrypt database keys. It is kept separately from the database keys and is managed externally in the key management store.

!!! note
For more information on managing and storing principal keys externally, including supported key management systems and the local keyring option, see [Key management overview](../global-key-provider-configuration/overview.md).

For more information on managing and storing principal keys externally, see [Configure Global Key Provider](../global-key-provider-configuration/overview.md).

You have the following options to store and manage principal keys externally:

* Use the HashiCorp Vault server. Only the back end KV Secrets Engine - Version 2 (API) is supported.
* Use the KMIP-compatible server. `pg_tde` has been tested with the [PyKMIP](https://pykmip.readthedocs.io/en/latest/server.html) server and [the HashiCorp Vault Enterprise KMIP Secrets Engine](https://www.vaultproject.io/docs/secrets/kmip).

The encryption process is the following:
The encryption process works as follows:

![image](../_images/tde-flow.png)

Expand Down
97 changes: 94 additions & 3 deletions contrib/pg_tde/documentation/docs/index/tde-limitations.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,99 @@
# Limitations of pg_tde

The following are current limitations of `pg_tde`:
Limitations of `pg_tde` {{release}}:

* System tables, which include statistics data and database statistics, are currently **not encrypted**.
* The WAL encryption feature is currently in beta and is not effective unless explicitly enabled. It is not yet production ready. **Do not enable this feature in production environments**.
* PostgreSQL’s internal system tables, which include statistics and metadata, are not encrypted.
* Temporary files created when queries exceed `work_mem` are not encrypted. These files may persist during long-running queries or after a server crash which can expose sensitive data in plaintext on disk.

## Currently unsupported WAL tools

The following tools are currently unsupported with `pg_tde` WAL encryption:

* `pg_createsubscriber`
* `pg_verifybackup` (checksum mismatch with encrypted WAL)

The following tools and extensions in Percona Distribution for PostgreSQL have been tested and verified to work with `pg_tde` WAL encryption:

## Supported WAL tools

The following tools have been tested and verified by Percona to work with `pg_tde` WAL encryption:

* Patroni, for an example configuration see the following [Patroni configuration file](#example-patroni-configuration)
* `pg_basebackup` (with `--wal-method=stream` or `--wal-method=none`), for details on using `pg_basebackup` with WAL encryption, see [Backup with WAL encryption enabled](../how-to/backup-wal-enabled.md)
* `pg_resetwal`
* `pg_rewind`
* `pg_upgrade`
* `pg_waldump`
* pgBackRest

## Example Patroni configuration

The following is a Percona-tested example configuration.

??? example "Click to expand the Percona-tested Patroni configuration"
```yaml
# Example Patroni configuration file maintained by Percona
# Source: https://github.com/jobinau/pgscripts/blob/main/patroni/patroni.yml
scope: postgres
namespace: /db/
name: postgresql0

restapi:
listen: 0.0.0.0:8008
connect_address: 127.0.0.1:8008

etcd:
host: 127.0.0.1:2379

bootstrap:
dcs:
ttl: 30
loop_wait: 10
retry_timeout: 10
maximum_lag_on_failover: 1048576
postgresql:
use_pg_rewind: true
use_slots: true
parameters:
max_connections: 100
shared_buffers: 1GB
wal_level: replica
hot_standby: "on"
wal_keep_size: 256MB
max_wal_senders: 10
max_replication_slots: 10

initdb:
- encoding: UTF8
- data-checksums

pg_hba:
- host replication replicator 127.0.0.1/32 md5
- host all all 0.0.0.0/0 md5

postgresql:
listen: 0.0.0.0:5432
connect_address: 127.0.0.1:5432
data_dir: /var/lib/postgresql/data
bin_dir: /usr/lib/postgresql/14/bin
authentication:
replication:
username: replicator
password: rep-pass
superuser:
username: postgres
password: secretpassword
```

!!! warning
The above example is Percona-tested, but Patroni versions differ, especially with discovery backends such as `etcd`. Ensure you adjust the configuration to match your environment, version, and security requirements.

## Next steps

Check which PostgreSQL versions and deployment types are compatible with `pg_tde` before planning your installation.

[View the versions and supported deployments :material-arrow-right:](supported-versions.md){.md-button}

Begin the installation process when you're ready to set up encryption.

[Start installing `pg_tde`](../install.md){.md-button}
2 changes: 1 addition & 1 deletion contrib/pg_tde/documentation/docs/test.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,4 +59,4 @@ ALTER TABLE table_name SET ACCESS METHOD tde_heap;

## Next steps

[Configure WAL Encryption (tech preview) :material-arrow-right:](wal-encryption.md){.md-button}
[Configure WAL encryption :material-arrow-right:](wal-encryption.md){.md-button}
7 changes: 3 additions & 4 deletions contrib/pg_tde/documentation/docs/wal-encryption.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,4 @@
# Configure WAL Encryption (tech preview)

!!! warning
The WAL encryption feature is currently in beta and is not effective unless explicitly enabled. It is not yet production ready. **Do not enable this feature in production environments**.
# Configure WAL encryption

Before enabling WAL encryption, follow the steps below to create a principal key and configure it for WAL:

Expand Down Expand Up @@ -118,3 +115,5 @@ Now WAL files start to be encrypted for both encrypted and unencrypted tables.

For more technical references related to architecture, variables or functions, see:
[Technical Reference](advanced-topics/tech-reference.md){.md-button}

💬 Need help customizing this for your infrastructure? [Contact Percona support :octicons-link-external-16:](get-help.md)
2 changes: 1 addition & 1 deletion contrib/pg_tde/documentation/mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -185,7 +185,7 @@ nav:
- "Keyring file configuration": global-key-provider-configuration/keyring.md
- "2.2 Global Principal Key configuration": global-key-provider-configuration/set-principal-key.md
- "3. Validate encryption with pg_tde": test.md
- "4. Configure WAL encryption (tech preview)": wal-encryption.md
- "4. Configure WAL encryption": wal-encryption.md
- "Technical reference":
- "Overview": advanced-topics/tech-reference.md
- "Architecture": architecture/architecture.md
Expand Down