Skip to content

Commit 978f869

Browse files
bmomjianbmomjian
committed
Add key management system
This adds a key management system that stores (currently) two data encryption keys of length 128, 192, or 256 bits. The data keys are AES256 encrypted using a key encryption key, and validated via GCM cipher mode. A command to obtain the key encryption key must be specified at initdb time, and will be run at every database server start. New parameters allow a file descriptor open to the terminal to be passed. pg_upgrade support has also been added. Discussion: https://postgr.es/m/CA+fd4k7q5o6Nc_AaX6BcYM9yqTbC6_pnH-6nSD=54Zp6NBQTCQ@mail.gmail.com Discussion: https://postgr.es/m/20201202213814.GG20285@momjian.us Author: Masahiko Sawada, me, Stephen Frost
1 parent 5c31afc commit 978f869

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

49 files changed

+2091
-35
lines changed

‎doc/src/sgml/config.sgml

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7816,6 +7816,52 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
78167816
</variablelist>
78177817
</sect1>
78187818

7819+
<sect1 id="runtime-config-encryption">
7820+
<title>Cluster File Encryption</title>
7821+
7822+
<variablelist>
7823+
<varlistentry id="guc-cluster-key-command" xreflabel="cluster_key_command">
7824+
<term><varname>cluster_key_command</varname> (<type>string</type>)
7825+
<indexterm>
7826+
<primary><varname>cluster_key_command</varname> configuration parameter</primary>
7827+
</indexterm>
7828+
</term>
7829+
<listitem>
7830+
<para>
7831+
This option specifies an external command to obtain the cluster-level
7832+
key for cluster file encryption during server initialization and
7833+
server start.
7834+
</para>
7835+
<para>
7836+
The command must print the cluster key to the standard output as
7837+
64 hexadecimal characters, and exit with code 0. The command
7838+
can prompt for the passphrase or PIN from the terminal if
7839+
<option>--authprompt</option> is used. In the parameter value,
7840+
<literal>%R</literal> represents the file descriptor number opened
7841+
to the terminal that started the server. A file descriptor is only
7842+
available if enabled at server start. If <literal>%R</literal>
7843+
is used and no file descriptor is available, the server will not
7844+
start. Value <literal>%p</literal> is replaced by a pre-defined
7845+
prompt string. Value <literal>%d</literal> is replaced by the
7846+
directory containing the keys; this is useful if the command
7847+
must create files with the keys, e.g., to store a cluster-level
7848+
key encryped by a key stored in a hardware security module.
7849+
(Write <literal>%%</literal> for a literal <literal>%</literal>.)
7850+
Note that the prompt string will probably contain whitespace,
7851+
so be sure to quote its use adequately. Newlines are stripped
7852+
from the end of the output if present.
7853+
</para>
7854+
<para>
7855+
This parameter can only be set by
7856+
<application>initdb</application>, in the
7857+
<filename>postgresql.conf</filename> file, or on the server
7858+
command line.
7859+
</para>
7860+
</listitem>
7861+
</varlistentry>
7862+
</variablelist>
7863+
</sect1>
7864+
78197865
<sect1 id="runtime-config-client">
78207866
<title>Client Connection Defaults</title>
78217867

@@ -9637,6 +9683,22 @@ dynamic_library_path = 'C:\tools\postgresql;H:\my_project\lib;$libdir'
96379683
</listitem>
96389684
</varlistentry>
96399685

9686+
<varlistentry id="guc-file-encryption-keylen" xreflabel="file_encryption_keylen">
9687+
<term><varname>file_encryption_keylen</varname> (<type>boolean</type>)
9688+
<indexterm>
9689+
<primary>Cluster file encryption key length</primary>
9690+
</indexterm>
9691+
</term>
9692+
<listitem>
9693+
<para>
9694+
Reports the bit length of the cluster file
9695+
encryption key, or zero if disabled. See <xref
9696+
linkend="app-initdb-cluster-key-command"/> for more
9697+
information.
9698+
</para>
9699+
</listitem>
9700+
</varlistentry>
9701+
96409702
<varlistentry id="guc-data-directory-mode" xreflabel="data_directory_mode">
96419703
<term><varname>data_directory_mode</varname> (<type>integer</type>)
96429704
<indexterm>

‎doc/src/sgml/database-encryption.sgml

Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
<!-- doc/src/sgml/database-encryption.sgml -->
2+
3+
<chapter id="database-file-encryption">
4+
<title>Cluster File Encryption</title>
5+
6+
<indexterm zone="database-file-encryption">
7+
<primary>Cluster File Encryption</primary>
8+
</indexterm>
9+
10+
<para>
11+
The purpose of cluster file encryption is to prevent users with read
12+
access to the directories used to store database files and write-ahead
13+
log from being able to access the data stored in those files.
14+
For example, when using cluster file encryption, users who have read
15+
access to the cluster directories for backup purposes will not be able
16+
to decrypt the data stored in the these files.
17+
</para>
18+
19+
<para>
20+
Cluster file encryption uses two levels of encryption. The first level
21+
is data encryption keys, specifically keys zero and one. Key zero is
22+
the key used to encrypt database heap and index files which are stored in
23+
the file system, plus temporary files created during database operation.
24+
Key one is used to encrypt write-ahead log (WAL) files. Two different
25+
keys are used so that primary and standby servers can use different zero
26+
(heap/index/temp) keys, but the same one (WAL) key, so that these keys
27+
can eventually be rotated by switching the primary to the standby as
28+
and then changing the WAL key.
29+
</para>
30+
31+
<para>
32+
The second level of encryption is a key used to encrypt first-level
33+
keys. This type of key is often referred to as a Key Encryption Key
34+
(<acronym>KEK</acronym>). This key is <emphasis>not</emphasis> stored
35+
in the file system, but provided at <command>initdb</command> time and
36+
each time the server is started. This key prevents anyone with access
37+
to the database directories from decrypting the data because they do
38+
not know the second-level key which encrypted the first-level keys
39+
which encrypted the database cluster files.
40+
</para>
41+
42+
<sect1 id="encryption-file-encryption">
43+
<title>Initialization</title>
44+
45+
<para>
46+
Cluster file encryption is enabled when
47+
<productname>PostgreSQL</productname> is built
48+
with <literal>--with-openssl</literal> and <xref
49+
linkend="app-initdb-cluster-key-command"/> is specified
50+
during <command>initdb</command>. The cluster key
51+
provided by the <option>--cluster-key-command</option>
52+
option during <command>initdb</command> and the one generated
53+
by <xref linkend="guc-cluster-key-command"/> in the
54+
<filename>postgresql.conf</filename> must match for the database
55+
cluster to start. Note that the cluster key command
56+
passed to <command>initdb</command> must return a key of
57+
64 hexadecimal characters. For example.
58+
<programlisting>
59+
initdb -D dbname --cluster-key-command='ckey_passphrase.sh'
60+
</programlisting>
61+
</para>
62+
</sect1>
63+
64+
<sect1 id="key-encryption-key">
65+
<title>Internals</title>
66+
67+
<para>
68+
During the <command>initdb</command> process, if
69+
<option>--cluster-key-command</option> is specified, two data-level
70+
encryption keys are created. These two keys are then encrypted with
71+
the key enryption key (KEK) supplied by the cluster key command before
72+
being stored in the database directory. The key or passphrase that
73+
derives the key must be supplied from the terminal or stored in a
74+
trusted key store, such as key vault software, hardware security module.
75+
</para>
76+
77+
<para>
78+
If the <productname>PostgreSQL</productname> server has
79+
been initialized to require a cluster key, each time the
80+
server starts the <filename>postgresql.conf</filename>
81+
<varname>cluster_key_command</varname> command will be executed
82+
and the cluster key retrieved. The data encryption keys in the
83+
<filename>pg_cryptokeys</filename> directory will then be decrypted
84+
using the supplied key and integrity-checked to ensure it
85+
matches the initdb-supplied key. If this check fails, the
86+
server will refuse to start.
87+
</para>
88+
89+
<para>
90+
The data encryption keys are randomly generated and are of 128, 192,
91+
or 256-bits in length. They are encrypted by the key encryption key
92+
(KEK) using Advanced Encryption Standard (<acronym>AES256</acronym>)
93+
encryption in Galois/Counter Mode (<acronym>GCM</acronym>), which also
94+
provides KEK authentication.
95+
</para>
96+
</sect1>
97+
</chapter>

‎doc/src/sgml/filelist.sgml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,7 @@
4949
<!ENTITY wal SYSTEM "wal.sgml">
5050
<!ENTITY logical-replication SYSTEM "logical-replication.sgml">
5151
<!ENTITY jit SYSTEM "jit.sgml">
52+
<!ENTITY database-encryption SYSTEM "database-encryption.sgml">
5253

5354
<!-- programmer's guide -->
5455
<!ENTITY bgworker SYSTEM "bgworker.sgml">

‎doc/src/sgml/installation.sgml

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -976,8 +976,9 @@ build-postgresql:
976976
<listitem>
977977
<para>
978978
Build with support for <acronym>SSL</acronym> (encrypted)
979-
connections. This requires the <productname>OpenSSL</productname>
980-
package to be installed. <filename>configure</filename> will check
979+
connections and cluster file encryption. This requires the
980+
<productname>OpenSSL</productname> package to be installed.
981+
<filename>configure</filename> will check
981982
for the required header files and libraries to make sure that
982983
your <productname>OpenSSL</productname> installation is sufficient
983984
before proceeding.

‎doc/src/sgml/postgres.sgml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -171,6 +171,7 @@ break is not needed in a wider output rendering.
171171
&wal;
172172
&logical-replication;
173173
&jit;
174+
&database-encryption;
174175
&regress;
175176

176177
</part>

‎doc/src/sgml/ref/initdb.sgml

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -163,6 +163,17 @@ PostgreSQL documentation
163163
</listitem>
164164
</varlistentry>
165165

166+
<varlistentry id="app-initdb-cluster-key-command" xreflabel="cluster key command">
167+
<term><option>--cluster-key-command=<replaceable class="parameter">command</replaceable></option></term>
168+
<listitem>
169+
<para>
170+
This option specifies an external command to obtain the cluster-level
171+
key for cluster file encryption during server initialization and
172+
server start; see <xref linkend="guc-cluster-key-command"/> for details.
173+
</para>
174+
</listitem>
175+
</varlistentry>
176+
166177
<varlistentry>
167178
<term><option>-D <replaceable class="parameter">directory</replaceable></option></term>
168179
<term><option>--pgdata=<replaceable class="parameter">directory</replaceable></option></term>
@@ -223,6 +234,18 @@ PostgreSQL documentation
223234
</listitem>
224235
</varlistentry>
225236

237+
<varlistentry id="app-initdb-file-encryption-keylen"
238+
xreflabel="file encryption">
239+
<term><option>-K</option></term>
240+
<term><option>--file-encryption-keylen</option></term>
241+
<listitem>
242+
<para>
243+
Specifies the number of bits for the file encryption keys. The
244+
default is 128 bits.
245+
</para>
246+
</listitem>
247+
</varlistentry>
248+
226249
<varlistentry>
227250
<term><option>--locale=<replaceable>locale</replaceable></option></term>
228251
<listitem>
@@ -285,6 +308,17 @@ PostgreSQL documentation
285308
</listitem>
286309
</varlistentry>
287310

311+
<varlistentry>
312+
<term><option>-R</option></term>
313+
<term><option>--authprompt</option></term>
314+
<listitem>
315+
<para>
316+
Allows the <option>--cluster-key-command</option> command
317+
to prompt for a passphrase or PIN.
318+
</para>
319+
</listitem>
320+
</varlistentry>
321+
288322
<varlistentry>
289323
<term><option>-S</option></term>
290324
<term><option>--sync-only</option></term>
@@ -307,6 +341,18 @@ PostgreSQL documentation
307341
</listitem>
308342
</varlistentry>
309343

344+
<varlistentry>
345+
<term><option>-u <replaceable>datadir</replaceable></option></term>
346+
<term><option>--copy-encryption-keys=<replaceable>datadir</replaceable></option></term>
347+
<listitem>
348+
<para>
349+
Copies cluster file encryption keys from another cluster; required
350+
when using <application>pg_upgrade</application> on a cluster
351+
with cluster file encryption enabled.
352+
</para>
353+
</listitem>
354+
</varlistentry>
355+
310356
<varlistentry>
311357
<term><option>-U <replaceable class="parameter">username</replaceable></option></term>
312358
<term><option>--username=<replaceable class="parameter">username</replaceable></option></term>

‎doc/src/sgml/ref/pg_ctl-ref.sgml

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,7 @@ PostgreSQL documentation
3838
<arg choice="opt"><option>-s</option></arg>
3939
<arg choice="opt"><option>-o</option> <replaceable>options</replaceable></arg>
4040
<arg choice="opt"><option>-p</option> <replaceable>path</replaceable></arg>
41+
<arg choice="opt"><option>-R</option></arg>
4142
<arg choice="opt"><option>-c</option></arg>
4243
</cmdsynopsis>
4344

@@ -72,6 +73,7 @@ PostgreSQL documentation
7273
<arg choice="opt"><option>-t</option> <replaceable>seconds</replaceable></arg>
7374
<arg choice="opt"><option>-s</option></arg>
7475
<arg choice="opt"><option>-o</option> <replaceable>options</replaceable></arg>
76+
<arg choice="opt"><option>-R</option></arg>
7577
<arg choice="opt"><option>-c</option></arg>
7678
</cmdsynopsis>
7779

@@ -373,6 +375,17 @@ PostgreSQL documentation
373375
</listitem>
374376
</varlistentry>
375377

378+
<varlistentry>
379+
<term><option>-R</option></term>
380+
<term><option>--authprompt</option></term>
381+
<listitem>
382+
<para>
383+
Allows the <option>--cluster-key-command</option> command
384+
to prompt for a passphrase or PIN.
385+
</para>
386+
</listitem>
387+
</varlistentry>
388+
376389
<varlistentry>
377390
<term><option>-s</option></term>
378391
<term><option>--silent</option></term>

‎doc/src/sgml/ref/pgupgrade.sgml

Lines changed: 17 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -167,6 +167,13 @@ PostgreSQL documentation
167167
</para></listitem>
168168
</varlistentry>
169169

170+
<varlistentry>
171+
<term><option>-R</option></term>
172+
<term><option>--authprompt</option></term>
173+
<listitem><para>allows prompting for a passphrase or PIN
174+
</para></listitem>
175+
</varlistentry>
176+
170177
<varlistentry>
171178
<term><option>-s</option> <replaceable>dir</replaceable></term>
172179
<term><option>--socketdir=</option><replaceable>dir</replaceable></term>
@@ -309,7 +316,9 @@ make prefix=/usr/local/pgsql.new install
309316
Again, use compatible <command>initdb</command>
310317
flags that match the old cluster. Many
311318
prebuilt installers do this step automatically. There is no need to
312-
start the new cluster.
319+
start the new cluster. If upgrading a cluster that uses
320+
cluster file encryption, the <command>initdb</command> option
321+
<option>--copy-encryption-keys</option> must be specified.
313322
</para>
314323
</step>
315324

@@ -838,6 +847,13 @@ psql --username=postgres --file=script.sql postgres
838847
is down.
839848
</para>
840849

850+
<para>
851+
If the old cluster uses file encryption, the new cluster must use
852+
the same keys, so <command>pg_upgrade</command> copies them to the
853+
new cluster. It is necessary to initialize the new cluster with
854+
the same <varname>cluster_key_command</varname> and the same
855+
file encryption key length.
856+
</para>
841857
</refsect1>
842858

843859
<refsect1>

‎doc/src/sgml/ref/postgres-ref.sgml

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -297,6 +297,19 @@ PostgreSQL documentation
297297
</listitem>
298298
</varlistentry>
299299

300+
<varlistentry>
301+
<term><option>-R <replaceable class="parameter">file-descriptor</replaceable></option></term>
302+
<listitem>
303+
<para>
304+
Makes <command>postgres</command> prompt for a passphrase or PIN
305+
from the specified open numeric file descriptor. The descriptor
306+
is closed after the key is read. The file descriptor number
307+
<literal>-1</literal> duplicates standard error for the terminal;
308+
this is useful for single-user mode.
309+
</para>
310+
</listitem>
311+
</varlistentry>
312+
300313
<varlistentry>
301314
<term><option>-s</option></term>
302315
<listitem>

‎doc/src/sgml/storage.sgml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -77,6 +77,11 @@ Item
7777
<entry>Subdirectory containing transaction commit timestamp data</entry>
7878
</row>
7979

80+
<row>
81+
<entry><filename>pg_cryptokeys</filename></entry>
82+
<entry>Subdirectory containing file encryption keys</entry>
83+
</row>
84+
8085
<row>
8186
<entry><filename>pg_dynshmem</filename></entry>
8287
<entry>Subdirectory containing files used by the dynamic shared memory

‎src/backend/Makefile

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@ SUBDIRS = access bootstrap catalog parser commands executor foreign lib libpq \
2121
main nodes optimizer partitioning port postmaster \
2222
regex replication rewrite \
2323
statistics storage tcop tsearch utils $(top_builddir)/src/timezone \
24-
jit
24+
jit crypto
2525

2626
include $(srcdir)/common.mk
2727

0 commit comments

Comments
 (0)