Skip to content

Commit 1fde38b

Browse files
mhagandermhagander
committed
Allow on-line enabling and disabling of data checksums
This makes it possible to turn checksums on in a live cluster, without the previous need for dump/reload or logical replication (and to turn it off). Enabling checkusm starts a background process in the form of a launcher/worker combination that goes through the entire database and recalculates checksums on each and every page. Only when all pages have been checksummed are they fully enabled in the cluster. Any failure of the process will revert to checksums off and the process has to be started. This adds a new WAL record that indicates the state of checksums, so the process works across replicated clusters. Authors: Magnus Hagander and Daniel Gustafsson Review: Tomas Vondra, Michael Banck, Heikki Linnakangas, Andrey Borodin
1 parent c39e903 commit 1fde38b

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

45 files changed

+2118
-34
lines changed

‎doc/src/sgml/func.sgml

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19540,6 +19540,71 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
1954019540

1954119541
</sect2>
1954219542

19543+
<sect2 id="functions-admin-checksum">
19544+
<title>Data Checksum Functions</title>
19545+
19546+
<para>
19547+
The functions shown in <xref linkend="functions-checksums-table" /> can
19548+
be used to enable or disable data checksums in a running cluster.
19549+
See <xref linkend="checksums" /> for details.
19550+
</para>
19551+
19552+
<table id="functions-checksums-table">
19553+
<title>Checksum <acronym>SQL</acronym> Functions</title>
19554+
<tgroup cols="3">
19555+
<thead>
19556+
<row>
19557+
<entry>Function</entry>
19558+
<entry>Return Type</entry>
19559+
<entry>Description</entry>
19560+
</row>
19561+
</thead>
19562+
<tbody>
19563+
<row>
19564+
<entry>
19565+
<indexterm>
19566+
<primary>pg_enable_data_checksums</primary>
19567+
</indexterm>
19568+
<literal><function>pg_enable_data_checksums(<optional><parameter>cost_delay</parameter> <type>int</type>, <parameter>cost_limit</parameter> <type>int</type></optional>)</function></literal>
19569+
</entry>
19570+
<entry>
19571+
void
19572+
</entry>
19573+
<entry>
19574+
<para>
19575+
Initiates data checksums for the cluster. This will switch the data checksums mode
19576+
to <literal>in progress</literal> and start a background worker that will process
19577+
all data in the database and enable checksums for it. When all data pages have had
19578+
checksums enabled, the cluster will automatically switch to checksums
19579+
<literal>on</literal>.
19580+
</para>
19581+
<para>
19582+
If <parameter>cost_delay</parameter> and <parameter>cost_limit</parameter> are
19583+
specified, the speed of the process is throttled using the same principles as
19584+
<link linkend="runtime-config-resource-vacuum-cost">Cost-based Vacuum Delay</link>.
19585+
</para>
19586+
</entry>
19587+
</row>
19588+
<row>
19589+
<entry>
19590+
<indexterm>
19591+
<primary>pg_disable_data_checksums</primary>
19592+
</indexterm>
19593+
<literal><function>pg_disable_data_checksums()</function></literal>
19594+
</entry>
19595+
<entry>
19596+
void
19597+
</entry>
19598+
<entry>
19599+
Disables data checksums for the cluster.
19600+
</entry>
19601+
</row>
19602+
</tbody>
19603+
</tgroup>
19604+
</table>
19605+
19606+
</sect2>
19607+
1954319608
<sect2 id="functions-admin-dbobject">
1954419609
<title>Database Object Management Functions</title>
1954519610

‎doc/src/sgml/ref/allfiles.sgml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -211,6 +211,7 @@ Complete list of usable sgml source files in this directory.
211211
<!ENTITY pgResetwal SYSTEM "pg_resetwal.sgml">
212212
<!ENTITY pgRestore SYSTEM "pg_restore.sgml">
213213
<!ENTITY pgRewind SYSTEM "pg_rewind.sgml">
214+
<!ENTITY pgVerifyChecksums SYSTEM "pg_verify_checksums.sgml">
214215
<!ENTITY pgtestfsync SYSTEM "pgtestfsync.sgml">
215216
<!ENTITY pgtesttiming SYSTEM "pgtesttiming.sgml">
216217
<!ENTITY pgupgrade SYSTEM "pgupgrade.sgml">

‎doc/src/sgml/ref/initdb.sgml

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -195,9 +195,9 @@ PostgreSQL documentation
195195
<para>
196196
Use checksums on data pages to help detect corruption by the
197197
I/O system that would otherwise be silent. Enabling checksums
198-
may incur a noticeable performance penalty. This option can only
199-
be set during initialization, and cannot be changed later. If
200-
set, checksums are calculated for all objects, in all databases.
198+
may incur a noticeable performance penalty. If set, checksums
199+
are calculated for all objects, in all databases. See
200+
<xref linkend="checksums" /> for details.
201201
</para>
202202
</listitem>
203203
</varlistentry>

‎doc/src/sgml/ref/pg_verify_checksums.sgml

Lines changed: 112 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,112 @@
1+
<!--
2+
doc/src/sgml/ref/pg_verify_checksums.sgml
3+
PostgreSQL documentation
4+
-->
5+
6+
<refentry id="pgverifychecksums">
7+
<indexterm zone="pgverifychecksums">
8+
<primary>pg_verify_checksums</primary>
9+
</indexterm>
10+
11+
<refmeta>
12+
<refentrytitle><application>pg_verify_checksums</application></refentrytitle>
13+
<manvolnum>1</manvolnum>
14+
<refmiscinfo>Application</refmiscinfo>
15+
</refmeta>
16+
17+
<refnamediv>
18+
<refname>pg_verify_checksums</refname>
19+
<refpurpose>verify data checksums in an offline <productname>PostgreSQL</productname> database cluster</refpurpose>
20+
</refnamediv>
21+
22+
<refsynopsisdiv>
23+
<cmdsynopsis>
24+
<command>pg_verify_checksums</command>
25+
<arg choice="opt"><replaceable class="parameter">option</replaceable></arg>
26+
<arg choice="opt"><arg choice="opt"><option>-D</option></arg> <replaceable class="parameter">datadir</replaceable></arg>
27+
</cmdsynopsis>
28+
</refsynopsisdiv>
29+
30+
<refsect1 id="r1-app-pg_verify_checksums-1">
31+
<title>Description</title>
32+
<para>
33+
<command>pg_verify_checksums</command> verifies data checksums in a PostgreSQL
34+
cluster. It must be run against a cluster that's offline.
35+
</para>
36+
</refsect1>
37+
38+
<refsect1>
39+
<title>Options</title>
40+
41+
<para>
42+
The following command-line options are available:
43+
44+
<variablelist>
45+
46+
<varlistentry>
47+
<term><option>-r <replaceable>relfilenode</replaceable></option></term>
48+
<listitem>
49+
<para>
50+
Only validate checksums in the relation with specified relfilenode.
51+
</para>
52+
</listitem>
53+
</varlistentry>
54+
55+
<varlistentry>
56+
<term><option>-f</option></term>
57+
<listitem>
58+
<para>
59+
Force check even if checksums are disabled on cluster.
60+
</para>
61+
</listitem>
62+
</varlistentry>
63+
64+
<varlistentry>
65+
<term><option>-d</option></term>
66+
<listitem>
67+
<para>
68+
Enable debug output. Lists all checked blocks and their checksum.
69+
</para>
70+
</listitem>
71+
</varlistentry>
72+
73+
<varlistentry>
74+
<term><option>-V</option></term>
75+
<term><option>--version</option></term>
76+
<listitem>
77+
<para>
78+
Print the <application>pg_verify_checksums</application> version and exit.
79+
</para>
80+
</listitem>
81+
</varlistentry>
82+
83+
<varlistentry>
84+
<term><option>-?</option></term>
85+
<term><option>--help</option></term>
86+
<listitem>
87+
<para>
88+
Show help about <application>pg_verify_checksums</application> command line
89+
arguments, and exit.
90+
</para>
91+
</listitem>
92+
</varlistentry>
93+
</variablelist>
94+
</para>
95+
</refsect1>
96+
97+
<refsect1>
98+
<title>Notes</title>
99+
<para>
100+
Can only be run when the server is offline.
101+
</para>
102+
</refsect1>
103+
104+
<refsect1>
105+
<title>See Also</title>
106+
107+
<simplelist type="inline">
108+
<member><xref linkend="checksums"/></member>
109+
</simplelist>
110+
</refsect1>
111+
112+
</refentry>

‎doc/src/sgml/reference.sgml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -284,6 +284,7 @@
284284
&pgtestfsync;
285285
&pgtesttiming;
286286
&pgupgrade;
287+
&pgVerifyChecksums;
287288
&pgwaldump;
288289
&postgres;
289290
&postmaster;

‎doc/src/sgml/wal.sgml

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -230,6 +230,87 @@
230230
</para>
231231
</sect1>
232232

233+
<sect1 id="checksums">
234+
<title>Data checksums</title>
235+
<indexterm>
236+
<primary>checksums</primary>
237+
</indexterm>
238+
239+
<para>
240+
Data pages are not checksum protected by default, but this can optionally be enabled for a cluster.
241+
When enabled, each data page will be assigned a checksum that is updated when the page is
242+
written and verified every time the page is read. Only data pages are protected by checksums,
243+
internal data structures and temporary files are not.
244+
</para>
245+
246+
<para>
247+
Checksums are normally enabled when the cluster is initialized using
248+
<link linkend="app-initdb-data-checksums"><application>initdb</application></link>. They
249+
can also be enabled or disabled at runtime. In all cases, checksums are enabled or disabled
250+
at the full cluster level, and cannot be specified individually for databases or tables.
251+
</para>
252+
253+
<para>
254+
The current state of checksums in the cluster can be verified by viewing the value
255+
of the read-only configuration variable <xref linkend="guc-data-checksums" /> by
256+
issuing the command <command>SHOW data_checksums</command>.
257+
</para>
258+
259+
<para>
260+
When attempting to recover from corrupt data it may be necessary to bypass the checksum
261+
protection in order to recover data. To do this, temporarily set the configuration parameter
262+
<xref linkend="guc-ignore-checksum-failure" />.
263+
</para>
264+
265+
<sect2 id="checksums-enable-disable">
266+
<title>On-line enabling of checksums</title>
267+
268+
<para>
269+
Checksums can be enabled or disabled online, by calling the appropriate
270+
<link linkend="functions-admin-checksum">functions</link>.
271+
Disabling of checksums takes effect immediately when the function is called.
272+
</para>
273+
274+
<para>
275+
Enabling checksums will put the cluster in <literal>inprogress</literal> mode.
276+
During this time, checksums will be written but not verified. In addition to
277+
this, a background worker process is started that enables checksums on all
278+
existing data in the cluster. Once this worker has completed processing all
279+
databases in the cluster, the checksum mode will automatically switch to
280+
<literal>on</literal>.
281+
</para>
282+
283+
<para>
284+
The process will initially wait for all open transactions to finish before
285+
it starts, so that it can be certain that there are no tables that have been
286+
created inside a transaction that has not committed yet and thus would not
287+
be visible to the process enabling checksums. It will also, for each database,
288+
wait for all pre-existing temporary tables to get removed before it finishes.
289+
If long-lived temporary tables are used in the application it may be necessary
290+
to terminate these application connections to allow the process to complete.
291+
Information about open transactions and connections with temporary tables is
292+
written to log.
293+
</para>
294+
295+
<para>
296+
If the cluster is stopped while in <literal>inprogress</literal> mode, for
297+
any reason, then this process must be restarted manually. To do this,
298+
re-execute the function <function>pg_enable_data_checksums()</function>
299+
once the cluster has been restarted. It is not possible to resume the work,
300+
the process has to start over and re-process the cluster.
301+
</para>
302+
303+
<note>
304+
<para>
305+
Enabling checksums can cause significant I/O to the system, as most of the
306+
database pages will need to be rewritten, and will be written both to the
307+
data files and the WAL.
308+
</para>
309+
</note>
310+
311+
</sect2>
312+
</sect1>
313+
233314
<sect1 id="wal-intro">
234315
<title>Write-Ahead Logging (<acronym>WAL</acronym>)</title>
235316

‎src/backend/access/rmgrdesc/xlogdesc.c

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@
1717
#include "access/xlog.h"
1818
#include "access/xlog_internal.h"
1919
#include "catalog/pg_control.h"
20+
#include "storage/bufpage.h"
2021
#include "utils/guc.h"
2122
#include "utils/timestamp.h"
2223

@@ -137,6 +138,18 @@ xlog_desc(StringInfo buf, XLogReaderState *record)
137138
xlrec.ThisTimeLineID, xlrec.PrevTimeLineID,
138139
timestamptz_to_str(xlrec.end_time));
139140
}
141+
else if (info == XLOG_CHECKSUMS)
142+
{
143+
xl_checksum_state xlrec;
144+
145+
memcpy(&xlrec, rec, sizeof(xl_checksum_state));
146+
if (xlrec.new_checksumtype == PG_DATA_CHECKSUM_VERSION)
147+
appendStringInfo(buf, "on");
148+
else if (xlrec.new_checksumtype == PG_DATA_CHECKSUM_INPROGRESS_VERSION)
149+
appendStringInfo(buf, "inprogress");
150+
else
151+
appendStringInfo(buf, "off");
152+
}
140153
}
141154

142155
const char *
@@ -182,6 +195,9 @@ xlog_identify(uint8 info)
182195
case XLOG_FPI_FOR_HINT:
183196
id = "FPI_FOR_HINT";
184197
break;
198+
case XLOG_CHECKSUMS:
199+
id = "CHECKSUMS";
200+
break;
185201
}
186202

187203
return id;

0 commit comments

Comments
 (0)