Skip to content

Commit 174c480

Browse files
robertmhaasrobertmhaas
committed
Add a new WAL summarizer process.
When active, this process writes WAL summary files to $PGDATA/pg_wal/summaries. Each summary file contains information for a certain range of LSNs on a certain TLI. For each relation, it stores a "limit block" which is 0 if a relation is created or destroyed within a certain range of WAL records, or otherwise the shortest length to which the relation was truncated during that range of WAL records, or otherwise InvalidBlockNumber. In addition, it stores a list of blocks which have been modified during that range of WAL records, but excluding blocks which were removed by truncation after they were modified and never subsequently modified again. In other words, it tells us which blocks need to copied in case of an incremental backup covering that range of WAL records. But this doesn't yet add the capability to actually perform an incremental backup; the next patch will do that. A new parameter summarize_wal enables or disables this new background process. The background process also automatically deletes summary files that are older than wal_summarize_keep_time, if that parameter has a non-zero value and the summarizer is configured to run. Patch by me, with some design help from Dilip Kumar and Andres Freund. Reviewed by Matthias van de Meent, Dilip Kumar, Jakub Wartak, Peter Eisentraut, and Álvaro Herrera. Discussion: http://postgr.es/m/CA+TgmoYOYZfMCyOXFyC-P+-mdrZqm5pP2N7S-r0z3_402h9rsA@mail.gmail.com
1 parent 00498b7 commit 174c480

30 files changed

+3743
-11
lines changed

‎doc/src/sgml/config.sgml

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4150,6 +4150,67 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
41504150
</variablelist>
41514151
</sect2>
41524152

4153+
<sect2 id="runtime-config-wal-summarization">
4154+
<title>WAL Summarization</title>
4155+
4156+
<!--
4157+
<para>
4158+
These settings control WAL summarization, a feature which must be
4159+
enabled in order to perform an
4160+
<link linkend="backup-incremental-backup">incremental backup</link>.
4161+
</para>
4162+
-->
4163+
4164+
<variablelist>
4165+
<varlistentry id="guc-summarize-wal" xreflabel="summarize_wal">
4166+
<term><varname>summarize_wal</varname> (<type>boolean</type>)
4167+
<indexterm>
4168+
<primary><varname>summarize_wal</varname> configuration parameter</primary>
4169+
</indexterm>
4170+
</term>
4171+
<listitem>
4172+
<para>
4173+
Enables the WAL summarizer process. Note that WAL summarization can
4174+
be enabled either on a primary or on a standby. WAL summarization
4175+
cannot be enabled when <varname>wal_level</varname> is set to
4176+
<literal>minimal</literal>. This parameter can only be set in the
4177+
<filename>postgresql.conf</filename> file or on the server command line.
4178+
The default is <literal>off</literal>.
4179+
</para>
4180+
</listitem>
4181+
</varlistentry>
4182+
4183+
<varlistentry id="guc-wal-summary-keep-time" xreflabel="wal_summary_keep_time">
4184+
<term><varname>wal_summary_keep_time</varname> (<type>boolean</type>)
4185+
<indexterm>
4186+
<primary><varname>wal_summary_keep_time</varname> configuration parameter</primary>
4187+
</indexterm>
4188+
</term>
4189+
<listitem>
4190+
<para>
4191+
Configures the amount of time after which the WAL summarizer
4192+
automatically removes old WAL summaries. The file timestamp is used to
4193+
determine which files are old enough to remove. Typically, you should set
4194+
this comfortably higher than the time that could pass between a backup
4195+
and a later incremental backup that depends on it. WAL summaries must
4196+
be available for the entire range of WAL records between the preceding
4197+
backup and the new one being taken; if not, the incremental backup will
4198+
fail. If this parameter is set to zero, WAL summaries will not be
4199+
automatically deleted, but it is safe to manually remove files that you
4200+
know will not be required for future incremental backups.
4201+
This parameter can only be set in the
4202+
<filename>postgresql.conf</filename> file or on the server command line.
4203+
The default is 10 days. If <literal>summarize_wal = off</literal>,
4204+
existing WAL summaries will not be removed regardless of the value of
4205+
this parameter, because the WAL summarizer will not run.
4206+
</para>
4207+
</listitem>
4208+
</varlistentry>
4209+
4210+
</variablelist>
4211+
4212+
</sect2>
4213+
41534214
</sect1>
41544215

41554216
<sect1 id="runtime-config-replication">

‎src/backend/access/transam/xlog.c

Lines changed: 96 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -77,6 +77,7 @@
7777
#include "port/pg_iovec.h"
7878
#include "postmaster/bgwriter.h"
7979
#include "postmaster/startup.h"
80+
#include "postmaster/walsummarizer.h"
8081
#include "postmaster/walwriter.h"
8182
#include "replication/logical.h"
8283
#include "replication/origin.h"
@@ -3592,6 +3593,43 @@ XLogGetLastRemovedSegno(void)
35923593
return lastRemovedSegNo;
35933594
}
35943595

3596+
/*
3597+
* Return the oldest WAL segment on the given TLI that still exists in
3598+
* XLOGDIR, or 0 if none.
3599+
*/
3600+
XLogSegNo
3601+
XLogGetOldestSegno(TimeLineID tli)
3602+
{
3603+
DIR *xldir;
3604+
struct dirent *xlde;
3605+
XLogSegNo oldest_segno = 0;
3606+
3607+
xldir = AllocateDir(XLOGDIR);
3608+
while ((xlde = ReadDir(xldir, XLOGDIR)) != NULL)
3609+
{
3610+
TimeLineID file_tli;
3611+
XLogSegNo file_segno;
3612+
3613+
/* Ignore files that are not XLOG segments. */
3614+
if (!IsXLogFileName(xlde->d_name))
3615+
continue;
3616+
3617+
/* Parse filename to get TLI and segno. */
3618+
XLogFromFileName(xlde->d_name, &file_tli, &file_segno,
3619+
wal_segment_size);
3620+
3621+
/* Ignore anything that's not from the TLI of interest. */
3622+
if (tli != file_tli)
3623+
continue;
3624+
3625+
/* If it's the oldest so far, update oldest_segno. */
3626+
if (oldest_segno == 0 || file_segno < oldest_segno)
3627+
oldest_segno = file_segno;
3628+
}
3629+
3630+
FreeDir(xldir);
3631+
return oldest_segno;
3632+
}
35953633

35963634
/*
35973635
* Update the last removed segno pointer in shared memory, to reflect that the
@@ -3872,8 +3910,8 @@ RemoveXlogFile(const struct dirent *segment_de,
38723910
}
38733911

38743912
/*
3875-
* Verify whether pg_wal and pg_wal/archive_status exist.
3876-
* If the latter does not exist, recreate it.
3913+
* Verify whether pg_wal, pg_wal/archive_status, and pg_wal/summaries exist.
3914+
* If the latter do not exist, recreate them.
38773915
*
38783916
* It is not the goal of this function to verify the contents of these
38793917
* directories, but to help in cases where someone has performed a cluster
@@ -3916,6 +3954,26 @@ ValidateXLOGDirectoryStructure(void)
39163954
(errmsg("could not create missing directory \"%s\": %m",
39173955
path)));
39183956
}
3957+
3958+
/* Check for summaries */
3959+
snprintf(path, MAXPGPATH, XLOGDIR "/summaries");
3960+
if (stat(path, &stat_buf) == 0)
3961+
{
3962+
/* Check for weird cases where it exists but isn't a directory */
3963+
if (!S_ISDIR(stat_buf.st_mode))
3964+
ereport(FATAL,
3965+
(errmsg("required WAL directory \"%s\" does not exist",
3966+
path)));
3967+
}
3968+
else
3969+
{
3970+
ereport(LOG,
3971+
(errmsg("creating missing WAL directory \"%s\"", path)));
3972+
if (MakePGDirectory(path) < 0)
3973+
ereport(FATAL,
3974+
(errmsg("could not create missing directory \"%s\": %m",
3975+
path)));
3976+
}
39193977
}
39203978

39213979
/*
@@ -5243,9 +5301,9 @@ StartupXLOG(void)
52435301
#endif
52445302

52455303
/*
5246-
* Verify that pg_wal and pg_wal/archive_status exist. In cases where
5247-
* someone has performed a copy for PITR, these directories may have been
5248-
* excluded and need to be re-created.
5304+
* Verify that pg_wal, pg_wal/archive_status, and pg_wal/summaries exist.
5305+
* In cases where someone has performed a copy for PITR, these directories
5306+
* may have been excluded and need to be re-created.
52495307
*/
52505308
ValidateXLOGDirectoryStructure();
52515309

@@ -6962,6 +7020,25 @@ CreateCheckPoint(int flags)
69627020
*/
69637021
END_CRIT_SECTION();
69647022

7023+
/*
7024+
* WAL summaries end when the next XLOG_CHECKPOINT_REDO or
7025+
* XLOG_CHECKPOINT_SHUTDOWN record is reached. This is the first point
7026+
* where (a) we're not inside of a critical section and (b) we can be
7027+
* certain that the relevant record has been flushed to disk, which must
7028+
* happen before it can be summarized.
7029+
*
7030+
* If this is a shutdown checkpoint, then this happens reasonably
7031+
* promptly: we've only just inserted and flushed the
7032+
* XLOG_CHECKPOINT_SHUTDOWN record. If this is not a shutdown checkpoint,
7033+
* then this might not be very prompt at all: the XLOG_CHECKPOINT_REDO
7034+
* record was written before we began flushing data to disk, and that
7035+
* could be many minutes ago at this point. However, we don't XLogFlush()
7036+
* after inserting that record, so we're not guaranteed that it's on disk
7037+
* until after the above call that flushes the XLOG_CHECKPOINT_ONLINE
7038+
* record.
7039+
*/
7040+
SetWalSummarizerLatch();
7041+
69657042
/*
69667043
* Let smgr do post-checkpoint cleanup (eg, deleting old files).
69677044
*/
@@ -7636,6 +7713,20 @@ KeepLogSeg(XLogRecPtr recptr, XLogSegNo *logSegNo)
76367713
}
76377714
}
76387715

7716+
/*
7717+
* If WAL summarization is in use, don't remove WAL that has yet to be
7718+
* summarized.
7719+
*/
7720+
keep = GetOldestUnsummarizedLSN(NULL, NULL, false);
7721+
if (keep != InvalidXLogRecPtr)
7722+
{
7723+
XLogSegNo unsummarized_segno;
7724+
7725+
XLByteToSeg(keep, unsummarized_segno, wal_segment_size);
7726+
if (unsummarized_segno < segno)
7727+
segno = unsummarized_segno;
7728+
}
7729+
76397730
/* but, keep at least wal_keep_size if that's set */
76407731
if (wal_keep_size_mb > 0)
76417732
{

‎src/backend/backup/Makefile

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,8 @@ OBJS = \
2525
basebackup_server.o \
2626
basebackup_sink.o \
2727
basebackup_target.o \
28-
basebackup_throttle.o
28+
basebackup_throttle.o \
29+
walsummary.o \
30+
walsummaryfuncs.o
2931

3032
include $(top_srcdir)/src/backend/common.mk

‎src/backend/backup/meson.build

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,4 +12,6 @@ backend_sources += files(
1212
'basebackup_target.c',
1313
'basebackup_throttle.c',
1414
'basebackup_zstd.c',
15+
'walsummary.c',
16+
'walsummaryfuncs.c',
1517
)

0 commit comments

Comments
 (0)