Skip to content

Commit 042a923

Browse files
michaelpqmichaelpq
committed
Rework compression options of pg_receivewal
Since babbbb5 and the introduction of LZ4 in pg_receivewal, the compression of the WAL archived is controlled by two options: - --compression-method with "gzip", "none" or "lz4" as possible value. - --compress=N to specify a compression level. This includes a backward-incompatible change where a value of 0 leads to a failure instead of no compression enforced. This commit takes advantage of a4b5754 and 3603f7c to rework the compression options of pg_receivewal, as of: - The removal of --compression-method. - The extenction of --compress to use the same grammar as pg_basebackup, with a METHOD:DETAIL format, where a METHOD is "gzip", "none" or "lz4" and a DETAIL is a comma-separated list of options, the only keyword supported is now "level" to control the compression level. If only an integer is specified as value of this option, "none" is implied on 0 and "gzip" is implied otherwise. This brings back --compress to be backward-compatible with ~14, while still supporting LZ4. This has also the advantage of centralizing the set of checks used by pg_receivewal to validate its compression options. Author: Michael Paquier Reviewed-by: Robert Haas, Georgios Kokolatos Discussion: https://postgr.es/m/YlPQGNAAa04raObK@paquier.xyz
1 parent d27323d commit 042a923

File tree

3 files changed

+121
-79
lines changed

3 files changed

+121
-79
lines changed

‎doc/src/sgml/ref/pg_receivewal.sgml

Lines changed: 22 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -263,37 +263,36 @@ PostgreSQL documentation
263263
</varlistentry>
264264

265265
<varlistentry>
266-
<term><option>--compression-method=<replaceable class="parameter">method</replaceable></option></term>
266+
<term><option>-Z <replaceable class="parameter">level</replaceable></option></term>
267+
<term><option>-Z <replaceable class="parameter">method</replaceable>[:<replaceable>detail</replaceable>]</option></term>
268+
<term><option>--compress=<replaceable class="parameter">level</replaceable></option></term>
269+
<term><option>--compress=<replaceable class="parameter">method</replaceable>[:<replaceable>detail</replaceable>]</option></term>
267270
<listitem>
268271
<para>
269-
Enables compression of write-ahead logs using the specified method.
270-
Supported values are <literal>gzip</literal>, <literal>lz4</literal>
271-
(if <productname>PostgreSQL</productname> was compiled with
272-
<option>--with-lz4</option>), and <literal>none</literal>.
272+
Enables compression of write-ahead logs.
273273
</para>
274-
275274
<para>
276-
The suffix <filename>.gz</filename> will automatically be added to
277-
all filenames when using <literal>gzip</literal>, and the suffix
278-
<filename>.lz4</filename> is added when using <literal>lz4</literal>.
275+
The compression method can be set to <literal>gzip</literal>,
276+
<literal>lz4</literal> (if <productname>PostgreSQL</productname>
277+
was compiled with <option>--with-lz4</option>) or
278+
<literal>none</literal> for no compression.
279+
A compression detail string can optionally be specified. If the
280+
detail string is an integer, it specifies the compression level.
281+
Otherwise, it should be a comma-separated list of items, each of the
282+
form <literal>keyword</literal> or <literal>keyword=value</literal>.
283+
Currently, the only supported keyword is <literal>level</literal>.
279284
</para>
280-
</listitem>
281-
</varlistentry>
282-
283-
<varlistentry>
284-
<term><option>-Z <replaceable class="parameter">level</replaceable></option></term>
285-
<term><option>--compress=<replaceable class="parameter">level</replaceable></option></term>
286-
<listitem>
287285
<para>
288-
Specifies the compression level (<literal>1</literal> through
289-
<literal>9</literal>, <literal>1</literal> being worst compression
290-
and <literal>9</literal> being best compression) for WAL segments
291-
compressed with <application>gzip</application>.
286+
If no compression level is specified, the default compression level
287+
will be used. If only a level is specified without mentioning an
288+
algorithm, <literal>gzip</literal> compression will be used if the
289+
level is greater than 0, and no compression will be used if the level
290+
is 0.
292291
</para>
293-
294292
<para>
295-
This option requires <option>--compression-method</option> to be
296-
specified with <literal>gzip</literal>.
293+
The suffix <filename>.gz</filename> will automatically be added to
294+
all filenames when using <literal>gzip</literal>, and the suffix
295+
<filename>.lz4</filename> is added when using <literal>lz4</literal>.
297296
</para>
298297
</listitem>
299298
</varlistentry>

‎src/bin/pg_basebackup/pg_receivewal.c

Lines changed: 90 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,8 @@ static XLogRecPtr endpos = InvalidXLogRecPtr;
5757

5858

5959
static void usage(void);
60+
static void parse_compress_options(char *option, char **algorithm,
61+
char **detail);
6062
static DIR *get_destination_dir(char *dest_folder);
6163
static void close_destination_dir(DIR *dest_dir, char *dest_folder);
6264
static XLogRecPtr FindStreamingStart(uint32 *tli);
@@ -90,9 +92,8 @@ usage(void)
9092
printf(_(" --synchronous flush write-ahead log immediately after writing\n"));
9193
printf(_(" -v, --verbose output verbose messages\n"));
9294
printf(_(" -V, --version output version information, then exit\n"));
93-
printf(_(" --compression-method=METHOD\n"
94-
" method to compress logs\n"));
95-
printf(_(" -Z, --compress=1-9 compress logs with given compression level\n"));
95+
printf(_(" -Z, --compress=METHOD[:DETAIL]\n"
96+
" compress as specified\n"));
9697
printf(_(" -?, --help show this help, then exit\n"));
9798
printf(_("\nConnection options:\n"));
9899
printf(_(" -d, --dbname=CONNSTR connection string\n"));
@@ -108,6 +109,66 @@ usage(void)
108109
printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
109110
}
110111

112+
/*
113+
* Basic parsing of a value specified for -Z/--compress
114+
*
115+
* The parsing consists of a METHOD:DETAIL string fed later on to a more
116+
* advanced routine in charge of proper validation checks. This only extracts
117+
* METHOD and DETAIL. If only an integer is found, the method is implied by
118+
* the value specified.
119+
*/
120+
static void
121+
parse_compress_options(char *option, char **algorithm, char **detail)
122+
{
123+
char *sep;
124+
char *endp;
125+
long result;
126+
127+
/*
128+
* Check whether the compression specification consists of a bare integer.
129+
*
130+
* For backward-compatibility, assume "none" if the integer found is zero
131+
* and "gzip" otherwise.
132+
*/
133+
result = strtol(option, &endp, 10);
134+
if (*endp == '\0')
135+
{
136+
if (result == 0)
137+
{
138+
*algorithm = pstrdup("none");
139+
*detail = NULL;
140+
}
141+
else
142+
{
143+
*algorithm = pstrdup("gzip");
144+
*detail = pstrdup(option);
145+
}
146+
return;
147+
}
148+
149+
/*
150+
* Check whether there is a compression detail following the algorithm
151+
* name.
152+
*/
153+
sep = strchr(option, ':');
154+
if (sep == NULL)
155+
{
156+
*algorithm = pstrdup(option);
157+
*detail = NULL;
158+
}
159+
else
160+
{
161+
char *alg;
162+
163+
alg = palloc((sep - option) + 1);
164+
memcpy(alg, option, sep - option);
165+
alg[sep - option] = '\0';
166+
167+
*algorithm = alg;
168+
*detail = pstrdup(sep + 1);
169+
}
170+
}
171+
111172
/*
112173
* Check if the filename looks like a WAL file, letting caller know if this
113174
* WAL segment is partial and/or compressed.
@@ -651,7 +712,6 @@ main(int argc, char **argv)
651712
{"if-not-exists", no_argument, NULL, 3},
652713
{"synchronous", no_argument, NULL, 4},
653714
{"no-sync", no_argument, NULL, 5},
654-
{"compression-method", required_argument, NULL, 6},
655715
{NULL, 0, NULL, 0}
656716
};
657717

@@ -660,6 +720,10 @@ main(int argc, char **argv)
660720
char *db_name;
661721
uint32 hi,
662722
lo;
723+
pg_compress_specification compression_spec;
724+
char *compression_detail = NULL;
725+
char *compression_algorithm_str = "none";
726+
char *error_detail = NULL;
663727

664728
pg_logging_init(argv[0]);
665729
progname = get_progname(argv[0]);
@@ -728,9 +792,8 @@ main(int argc, char **argv)
728792
verbose++;
729793
break;
730794
case 'Z':
731-
if (!option_parse_int(optarg, "-Z/--compress", 1, 9,
732-
&compresslevel))
733-
exit(1);
795+
parse_compress_options(optarg, &compression_algorithm_str,
796+
&compression_detail);
734797
break;
735798
/* action */
736799
case 1:
@@ -748,17 +811,6 @@ main(int argc, char **argv)
748811
case 5:
749812
do_sync = false;
750813
break;
751-
case 6:
752-
if (pg_strcasecmp(optarg, "gzip") == 0)
753-
compression_algorithm = PG_COMPRESSION_GZIP;
754-
else if (pg_strcasecmp(optarg, "lz4") == 0)
755-
compression_algorithm = PG_COMPRESSION_LZ4;
756-
else if (pg_strcasecmp(optarg, "none") == 0)
757-
compression_algorithm = PG_COMPRESSION_NONE;
758-
else
759-
pg_fatal("invalid value \"%s\" for option %s",
760-
optarg, "--compression-method");
761-
break;
762814
default:
763815
/* getopt_long already emitted a complaint */
764816
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
@@ -810,24 +862,33 @@ main(int argc, char **argv)
810862
exit(1);
811863
}
812864

813-
814865
/*
815-
* Compression-related options.
866+
* Compression options
816867
*/
868+
if (!parse_compress_algorithm(compression_algorithm_str,
869+
&compression_algorithm))
870+
pg_fatal("unrecognized compression algorithm \"%s\"",
871+
compression_algorithm_str);
872+
873+
parse_compress_specification(compression_algorithm, compression_detail,
874+
&compression_spec);
875+
error_detail = validate_compress_specification(&compression_spec);
876+
if (error_detail != NULL)
877+
pg_fatal("invalid compression specification: %s",
878+
error_detail);
879+
880+
/* Extract the compression level, if found in the specification */
881+
if ((compression_spec.options & PG_COMPRESSION_OPTION_LEVEL) != 0)
882+
compresslevel = compression_spec.level;
883+
817884
switch (compression_algorithm)
818885
{
819886
case PG_COMPRESSION_NONE:
820-
if (compresslevel != 0)
821-
{
822-
pg_log_error("cannot use --compress with --compression-method=%s",
823-
"none");
824-
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
825-
exit(1);
826-
}
887+
/* nothing to do */
827888
break;
828889
case PG_COMPRESSION_GZIP:
829890
#ifdef HAVE_LIBZ
830-
if (compresslevel == 0)
891+
if ((compression_spec.options & PG_COMPRESSION_OPTION_LEVEL) == 0)
831892
{
832893
pg_log_info("no value specified for --compress, switching to default");
833894
compresslevel = Z_DEFAULT_COMPRESSION;
@@ -838,15 +899,7 @@ main(int argc, char **argv)
838899
#endif
839900
break;
840901
case PG_COMPRESSION_LZ4:
841-
#ifdef USE_LZ4
842-
if (compresslevel != 0)
843-
{
844-
pg_log_error("cannot use --compress with --compression-method=%s",
845-
"lz4");
846-
pg_log_error_hint("Try \"%s --help\" for more information.", progname);
847-
exit(1);
848-
}
849-
#else
902+
#ifndef USE_LZ4
850903
pg_fatal("this build does not support compression with %s",
851904
"LZ4");
852905
#endif

‎src/bin/pg_basebackup/t/020_pg_receivewal.pl

Lines changed: 9 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -34,12 +34,9 @@
3434
[ 'pg_receivewal', '-D', $stream_dir, '--synchronous', '--no-sync' ],
3535
'failure if --synchronous specified with --no-sync');
3636
$primary->command_fails_like(
37-
[
38-
'pg_receivewal', '-D', $stream_dir, '--compression-method', 'none',
39-
'--compress', '1'
40-
],
41-
qr/\Qpg_receivewal: error: cannot use --compress with --compression-method=none/,
42-
'failure if --compress specified with --compression-method=none');
37+
[ 'pg_receivewal', '-D', $stream_dir, '--compress', 'none:1', ],
38+
qr/\Qpg_receivewal: error: invalid compression specification: compression algorithm "none" does not accept a compression level/,
39+
'failure if --compress none:N (where N > 0)');
4340

4441
# Slot creation and drop
4542
my $slot_name = 'test';
@@ -48,7 +45,7 @@
4845
'creating a replication slot');
4946
my $slot = $primary->slot($slot_name);
5047
is($slot->{'slot_type'}, 'physical', 'physical replication slot was created');
51-
is($slot->{'restart_lsn'}, '', 'restart LSN of new slot is null');
48+
is($slot->{'restart_lsn'}, '', 'restart LSN of new slot is null');
5249
$primary->command_ok([ 'pg_receivewal', '--slot', $slot_name, '--drop-slot' ],
5350
'dropping a replication slot');
5451
is($primary->slot($slot_name)->{'slot_type'},
@@ -93,15 +90,10 @@
9390
chomp($nextlsn);
9491
$primary->psql('postgres', 'INSERT INTO test_table VALUES (2);');
9592

96-
# Note the trailing whitespace after the value of --compress, that is
97-
# a valid value.
9893
$primary->command_ok(
9994
[
100-
'pg_receivewal', '-D',
101-
$stream_dir, '--verbose',
102-
'--endpos', $nextlsn,
103-
'--compression-method', 'gzip',
104-
'--compress', '1 ',
95+
'pg_receivewal', '-D', $stream_dir, '--verbose',
96+
'--endpos', $nextlsn, '--compress', 'gzip:1',
10597
'--no-loop'
10698
],
10799
"streaming some WAL using ZLIB compression");
@@ -153,13 +145,11 @@
153145
# Stream up to the given position.
154146
$primary->command_ok(
155147
[
156-
'pg_receivewal', '-D',
157-
$stream_dir, '--verbose',
158-
'--endpos', $nextlsn,
159-
'--no-loop', '--compression-method',
148+
'pg_receivewal', '-D', $stream_dir, '--verbose',
149+
'--endpos', $nextlsn, '--no-loop', '--compress',
160150
'lz4'
161151
],
162-
'streaming some WAL using --compression-method=lz4');
152+
'streaming some WAL using --compress=lz4');
163153

164154
# Verify that the stored files are generated with their expected
165155
# names.

0 commit comments

Comments
 (0)