Skip to content

Commit ffd5365

Browse files
robertmhaasrobertmhaas
committed
Replace BASE_BACKUP COMPRESSION_LEVEL option with COMPRESSION_DETAIL.
There are more compression parameters that can be specified than just an integer compression level, so rename the new COMPRESSION_LEVEL option to COMPRESSION_DETAIL before it gets released. Introduce a flexible syntax for that option to allow arbitrary options to be specified without needing to adjust the main replication grammar, and common code to parse it that is shared between the client and the server. This commit doesn't actually add any new compression parameters, so the only user-visible change is that you can now type something like pg_basebackup --compress gzip:level=5 instead of writing just pg_basebackup --compress gzip:5. However, it should make it easy to add new options. If for example gzip starts offering fries, we can support pg_basebackup --compress gzip:level=5,fries=true for the benefit of users who want fries with that. Along the way, this fixes a few things in pg_basebackup so that the pg_basebackup can be used with a server-side compression algorithm that pg_basebackup itself does not understand. For example, pg_basebackup --compress server-lz4 could still succeed even if only the server and not the client has LZ4 support, provided that the other options to pg_basebackup don't require the client to decompress the archive. Patch by me. Reviewed by Justin Pryzby and Dagfinn Ilmari Mannsåker. Discussion: http://postgr.es/m/CA+TgmoYvpetyRAbbg1M8b3-iHsaN4nsgmWPjOENu5-doHuJ7fA@mail.gmail.com
1 parent 4a39f87 commit ffd5365

17 files changed

+662
-331
lines changed

‎doc/src/sgml/protocol.sgml

+16-6
Original file line numberDiff line numberDiff line change
@@ -2731,14 +2731,24 @@ The commands accepted in replication mode are:
27312731
</varlistentry>
27322732

27332733
<varlistentry>
2734-
<term><literal>COMPRESSION_LEVEL</literal> <replaceable>level</replaceable></term>
2734+
<term><literal>COMPRESSION_DETAIL</literal> <replaceable>detail</replaceable></term>
27352735
<listitem>
27362736
<para>
2737-
Specifies the compression level to be used. This should only be
2738-
used in conjunction with the <literal>COMPRESSION</literal> option.
2739-
For <literal>gzip</literal> the value should be an integer between 1
2740-
and 9, for <literal>lz4</literal> between 1 and 12, and for
2741-
<literal>zstd</literal> it should be between 1 and 22.
2737+
Specifies details for the chosen compression method. This should only
2738+
be used in conjunction with the <literal>COMPRESSION</literal>
2739+
option. If the value is an integer, it specifies the compression
2740+
level. Otherwise, it should be a comma-separated list of items,
2741+
each of the form <literal>keyword</literal> or
2742+
<literal>keyword=value</literal>. Currently, the only supported
2743+
keyword is <literal>level</literal>, which sets the compression
2744+
level.
2745+
</para>
2746+
2747+
<para>
2748+
For <literal>gzip</literal> the compression level should be an
2749+
integer between 1 and 9, for <literal>lz4</literal> an integer
2750+
between 1 and 12, and for <literal>zstd</literal> an integer
2751+
between 1 and 22.
27422752
</para>
27432753
</listitem>
27442754
</varlistentry>

‎doc/src/sgml/ref/pg_basebackup.sgml

+16-9
Original file line numberDiff line numberDiff line change
@@ -399,9 +399,9 @@ PostgreSQL documentation
399399

400400
<varlistentry>
401401
<term><option>-Z <replaceable class="parameter">level</replaceable></option></term>
402-
<term><option>-Z [{client|server}-]<replaceable class="parameter">method</replaceable></option>[:<replaceable>level</replaceable>]</term>
402+
<term><option>-Z [{client|server}-]<replaceable class="parameter">method</replaceable>[:<replaceable>detail</replaceable>]</option></term>
403403
<term><option>--compress=<replaceable class="parameter">level</replaceable></option></term>
404-
<term><option>--compress=[{client|server}-]<replaceable class="parameter">method</replaceable></option>[:<replaceable>level</replaceable>]</term>
404+
<term><option>--compress=[{client|server}-]<replaceable class="parameter">method</replaceable>[:<replaceable>detail</replaceable>]</option></term>
405405
<listitem>
406406
<para>
407407
Requests compression of the backup. If <literal>client</literal> or
@@ -419,13 +419,20 @@ PostgreSQL documentation
419419
<para>
420420
The compression method can be set to <literal>gzip</literal>,
421421
<literal>lz4</literal>, <literal>zstd</literal>, or
422-
<literal>none</literal> for no compression. A compression level can
423-
optionally be specified, by appending the level number after a colon
424-
(<literal>:</literal>). If no level is specified, the default
425-
compression level will be used. If only a level is specified without
426-
mentioning an algorithm, <literal>gzip</literal> compression will be
427-
used if the level is greater than 0, and no compression will be used if
428-
the level is 0.
422+
<literal>none</literal> for no compression. A compression detail
423+
string can optionally be specified. If the detail string is an
424+
integer, it specifies the compression level. Otherwise, it should be
425+
a comma-separated list of items, each of the form
426+
<literal>keyword</literal> or <literal>keyword=value</literal>.
427+
Currently, the only supported keyword is <literal>level</literal>,
428+
which sets the compression level.
429+
</para>
430+
<para>
431+
If no compression level is specified, the default compression level
432+
will be used. If only a level is specified without mentioning an
433+
algorithm, <literal>gzip</literal> compression will be used if the
434+
level is greater than 0, and no compression will be used if the level
435+
is 0.
429436
</para>
430437
<para>
431438
When the tar format is used with <literal>gzip</literal>,

‎src/backend/replication/basebackup.c

+32-30
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@
1717
#include <time.h>
1818

1919
#include "access/xlog_internal.h" /* for pg_start/stop_backup */
20+
#include "common/backup_compression.h"
2021
#include "common/file_perm.h"
2122
#include "commands/defrem.h"
2223
#include "lib/stringinfo.h"
@@ -54,14 +55,6 @@
5455
*/
5556
#define SINK_BUFFER_LENGTH Max(32768, BLCKSZ)
5657

57-
typedef enum
58-
{
59-
BACKUP_COMPRESSION_NONE,
60-
BACKUP_COMPRESSION_GZIP,
61-
BACKUP_COMPRESSION_LZ4,
62-
BACKUP_COMPRESSION_ZSTD
63-
} basebackup_compression_type;
64-
6558
typedef struct
6659
{
6760
const char *label;
@@ -75,8 +68,8 @@ typedef struct
7568
bool use_copytblspc;
7669
BaseBackupTargetHandle *target_handle;
7770
backup_manifest_option manifest;
78-
basebackup_compression_type compression;
79-
int compression_level;
71+
bc_algorithm compression;
72+
bc_specification compression_specification;
8073
pg_checksum_type manifest_checksum_type;
8174
} basebackup_options;
8275

@@ -713,12 +706,14 @@ parse_basebackup_options(List *options, basebackup_options *opt)
713706
char *target_str = NULL;
714707
char *target_detail_str = NULL;
715708
bool o_compression = false;
716-
bool o_compression_level = false;
709+
bool o_compression_detail = false;
710+
char *compression_detail_str = NULL;
717711

718712
MemSet(opt, 0, sizeof(*opt));
719713
opt->manifest = MANIFEST_OPTION_NO;
720714
opt->manifest_checksum_type = CHECKSUM_TYPE_CRC32C;
721715
opt->compression = BACKUP_COMPRESSION_NONE;
716+
opt->compression_specification.algorithm = BACKUP_COMPRESSION_NONE;
722717

723718
foreach(lopt, options)
724719
{
@@ -885,29 +880,21 @@ parse_basebackup_options(List *options, basebackup_options *opt)
885880
ereport(ERROR,
886881
(errcode(ERRCODE_SYNTAX_ERROR),
887882
errmsg("duplicate option \"%s\"", defel->defname)));
888-
if (strcmp(optval, "none") == 0)
889-
opt->compression = BACKUP_COMPRESSION_NONE;
890-
else if (strcmp(optval, "gzip") == 0)
891-
opt->compression = BACKUP_COMPRESSION_GZIP;
892-
else if (strcmp(optval, "lz4") == 0)
893-
opt->compression = BACKUP_COMPRESSION_LZ4;
894-
else if (strcmp(optval, "zstd") == 0)
895-
opt->compression = BACKUP_COMPRESSION_ZSTD;
896-
else
883+
if (!parse_bc_algorithm(optval, &opt->compression))
897884
ereport(ERROR,
898885
(errcode(ERRCODE_SYNTAX_ERROR),
899-
errmsg("unrecognized compression algorithm: \"%s\"",
886+
errmsg("unrecognized compression algorithm \"%s\"",
900887
optval)));
901888
o_compression = true;
902889
}
903-
else if (strcmp(defel->defname, "compression_level") == 0)
890+
else if (strcmp(defel->defname, "compression_detail") == 0)
904891
{
905-
if (o_compression_level)
892+
if (o_compression_detail)
906893
ereport(ERROR,
907894
(errcode(ERRCODE_SYNTAX_ERROR),
908895
errmsg("duplicate option \"%s\"", defel->defname)));
909-
opt->compression_level = defGetInt32(defel);
910-
o_compression_level = true;
896+
compression_detail_str = defGetString(defel);
897+
o_compression_detail = true;
911898
}
912899
else
913900
ereport(ERROR,
@@ -949,10 +936,25 @@ parse_basebackup_options(List *options, basebackup_options *opt)
949936
opt->target_handle =
950937
BaseBackupGetTargetHandle(target_str, target_detail_str);
951938

952-
if (o_compression_level && !o_compression)
939+
if (o_compression_detail && !o_compression)
953940
ereport(ERROR,
954941
(errcode(ERRCODE_SYNTAX_ERROR),
955-
errmsg("compression level requires compression")));
942+
errmsg("compression detail requires compression")));
943+
944+
if (o_compression)
945+
{
946+
char *error_detail;
947+
948+
parse_bc_specification(opt->compression, compression_detail_str,
949+
&opt->compression_specification);
950+
error_detail =
951+
validate_bc_specification(&opt->compression_specification);
952+
if (error_detail != NULL)
953+
ereport(ERROR,
954+
errcode(ERRCODE_SYNTAX_ERROR),
955+
errmsg("invalid compression specification: %s",
956+
error_detail));
957+
}
956958
}
957959

958960

@@ -998,11 +1000,11 @@ SendBaseBackup(BaseBackupCmd *cmd)
9981000

9991001
/* Set up server-side compression, if client requested it */
10001002
if (opt.compression == BACKUP_COMPRESSION_GZIP)
1001-
sink = bbsink_gzip_new(sink, opt.compression_level);
1003+
sink = bbsink_gzip_new(sink, &opt.compression_specification);
10021004
else if (opt.compression == BACKUP_COMPRESSION_LZ4)
1003-
sink = bbsink_lz4_new(sink, opt.compression_level);
1005+
sink = bbsink_lz4_new(sink, &opt.compression_specification);
10041006
else if (opt.compression == BACKUP_COMPRESSION_ZSTD)
1005-
sink = bbsink_zstd_new(sink, opt.compression_level);
1007+
sink = bbsink_zstd_new(sink, &opt.compression_specification);
10061008

10071009
/* Set up progress reporting. */
10081010
sink = bbsink_progress_new(sink, opt.progress);

‎src/backend/replication/basebackup_gzip.c

+10-10
Original file line numberDiff line numberDiff line change
@@ -56,12 +56,13 @@ const bbsink_ops bbsink_gzip_ops = {
5656
#endif
5757

5858
/*
59-
* Create a new basebackup sink that performs gzip compression using the
60-
* designated compression level.
59+
* Create a new basebackup sink that performs gzip compression.
6160
*/
6261
bbsink *
63-
bbsink_gzip_new(bbsink *next, int compresslevel)
62+
bbsink_gzip_new(bbsink *next, bc_specification *compress)
6463
{
64+
int compresslevel;
65+
6566
#ifndef HAVE_LIBZ
6667
ereport(ERROR,
6768
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
@@ -71,15 +72,14 @@ bbsink_gzip_new(bbsink *next, int compresslevel)
7172
bbsink_gzip *sink;
7273

7374
Assert(next != NULL);
74-
Assert(compresslevel >= 0 && compresslevel <= 9);
7575

76-
if (compresslevel == 0)
76+
if ((compress->options & BACKUP_COMPRESSION_OPTION_LEVEL) == 0)
7777
compresslevel = Z_DEFAULT_COMPRESSION;
78-
else if (compresslevel < 0 || compresslevel > 9)
79-
ereport(ERROR,
80-
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
81-
errmsg("gzip compression level %d is out of range",
82-
compresslevel)));
78+
else
79+
{
80+
compresslevel = compress->level;
81+
Assert(compresslevel >= 1 && compresslevel <= 9);
82+
}
8383

8484
sink = palloc0(sizeof(bbsink_gzip));
8585
*((const bbsink_ops **) &sink->base.bbs_ops) = &bbsink_gzip_ops;

‎src/backend/replication/basebackup_lz4.c

+11-8
Original file line numberDiff line numberDiff line change
@@ -56,12 +56,13 @@ const bbsink_ops bbsink_lz4_ops = {
5656
#endif
5757

5858
/*
59-
* Create a new basebackup sink that performs lz4 compression using the
60-
* designated compression level.
59+
* Create a new basebackup sink that performs lz4 compression.
6160
*/
6261
bbsink *
63-
bbsink_lz4_new(bbsink *next, int compresslevel)
62+
bbsink_lz4_new(bbsink *next, bc_specification *compress)
6463
{
64+
int compresslevel;
65+
6566
#ifndef USE_LZ4
6667
ereport(ERROR,
6768
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
@@ -72,11 +73,13 @@ bbsink_lz4_new(bbsink *next, int compresslevel)
7273

7374
Assert(next != NULL);
7475

75-
if (compresslevel < 0 || compresslevel > 12)
76-
ereport(ERROR,
77-
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
78-
errmsg("lz4 compression level %d is out of range",
79-
compresslevel)));
76+
if ((compress->options & BACKUP_COMPRESSION_OPTION_LEVEL) == 0)
77+
compresslevel = 0;
78+
else
79+
{
80+
compresslevel = compress->level;
81+
Assert(compresslevel >= 1 && compresslevel <= 12);
82+
}
8083

8184
sink = palloc0(sizeof(bbsink_lz4));
8285
*((const bbsink_ops **) &sink->base.bbs_ops) = &bbsink_lz4_ops;

‎src/backend/replication/basebackup_zstd.c

+11-8
Original file line numberDiff line numberDiff line change
@@ -55,12 +55,13 @@ const bbsink_ops bbsink_zstd_ops = {
5555
#endif
5656

5757
/*
58-
* Create a new basebackup sink that performs zstd compression using the
59-
* designated compression level.
58+
* Create a new basebackup sink that performs zstd compression.
6059
*/
6160
bbsink *
62-
bbsink_zstd_new(bbsink *next, int compresslevel)
61+
bbsink_zstd_new(bbsink *next, bc_specification *compress)
6362
{
63+
int compresslevel;
64+
6465
#ifndef USE_ZSTD
6566
ereport(ERROR,
6667
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
@@ -71,11 +72,13 @@ bbsink_zstd_new(bbsink *next, int compresslevel)
7172

7273
Assert(next != NULL);
7374

74-
if (compresslevel < 0 || compresslevel > 22)
75-
ereport(ERROR,
76-
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
77-
errmsg("zstd compression level %d is out of range",
78-
compresslevel)));
75+
if ((compress->options & BACKUP_COMPRESSION_OPTION_LEVEL) == 0)
76+
compresslevel = 0;
77+
else
78+
{
79+
compresslevel = compress->level;
80+
Assert(compresslevel >= 1 && compresslevel <= 22);
81+
}
7982

8083
sink = palloc0(sizeof(bbsink_zstd));
8184
*((const bbsink_ops **) &sink->base.bbs_ops) = &bbsink_zstd_ops;

‎src/bin/pg_basebackup/bbstreamer.h

+4-3
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@
2222
#ifndef BBSTREAMER_H
2323
#define BBSTREAMER_H
2424

25+
#include "common/backup_compression.h"
2526
#include "lib/stringinfo.h"
2627
#include "pqexpbuffer.h"
2728

@@ -200,17 +201,17 @@ bbstreamer_buffer_until(bbstreamer *streamer, const char **data, int *len,
200201
*/
201202
extern bbstreamer *bbstreamer_plain_writer_new(char *pathname, FILE *file);
202203
extern bbstreamer *bbstreamer_gzip_writer_new(char *pathname, FILE *file,
203-
int compresslevel);
204+
bc_specification *compress);
204205
extern bbstreamer *bbstreamer_extractor_new(const char *basepath,
205206
const char *(*link_map) (const char *),
206207
void (*report_output_file) (const char *));
207208

208209
extern bbstreamer *bbstreamer_gzip_decompressor_new(bbstreamer *next);
209210
extern bbstreamer *bbstreamer_lz4_compressor_new(bbstreamer *next,
210-
int compresslevel);
211+
bc_specification *compress);
211212
extern bbstreamer *bbstreamer_lz4_decompressor_new(bbstreamer *next);
212213
extern bbstreamer *bbstreamer_zstd_compressor_new(bbstreamer *next,
213-
int compresslevel);
214+
bc_specification *compress);
214215
extern bbstreamer *bbstreamer_zstd_decompressor_new(bbstreamer *next);
215216
extern bbstreamer *bbstreamer_tar_parser_new(bbstreamer *next);
216217
extern bbstreamer *bbstreamer_tar_terminator_new(bbstreamer *next);

‎src/bin/pg_basebackup/bbstreamer_gzip.c

+4-3
Original file line numberDiff line numberDiff line change
@@ -76,7 +76,8 @@ const bbstreamer_ops bbstreamer_gzip_decompressor_ops = {
7676
* closed so that the data may be written there.
7777
*/
7878
bbstreamer *
79-
bbstreamer_gzip_writer_new(char *pathname, FILE *file, int compresslevel)
79+
bbstreamer_gzip_writer_new(char *pathname, FILE *file,
80+
bc_specification *compress)
8081
{
8182
#ifdef HAVE_LIBZ
8283
bbstreamer_gzip_writer *streamer;
@@ -115,11 +116,11 @@ bbstreamer_gzip_writer_new(char *pathname, FILE *file, int compresslevel)
115116
}
116117
}
117118

118-
if (gzsetparams(streamer->gzfile, compresslevel,
119+
if (gzsetparams(streamer->gzfile, compress->level,
119120
Z_DEFAULT_STRATEGY) != Z_OK)
120121
{
121122
pg_log_error("could not set compression level %d: %s",
122-
compresslevel, get_gz_error(streamer->gzfile));
123+
compress->level, get_gz_error(streamer->gzfile));
123124
exit(1);
124125
}
125126

‎src/bin/pg_basebackup/bbstreamer_lz4.c

+2-2
Original file line numberDiff line numberDiff line change
@@ -67,7 +67,7 @@ const bbstreamer_ops bbstreamer_lz4_decompressor_ops = {
6767
* blocks.
6868
*/
6969
bbstreamer *
70-
bbstreamer_lz4_compressor_new(bbstreamer *next, int compresslevel)
70+
bbstreamer_lz4_compressor_new(bbstreamer *next, bc_specification *compress)
7171
{
7272
#ifdef USE_LZ4
7373
bbstreamer_lz4_frame *streamer;
@@ -89,7 +89,7 @@ bbstreamer_lz4_compressor_new(bbstreamer *next, int compresslevel)
8989
prefs = &streamer->prefs;
9090
memset(prefs, 0, sizeof(LZ4F_preferences_t));
9191
prefs->frameInfo.blockSizeID = LZ4F_max256KB;
92-
prefs->compressionLevel = compresslevel;
92+
prefs->compressionLevel = compress->level;
9393

9494
/*
9595
* Find out the compression bound, it specifies the minimum destination

‎src/bin/pg_basebackup/bbstreamer_zstd.c

+2-2
Original file line numberDiff line numberDiff line change
@@ -63,7 +63,7 @@ const bbstreamer_ops bbstreamer_zstd_decompressor_ops = {
6363
* blocks.
6464
*/
6565
bbstreamer *
66-
bbstreamer_zstd_compressor_new(bbstreamer *next, int compresslevel)
66+
bbstreamer_zstd_compressor_new(bbstreamer *next, bc_specification *compress)
6767
{
6868
#ifdef USE_ZSTD
6969
bbstreamer_zstd_frame *streamer;
@@ -85,7 +85,7 @@ bbstreamer_zstd_compressor_new(bbstreamer *next, int compresslevel)
8585

8686
/* Initialize stream compression preferences */
8787
ZSTD_CCtx_setParameter(streamer->cctx, ZSTD_c_compressionLevel,
88-
compresslevel);
88+
compress->level);
8989

9090
/* Initialize the ZSTD output buffer. */
9191
streamer->zstd_outBuf.dst = streamer->base.bbs_buffer.data;

0 commit comments

Comments
 (0)