Skip to content

Commit e6241d8

Browse files
tglsfdctglsfdc
committed
Rethink definition of pg_attribute.attcompression.
Redefine '\0' (InvalidCompressionMethod) as meaning "if we need to compress, use the current setting of default_toast_compression". This allows '\0' to be a suitable default choice regardless of datatype, greatly simplifying code paths that initialize tupledescs and the like. It seems like a more user-friendly approach as well, because now the default compression choice doesn't migrate into table definitions, meaning that changing default_toast_compression is usually sufficient to flip an installation's behavior; one needn't tediously issue per-column ALTER SET COMPRESSION commands. Along the way, fix a few minor bugs and documentation issues with the per-column-compression feature. Adopt more robust APIs for SetIndexStorageProperties and GetAttributeCompression. Bump catversion because typical contents of attcompression will now be different. We could get away without doing that, but it seems better to ensure v14 installations all agree on this. (We already forced initdb for beta2, anyway.) Discussion: https://postgr.es/m/626613.1621787110@sss.pgh.pa.us
1 parent a717e5c commit e6241d8

29 files changed

+257
-380
lines changed

‎doc/src/sgml/catalogs.sgml

+8-4
Original file line numberDiff line numberDiff line change
@@ -1261,10 +1261,14 @@
12611261
<structfield>attcompression</structfield> <type>char</type>
12621262
</para>
12631263
<para>
1264-
The current compression method of the column. If it is an invalid
1265-
compression method (<literal>'\0'</literal>) then column data will not
1266-
be compressed. Otherwise, <literal>'p'</literal> = pglz compression or
1267-
<literal>'l'</literal> = <productname>LZ4</productname> compression.
1264+
The current compression method of the column. Typically this is
1265+
<literal>'\0'</literal> to specify use of the current default setting
1266+
(see <xref linkend="guc-default-toast-compression"/>). Otherwise,
1267+
<literal>'p'</literal> selects pglz compression, while
1268+
<literal>'l'</literal> selects <productname>LZ4</productname>
1269+
compression. However, this field is ignored
1270+
whenever <structfield>attstorage</structfield> does not allow
1271+
compression.
12681272
</para></entry>
12691273
</row>
12701274

‎doc/src/sgml/config.sgml

+8-7
Original file line numberDiff line numberDiff line change
@@ -8256,13 +8256,14 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv;
82568256
<para>
82578257
This variable sets the default
82588258
<link linkend="storage-toast">TOAST</link>
8259-
compression method for columns of newly-created tables. The
8260-
<command>CREATE TABLE</command> statement can override this default
8261-
by specifying the <literal>COMPRESSION</literal> column option.
8262-
8263-
The supported compression methods are <literal>pglz</literal> and,
8264-
if <productname>PostgreSQL</productname> was compiled with
8265-
<literal>--with-lz4</literal>, <literal>lz4</literal>.
8259+
compression method for values of compressible columns.
8260+
(This can be overridden for individual columns by setting
8261+
the <literal>COMPRESSION</literal> column option in
8262+
<command>CREATE TABLE</command> or
8263+
<command>ALTER TABLE</command>.)
8264+
The supported compression methods are <literal>pglz</literal> and
8265+
(if <productname>PostgreSQL</productname> was compiled with
8266+
<option>--with-lz4</option>) <literal>lz4</literal>.
82668267
The default is <literal>pglz</literal>.
82678268
</para>
82688269
</listitem>

‎doc/src/sgml/func.sgml

+2-2
Original file line numberDiff line numberDiff line change
@@ -26253,10 +26253,10 @@ postgres=# SELECT * FROM pg_walfile_name_offset(pg_stop_backup());
2625326253
<primary>pg_column_compression</primary>
2625426254
</indexterm>
2625526255
<function>pg_column_compression</function> ( <type>"any"</type> )
26256-
<returnvalue>integer</returnvalue>
26256+
<returnvalue>text</returnvalue>
2625726257
</para>
2625826258
<para>
26259-
Shows the compression algorithm that was used to compress a
26259+
Shows the compression algorithm that was used to compress
2626026260
an individual variable-length value. Returns <literal>NULL</literal>
2626126261
if the value is not compressed.
2626226262
</para></entry>

‎doc/src/sgml/ref/alter_table.sgml

+16-14
Original file line numberDiff line numberDiff line change
@@ -104,7 +104,6 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
104104
GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( <replaceable>sequence_options</replaceable> ) ] |
105105
UNIQUE <replaceable class="parameter">index_parameters</replaceable> |
106106
PRIMARY KEY <replaceable class="parameter">index_parameters</replaceable> |
107-
COMPRESSION <replaceable class="parameter">compression_method</replaceable> |
108107
REFERENCES <replaceable class="parameter">reftable</replaceable> [ ( <replaceable class="parameter">refcolumn</replaceable> ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ]
109108
[ ON DELETE <replaceable class="parameter">referential_action</replaceable> ] [ ON UPDATE <replaceable class="parameter">referential_action</replaceable> ] }
110109
[ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ]
@@ -391,24 +390,27 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
391390
</term>
392391
<listitem>
393392
<para>
394-
This sets the compression method to be used for data inserted into a column.
395-
393+
This form sets the compression method for a column, determining how
394+
values inserted in future will be compressed (if the storage mode
395+
permits compression at all).
396396
This does not cause the table to be rewritten, so existing data may still
397397
be compressed with other compression methods. If the table is rewritten with
398398
<command>VACUUM FULL</command> or <command>CLUSTER</command>, or restored
399-
with <application>pg_restore</application>, then all tuples are rewritten
400-
with the configured compression methods.
401-
402-
Also, note that when data is inserted from another relation (for example,
403-
by <command>INSERT ... SELECT</command>), tuples from the source data are
404-
not necessarily detoasted, and any previously compressed data is retained
405-
with its existing compression method, rather than recompressing with the
406-
compression methods of the target columns.
407-
399+
with <application>pg_restore</application>, then all values are rewritten
400+
with the configured compression method.
401+
However, when data is inserted from another relation (for example,
402+
by <command>INSERT ... SELECT</command>), values from the source table are
403+
not necessarily detoasted, so any previously compressed data may retain
404+
its existing compression method, rather than being recompressed with the
405+
compression method of the target column.
408406
The supported compression
409407
methods are <literal>pglz</literal> and <literal>lz4</literal>.
410-
<literal>lz4</literal> is available only if <literal>--with-lz4</literal>
411-
was used when building <productname>PostgreSQL</productname>.
408+
(<literal>lz4</literal> is available only if <option>--with-lz4</option>
409+
was used when building <productname>PostgreSQL</productname>.) In
410+
addition, <replaceable class="parameter">compression_method</replaceable>
411+
can be <literal>default</literal>, which selects the default behavior of
412+
consulting the <xref linkend="guc-default-toast-compression"/> setting
413+
at the time of data insertion to determine the method to use.
412414
</para>
413415
</listitem>
414416
</varlistentry>

‎doc/src/sgml/ref/create_table.sgml

+15-10
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ PostgreSQL documentation
2222
<refsynopsisdiv>
2323
<synopsis>
2424
CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] <replaceable class="parameter">table_name</replaceable> ( [
25-
{ <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COLLATE <replaceable>collation</replaceable> ] [ COMPRESSION <replaceable>compression_method</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ]
25+
{ <replaceable class="parameter">column_name</replaceable> <replaceable class="parameter">data_type</replaceable> [ COMPRESSION <replaceable>compression_method</replaceable> ] [ COLLATE <replaceable>collation</replaceable> ] [ <replaceable class="parameter">column_constraint</replaceable> [ ... ] ]
2626
| <replaceable>table_constraint</replaceable>
2727
| LIKE <replaceable>source_table</replaceable> [ <replaceable>like_option</replaceable> ... ] }
2828
[, ... ]
@@ -293,17 +293,22 @@ WITH ( MODULUS <replaceable class="parameter">numeric_literal</replaceable>, REM
293293
<listitem>
294294
<para>
295295
The <literal>COMPRESSION</literal> clause sets the compression method
296-
for a column. Compression is supported only for variable-width data
297-
types, and is used only for columns whose storage type is main or
298-
extended. (See <xref linkend="sql-altertable"/> for information on
299-
column storage types.) Setting this property for a partitioned table
296+
for the column. Compression is supported only for variable-width data
297+
types, and is used only when the column's storage mode
298+
is <literal>main</literal> or <literal>extended</literal>.
299+
(See <xref linkend="sql-altertable"/> for information on
300+
column storage modes.) Setting this property for a partitioned table
300301
has no direct effect, because such tables have no storage of their own,
301-
but the configured value is inherited by newly-created partitions.
302+
but the configured value will be inherited by newly-created partitions.
302303
The supported compression methods are <literal>pglz</literal> and
303-
<literal>lz4</literal>. <literal>lz4</literal> is available only if
304-
<literal>--with-lz4</literal> was used when building
305-
<productname>PostgreSQL</productname>. The default
306-
is <literal>pglz</literal>.
304+
<literal>lz4</literal>. (<literal>lz4</literal> is available only if
305+
<option>--with-lz4</option> was used when building
306+
<productname>PostgreSQL</productname>.) In addition,
307+
<replaceable class="parameter">compression_method</replaceable>
308+
can be <literal>default</literal> to explicitly specify the default
309+
behavior, which is to consult the
310+
<xref linkend="guc-default-toast-compression"/> setting at the time of
311+
data insertion to determine the method to use.
307312
</para>
308313
</listitem>
309314
</varlistentry>

‎doc/src/sgml/ref/pg_dump.sgml

+2-2
Original file line numberDiff line numberDiff line change
@@ -975,8 +975,8 @@ PostgreSQL documentation
975975
<para>
976976
Do not output commands to set <acronym>TOAST</acronym> compression
977977
methods.
978-
With this option, all objects will be created using whichever
979-
compression method is the default during restore.
978+
With this option, all columns will be restored with the default
979+
compression setting.
980980
</para>
981981
</listitem>
982982
</varlistentry>

‎doc/src/sgml/ref/pg_dumpall.sgml

+3-3
Original file line numberDiff line numberDiff line change
@@ -464,12 +464,12 @@ PostgreSQL documentation
464464
<para>
465465
Do not output commands to set <acronym>TOAST</acronym> compression
466466
methods.
467-
With this option, all objects will be created using whichever
468-
compression method is the default during restore.
467+
With this option, all columns will be restored with the default
468+
compression setting.
469469
</para>
470470
</listitem>
471471
</varlistentry>
472-
472+
473473
<varlistentry>
474474
<term><option>--no-unlogged-table-data</option></term>
475475
<listitem>

‎doc/src/sgml/storage.sgml

+10-7
Original file line numberDiff line numberDiff line change
@@ -376,6 +376,16 @@ but the varlena header does not tell whether it has occurred &mdash;
376376
the content of the <acronym>TOAST</acronym> pointer tells that, instead.
377377
</para>
378378

379+
<para>
380+
The compression technique used for either in-line or out-of-line compressed
381+
data can be selected for each column by setting
382+
the <literal>COMPRESSION</literal> column option in <command>CREATE
383+
TABLE</command> or <command>ALTER TABLE</command>. The default for columns
384+
with no explicit setting is to consult the
385+
<xref linkend="guc-default-toast-compression"/> parameter at the time data is
386+
inserted.
387+
</para>
388+
379389
<para>
380390
As mentioned, there are multiple types of <acronym>TOAST</acronym> pointer datums.
381391
The oldest and most common type is a pointer to out-of-line data stored in
@@ -392,13 +402,6 @@ useful for avoiding copying and redundant processing of large data values.
392402
Further details appear in <xref linkend="storage-toast-inmemory"/>.
393403
</para>
394404

395-
<para>
396-
The compression technique used for either in-line or out-of-line compressed
397-
data can be selected using the <literal>COMPRESSION</literal> option on a per-column
398-
basis when creating a table. The default for columns with no explicit setting
399-
is taken from the value of <xref linkend="guc-default-toast-compression" />.
400-
</para>
401-
402405
<sect2 id="storage-toast-ondisk">
403406
<title>Out-of-Line, On-Disk TOAST Storage</title>
404407

‎src/backend/access/brin/brin_tuple.c

+2-3
Original file line numberDiff line numberDiff line change
@@ -232,11 +232,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple,
232232
* same compression method. Otherwise we have to use the
233233
* default method.
234234
*/
235-
if (att->atttypid == atttype->type_id &&
236-
CompressionMethodIsValid(att->attcompression))
235+
if (att->atttypid == atttype->type_id)
237236
compression = att->attcompression;
238237
else
239-
compression = GetDefaultToastCompression();
238+
compression = InvalidCompressionMethod;
240239

241240
cvalue = toast_compress_datum(value, compression);
242241

‎src/backend/access/common/indextuple.c

+2-11
Original file line numberDiff line numberDiff line change
@@ -104,18 +104,9 @@ index_form_tuple(TupleDesc tupleDescriptor,
104104
att->attstorage == TYPSTORAGE_MAIN))
105105
{
106106
Datum cvalue;
107-
char compression = att->attcompression;
108107

109-
/*
110-
* If the compression method is not valid, use the default. We
111-
* don't expect this to happen for regular index columns, which
112-
* inherit the setting from the corresponding table column, but we
113-
* do expect it to happen whenever an expression is indexed.
114-
*/
115-
if (!CompressionMethodIsValid(compression))
116-
compression = GetDefaultToastCompression();
117-
118-
cvalue = toast_compress_datum(untoasted_values[i], compression);
108+
cvalue = toast_compress_datum(untoasted_values[i],
109+
att->attcompression);
119110

120111
if (DatumGetPointer(cvalue) != NULL)
121112
{

‎src/backend/access/common/toast_internals.c

+4-2
Original file line numberDiff line numberDiff line change
@@ -53,10 +53,12 @@ toast_compress_datum(Datum value, char cmethod)
5353
Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value)));
5454
Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value)));
5555

56-
Assert(CompressionMethodIsValid(cmethod));
57-
5856
valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value));
5957

58+
/* If the compression method is not valid, use the current default */
59+
if (!CompressionMethodIsValid(cmethod))
60+
cmethod = default_toast_compression;
61+
6062
/*
6163
* Call appropriate compression routine for the compression method.
6264
*/

‎src/backend/access/common/tupdesc.c

+2-5
Original file line numberDiff line numberDiff line change
@@ -642,10 +642,7 @@ TupleDescInitEntry(TupleDesc desc,
642642
att->attbyval = typeForm->typbyval;
643643
att->attalign = typeForm->typalign;
644644
att->attstorage = typeForm->typstorage;
645-
if (IsStorageCompressible(typeForm->typstorage))
646-
att->attcompression = GetDefaultToastCompression();
647-
else
648-
att->attcompression = InvalidCompressionMethod;
645+
att->attcompression = InvalidCompressionMethod;
649646
att->attcollation = typeForm->typcollation;
650647

651648
ReleaseSysCache(tuple);
@@ -711,7 +708,7 @@ TupleDescInitBuiltinEntry(TupleDesc desc,
711708
att->attbyval = false;
712709
att->attalign = TYPALIGN_INT;
713710
att->attstorage = TYPSTORAGE_EXTENDED;
714-
att->attcompression = GetDefaultToastCompression();
711+
att->attcompression = InvalidCompressionMethod;
715712
att->attcollation = DEFAULT_COLLATION_OID;
716713
break;
717714

‎src/backend/access/heap/heapam_handler.c

+9-3
Original file line numberDiff line numberDiff line change
@@ -2483,10 +2483,10 @@ reform_and_rewrite_tuple(HeapTuple tuple,
24832483
* perform the compression here; we just need to decompress. That
24842484
* will trigger recompression later on.
24852485
*/
2486-
24872486
struct varlena *new_value;
24882487
ToastCompressionId cmid;
24892488
char cmethod;
2489+
char targetmethod;
24902490

24912491
new_value = (struct varlena *) DatumGetPointer(values[i]);
24922492
cmid = toast_get_compression_id(new_value);
@@ -2495,7 +2495,7 @@ reform_and_rewrite_tuple(HeapTuple tuple,
24952495
if (cmid == TOAST_INVALID_COMPRESSION_ID)
24962496
continue;
24972497

2498-
/* convert compression id to compression method */
2498+
/* convert existing compression id to compression method */
24992499
switch (cmid)
25002500
{
25012501
case TOAST_PGLZ_COMPRESSION_ID:
@@ -2506,10 +2506,16 @@ reform_and_rewrite_tuple(HeapTuple tuple,
25062506
break;
25072507
default:
25082508
elog(ERROR, "invalid compression method id %d", cmid);
2509+
cmethod = '\0'; /* keep compiler quiet */
25092510
}
25102511

2512+
/* figure out what the target method is */
2513+
targetmethod = TupleDescAttr(newTupDesc, i)->attcompression;
2514+
if (!CompressionMethodIsValid(targetmethod))
2515+
targetmethod = default_toast_compression;
2516+
25112517
/* if compression method doesn't match then detoast the value */
2512-
if (TupleDescAttr(newTupDesc, i)->attcompression != cmethod)
2518+
if (targetmethod != cmethod)
25132519
{
25142520
values[i] = PointerGetDatum(detoast_attr(new_value));
25152521
values_free[i] = true;

‎src/backend/bootstrap/bootstrap.c

+2-5
Original file line numberDiff line numberDiff line change
@@ -701,6 +701,7 @@ DefineAttr(char *name, char *type, int attnum, int nullness)
701701
attrtypes[attnum]->attbyval = Ap->am_typ.typbyval;
702702
attrtypes[attnum]->attalign = Ap->am_typ.typalign;
703703
attrtypes[attnum]->attstorage = Ap->am_typ.typstorage;
704+
attrtypes[attnum]->attcompression = InvalidCompressionMethod;
704705
attrtypes[attnum]->attcollation = Ap->am_typ.typcollation;
705706
/* if an array type, assume 1-dimensional attribute */
706707
if (Ap->am_typ.typelem != InvalidOid && Ap->am_typ.typlen < 0)
@@ -715,6 +716,7 @@ DefineAttr(char *name, char *type, int attnum, int nullness)
715716
attrtypes[attnum]->attbyval = TypInfo[typeoid].byval;
716717
attrtypes[attnum]->attalign = TypInfo[typeoid].align;
717718
attrtypes[attnum]->attstorage = TypInfo[typeoid].storage;
719+
attrtypes[attnum]->attcompression = InvalidCompressionMethod;
718720
attrtypes[attnum]->attcollation = TypInfo[typeoid].collation;
719721
/* if an array type, assume 1-dimensional attribute */
720722
if (TypInfo[typeoid].elem != InvalidOid &&
@@ -724,11 +726,6 @@ DefineAttr(char *name, char *type, int attnum, int nullness)
724726
attrtypes[attnum]->attndims = 0;
725727
}
726728

727-
if (IsStorageCompressible(attrtypes[attnum]->attstorage))
728-
attrtypes[attnum]->attcompression = GetDefaultToastCompression();
729-
else
730-
attrtypes[attnum]->attcompression = InvalidCompressionMethod;
731-
732729
/*
733730
* If a system catalog column is collation-aware, force it to use C
734731
* collation, so that its behavior is independent of the database's

‎src/backend/catalog/genbki.pl

+1-3
Original file line numberDiff line numberDiff line change
@@ -899,9 +899,7 @@ sub morph_row_for_pgattr
899899
$row->{attbyval} = $type->{typbyval};
900900
$row->{attalign} = $type->{typalign};
901901
$row->{attstorage} = $type->{typstorage};
902-
903-
$row->{attcompression} =
904-
$type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0';
902+
$row->{attcompression} = '\0';
905903

906904
# set attndims if it's an array type
907905
$row->{attndims} = $type->{typcategory} eq 'A' ? '1' : '0';

‎src/backend/catalog/heap.c

-2
Original file line numberDiff line numberDiff line change
@@ -1719,8 +1719,6 @@ RemoveAttributeById(Oid relid, AttrNumber attnum)
17191719
/* Unset this so no one tries to look up the generation expression */
17201720
attStruct->attgenerated = '\0';
17211721

1722-
attStruct->attcompression = InvalidCompressionMethod;
1723-
17241722
/*
17251723
* Change the column name to something that isn't likely to conflict
17261724
*/

0 commit comments

Comments
 (0)