Skip to content

Commit a209184

Browse files
tglsfdctglsfdc
committed
Create common infrastructure for cross-version upgrade testing.
To test pg_upgrade across major PG versions, we have to be able to modify or drop any old objects with no-longer-supported properties, and we have to be able to deal with cosmetic changes in pg_dump output. Up to now, the buildfarm and pg_upgrade's own test infrastructure had separate implementations of the former, and we had nothing but very ad-hoc rules for the latter (including an arbitrary threshold on how many lines of unchecked diff were okay!). This patch creates a Perl module that can be shared by both those use-cases, and adds logic that deals with pg_dump output diffs in a much more tightly defined fashion. This largely supersedes previous efforts in commits 0df9641, 9814ff5, and 62be9e4, which developed a SQL-script-based solution for the task of dropping old objects. There was nothing fundamentally wrong with that work in itself, but it had no basis for solving the output-formatting problem. The most plausible way to deal with formatting is to build a Perl module that can perform editing on the dump files; and once we commit to that, it makes more sense for the same module to also embed the knowledge of what has to be done for dropping old objects. Back-patch versions of the helper module as far as 9.2, to support buildfarm animals that still test that far back. It's also necessary to back-patch PostgreSQL/Version.pm, because the new code depends on that. I fixed up pg_upgrade's 002_pg_upgrade.pl in v15, but did not look into back-patching it further than that. Tom Lane and Andrew Dunstan Discussion: https://postgr.es/m/891521.1673657296@sss.pgh.pa.us
1 parent c58c077 commit a209184

File tree

2 files changed

+382
-0
lines changed

2 files changed

+382
-0
lines changed

‎src/test/perl/PostgreSQL/Test/AdjustUpgrade.pm

Lines changed: 215 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,215 @@
1+
2+
# Copyright (c) 2023, PostgreSQL Global Development Group
3+
4+
=pod
5+
6+
=head1 NAME
7+
8+
PostgreSQL::Test::AdjustUpgrade - helper module for cross-version upgrade tests
9+
10+
=head1 SYNOPSIS
11+
12+
use PostgreSQL::Test::AdjustUpgrade;
13+
14+
# Build commands to adjust contents of old-version database before dumping
15+
$statements = adjust_database_contents($old_version, %dbnames);
16+
17+
# Adjust contents of old pg_dumpall output file to match newer version
18+
$dump = adjust_old_dumpfile($old_version, $dump);
19+
20+
# Adjust contents of new pg_dumpall output file to match older version
21+
$dump = adjust_new_dumpfile($old_version, $dump);
22+
23+
=head1 DESCRIPTION
24+
25+
C<PostgreSQL::Test::AdjustUpgrade> encapsulates various hacks needed to
26+
compare the results of cross-version upgrade tests.
27+
28+
=cut
29+
30+
package PostgreSQL::Test::AdjustUpgrade;
31+
32+
use strict;
33+
use warnings;
34+
35+
use Exporter 'import';
36+
use PostgreSQL::Version;
37+
38+
our @EXPORT = qw(
39+
adjust_database_contents
40+
adjust_old_dumpfile
41+
adjust_new_dumpfile
42+
);
43+
44+
=pod
45+
46+
=head1 ROUTINES
47+
48+
=over
49+
50+
=item $statements = adjust_database_contents($old_version, %dbnames)
51+
52+
Generate SQL commands to perform any changes to an old-version installation
53+
that are needed before we can pg_upgrade it into the current PostgreSQL
54+
version.
55+
56+
Typically this involves dropping or adjusting no-longer-supported objects.
57+
58+
Arguments:
59+
60+
=over
61+
62+
=item C<old_version>: Branch we are upgrading from, represented as a
63+
PostgreSQL::Version object.
64+
65+
=item C<dbnames>: Hash of database names present in the old installation.
66+
67+
=back
68+
69+
Returns a reference to a hash, wherein the keys are database names and the
70+
values are arrayrefs to lists of statements to be run in those databases.
71+
72+
=cut
73+
74+
sub adjust_database_contents
75+
{
76+
my ($old_version, %dbnames) = @_;
77+
my $result = {};
78+
79+
# remove dbs of modules known to cause pg_upgrade to fail
80+
# anything not builtin and incompatible should clean up its own db
81+
foreach my $bad_module ('test_ddl_deparse', 'tsearch2')
82+
{
83+
if ($dbnames{"contrib_regression_$bad_module"})
84+
{
85+
_add_st($result, 'postgres',
86+
"drop database contrib_regression_$bad_module");
87+
delete($dbnames{"contrib_regression_$bad_module"});
88+
}
89+
}
90+
91+
# avoid version number issues with test_ext7
92+
if ($dbnames{contrib_regression_test_extensions})
93+
{
94+
_add_st(
95+
$result,
96+
'contrib_regression_test_extensions',
97+
'drop extension if exists test_ext7');
98+
}
99+
100+
# get rid of dblink's dependencies on regress.so
101+
my $regrdb =
102+
$old_version le '9.4'
103+
? 'contrib_regression'
104+
: 'contrib_regression_dblink';
105+
106+
if ($dbnames{$regrdb})
107+
{
108+
_add_st(
109+
$result, $regrdb,
110+
'drop function if exists public.putenv(text)',
111+
'drop function if exists public.wait_pid(integer)');
112+
}
113+
114+
return $result;
115+
}
116+
117+
# Internal subroutine to add statement(s) to the list for the given db.
118+
sub _add_st
119+
{
120+
my ($result, $db, @st) = @_;
121+
122+
$result->{$db} ||= [];
123+
push(@{ $result->{$db} }, @st);
124+
}
125+
126+
=pod
127+
128+
=item adjust_old_dumpfile($old_version, $dump)
129+
130+
Edit a dump output file, taken from the adjusted old-version installation
131+
by current-version C<pg_dumpall -s>, so that it will match the results of
132+
C<pg_dumpall -s> on the pg_upgrade'd installation.
133+
134+
Typically this involves coping with cosmetic differences in the output
135+
of backend subroutines used by pg_dump.
136+
137+
Arguments:
138+
139+
=over
140+
141+
=item C<old_version>: Branch we are upgrading from, represented as a
142+
PostgreSQL::Version object.
143+
144+
=item C<dump>: Contents of dump file
145+
146+
=back
147+
148+
Returns the modified dump text.
149+
150+
=cut
151+
152+
sub adjust_old_dumpfile
153+
{
154+
my ($old_version, $dump) = @_;
155+
156+
# use Unix newlines
157+
$dump =~ s/\r\n/\n/g;
158+
159+
# Version comments will certainly not match.
160+
$dump =~ s/^-- Dumped from database version.*\n//mg;
161+
162+
# Suppress blank lines, as some places in pg_dump emit more or fewer.
163+
$dump =~ s/\n\n+/\n/g;
164+
165+
return $dump;
166+
}
167+
168+
=pod
169+
170+
=item adjust_new_dumpfile($old_version, $dump)
171+
172+
Edit a dump output file, taken from the pg_upgrade'd installation
173+
by current-version C<pg_dumpall -s>, so that it will match the old
174+
dump output file as adjusted by C<adjust_old_dumpfile>.
175+
176+
Typically this involves deleting data not present in the old installation.
177+
178+
Arguments:
179+
180+
=over
181+
182+
=item C<old_version>: Branch we are upgrading from, represented as a
183+
PostgreSQL::Version object.
184+
185+
=item C<dump>: Contents of dump file
186+
187+
=back
188+
189+
Returns the modified dump text.
190+
191+
=cut
192+
193+
sub adjust_new_dumpfile
194+
{
195+
my ($old_version, $dump) = @_;
196+
197+
# use Unix newlines
198+
$dump =~ s/\r\n/\n/g;
199+
200+
# Version comments will certainly not match.
201+
$dump =~ s/^-- Dumped from database version.*\n//mg;
202+
203+
# Suppress blank lines, as some places in pg_dump emit more or fewer.
204+
$dump =~ s/\n\n+/\n/g;
205+
206+
return $dump;
207+
}
208+
209+
=pod
210+
211+
=back
212+
213+
=cut
214+
215+
1;

‎src/test/perl/PostgreSQL/Version.pm

Lines changed: 167 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,167 @@
1+
############################################################################
2+
#
3+
# PostgreSQL/Version.pm
4+
#
5+
# Module encapsulating Postgres Version numbers
6+
#
7+
# Copyright (c) 2021-2022, PostgreSQL Global Development Group
8+
#
9+
############################################################################
10+
11+
=pod
12+
13+
=head1 NAME
14+
15+
PostgreSQL::Version - class representing PostgreSQL version numbers
16+
17+
=head1 SYNOPSIS
18+
19+
use PostgreSQL::Version;
20+
21+
my $version = PostgreSQL::Version->new($version_arg);
22+
23+
# compare two versions
24+
my $bool = $version1 <= $version2;
25+
26+
# or compare with a number
27+
$bool = $version < 12;
28+
29+
# or with a string
30+
$bool = $version lt "13.1";
31+
32+
# interpolate in a string
33+
my $stringyval = "version: $version";
34+
35+
# get the major version
36+
my $maj = $version->major;
37+
38+
=head1 DESCRIPTION
39+
40+
PostgreSQL::Version encapsulates Postgres version numbers, providing parsing
41+
of common version formats and comparison operations.
42+
43+
=cut
44+
45+
package PostgreSQL::Version;
46+
47+
use strict;
48+
use warnings;
49+
50+
use Scalar::Util qw(blessed);
51+
52+
use overload
53+
'<=>' => \&_version_cmp,
54+
'cmp' => \&_version_cmp,
55+
'""' => \&_stringify;
56+
57+
=pod
58+
59+
=head1 METHODS
60+
61+
=over
62+
63+
=item PostgreSQL::Version->new($version)
64+
65+
Create a new PostgreSQL::Version instance.
66+
67+
The argument can be a number like 12, or a string like '12.2' or the output
68+
of a Postgres command like `psql --version` or `pg_config --version`;
69+
70+
=back
71+
72+
=cut
73+
74+
sub new
75+
{
76+
my $class = shift;
77+
my $arg = shift;
78+
79+
chomp $arg;
80+
81+
# Accept standard formats, in case caller has handed us the output of a
82+
# postgres command line tool
83+
my $devel;
84+
($arg, $devel) = ($1, $2)
85+
if (
86+
$arg =~ m!^ # beginning of line
87+
(?:\(?PostgreSQL\)?\s)? # ignore PostgreSQL marker
88+
(\d+(?:\.\d+)*) # version number, dotted notation
89+
(devel|(?:alpha|beta|rc)\d+)? # dev marker - see version_stamp.pl
90+
!x);
91+
92+
# Split into an array
93+
my @numbers = split(/\./, $arg);
94+
95+
# Treat development versions as having a minor/micro version one less than
96+
# the first released version of that branch.
97+
push @numbers, -1 if ($devel);
98+
99+
$devel ||= "";
100+
101+
return bless { str => "$arg$devel", num => \@numbers }, $class;
102+
}
103+
104+
# Routine which compares the _pg_version_array obtained for the two
105+
# arguments and returns -1, 0, or 1, allowing comparison between two
106+
# PostgreSQL::Version objects or a PostgreSQL::Version and a version string or number.
107+
#
108+
# If the second argument is not a blessed object we call the constructor
109+
# to make one.
110+
#
111+
# Because we're overloading '<=>' and 'cmp' this function supplies us with
112+
# all the comparison operators ('<' and friends, 'gt' and friends)
113+
#
114+
sub _version_cmp
115+
{
116+
my ($a, $b, $swapped) = @_;
117+
118+
$b = __PACKAGE__->new($b) unless blessed($b);
119+
120+
($a, $b) = ($b, $a) if $swapped;
121+
122+
my ($an, $bn) = ($a->{num}, $b->{num});
123+
124+
for (my $idx = 0;; $idx++)
125+
{
126+
return 0
127+
if ($idx >= @$an && $idx >= @$bn);
128+
# treat a missing number as 0
129+
my ($anum, $bnum) = ($an->[$idx] || 0, $bn->[$idx] || 0);
130+
return $anum <=> $bnum
131+
if ($anum <=> $bnum);
132+
}
133+
}
134+
135+
# Render the version number using the saved string.
136+
sub _stringify
137+
{
138+
my $self = shift;
139+
return $self->{str};
140+
}
141+
142+
=pod
143+
144+
=over
145+
146+
=item major([separator => 'char'])
147+
148+
Returns the major version. For versions before 10 the parts are separated by
149+
a dot unless the separator argument is given.
150+
151+
=back
152+
153+
=cut
154+
155+
sub major
156+
{
157+
my ($self, %params) = @_;
158+
my $result = $self->{num}->[0];
159+
if ($result + 0 < 10)
160+
{
161+
my $sep = $params{separator} || '.';
162+
$result .= "$sep$self->{num}->[1]";
163+
}
164+
return $result;
165+
}
166+
167+
1;

0 commit comments

Comments
 (0)