Skip to content

Commit 21bfc74

Browse files
tglsfdctglsfdc
committed
Update regression testing instructions.
This documentation never got the word about the existence of check-world or installcheck-world. Revise to recommend use of those, and document all the subsidiary test suites. Do some minor wordsmithing elsewhere, too. In passing, remove markup related to generation of plain-text regression test instructions, since we don't do that anymore. Back-patch to 9.1 where check-world was added. (installcheck-world exists in 9.0; but since check-world doesn't, this patch would need additional work to cover that branch, and it doesn't seem worth the effort.)
1 parent c40b2d5 commit 21bfc74

File tree

1 file changed

+149
-69
lines changed

1 file changed

+149
-69
lines changed

‎doc/src/sgml/regress.sgml

Lines changed: 149 additions & 69 deletions
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@
2828
<quote>sequential</quote> mode for running the tests. The
2929
sequential method runs each test script alone, while the
3030
parallel method starts up multiple server processes to run groups
31-
of tests in parallel. Parallel testing gives confidence that
31+
of tests in parallel. Parallel testing adds confidence that
3232
interprocess communication and locking are working correctly.
3333
</para>
3434

@@ -43,9 +43,7 @@ gmake check
4343
</screen>
4444
in the top-level directory. (Or you can change to
4545
<filename>src/test/regress</filename> and run the command there.)
46-
This will first build several auxiliary files, such as
47-
sample user-defined trigger functions, and then run the test driver
48-
script. At the end you should see something like:
46+
At the end you should see something like:
4947
<screen>
5048
<computeroutput>
5149
=======================
@@ -113,9 +111,9 @@ gmake MAX_CONNECTIONS=10 check
113111
<title>Running the Tests Against an Existing Installation</title>
114112

115113
<para>
116-
To run the tests after installation<![%standalone-ignore;[ (see <xref linkend="installation">)]]>,
114+
To run the tests after installation (see <xref linkend="installation">),
117115
initialize a data area and start the
118-
server, <![%standalone-ignore;[as explained in <xref linkend="runtime">, ]]> then type:
116+
server as explained in <xref linkend="runtime">, then type:
119117
<screen>
120118
gmake installcheck
121119
</screen>
@@ -125,81 +123,97 @@ gmake installcheck-parallel
125123
</screen>
126124
The tests will expect to contact the server at the local host and the
127125
default port number, unless directed otherwise by <envar>PGHOST</envar> and
128-
<envar>PGPORT</envar> environment variables.
129-
</para>
130-
131-
<para>
132-
The source distribution also contains regression tests for the optional
133-
procedural languages and for some of the <filename>contrib</> modules.
134-
At present, these tests can be used only against an already-installed
135-
server. To run the tests for all procedural languages that have been
136-
built and installed, change to the <filename>src/pl</> directory of the
137-
build tree and type:
138-
<screen>
139-
gmake installcheck
140-
</screen>
141-
You can also do this in any of the subdirectories of <filename>src/pl</>
142-
to run tests for just one procedural language. To run the tests for all
143-
<filename>contrib</> modules that have them, change to the
144-
<filename>contrib</> directory of the build tree and type:
145-
<screen>
146-
gmake installcheck
147-
</screen>
148-
The <filename>contrib</> modules must have been built and installed first.
149-
You can also do this in a subdirectory of <filename>contrib</> to run
150-
the tests for just one module.
126+
<envar>PGPORT</envar> environment variables. The tests will be run in a
127+
database named <literal>regression</>; any existing database by this name
128+
will be dropped.
129+
The tests will also transiently create some cluster-wide objects, such as
130+
user identities named <literal>regressuser<replaceable>N</></literal>.
151131
</para>
152132
</sect2>
153133

154134
<sect2>
155-
<title>Testing Hot Standby</title>
135+
<title>Additional Test Suites</title>
156136

157137
<para>
158-
The source distribution also contains regression tests of the static
159-
behaviour of Hot Standby. These tests require a running primary server
160-
and a running standby server that is accepting new WAL changes from the
161-
primary using either file-based log shipping or streaming replication.
162-
Those servers are not automatically created for you, nor is the setup
163-
documented here. Please check the various sections of the documentation already
164-
devoted to the required commands and related issues.
138+
The <literal>gmake check</> and <literal>gmake installcheck</> commands
139+
run only the <quote>core</> regression tests, which test built-in
140+
functionality of the <productname>PostgreSQL</> server. The source
141+
distribution also contains additional test suites, most of them having
142+
to do with add-on functionality such as optional procedural languages.
165143
</para>
166144

167145
<para>
168-
First create a database called "regression" on the primary.
169-
<screen>
170-
psql -h primary -c "CREATE DATABASE regression"
171-
</screen>
172-
Next, run a preparatory script on the primary in the regression database:
173-
<filename>src/test/regress/sql/hs_primary_setup.sql</filename>, and
174-
allow for the changes to propagate to the standby, for example
175-
<screen>
176-
psql -h primary -f src/test/regress/sql/hs_primary_setup.sql regression
177-
</screen>
178-
Now confirm that the default connection for the tester is the standby
179-
server under test and then run the <literal>standbycheck</> target from the regression
180-
directory:
146+
To run all test suites applicable to the modules that have been selected
147+
to be built, including the core tests, type one of these commands at the
148+
top of the build tree:
181149
<screen>
182-
cd src/test/regress
183-
gmake standbycheck
150+
gmake check-world
151+
gmake installcheck-world
184152
</screen>
153+
These commands run the tests using temporary servers or an
154+
already-installed server, respectively, just as previously explained
155+
for <literal>gmake check</> and <literal>gmake installcheck</>. Other
156+
considerations are the same as previously explained for each method.
157+
Note that <literal>gmake check-world</> builds a separate temporary
158+
installation tree for each tested module, so it requires a great deal
159+
more time and disk space than <literal>gmake installcheck-world</>.
185160
</para>
186161

187162
<para>
188-
Some extreme behaviours can also be generated on the primary using the
189-
script: <filename>src/test/regress/sql/hs_primary_extremes.sql</filename>
190-
to allow the behaviour of the standby to be tested.
163+
Alternatively, you can run individual test suites by typing
164+
<literal>gmake check</> or <literal>gmake installcheck</> in the appropriate
165+
subdirectory of the build tree. Keep in mind that <literal>gmake
166+
installcheck</> assumes you've installed the relevant module(s), not
167+
only the core server.
191168
</para>
192169

193170
<para>
194-
Additional automated testing may be available in later releases.
171+
The additional tests that can be invoked this way include:
172+
</para>
173+
174+
<itemizedlist>
175+
<listitem>
176+
<para>
177+
Regression tests for optional procedural languages (other than
178+
<application>PL/pgSQL</>, which is tested by the core tests).
179+
These are located under <filename>src/pl</>.
180+
</para>
181+
</listitem>
182+
<listitem>
183+
<para>
184+
Regression tests for <filename>contrib</> modules,
185+
located under <filename>contrib</>.
186+
Not all <filename>contrib</> modules have tests.
187+
</para>
188+
</listitem>
189+
<listitem>
190+
<para>
191+
Regression tests for the ECPG interface library,
192+
located in <filename>src/interfaces/ecpg/test</>.
193+
</para>
194+
</listitem>
195+
<listitem>
196+
<para>
197+
Tests stressing behavior of concurrent sessions,
198+
located in <filename>src/test/isolation</>.
199+
</para>
200+
</listitem>
201+
</itemizedlist>
202+
203+
<para>
204+
When using <literal>installcheck</> mode, these tests will destroy any
205+
existing databases named <literal>pl_regression</>,
206+
<literal>contrib_regression</>, <literal>isolationtest</>,
207+
<literal>regress1</>, or <literal>connectdb</>, as well as
208+
<literal>regression</>.
195209
</para>
196210
</sect2>
197211

198212
<sect2>
199213
<title>Locale and Encoding</title>
200214

201215
<para>
202-
By default, the tests against a temporary installation use the
216+
By default, tests using a temporary installation use the
203217
locale defined in the current environment and the corresponding
204218
database encoding as determined by <command>initdb</command>. It
205219
can be useful to test different locales by setting the appropriate
@@ -232,16 +246,17 @@ gmake check LANG=C ENCODING=EUC_JP
232246
</para>
233247

234248
<para>
235-
The encoding can be set for tests against a temporary or an
236-
existing installation.
249+
The database encoding can be set for tests against either a temporary or
250+
an existing installation, though in the latter case it must be
251+
compatible with the installation's locale.
237252
</para>
238253
</sect2>
239254

240255
<sect2>
241256
<title>Extra Tests</title>
242257

243258
<para>
244-
The regression test suite contains a few test files that are not
259+
The core regression test suite contains a few test files that are not
245260
run by default, because they might be platform-dependent or take a
246261
very long time to run. You can run these or other extra test
247262
files by setting the variable <envar>EXTRA_TESTS</envar>. For
@@ -257,6 +272,52 @@ gmake check EXTRA_TESTS=collate.linux.utf8 LANG=en_US.utf8
257272
platforms, and only when run in a database that uses UTF-8 encoding.
258273
</para>
259274
</sect2>
275+
276+
<sect2>
277+
<title>Testing Hot Standby</title>
278+
279+
<para>
280+
The source distribution also contains regression tests for the static
281+
behavior of Hot Standby. These tests require a running primary server
282+
and a running standby server that is accepting new WAL changes from the
283+
primary (using either file-based log shipping or streaming replication).
284+
Those servers are not automatically created for you, nor is replication
285+
setup documented here. Please check the various sections of the
286+
documentation devoted to the required commands and related issues.
287+
</para>
288+
289+
<para>
290+
To run the Hot Standby tests, first create a database
291+
called <literal>regression</> on the primary:
292+
<screen>
293+
psql -h primary -c "CREATE DATABASE regression"
294+
</screen>
295+
Next, run the preparatory script
296+
<filename>src/test/regress/sql/hs_primary_setup.sql</filename>
297+
on the primary in the regression database, for example:
298+
<screen>
299+
psql -h primary -f src/test/regress/sql/hs_primary_setup.sql regression
300+
</screen>
301+
Allow these changes to propagate to the standby.
302+
</para>
303+
304+
<para>
305+
Now arrange for the default database connection to be to the standby
306+
server under test (for example, by setting the <envar>PGHOST</envar> and
307+
<envar>PGPORT</envar> environment variables).
308+
Finally, run <literal>gmake standbycheck</> in the regression directory:
309+
<screen>
310+
cd src/test/regress
311+
gmake standbycheck
312+
</screen>
313+
</para>
314+
315+
<para>
316+
Some extreme behaviors can also be generated on the primary using the
317+
script <filename>src/test/regress/sql/hs_primary_extremes.sql</filename>
318+
to allow the behavior of the standby to be tested.
319+
</para>
320+
</sect2>
260321
</sect1>
261322

262323
<sect1 id="regress-evaluation">
@@ -285,8 +346,10 @@ gmake check EXTRA_TESTS=collate.linux.utf8 LANG=en_US.utf8
285346
file against the reference outputs stored in the
286347
<filename>src/test/regress/expected</filename> directory. Any
287348
differences are saved for your inspection in
288-
<filename>src/test/regress/regression.diffs</filename>. (Or you
289-
can run <command>diff</command> yourself, if you prefer.)
349+
<filename>src/test/regress/regression.diffs</filename>.
350+
(When running a test suite other than the core tests, these files
351+
of course appear in the relevant subdirectory,
352+
not <filename>src/test/regress</>.)
290353
</para>
291354

292355
<para>
@@ -328,7 +391,7 @@ gmake check EXTRA_TESTS=collate.linux.utf8 LANG=en_US.utf8
328391
To run the tests in a different locale when using the
329392
temporary-installation method, pass the appropriate
330393
locale-related environment variables on
331-
the <command>make</command> command line, for example:
394+
the <command>gmake</command> command line, for example:
332395
<programlisting>
333396
gmake check LANG=de_DE.utf8
334397
</programlisting>
@@ -348,7 +411,7 @@ gmake check NO_LOCALE=1
348411
</para>
349412

350413
<para>
351-
In general, it is nevertheless advisable to try to run the
414+
In general, it is advisable to try to run the
352415
regression tests in the locale setup that is wanted for
353416
production use, as this will exercise the locale- and
354417
encoding-related code portions that will actually be used in
@@ -382,7 +445,7 @@ gmake check NO_LOCALE=1
382445
results involving mathematical functions of <type>double
383446
precision</type> columns have been observed. The <literal>float8</> and
384447
<literal>geometry</> tests are particularly prone to small differences
385-
across platforms, or even with different compiler optimization setting.
448+
across platforms, or even with different compiler optimization settings.
386449
Human eyeball comparison is needed to determine the real
387450
significance of these differences which are usually 10 places to
388451
the right of the decimal point.
@@ -444,22 +507,27 @@ exclusion of those that don't.
444507
If the <literal>errors</literal> test results in a server crash
445508
at the <literal>select infinite_recurse()</> command, it means that
446509
the platform's limit on process stack size is smaller than the
447-
<![%standalone-ignore;[<xref linkend="guc-max-stack-depth">]]>
448-
<![%standalone-include;[<literal>max_stack_depth</literal>]]>
449-
parameter indicates. This
510+
<xref linkend="guc-max-stack-depth"> parameter indicates. This
450511
can be fixed by running the server under a higher stack
451512
size limit (4MB is recommended with the default value of
452513
<varname>max_stack_depth</>). If you are unable to do that, an
453514
alternative is to reduce the value of <varname>max_stack_depth</>.
454515
</para>
516+
517+
<para>
518+
On platforms supporting <function>getrlimit()</>, the server should
519+
automatically choose a safe value of <varname>max_stack_depth</>;
520+
so unless you've manually overridden this setting, a failure of this
521+
kind is a reportable bug.
522+
</para>
455523
</sect2>
456524

457525
<sect2>
458526
<title>The <quote>random</quote> Test</title>
459527

460528
<para>
461529
The <literal>random</literal> test script is intended to produce
462-
random results. In rare cases, this causes the random regression
530+
random results. In very rare cases, this causes that regression
463531
test to fail. Typing:
464532
<programlisting>
465533
diff results/random.out expected/random.out
@@ -468,6 +536,18 @@ diff results/random.out expected/random.out
468536
not worry unless the random test fails repeatedly.
469537
</para>
470538
</sect2>
539+
540+
<sect2>
541+
<title>Configuration Parameters</title>
542+
543+
<para>
544+
When running the tests against an existing installation, some non-default
545+
parameter settings could cause the tests to fail. For example, changing
546+
parameters such as <varname>enable_seqscan</varname> or
547+
<varname>enable_indexscan</varname> could cause plan changes that would
548+
affect the results of tests that use <command>EXPLAIN</>.
549+
</para>
550+
</sect2>
471551
</sect1>
472552

473553
<!-- We might want to move the following section into the developer's guide. -->

0 commit comments

Comments
 (0)