Skip to content

Commit bef47ff

Browse files
robertmhaasrobertmhaas
committed
Introduce 'bbsink' abstraction to modularize base backup code.
The base backup code has accumulated a healthy number of new features over the years, but it's becoming increasingly difficult to maintain and further enhance that code because there's no real separation of concerns. For example, the code that understands knows the details of how we send data to the client using the libpq protocol is scattered throughout basebackup.c, rather than being centralized in one place. To try to improve this situation, introduce a new 'bbsink' object which acts as a recipient for archives generated during the base backup progress and also for the backup manifest. This commit introduces three types of bbsink: a 'copytblspc' bbsink forwards the backup to the client using one COPY OUT operation per tablespace and another for the manifest, a 'progress' bbsink performs command progress reporting, and a 'throttle' bbsink performs rate-limiting. The 'progress' and 'throttle' bbsink types also forward the data to a successor bbsink; at present, the last bbsink in the chain will always be of type 'copytblspc'. There are plans to add more types of 'bbsink' in future commits. This abstraction is a bit leaky in the case of progress reporting, but this still seems cleaner than what we had before. Patch by me, reviewed and tested by Andres Freund, Sumanta Mukherjee, Dilip Kumar, Suraj Kharage, Dipesh Pandit, Tushar Ahuja, Mark Dilger, and Jeevan Ladhe. Discussion: https://postgr.es/m/CA+TgmoZGwR=ZVWFeecncubEyPdwghnvfkkdBe9BLccLSiqdf9Q@mail.gmail.com Discussion: https://postgr.es/m/CA+TgmoZvqk7UuzxsX1xjJRmMGkqoUGYTZLDCH8SmU1xTPr1Xig@mail.gmail.com
1 parent bd807be commit bef47ff

File tree

9 files changed

+1414
-516
lines changed

9 files changed

+1414
-516
lines changed

‎src/backend/replication/Makefile

+4
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,10 @@ override CPPFLAGS := -I. -I$(srcdir) $(CPPFLAGS)
1717
OBJS = \
1818
backup_manifest.o \
1919
basebackup.o \
20+
basebackup_copy.o \
21+
basebackup_progress.o \
22+
basebackup_sink.o \
23+
basebackup_throttle.o \
2024
repl_gram.o \
2125
slot.o \
2226
slotfuncs.o \

‎src/backend/replication/backup_manifest.c

+9-19
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@
1717
#include "libpq/pqformat.h"
1818
#include "mb/pg_wchar.h"
1919
#include "replication/backup_manifest.h"
20+
#include "replication/basebackup_sink.h"
2021
#include "utils/builtins.h"
2122
#include "utils/json.h"
2223

@@ -310,9 +311,8 @@ AddWALInfoToBackupManifest(backup_manifest_info *manifest, XLogRecPtr startptr,
310311
* Finalize the backup manifest, and send it to the client.
311312
*/
312313
void
313-
SendBackupManifest(backup_manifest_info *manifest)
314+
SendBackupManifest(backup_manifest_info *manifest, bbsink *sink)
314315
{
315-
StringInfoData protobuf;
316316
uint8 checksumbuf[PG_SHA256_DIGEST_LENGTH];
317317
char checksumstringbuf[PG_SHA256_DIGEST_STRING_LENGTH];
318318
size_t manifest_bytes_done = 0;
@@ -352,38 +352,28 @@ SendBackupManifest(backup_manifest_info *manifest)
352352
(errcode_for_file_access(),
353353
errmsg("could not rewind temporary file")));
354354

355-
/* Send CopyOutResponse message */
356-
pq_beginmessage(&protobuf, 'H');
357-
pq_sendbyte(&protobuf, 0); /* overall format */
358-
pq_sendint16(&protobuf, 0); /* natts */
359-
pq_endmessage(&protobuf);
360355

361356
/*
362-
* Send CopyData messages.
363-
*
364-
* We choose to read back the data from the temporary file in chunks of
365-
* size BLCKSZ; this isn't necessary, but buffile.c uses that as the I/O
366-
* size, so it seems to make sense to match that value here.
357+
* Send the backup manifest.
367358
*/
359+
bbsink_begin_manifest(sink);
368360
while (manifest_bytes_done < manifest->manifest_size)
369361
{
370-
char manifestbuf[BLCKSZ];
371362
size_t bytes_to_read;
372363
size_t rc;
373364

374-
bytes_to_read = Min(sizeof(manifestbuf),
365+
bytes_to_read = Min(sink->bbs_buffer_length,
375366
manifest->manifest_size - manifest_bytes_done);
376-
rc = BufFileRead(manifest->buffile, manifestbuf, bytes_to_read);
367+
rc = BufFileRead(manifest->buffile, sink->bbs_buffer,
368+
bytes_to_read);
377369
if (rc != bytes_to_read)
378370
ereport(ERROR,
379371
(errcode_for_file_access(),
380372
errmsg("could not read from temporary file: %m")));
381-
pq_putmessage('d', manifestbuf, bytes_to_read);
373+
bbsink_manifest_contents(sink, bytes_to_read);
382374
manifest_bytes_done += bytes_to_read;
383375
}
384-
385-
/* No more data, so send CopyDone message */
386-
pq_putemptymessage('c');
376+
bbsink_end_manifest(sink);
387377

388378
/* Release resources */
389379
BufFileClose(manifest->buffile);

0 commit comments

Comments
 (0)