Skip to content

Commit 23a1c65

Browse files
robertmhaasrobertmhaas
committed
Introduce 'bbstreamer' abstraction to modularize pg_basebackup.
pg_basebackup knows how to do quite a few things with a backup that it gets from the server, like just write out the files, or compress them first, or even parse the tar format and inject a modified postgresql.auto.conf file into the archive generated by the server. Unforatunely, this makes pg_basebackup.c a very large source file, and also somewhat difficult to enhance, because for example the knowledge that the server is sending us a 'tar' file rather than some other sort of archive is spread all over the place rather than centralized. In an effort to improve this situation, this commit invents a new 'bbstreamer' abstraction. Each archive received from the server is fed to a bbstreamer which may choose to dispose of it or pass it along to some other bbstreamer. Chunks may also be "labelled" according to whether they are part of the payload data of a file in the archive or part of the archive metadata. So, for example, if we want to take a tar file, modify the postgresql.auto.conf file it contains, and the gzip the result and write it out, we can use a bbstreamer_tar_parser to parse the tar file received from the server, a bbstreamer_recovery_injector to modify the contents of postgresql.auto.conf, a bbstreamer_tar_archiver to replace the tar headers for the file modified in the previous step with newly-built ones that are correct for the modified file, and a bbstreamer_gzip_writer to gzip and write the resulting data. Only the objects with "tar" in the name know anything about the tar archive format, and in theory we could re-archive using some other format rather than "tar" if somebody wanted to write the code. These chances do add a substantial amount of code, but I think the result is a lot more maintainable and extensible. pg_basebackup.c itself shrinks by roughly a third, with a lot of the complexity previously contained there moving into the newly-added files. Patch by me. The larger patch series of which this is a part has been reviewed and tested at various times by Andres Freund, Sumanta Mukherjee, Dilip Kumar, Suraj Kharage, Dipesh Pandit, Tushar Ahuja, Mark Dilger, Sergei Kornilov, and Jeevan Ladhe. Discussion: https://postgr.es/m/CA+TgmoZGwR=ZVWFeecncubEyPdwghnvfkkdBe9BLccLSiqdf9Q@mail.gmail.com Discussion: https://postgr.es/m/CA+TgmoZvqk7UuzxsX1xjJRmMGkqoUGYTZLDCH8SmU1xTPr1Xig@mail.gmail.com
1 parent 00a354a commit 23a1c65

File tree

6 files changed

+1687
-727
lines changed

6 files changed

+1687
-727
lines changed

‎src/bin/pg_basebackup/Makefile

+9-3
Original file line numberDiff line numberDiff line change
@@ -35,10 +35,16 @@ OBJS = \
3535
streamutil.o \
3636
walmethods.o
3737

38+
BBOBJS = \
39+
pg_basebackup.o \
40+
bbstreamer_file.o \
41+
bbstreamer_inject.o \
42+
bbstreamer_tar.o
43+
3844
all: pg_basebackup pg_receivewal pg_recvlogical
3945

40-
pg_basebackup: pg_basebackup.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
41-
$(CC) $(CFLAGS) pg_basebackup.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
46+
pg_basebackup: $(BBOBJS) $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
47+
$(CC) $(CFLAGS) $(BBOBJS) $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
4248

4349
pg_receivewal: pg_receivewal.o $(OBJS) | submake-libpq submake-libpgport submake-libpgfeutils
4450
$(CC) $(CFLAGS) pg_receivewal.o $(OBJS) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
@@ -61,7 +67,7 @@ uninstall:
6167

6268
clean distclean maintainer-clean:
6369
rm -f pg_basebackup$(X) pg_receivewal$(X) pg_recvlogical$(X) \
64-
pg_basebackup.o pg_receivewal.o pg_recvlogical.o \
70+
$(BBOBJS) pg_receivewal.o pg_recvlogical.o \
6571
$(OBJS)
6672
rm -rf tmp_check
6773

‎src/bin/pg_basebackup/bbstreamer.h

+217
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,217 @@
1+
/*-------------------------------------------------------------------------
2+
*
3+
* bbstreamer.h
4+
*
5+
* Each tar archive returned by the server is passed to one or more
6+
* bbstreamer objects for further processing. The bbstreamer may do
7+
* something simple, like write the archive to a file, perhaps after
8+
* compressing it, but it can also do more complicated things, like
9+
* annotating the byte stream to indicate which parts of the data
10+
* correspond to tar headers or trailing padding, vs. which parts are
11+
* payload data. A subsequent bbstreamer may use this information to
12+
* make further decisions about how to process the data; for example,
13+
* it might choose to modify the archive contents.
14+
*
15+
* Portions Copyright (c) 1996-2021, PostgreSQL Global Development Group
16+
*
17+
* IDENTIFICATION
18+
* src/bin/pg_basebackup/bbstreamer.h
19+
*-------------------------------------------------------------------------
20+
*/
21+
22+
#ifndef BBSTREAMER_H
23+
#define BBSTREAMER_H
24+
25+
#include "lib/stringinfo.h"
26+
#include "pqexpbuffer.h"
27+
28+
struct bbstreamer;
29+
struct bbstreamer_ops;
30+
typedef struct bbstreamer bbstreamer;
31+
typedef struct bbstreamer_ops bbstreamer_ops;
32+
33+
/*
34+
* Each chunk of archive data passed to a bbstreamer is classified into one
35+
* of these categories. When data is first received from the remote server,
36+
* each chunk will be categorized as BBSTREAMER_UNKNOWN, and the chunks will
37+
* be of whatever size the remote server chose to send.
38+
*
39+
* If the archive is parsed (e.g. see bbstreamer_tar_parser_new()), then all
40+
* chunks should be labelled as one of the other types listed here. In
41+
* addition, there should be exactly one BBSTREAMER_MEMBER_HEADER chunk and
42+
* exactly one BBSTREAMER_MEMBER_TRAILER chunk per archive member, even if
43+
* that means a zero-length call. There can be any number of
44+
* BBSTREAMER_MEMBER_CONTENTS chunks in between those calls. There
45+
* should exactly BBSTREAMER_ARCHIVE_TRAILER chunk, and it should follow the
46+
* last BBSTREAMER_MEMBER_TRAILER chunk.
47+
*
48+
* In theory, we could need other classifications here, such as a way of
49+
* indicating an archive header, but the "tar" format doesn't need anything
50+
* else, so for the time being there's no point.
51+
*/
52+
typedef enum
53+
{
54+
BBSTREAMER_UNKNOWN,
55+
BBSTREAMER_MEMBER_HEADER,
56+
BBSTREAMER_MEMBER_CONTENTS,
57+
BBSTREAMER_MEMBER_TRAILER,
58+
BBSTREAMER_ARCHIVE_TRAILER
59+
} bbstreamer_archive_context;
60+
61+
/*
62+
* Each chunk of data that is classified as BBSTREAMER_MEMBER_HEADER,
63+
* BBSTREAMER_MEMBER_CONTENTS, or BBSTREAMER_MEMBER_TRAILER should also
64+
* pass a pointer to an instance of this struct. The details are expected
65+
* to be present in the archive header and used to fill the struct, after
66+
* which all subsequent calls for the same archive member are expected to
67+
* pass the same details.
68+
*/
69+
typedef struct
70+
{
71+
char pathname[MAXPGPATH];
72+
pgoff_t size;
73+
mode_t mode;
74+
uid_t uid;
75+
gid_t gid;
76+
bool is_directory;
77+
bool is_link;
78+
char linktarget[MAXPGPATH];
79+
} bbstreamer_member;
80+
81+
/*
82+
* Generally, each type of bbstreamer will define its own struct, but the
83+
* first element should be 'bbstreamer base'. A bbstreamer that does not
84+
* require any additional private data could use this structure directly.
85+
*
86+
* bbs_ops is a pointer to the bbstreamer_ops object which contains the
87+
* function pointers appropriate to this type of bbstreamer.
88+
*
89+
* bbs_next is a pointer to the successor bbstreamer, for those types of
90+
* bbstreamer which forward data to a successor. It need not be used and
91+
* should be set to NULL when not relevant.
92+
*
93+
* bbs_buffer is a buffer for accumulating data for temporary storage. Each
94+
* type of bbstreamer makes its own decisions about whether and how to use
95+
* this buffer.
96+
*/
97+
struct bbstreamer
98+
{
99+
const bbstreamer_ops *bbs_ops;
100+
bbstreamer *bbs_next;
101+
StringInfoData bbs_buffer;
102+
};
103+
104+
/*
105+
* There are three callbacks for a bbstreamer. The 'content' callback is
106+
* called repeatedly, as described in the bbstreamer_archive_context comments.
107+
* Then, the 'finalize' callback is called once at the end, to give the
108+
* bbstreamer a chance to perform cleanup such as closing files. Finally,
109+
* because this code is running in a frontend environment where, as of this
110+
* writing, there are no memory contexts, the 'free' callback is called to
111+
* release memory. These callbacks should always be invoked using the static
112+
* inline functions defined below.
113+
*/
114+
struct bbstreamer_ops
115+
{
116+
void (*content) (bbstreamer *streamer, bbstreamer_member *member,
117+
const char *data, int len,
118+
bbstreamer_archive_context context);
119+
void (*finalize) (bbstreamer *streamer);
120+
void (*free) (bbstreamer *streamer);
121+
};
122+
123+
/* Send some content to a bbstreamer. */
124+
static inline void
125+
bbstreamer_content(bbstreamer *streamer, bbstreamer_member *member,
126+
const char *data, int len,
127+
bbstreamer_archive_context context)
128+
{
129+
Assert(streamer != NULL);
130+
streamer->bbs_ops->content(streamer, member, data, len, context);
131+
}
132+
133+
/* Finalize a bbstreamer. */
134+
static inline void
135+
bbstreamer_finalize(bbstreamer *streamer)
136+
{
137+
Assert(streamer != NULL);
138+
streamer->bbs_ops->finalize(streamer);
139+
}
140+
141+
/* Free a bbstreamer. */
142+
static inline void
143+
bbstreamer_free(bbstreamer *streamer)
144+
{
145+
Assert(streamer != NULL);
146+
streamer->bbs_ops->free(streamer);
147+
}
148+
149+
/*
150+
* This is a convenience method for use when implementing a bbstreamer; it is
151+
* not for use by outside callers. It adds the amount of data specified by
152+
* 'nbytes' to the bbstreamer's buffer and adjusts '*len' and '*data'
153+
* accordingly.
154+
*/
155+
static inline void
156+
bbstreamer_buffer_bytes(bbstreamer *streamer, const char **data, int *len,
157+
int nbytes)
158+
{
159+
Assert(nbytes <= *len);
160+
161+
appendBinaryStringInfo(&streamer->bbs_buffer, *data, nbytes);
162+
*len -= nbytes;
163+
*data += nbytes;
164+
}
165+
166+
/*
167+
* This is a convenence method for use when implementing a bbstreamer; it is
168+
* not for use by outsider callers. It attempts to add enough data to the
169+
* bbstreamer's buffer to reach a length of target_bytes and adjusts '*len'
170+
* and '*data' accordingly. It returns true if the target length has been
171+
* reached and false otherwise.
172+
*/
173+
static inline bool
174+
bbstreamer_buffer_until(bbstreamer *streamer, const char **data, int *len,
175+
int target_bytes)
176+
{
177+
int buflen = streamer->bbs_buffer.len;
178+
179+
if (buflen >= target_bytes)
180+
{
181+
/* Target length already reached; nothing to do. */
182+
return true;
183+
}
184+
185+
if (buflen + *len < target_bytes)
186+
{
187+
/* Not enough data to reach target length; buffer all of it. */
188+
bbstreamer_buffer_bytes(streamer, data, len, *len);
189+
return false;
190+
}
191+
192+
/* Buffer just enough to reach the target length. */
193+
bbstreamer_buffer_bytes(streamer, data, len, target_bytes - buflen);
194+
return true;
195+
}
196+
197+
/*
198+
* Functions for creating bbstreamer objects of various types. See the header
199+
* comments for each of these functions for details.
200+
*/
201+
extern bbstreamer *bbstreamer_plain_writer_new(char *pathname, FILE *file);
202+
extern bbstreamer *bbstreamer_gzip_writer_new(char *pathname, FILE *file,
203+
int compresslevel);
204+
extern bbstreamer *bbstreamer_extractor_new(const char *basepath,
205+
const char *(*link_map) (const char *),
206+
void (*report_output_file) (const char *));
207+
208+
extern bbstreamer *bbstreamer_tar_parser_new(bbstreamer *next);
209+
extern bbstreamer *bbstreamer_tar_archiver_new(bbstreamer *next);
210+
211+
extern bbstreamer *bbstreamer_recovery_injector_new(bbstreamer *next,
212+
bool is_recovery_guc_supported,
213+
PQExpBuffer recoveryconfcontents);
214+
extern void bbstreamer_inject_file(bbstreamer *streamer, char *pathname,
215+
char *data, int len);
216+
217+
#endif

0 commit comments

Comments
 (0)