Skip to content

Commit 651dc0f

Browse files
author
Alexander Korotkov
committed
Add description and howto about incremental backup.
1 parent 4890370 commit 651dc0f

File tree

1 file changed

+113
-3
lines changed

1 file changed

+113
-3
lines changed

‎README.md

Lines changed: 113 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,8 @@
1-
pg_arman
2-
========
1+
pg_arman fork from Postgres Professional
2+
========================================
3+
4+
This repository contains fork of pg_arman by Postgres Professional with
5+
block level incremental backup support.
36

47
pg_arman is a backup and recovery manager for PostgreSQL servers able to do
58
differential and full backup as well as restore a cluster to a
@@ -14,7 +17,8 @@ Download
1417
--------
1518

1619
The latest version of this software can be found on the project website at
17-
https://github.com/michaelpq/pg_arman.
20+
https://github.com/postgrespro/pg_arman. Original fork of pg_arman can be
21+
found at https://github.com/michaelpq/pg_arman.
1822

1923
Installation
2024
------------
@@ -48,6 +52,112 @@ launched in a way similar to common PostgreSQL extensions and modules:
4852

4953
make installcheck
5054

55+
Block level incremental backup
56+
------------------------------
57+
58+
Idea of block level incremental backup is that you may backup only blocks
59+
changed since last full backup. It gives two major benefits: taking backups
60+
faster and making backups smaller.
61+
62+
The major question here is how to get the list of changed blocks. Since
63+
each block contains LSN number, changed blocks could be retrieved by full scan
64+
of all the blocks. But this approach consumes as much server IO as full
65+
backup.
66+
67+
This is why we implemented alternative approaches to retrieve
68+
list of changed blocks.
69+
70+
1. Scan WAL archive and extract changed blocks from it. However, shortcoming
71+
of these approach is requirement to have WAL archive.
72+
73+
2. Track bitmap of changes blocks inside PostgreSQL (ptrack). It introduces
74+
some overhead to PostgreSQL performance. On our experiments it appears to be
75+
less than 3%.
76+
77+
These two approaches were implemented in this fork of pg_arman. The second
78+
approach requires [patch for PostgreSQL](https://gist.github.com/stalkerg/7ce2394c4f3b36f995895190a633b4fa).
79+
80+
Testing block level incremental backup
81+
--------------------------------------
82+
83+
You need build and install [PGPRO9_5_ptrack branch of PostgreSQL](https://github.com/postgrespro/postgrespro/tree/PGPRO9_5_ptrack) or [apply this patch to PostgreSQL 9.5](https://gist.github.com/stalkerg/7ce2394c4f3b36f995895190a633b4fa).
84+
85+
### Retrieving changed blocks from WAL archive
86+
87+
You need to enable WAL archive by adding following lines to postgresql.conf:
88+
89+
```
90+
wal_level = archive
91+
archive_command = 'test ! -f /home/postgres/backup/arman/wal/%f && cp %p /home/postgres/backup/arman/wal/%f'
92+
wal_log_hints = on
93+
```
94+
95+
Example backup (assuming PostgreSQL is running):
96+
```bash
97+
# Init pg_aramn backup folder
98+
pg_arman init -B /home/postgres/backup/pgarman
99+
cat << __EOF__ >> /home/postgres/backup/pgarman/pg_arman.ini
100+
ARCLOG_PATH = '/home/postgres/backup/arman/wal'
101+
__EOF__
102+
# Make full backup with 2 thread and verbose mode.
103+
pg_arman backup -B /home/postgres/backup/pgarman -D /home/postgres/pgdata/arman -b full -v -j 2
104+
# Validate backup
105+
pg_arman validate -B /home/postgres/backup/pgarman -D /home/postgres/pgdata/arman
106+
# Show backups information
107+
pg_arman show -B /home/postgres/backup/pgarman
108+
109+
# Now you can insert or update some data in your database
110+
111+
# Then start the incremental backup.
112+
pg_arman backup -B /home/postgres/backup/pgarman -D /home/postgres/pgdata/arman -b page -v -j 2
113+
pg_arman validate -B /home/postgres/backup/pgarman -D /home/postgres/pgdata/arman
114+
# You should see that increment is really small
115+
pg_arman show -B /home/postgres/backup/pgarman
116+
```
117+
118+
For restore after remove your pgdata you can use:
119+
```
120+
pg_arman restore -B /home/postgres/backup/pgarman -D /home/postgres/pgdata/arman -j 4 --verbose
121+
```
122+
123+
### Retrieving changed blocks from ptrack
124+
125+
The advantage of this approach is that you don't have to save WAL archive. You will need to enable ptrack in postgresql.conf (restart required).
126+
127+
```
128+
ptrack_enable = on
129+
```
130+
131+
Also, some WALs still need to be fetched in order to get consistent backup. pg_arman can fetch them trough the streaming replication protocol. Thus, you also need to [enable streaming replication connection](https://wiki.postgresql.org/wiki/Streaming_Replication).
132+
133+
Example backup (assuming PostgreSQL is running):
134+
```bash
135+
# Init pg_aramn backup folder
136+
pg_arman init -B /home/postgres/backup/pgarman
137+
cat << __EOF__ >> /home/postgres/backup/pgarman/pg_arman.ini
138+
ARCLOG_PATH = '/home/postgres/backup/arman/wal'
139+
__EOF__
140+
# Make full backup with 2 thread and verbose mode.
141+
pg_arman backup -B /home/postgres/backup/pgarman -D /home/postgres/pgdata/arman -b full -v -j 2 --stream
142+
# Validate backup
143+
pg_arman validate -B /home/postgres/backup/pgarman -D /home/postgres/pgdata/arman
144+
# Show backups information
145+
pg_arman show -B /home/postgres/backup/pgarman
146+
147+
# Now you can insert or update some data in your database
148+
149+
# Then start the incremental backup.
150+
pg_arman backup -B /home/postgres/backup/pgarman -D /home/postgres/pgdata/arman -b ptrack -v -j 2 --stream
151+
pg_arman validate -B /home/postgres/backup/pgarman -D /home/postgres/pgdata/arman
152+
# You should see that increment is really small
153+
pg_arman show -B /home/postgres/backup/pgarman
154+
```
155+
156+
For restore after remove your pgdata you can use:
157+
```
158+
pg_arman restore -B /home/postgres/backup/pgarman -D /home/postgres/pgdata/arman -j 4 --verbose --stream
159+
```
160+
51161
License
52162
-------
53163

0 commit comments

Comments
 (0)