Skip to content

Commit 558a916

Browse files
anarazelanarazel
committed
Compute XID horizon for page level index vacuum on primary.
Previously the xid horizon was only computed during WAL replay. That had two major problems: 1) It relied on knowing what the table pointed to looks like. That was easy enough before the introducing of tableam (we knew it had to be heap, although some trickery around logging the heap relfilenodes was required). But to properly handle table AMs we need per-database catalog access to look up the AM handler, which recovery doesn't allow. 2) Not knowing the xid horizon also makes it hard to support logical decoding on standbys. When on a catalog table, we need to be able to conflict with slots that have an xid horizon that's too old. But computing the horizon by visiting the heap only works once consistency is reached, but we always need to be able to detect conflicts. There's also a secondary problem, in that the current method performs redundant work on every standby. But that's counterbalanced by potentially computing the value when not necessary (either because there's no standby, or because there's no connected backends). Solve 1) and 2) by moving computation of the xid horizon to the primary and by involving tableam in the computation of the horizon. To address the potentially increased overhead, increase the efficiency of the xid horizon computation for heap by sorting the tids, and eliminating redundant buffer accesses. When prefetching is available, additionally perform prefetching of buffers. As this is more of a maintenance task, rather than something routinely done in every read only query, we add an arbitrary 10 to the effective concurrency - thereby using IO concurrency, when not globally enabled. That's possibly not the perfect formula, but seems good enough for now. Bumps WAL format, as latestRemovedXid is now part of the records, and the heap's relfilenode isn't anymore. Author: Andres Freund, Amit Khandekar, Robert Haas Reviewed-By: Robert Haas Discussion: https://postgr.es/m/20181212204154.nsxf3gzqv3gesl32@alap3.anarazel.de https://postgr.es/m/20181214014235.dal5ogljs3bmlq44@alap3.anarazel.de https://postgr.es/m/20180703070645.wchpu5muyto5n647@alap3.anarazel.de
1 parent 126d631 commit 558a916

File tree

16 files changed

+316
-321
lines changed

16 files changed

+316
-321
lines changed

‎src/backend/access/hash/hash_xlog.c

Lines changed: 1 addition & 152 deletions
Original file line numberDiff line numberDiff line change
@@ -969,155 +969,6 @@ hash_xlog_update_meta_page(XLogReaderState *record)
969969
UnlockReleaseBuffer(metabuf);
970970
}
971971

972-
/*
973-
* Get the latestRemovedXid from the heap pages pointed at by the index
974-
* tuples being deleted. See also btree_xlog_delete_get_latestRemovedXid,
975-
* on which this function is based.
976-
*/
977-
static TransactionId
978-
hash_xlog_vacuum_get_latestRemovedXid(XLogReaderState *record)
979-
{
980-
xl_hash_vacuum_one_page *xlrec;
981-
OffsetNumber *unused;
982-
Buffer ibuffer,
983-
hbuffer;
984-
Page ipage,
985-
hpage;
986-
RelFileNode rnode;
987-
BlockNumber blkno;
988-
ItemId iitemid,
989-
hitemid;
990-
IndexTuple itup;
991-
HeapTupleHeader htuphdr;
992-
BlockNumber hblkno;
993-
OffsetNumber hoffnum;
994-
TransactionId latestRemovedXid = InvalidTransactionId;
995-
int i;
996-
997-
xlrec = (xl_hash_vacuum_one_page *) XLogRecGetData(record);
998-
999-
/*
1000-
* If there's nothing running on the standby we don't need to derive a
1001-
* full latestRemovedXid value, so use a fast path out of here. This
1002-
* returns InvalidTransactionId, and so will conflict with all HS
1003-
* transactions; but since we just worked out that that's zero people,
1004-
* it's OK.
1005-
*
1006-
* XXX There is a race condition here, which is that a new backend might
1007-
* start just after we look. If so, it cannot need to conflict, but this
1008-
* coding will result in throwing a conflict anyway.
1009-
*/
1010-
if (CountDBBackends(InvalidOid) == 0)
1011-
return latestRemovedXid;
1012-
1013-
/*
1014-
* Check if WAL replay has reached a consistent database state. If not, we
1015-
* must PANIC. See the definition of
1016-
* btree_xlog_delete_get_latestRemovedXid for more details.
1017-
*/
1018-
if (!reachedConsistency)
1019-
elog(PANIC, "hash_xlog_vacuum_get_latestRemovedXid: cannot operate with inconsistent data");
1020-
1021-
/*
1022-
* Get index page. If the DB is consistent, this should not fail, nor
1023-
* should any of the heap page fetches below. If one does, we return
1024-
* InvalidTransactionId to cancel all HS transactions. That's probably
1025-
* overkill, but it's safe, and certainly better than panicking here.
1026-
*/
1027-
XLogRecGetBlockTag(record, 0, &rnode, NULL, &blkno);
1028-
ibuffer = XLogReadBufferExtended(rnode, MAIN_FORKNUM, blkno, RBM_NORMAL);
1029-
1030-
if (!BufferIsValid(ibuffer))
1031-
return InvalidTransactionId;
1032-
LockBuffer(ibuffer, HASH_READ);
1033-
ipage = (Page) BufferGetPage(ibuffer);
1034-
1035-
/*
1036-
* Loop through the deleted index items to obtain the TransactionId from
1037-
* the heap items they point to.
1038-
*/
1039-
unused = (OffsetNumber *) ((char *) xlrec + SizeOfHashVacuumOnePage);
1040-
1041-
for (i = 0; i < xlrec->ntuples; i++)
1042-
{
1043-
/*
1044-
* Identify the index tuple about to be deleted.
1045-
*/
1046-
iitemid = PageGetItemId(ipage, unused[i]);
1047-
itup = (IndexTuple) PageGetItem(ipage, iitemid);
1048-
1049-
/*
1050-
* Locate the heap page that the index tuple points at
1051-
*/
1052-
hblkno = ItemPointerGetBlockNumber(&(itup->t_tid));
1053-
hbuffer = XLogReadBufferExtended(xlrec->hnode, MAIN_FORKNUM,
1054-
hblkno, RBM_NORMAL);
1055-
1056-
if (!BufferIsValid(hbuffer))
1057-
{
1058-
UnlockReleaseBuffer(ibuffer);
1059-
return InvalidTransactionId;
1060-
}
1061-
LockBuffer(hbuffer, HASH_READ);
1062-
hpage = (Page) BufferGetPage(hbuffer);
1063-
1064-
/*
1065-
* Look up the heap tuple header that the index tuple points at by
1066-
* using the heap node supplied with the xlrec. We can't use
1067-
* heap_fetch, since it uses ReadBuffer rather than XLogReadBuffer.
1068-
* Note that we are not looking at tuple data here, just headers.
1069-
*/
1070-
hoffnum = ItemPointerGetOffsetNumber(&(itup->t_tid));
1071-
hitemid = PageGetItemId(hpage, hoffnum);
1072-
1073-
/*
1074-
* Follow any redirections until we find something useful.
1075-
*/
1076-
while (ItemIdIsRedirected(hitemid))
1077-
{
1078-
hoffnum = ItemIdGetRedirect(hitemid);
1079-
hitemid = PageGetItemId(hpage, hoffnum);
1080-
CHECK_FOR_INTERRUPTS();
1081-
}
1082-
1083-
/*
1084-
* If the heap item has storage, then read the header and use that to
1085-
* set latestRemovedXid.
1086-
*
1087-
* Some LP_DEAD items may not be accessible, so we ignore them.
1088-
*/
1089-
if (ItemIdHasStorage(hitemid))
1090-
{
1091-
htuphdr = (HeapTupleHeader) PageGetItem(hpage, hitemid);
1092-
HeapTupleHeaderAdvanceLatestRemovedXid(htuphdr, &latestRemovedXid);
1093-
}
1094-
else if (ItemIdIsDead(hitemid))
1095-
{
1096-
/*
1097-
* Conjecture: if hitemid is dead then it had xids before the xids
1098-
* marked on LP_NORMAL items. So we just ignore this item and move
1099-
* onto the next, for the purposes of calculating
1100-
* latestRemovedxids.
1101-
*/
1102-
}
1103-
else
1104-
Assert(!ItemIdIsUsed(hitemid));
1105-
1106-
UnlockReleaseBuffer(hbuffer);
1107-
}
1108-
1109-
UnlockReleaseBuffer(ibuffer);
1110-
1111-
/*
1112-
* If all heap tuples were LP_DEAD then we will be returning
1113-
* InvalidTransactionId here, which avoids conflicts. This matches
1114-
* existing logic which assumes that LP_DEAD tuples must already be older
1115-
* than the latestRemovedXid on the cleanup record that set them as
1116-
* LP_DEAD, hence must already have generated a conflict.
1117-
*/
1118-
return latestRemovedXid;
1119-
}
1120-
1121972
/*
1122973
* replay delete operation in hash index to remove
1123974
* tuples marked as DEAD during index tuple insertion.
@@ -1149,12 +1000,10 @@ hash_xlog_vacuum_one_page(XLogReaderState *record)
11491000
*/
11501001
if (InHotStandby)
11511002
{
1152-
TransactionId latestRemovedXid =
1153-
hash_xlog_vacuum_get_latestRemovedXid(record);
11541003
RelFileNode rnode;
11551004

11561005
XLogRecGetBlockTag(record, 0, &rnode, NULL, NULL);
1157-
ResolveRecoveryConflictWithSnapshot(latestRemovedXid, rnode);
1006+
ResolveRecoveryConflictWithSnapshot(xldata->latestRemovedXid, rnode);
11581007
}
11591008

11601009
action = XLogReadBufferForRedoExtended(record, 0, RBM_NORMAL, true, &buffer);

‎src/backend/access/hash/hashinsert.c

Lines changed: 11 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -23,8 +23,8 @@
2323
#include "storage/buf_internals.h"
2424
#include "storage/predicate.h"
2525

26-
static void _hash_vacuum_one_page(Relation rel, Buffer metabuf, Buffer buf,
27-
RelFileNode hnode);
26+
static void _hash_vacuum_one_page(Relation rel, Relation hrel,
27+
Buffer metabuf, Buffer buf);
2828

2929
/*
3030
* _hash_doinsert() -- Handle insertion of a single index tuple.
@@ -137,7 +137,7 @@ _hash_doinsert(Relation rel, IndexTuple itup, Relation heapRel)
137137

138138
if (IsBufferCleanupOK(buf))
139139
{
140-
_hash_vacuum_one_page(rel, metabuf, buf, heapRel->rd_node);
140+
_hash_vacuum_one_page(rel, heapRel, metabuf, buf);
141141

142142
if (PageGetFreeSpace(page) >= itemsz)
143143
break; /* OK, now we have enough space */
@@ -335,8 +335,7 @@ _hash_pgaddmultitup(Relation rel, Buffer buf, IndexTuple *itups,
335335
*/
336336

337337
static void
338-
_hash_vacuum_one_page(Relation rel, Buffer metabuf, Buffer buf,
339-
RelFileNode hnode)
338+
_hash_vacuum_one_page(Relation rel, Relation hrel, Buffer metabuf, Buffer buf)
340339
{
341340
OffsetNumber deletable[MaxOffsetNumber];
342341
int ndeletable = 0;
@@ -360,6 +359,12 @@ _hash_vacuum_one_page(Relation rel, Buffer metabuf, Buffer buf,
360359

361360
if (ndeletable > 0)
362361
{
362+
TransactionId latestRemovedXid;
363+
364+
latestRemovedXid =
365+
index_compute_xid_horizon_for_tuples(rel, hrel, buf,
366+
deletable, ndeletable);
367+
363368
/*
364369
* Write-lock the meta page so that we can decrement tuple count.
365370
*/
@@ -393,7 +398,7 @@ _hash_vacuum_one_page(Relation rel, Buffer metabuf, Buffer buf,
393398
xl_hash_vacuum_one_page xlrec;
394399
XLogRecPtr recptr;
395400

396-
xlrec.hnode = hnode;
401+
xlrec.latestRemovedXid = latestRemovedXid;
397402
xlrec.ntuples = ndeletable;
398403

399404
XLogBeginInsert();

0 commit comments

Comments
 (0)